== PS1 IPP Czar Logs for the week 2013.02.04 - 2013.02.11 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2013.02.04 ===
  
=== Tuesday : 2013.02.05 ===

 * 13:59: CZW: Moved ipp006 and ipp007 from "wave1" to "wave1_dvo" in the pantasks.hosts file.  I also re-enabled the compute3 machines for stdscience.  Restarting stdscience for these changes to take effect.

=== Wednesday : 2013.02.06 ===
 * 16:30 Serge: Queued 4 exposures for stdscience reprocessing (label: MopsSynthetics)

=== Thursday : 2013.02.07 ===
 * 09:25 Bill queued 5 filter staticsky runs for skycells with 4h < RA < 6h. Adding staticsky to the stack pantasks
 * 10:45 Bill reverted 4 stacks that got fault 2 due to nfs issues.
 * 10:50 Fixed some broken files
{{{
recovered lost instance of o5507g0448o.ota25.fits using deneb-locate.py
reran burntool for o5464g0516o.ota26
recovered lost instance of o5463g0418o.ota23.fits
reran burntool for o5170g0154o.ota32.fits 
}}}
 * 10:50 Serge: restarted stdscience 
 * 14:27 Bill: set stack and stdscience pantasks to stop in preparation for load rebalancing
 * 14:30 Bill: set 2 x compute3 to off in stdscience and added them to stack. Set 2 x compute2 to off in stack. Not adding them to stdscience for now since they are still overloaded with existing processing.
 * 14:38 Bill fixed burntool table for o5456g0481o.ota26.fits
 * 14:45 Bill is running some M31jobs on compute3 and wave4. It should only take a few minutes.
 * 16:05 Bill's jobs are done. Setting 2x compute3 back to on stdscience
 * 16:10 restarted distribution. The job counts were all mangled by timeouts.
=== Friday : 2013.02.08 ===
 * 09:45 Bill stopped stdscience. massive job failures. Looks like an nfs or nebulous problem
 * 09:51 Bill: The problem is that stsci00.2 is 100% full yet nebulous was still trying to put files there. Set it to repair. Leaving stdsciene stopped for a bit.
 * 10:05 Bill restarted stdscience. chip.off to allow last nights warps to have the cpus
 * 10:33 Bill: added a few lines to diff_skycell.pl to fix a missing prepare_output(). Warps are done, Turned chip back on. Set stack back to run
=== Saturday : 2013.02.09 ===

=== Sunday : 2013.02.10 ===