== PS1 IPP Czar Logs for the week 2011.01.31 - 2011.02.06 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2011.01.31 ===
 * Bill has magic cleanup is running cleaning up old magic runs on ipp053. This is the reason for the relatively higher load. pantasks directory is ~bills/magic_cleanup.

=== Tuesday : 2011.02.01 ===
 * Bill 06:55 Lots of red on the board this morning. This could be because there is a lot of update processing going on. 3 stack faults. Turned stack.revert on (since we fixed the known bugs it should be on all the time now.)
 * Bill 07:00 Once last night's data is through warp I will queue the next STS night from September. We still have several thousand STS exposures to deliver. This key project has been much neglected.
 * Lots of database problems (probably due to multiple processing streams) and nfs errors today.
 * Bill 17:24. Two distRuns fail to complete because of corrupted input files. dist_id: 409008 403036 Set state to 'hold' to stop the fail/revert/fail cycle
 * Set STS.201009 to be distributed.
 * 17:30 set update and distribution pantasks to off to see if the load drops significantly. Yep. Set them to run at 17:40
 * fixed a corrupted warp file with runwarpskyfile. I lost the ids. the diff that was failing completed.
 * 15:50 STS is through diff. So the only thing running is distribution and update). Still getting lots of nfs faults.
 * restarting distribution pantasks to reduce the number of enabled hosts.
 * we got an MDO07 exposure last night and we are threatened with MD06. Added these labels back into stdscience survey tasks
 * 17:00 Bill is investigating distribution and destreak failures. dist.revert.off and destreak.revert.off
 * all reverts back on
 * fixed one corrupt warp component with perl ~bills/ipp/tools/runwarpskycell.pl --redirect-output --warp_id 157463 --skycell_id skycell.298

=== Wednesday : 2011.02.02 ===
 * 05:50 (Bill) space page on czartool is mostly red. STS.20100909 is done except for 3 broken diff stage distRuns. Dropped those and set chip,warp,and diff data to be cleaned.

=== Thursday : 2011.02.03 ===

=== Friday : 2011.02.04 ===

=== Saturday : 2011.02.05 ===

=== Sunday : 2011.02.06 ===