PS1 IPP Czar Logs for the week 2011.01.31 - 2011.02.06

(Up to PS1 IPP Czar Logs)

Monday : 2011.01.31

• Bill has magic cleanup is running cleaning up old magic runs on ipp053. This is the reason for the relatively higher load. pantasks directory is ~bills/magic_cleanup.

Tuesday : 2011.02.01

• Bill 06:55 Lots of red on the board this morning. This could be because there is a lot of update processing going on. 3 stack faults. Turned stack.revert on (since we fixed the known bugs it should be on all the time now.)
• Bill 07:00 Once last night's data is through warp I will queue the next STS night from September. We still have several thousand STS exposures to deliver. This key project has been much neglected.
• Lots of database problems (probably due to multiple processing streams) and nfs errors today.
• Bill 17:24. Two distRuns fail to complete because of corrupted input files. dist_id: 409008 403036 Set state to 'hold' to stop the fail/revert/fail cycle
• Set STS.201009 to be distributed.
• 17:30 set update and distribution pantasks to off to see if the load drops significantly. Yep. Set them to run at 17:40
• fixed a corrupted warp file with runwarpskyfile. I lost the ids. the diff that was failing completed.
• 15:50 STS is through warp. Only a few diffs but lots of distribution running. Still lots of nfs faults.
• restarting distribution pantasks to reduce the number of enabled hosts.

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD