PS1 IPP Czar Logs for the week 2011.07.04 - 2011.07.10

(Up to PS1 IPP Czar Logs)

Monday : 2011.07.04

• 07:50 : Roy : Restarted summitcopy, which appeared to crash at 2am-ish
• 15:54 : Bill checked in to see whether the STS.2009.b data is finally done. Turned out that there were a couple of corrupt files that needed repair. Noticed that distribution pcontrrol was 1.3GB in size and not spawning the pending jobs very fast. Restarted it.

Tuesday : 2011-07-05

Serge is czar

• 10:41 Serge: stopping all servers
• 10:50 Serge: all servers stopped
• 12:00 Serge: processing resumed
• 13:15 Serge: chip.off
• 13:34 CZW: Serge noticed that diffs were not being queued. There was a registration fault this morning, and I suspect that fooled nightly_science into setting the nsDiffState to FINISHED_DIFFING. I launched most of the missing diffs with nightly_science.pl --date 2011-07-05 --verbose --queue_diffs. It looks like some diff inputs aren't ready, so this command will likely need to be run once more (alternatively, a restart of stdscience will reset the state).

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD