PS1 IPP Czar Logs for the week 2011.01.03 - 2011.01.09

(Up to PS1 IPP Czar Logs)

Monday : 2011.01.3

heather reverted (using regtool -revert...) burntool/registration.

bill and eugene have turned off processing because we are out of disk space.

Tuesday : 2011.01.04

bill is czar today

• It appears that all data from last night has had burtool applied.
• 12:30 Set stdscience to 'run' added ThreePi.nightlyscience back in
• 12:52 we seem to be getting a pretty decent rate of faults due to nfs errors

Wednesday : 2011.01.05

• (serge/07:40) cam revert on
• (serge/08:39) publishing restarted
• (bills/10:00) warp stuck lots of entries in warpPendingSkyCell book in done state. ipp049 not responding to ssh 4 warps stuck running there. Stopped everything for awhile let jobs finish. Then reset the books (warp.reset, chip.reset, etc)
• (bills/10:51) Gavin rebooted ipp049. publish was getting lots of faults. Stopped it and asked Serge to investigate)
• (bills/11:35) Turned off some reverts in order to debug the fault rate. Also set poll limit to 32 to reduce the load in order to get an idea whether that is the problem or not.

Thursday : 2011.01.06

Friday : 2011.01.07

Saturday : 2011.01.08

Sunday : 2011.01.09