[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2012.08.06 ===
Mark is czar 
 * 07:40 MEH:  nightly science slowly continues to download/process, ~100 exposures behind. restarted summitcopy to see if that helps.
  * ipp027.0 continues to decrease, 100% disk used for ~140GB remaining.
  * reports that 'IPP datastore is down, can't connect to backend MySQL server', i'm getting proxy error. can't find any notes on the proxy setup other than now on ippops1. /data/ippc17.0 was full, cleaned out /local/ipp/tmp and mysql happy again. pstamp pantasks having trouble now.
 * 08:55 (Serge): {{{pubtool -revert -dbname gpc1 -fault 2 -label ThreePi.nightlyscience}}} after ippc17 full disk incident
 * 09:10 (Serge): Killed rogue jobs and restarted publishing
 * 09:30 (Serge): Fixed LAP fault 2: gpc1/20100530/o5346g0039o/o5346g0039o.ota47.burn.tbl, gpc1/20100624/o5371g0063o/o5371g0063o.ota64.fits, gpc1/20100529/o5345g0048o/o5345g0048o.ota26.fits
 * 10:20 MEH: 12-18hr restarting of stdscience. seems to have picked up the remaining 3 3PI chips..
 * 10:30 rebooting ippc17 to try and possibly clear some weird behavior.
 * 11:10 (Serge): ippc63 is 68965 behind its master
=== Tuesday : YYYY.MM.DD ===

=== Wednesday : YYYY.MM.DD ===

=== Thursday : YYYY.MM.DD ===

=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===