[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2012-08-20 ===
 * 09:15 (Serge): {{{stdscience}}} apparently crashed. Restarted.
 * 10:10 (Serge): Recovered gpc1/20100527/o5343g0315o/o5343g0315o.ota26.fits. Temporarily set ippb00 to up in nebulous to manually replicate an instance on ippb00.2.


=== Tuesday : 2012-08-21 ===
Serge is czar
 * 06:30 (Serge): Nightly processing almost complete. Just a few 3pi at the warp stage.
 * 10:40 (Serge): LAP 
   * Lost:
     * gpc1/20090628/o5010g0074o/o5010g0074o.ota25.fits (XY25 82622 553606)
     * gpc1/20090628/o5010g0066o/o5010g0066o.ota44.fits (XY44 82614 553693)
     * gpc1/20090628/o5010g0077o/o5010g0077o.ota24.fits (XY24 82625 553724)
     * gpc1/20090628/o5010g0077o/o5010g0077o.ota31.fits (XY31 82625 553724)
   * Recovered:
     * gpc1/20100629/o5376g0035o/o5376g0035o.ota03.fits
     * gpc1/20100629/o5376g0038o/o5376g0038o.ota46.fits
     * gpc1/20100626/o5373g0186o/o5373g0186o.ota56.fits
     * gpc1/20100629/o5376g0045o/o5376g0045o.ota05.fits
     * gpc1/20100629/o5376g0041o/o5376g0041o.ota64.fits
     * gpc1/20100629/o5376g0042o/o5376g0042o.ota44.fits
     * gpc1/20100629/o5376g0042o/o5376g0042o.ota64.fits
     * gpc1/20100629/o5376g0054o/o5376g0054o.ota05.fits
   * Fixed:
     * gpc1/20100702/o5379g0103o/o5379g0103o.ota41.burn.tbl
     * gpc1/20100702/o5379g0120o/o5379g0120o.ota76.burn.tbl
     * gpc1/20100702/o5379g0104o/o5379g0104o.ota52.burn.tbl
     * gpc1/20100702/o5379g0119o/o5379g0119o.ota05.burn.tbl
     * gpc1/20100702/o5379g0105o/o5379g0105o.ota41.burn.tbl
     * gpc1/20100702/o5379g0122o/o5379g0122o.ota17.burn.tbl
     * gpc1/20100627/o5374g0246o/o5374g0246o.ota33.burn.tbl
     * gpc1/20100627/o5374g0237o/o5374g0237o.ota30.burn.tbl
     * gpc1/20100627/o5374g0204o/o5374g0204o.ota24.burn.tbl
     * gpc1/20100702/o5379g0155o/o5379g0155o.ota41.burn.tbl
 * 11:00 (Serge): All 4 ATRC nodes shut down (A/C check there)
 * 14:35 (Serge): The 4 ATRC nodes are back. All partitions set to repair but ippb00.2 which is up.
 * 15:00 (Serge): LAP
   * Fixed:
     * gpc1/20100702/o5379g0107o/o5379g0107o.ota76.burn.tbl
     * gpc1/20100702/o5379g0125o/o5379g0125o.ota31.burn.tbl
=== Wednesday : YYYY.MM.DD ===

=== Thursday : YYYY.MM.DD ===

=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===