== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2013.04.01 ===
mark is czar
 * 08:05 MEH: remaining few nightly science exposures finishing up
 * 09:38 Bill: ippc03 is running out of space in /tmp. The nebulous_server.log file had grown to 22G. Stopped apache, moved the file out of the way, restarted apache, then deleted it.
 * 11:10 MEH: ipp05,08,09 nebulous_sever.log >22G as well, leaving ~4G on / OS disk and should be cycled out as well -- done. ippc07 will be next, only ~9G left available
 * 19:00 MEH: MD09 deepstacks done so compute3 back to stdsci and stack until start MD08 refstack ~tomorrow
 * 22:20 MEH: appear to have lost connection from ifa to production cluster? ganglia also reporting all systems down..
=== Tuesday : YYYY.MM.DD ===
mark is czar
 * 07:00 MEH: network issue with summit, but also something odd with production cluster and looks like some systems are down and some were rebooted.
  * still down: 028,033,036,046,048,c17,c19,db02 and console not reachable -- was ganglia problem for some, ipp046,048,c17,c19 actually down/off network, cannot connect via console
  * checking if the stsci nodes were power cycled -- no, so probably good thing
  * ippdb00,01,03 have been so may need to check mysql
  * various data (non-stsci data) nodes have been rebooted it seems
 * 08:30 MEH: restarted screens and czarpoll on ippc11, will wait for roboczar until systems back running
 * 08:40 MEH: rebooted ipp036 to fix mount issues, didn't come back up cleanly after whatever happened last night. need to scan mounts on all machines
  * serge put ipp036 into neb-host repair -- need to take back out
  * ipp028,033,036 disks aren't always mounting 
  * some wave1 machines trouble mounting wave4
 * 09:30 MEH: ganglia is showing several machines high 1-m load but nothing running, gmond probably needs to be restarted on most/all machines
 * 09:40: MEH: ippc17 and ippc19 down, ippRequestServer (pss, datastore db and replication) so not possible for PSS to run until fixed
 * 10:00 MEH: ipp041 date is wrong
=== Wednesday : YYYY.MM.DD ===

=== Thursday : YYYY.MM.DD ===

=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===