== PS1 IPP Czar Logs for the week 2015.01.26 - 2015.02.01 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2015.01.26 ===
 * 04:30 EAM: ipp044 crashed, nothing on console.  rebooting.
 * 04:45 EAM: ipp044 having trouble on reboot.  I set it down in nebulous.
 * 05:00 EAM: ipp044 still not returning.  i've shutdown stdscience and i'm clearing hung nfs mounts
 * 05:40 EAM: ipp044 mounts on the storage nodes are clear, stdscience restart.  clearing computes as well
 * 05:50 EAM: looks like everything is clear, ipp044 is still down.  stdscience, stdlocal, stdlanl are all back up and running.
 * 07:30 MEH: manually had to clear exposure fault in registration for o7048g0547o
{{{
regtool -dbname gpc1 -revertprocessedexp -exp_id 863126
}}}
 * 08:00 EAM: stopping and restarting stdlocal (110k warps)
   * there were various hung jobs due to hung ipp044 mount points on the stdlocal x-nodes.  I've cleared them out.
 * 10:05 MEH: nightly finished, adding storage nodes to stdlocal storage.hosts.on
 * 11:35 MEH: ipp094 Haydn releasing for IPP use now, BBU not ready for data targeting but will put into group s5 for processing.
  * also adding for neb-host, may be ok off and on for non-targeted data 
 * 14:15 MEH: ipp090,093,095,082,080 typically have heavy loads and also seem to be doing some sort of raid verification, put to repair for now
  * is a read process, so unclear if that will help -- 080 almost finished, others still <60% since saturday midnight?
 * 15:45 MEH: modified nightly targeting in ipphosts.mhpcc.config to remove ipp083,084,086 since BBU replacement will be few more days at least -- restarted summitcopy, registration, stdsci
 * 15:55 MEH: ipp094 neb-host up but non-targeted, same for ipp083,084,086,092 -- monitor for loads but should be fine for non-targeted w/o BBU WB cache
=== Tuesday : YYYY.MM.DD ===

=== Wednesday : YYYY.MM.DD ===

=== Thursday : YYYY.MM.DD ===

=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===