== PS1 IPP Czar Logs for the week 2012-05-14 -- 2012-05-20 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2012.05.14 ===
Serge is czar

 * 00:05 Mark: wave1 hosts not very happy when system is full use, may be overloading. manually removing 1 group from stdscience to see how that helps. ipp021 was also being killed by a ppSub using >90% RAM (neb://ipp021.0/gpc1/OSS.nt/2012/05/14/RINGS.V3/skycell.1560.018/RINGS.V3.skycell.1560.018.dif.238554)
 * 06:30 Serge: Nightly processing almost complete (~50 3pi exposures). ipp002 is down. Gavin rebooted it around 7:30.
 * 09:00 Serge: Nightly processing complete.
 * 13:50 Serge: Ganglia hiccups pattern: restarting stdscience
 * 14:20 Serge: Killed 4 ppImage that were drowning ipp039
 * 15:19 Serge: ipp002 has a huge load because of apache(?)
 * 17:00 Bill: stopping pstamp, distribution, and publishing in preparation for rebuilding the ippRequestServer database
 * 18:00 Bill: testing re-ingest was taking too long. I will change the database tomorrow. pantasks' restarted


=== Tuesday : 2012.05.15 ===
 * 08:15 Bill pstamp, distribution and publishing set to stop. Waiting for mysql slave on ippc19 to catch up then we will rebuild the ippRequestServer database on ippc17 using InnoDB storage engine.
 * 10:11 CZW: Restarting stdscience to hopefully gain some speed and track down jobs taking longer than they should.
=== Wednesday : YYYY.MM.DD ===

=== Thursday : YYYY.MM.DD ===

=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===