PS1 IPP Czar Logs for the week 2012-05-14 -- 2012-05-20

(Up to PS1 IPP Czar Logs)

Monday : 2012.05.14

Serge is czar

• 00:05 Mark: wave1 hosts not very happy when system is full use, may be overloading. manually removing 1 group from stdscience to see how that helps. ipp021 was also being killed by a ppSub using >90% RAM (neb://ipp021.0/gpc1/OSS.nt/2012/05/14/RINGS.V3/skycell.1560.018/RINGS.V3.skycell.1560.018.dif.238554)
• 06:30 Serge: Nightly processing almost complete (~50 3pi exposures). ipp002 is down. Gavin rebooted it around 7:30.
• 09:00 Serge: Nightly processing complete.
• 13:50 Serge: Ganglia hiccups pattern: restarting stdscience
• 14:20 Serge: Killed 4 ppImage that were drowning ipp039
• 15:19 Serge: ipp002 has a huge load because of apache(?)
• 17:00 Bill: stopping pstamp, distribution, and publishing in preparation for rebuilding the ippRequestServer database
• 18:00 Bill: testing re-ingest was taking too long. I will change the database tomorrow. pantasks' restarted

Tuesday : 2012.05.15

• 08:15 Bill pstamp, distribution and publishing set to stop. Waiting for mysql slave on ippc19 to catch up then we will rebuild the ippRequestServer database on ippc17 using InnoDB storage engine.

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD