PS1 IPP Czar Logs for the week 2014.08.18 - 2014.08.24

(Up to PS1 IPP Czar Logs)

Monday : 2014.08.18

• 22:30 MEH: ipp036 crashed ~15 min ago, nothing on console. power cycled and back up before it wedged processing..

Tuesday : 2014.08.19

• HAF is czar today

Wednesday : 2014.08.20

• 13:32 Bill: stopping pstamp server in order to restart and to install scripts to support stamps of exptime and expnum images.
• 18:20 MEH: using ippsXX and compute3 nodes for ~1hr for MD stacks before nightly processing

Thursday : 2014.08.21

• 00:05 MEH: stdsci really needed regular restart
  • unwant normally 20, someone set to 10 and doesn't appear to be logged but suspect to ease overloading the few nodes with disk space.. suspect some nodes may have also been turned off but again not logged
  • rate from ~30/hr back up to 50/hr? yes and was able to maintain this even with most datanodes red -- if stdsci not regularly restarted then processing known to be slowed
• 00:20 MEH: observations stopped for humidity it looks like -- restart long running summitcopy and registration as well
• 01:35 MEH: ipp036 very unhappy -- neb-host repair and out of stdsci until cleared -- leave in repair since seems to be sensitive when low disk space
• 12:00 MEH: doing long neglected pantasks_sever log archiving..
• 12:15 MEH: little processing, flipping the nebulous log on ippc02 so free space similar to that on the other ippc0x machines -- WARNING something else using diskspace on ippc02 so only 18G free while others have >20-23G and will regularly run out first

Friday : 2014.08.22

Saturday : 2014.08.23

Sunday : 2014.08.24