PS1 IPP Czar Logs for the week 2010.01.24 - 2010.01.30

(Up to PS1 IPP Czar Logs)

Monday : 2011.01.24

• eam : warp seemed to be slow: no progress for about 1 hour. the book was full of DONE runs. I stopped processing, ran 'process_cleanup warpPendingSkyCell' to clear the book, and restarted processing. It ran fine after that
• eam : burntool seemed to have gotten stuck. I looked on the new burntool state ippMonitor page and found one of the imfiles did not seem to be making progress. Looking at the pantasks (control status), I noticed that the job for that cell had been running for a very long time (>500 sec). I went to the machine where it was running and noticed that it was hanging on access /data/ipp033.0 (which crashed over the weekend). I used force.umount to clear the mount point, and things moved along from there
• bills 15:53 Removed ipp053 from distribution host lists and restarted distribution pantasks. It seemed sluggish anyways.
• bills 16:00 cleared some magicDSRun revert faults. I need to automate this!
• bills 16:09 updated magic_destreak_cleanup.pl to *not* delete the original uncensored diff stage cmf files.
• bills 16:10 set label STS.201009 back to active.
• bills 16:25 Executed stacktool -updaterun -set_state drop -stack_id 216256 -set_note 'fails due to problem in ticket 1427'
• bills 19:37 several faults have appeared setting STS.201009 back to inactive
• bills 21:19 still lots of faults. I suspect that the rsync processes running on this node are related. I did neb-host --host ipp008 --state repair

Tuesday : 2011.01.25

Bill is czar today

• 04:21 Many many faults. ipp012 filesystem is read-only from many nodes. ssh is rejected. Stopping all processing for a few minutes to investigate.

Wednesday : 2011.01.26

Thursday : 2011.01.27

Friday : 2011.01.28

Saturday : 2011.01.29

Sunday : 2011.01.30