PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

Extra/Non-standard Processing

Continuing HAF suggestion to improve communication, so we know what's going on better -- a list in the czar pages additional (non-standard) processing - so that we all know what's going on.

Daily Czaring:

• currently there is a modified ops tag running diffs (WS labels only) as ippmops under ~ippmops/src/stdscience on ippc29 -- if problems (Njobs>100k, power loss on ippc29 etc), it will need to be restarted like a normal nightly processing pantasks

  ./start_server.pl stdscience
  

  • even with the modified ops tag, there are still various files MIA in nebulous and as such requires the daily czar to check and clear them as has been discussed before

(Up to PS1 IPP Czar Logs)

Monday : 2015.09.21

• 08:40 EAM : looks like we had a power event on cab1 : all machines in that cabinet rebooted spontaneously in the past 20 min. ipp013 is still coming up slowly.
• 18:15 EAM : we lost all power around 9:30 this morning. UPS failure. it took until around noon for power to come back. I am now restarting all ipp pantasks
• 22:45 MEH: restarted czarpoll and manually added WS while in another pantasks, but found labels not showing up. rebuild ippMonitor on db03 and fix it and cleared all the old labels in ippMonitor not needed. seems now new labels in stdsci actually show up on ippMonitor when added too..
  • nebdiskd started on ippdb08
  • WS diff pantasks run under ippmops stdscience also needed to be started

Tuesday : 2015.09.22

• 11:30 MEH: c2 nodes being used for extra processing for couple hours -- off in stdsci
  • c2 nodes back on in ipp:stdscience and ippmops:stdscience

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD