PS1 IPP Czar Logs for the week 2017-11-06 - 2017-11-13

(Up to PS1 IPP Czar Logs)

Monday : 2017-11-06

Tuesday : 2017-11-07

• 12:00 CZW: Starting MD database based shuffle check, using ~ipptest/replication as the pantasks server location (20 jobs using stare nodes). Should have minor impact as only newly created files/repaired files will receive md5sum calculations.

Wednesday : 2017.11.08

• MEH: extra QUB processing running on nightly nodes once nightly processing finished for next few days
• MEH: ipp121 and ippc118 not back up after afternoon failure of AC@ITC -- tried power cycle and no boot, power off in console and neb-host down for ipp121 and turning ippc118 in nightly pantasks
• MEH: ippc34,c39,c42,c43,c51 -- gen prot faults or stalled nfs requiring reboot -- ippc43 some reason doing full mem scan by default and takes a very long time
• MEH: ippc58 kernel panic -- power cycled and back up again

Thursday : 2017.11.09

• MEH: ippc36 gen prot fault and unresponsive -- power cycled and back up again
• MEH: ippc33,c40 as well -- power cycled and back up again
• 12:20 CZW: I'm going to power cycle stare02 and stare04, which have stopped responding to ssh.
  • And also stare00, which I didn't notice was doing the same thing.
• MEH: more compute nodes with kernel panics and gen prot faults showing up and need power cycling -- c32,c34,c35,c38,c44,c45,c46,c47,c48,c49,c50,c52,c56,c57
  • ippc39,c43,c51 now twice <24hr

Friday : 2017.11.10

• MEH: more GPF forcing power cycle on c30,c32,c37,c42,c45,c49,c53 -- ippc32,c45,c49 also needing reboot w/in <24hr with minor update processing load

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD