Changes between Version 71 and Version 72 of PS1_IPP_Czarlog_20121126

Timestamp:

Nov 29, 2012, 8:57:47 AM (14 years ago)

Author:

Mark Huber

Comment:

--

PS1_IPP_Czarlog_20121126

@@ -78 +78 @@
  * 16:40 MEH: Gene suggested adding the stsci nodes as hosts in stdscience and then turning them off to avoid disk/data access load spikes on those machines. Will be watching with LAP+MD01 processing.
  * 15:00 chip.off to push LAP warps for stacks until nightly science starts
-=== Thursday : YYYY.MM.DD ===
+=== Thursday : 2012.11.29 ===
  Mark is czar
  * 06:45 Bill: ipp011 is not a good nebulous host. Killed stuck registration jobs there. restarted registration. In the process of checking the burntool states
- * 07:00 MEH: ipp011 was another lm_modules/kernel issue crash [wiki:ipp011-crash-20121129]
+ * 07:00 MEH: ipp011 was another lm_sensors module/kernel issue crash [wiki:ipp011-crash-20121129] -- don't reboot until after stdscience is done. 
+  * tally of similar machines having this problem recently -- ipp011, 012, 010, 016, 013, 014, 018, may be others but not enough info added to ippXXX_log or czarlog pages to tell
  * 07:02 stopped pantasks. summit copy is totally clogged with stuck ipp011 jobs. Will restart once the jobs clear.
  * took ipp011 to repair in nebulous killed all glockfile processes
  * 07:14 registration and summit copy restarted. 
+ * 07:30 MEH: stdscience restart reveal typo in i had in pantasks_hosts.input for adding off the stsci nodes, they were back to load spiking again. fixed and restarted again. 
 === Friday : 2012.11.29 ===