Changes between Version 10 and Version 11 of PS1_IPP_Czarlog_20190218

Timestamp:

Feb 21, 2019, 12:26:43 PM (7 years ago)

Author:

tdeboer

Comment:

--

PS1_IPP_Czarlog_20190218

@@ -28 +28 @@
  * MEH: large warp update process continuing to run in ~ippmops/stdscience on the ippc31-c63, ipps00-14, ipp067-ipp123 nodes 
   * ps_ud% chips and warps label changed to update.hold20190219 so not to be cleaned by accident
+
+=== Wednesday : 2019.02.20 ===
 
  * TdB: John, Roger and myself went through the logfiles to figure out what went wrong yesterday with ipp113. We looked through the reports for ipp113 on Ganglia, noticing the spiking load after 18:00 until 20:00. We went into the serial console using:
@@ -83 +85 @@
 }}}
 
+=== Thursday : 2019.02.21 ===
+ * TdB: ipp125 experienced stalling after high load and high wait yesterday night. messages in /var/log/messages showed some issues related to the swapper and the raid. Conferring with Haydn, the raid looks fine and a reboot is warranted.
+   We went into the pantasks_client in ippqub:stdscience_ws to stop all the current processing:
+{{{
+pantasks_client
 
+stop
+}}}
+  To check that all current processing was properly synced, we did:
+{{{
+sudo sync
+}}}
+  This showed nothing, indicating everything is synced correctly. Following this, we logged in to the machine using the serial console and did a regular reboot using:
+{{{
+sudo reboot -f
+}}}
+  boot messages indicate the Superblock last mount time was in the future and the clock was not set. Haydn confirmed the system clock does not automatically set on this motherboard/kernel combination, and set the clock manually using:
+{{{
+ ntpdate -u galileo
+}}} 
+  which fixes the clock issue. The machine has subsequently been set back to repair to monitor its activity and load levels.
 
-=== Wednesday : YYYY.MM.DD ===
+ * TdB: several machines need a kernel update, among which ippx052. We removed it for current processing by going into the ps2 stdscience pantasks_client at ~ippps2/stdscience and doing:
+{{{
+controller status
+controller host off ippx052 (6 times)
+controller host delect ippx052 (6 times)
+}}}
+ After this, I let Gavin know the machineis ready for upgrade.
 
-=== Thursday : YYYY.MM.DD ===
 
 === Friday : YYYY.MM.DD ===