Changes between Version 12 and Version 13 of PS1_IPP_Czarlog_20160328

Timestamp:

Mar 31, 2016, 12:51:42 AM (10 years ago)

Author:

Mark Huber

Comment:

--

PS1_IPP_Czarlog_20160328

@@ -19 +19 @@
  * 22:00 HAF: nebulous hates me... all sorts of nebulous errors (continuing), investigating them now...
  * 22:37 CZW: o7478g0215o wasn't completing because the pztool -copydone command was hung waiting for the gpc1 database (based on looking at the distribution of running times in the summitcopy pantasks).  Killed this command, the job failed, and was properly retried.
-=== Thursday : YYYY.MM.DD ===
-
+=== Thursday : 2016.03.31 ===
+ * 00:46 MEH: ipp008,012,013,037 down for >30 minutes... seems like the bank of machines that sometimes power cycles on own but usually it is just ipp013 that stalls.. retry a power cycle on all and see if can recover -- otherwise all are in repair and processing should be recoverable setting neb-host down and killing all the outstanding tasks..
 === Friday : YYYY.MM.DD ===