Changes between Version 13 and Version 14 of PS1_IPP_CzarLog_20110103

Timestamp:

Jan 5, 2011, 2:09:34 PM (15 years ago)

Author:

bills

Comment:

--

PS1_IPP_CzarLog_20110103

@@ -19 +19 @@
 
 === Wednesday : 2011.01.05 ===
+Bill is czar today
 
- - (serge/07:40) cam revert on
- - (serge/08:39) publishing restarted
- - (bills/10:00) warp stuck lots of entries in warpPendingSkyCell book in done state. ipp049 not responding to ssh 4 warps stuck running there. Stopped everything for awhile let jobs finish. Then reset the books (warp.reset, chip.reset, etc)
- - (bills/10:51) Gavin rebooted ipp049. publish was getting lots of faults. Stopped it and asked Serge to investigate)
- - (bills/11:35) Turned off some reverts in order to debug the fault rate. Also set poll limit to 32 to reduce the load in order to get an idea whether that is the problem or not.
- - (bills/12:20) two chips failed repeatedly. Turned out to that the log files had a storage object but no instances. Fixed with neb-mv
+ * (serge/07:40) cam revert on
+ * (serge/08:39) publishing restarted
+ * 10:00 warp stuck lots of entries in warpPendingSkyCell book in done state. ipp049 not responding to ssh 4 warps stuck running there. Stopped everything for awhile let jobs finish. Then reset the books (warp.reset, chip.reset, etc)
+ * 10:51 Gavin rebooted ipp049. publish was getting lots of faults. Stopped it and asked Serge to investigate)
+ * 11:35 Turned off some reverts in order to debug the fault rate. Also set poll limit to 32 to reduce the load in order to get an idea whether that is the problem or not.
+ * 12:20 two chips failed repeatedly. Turned out to that the log files had a storage object but no instances. Fixed with neb-mv
    * Serge found the origin of the publishFailures (some runs got queued for a client with a non-existent data store)
    * turned warp off to allow the diffs to make better progress. poll limit is 64
- - (bills/12:41) turned warp back on. Still getting failures even with only a few jobs running. There is at least one corrupt camRun 152676. See http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/PS1_Operations/broken_files
- - (bills/13:29) found another corrupt file warp_id 142806 skycell.1162.062 . Increased poll limit to 128
+ * 12:41 turned warp back on. Still getting failures even with only a few jobs running. There is at least one corrupt camRun 152676. See http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/PS1_Operations/broken_files
+ * 13:29 found another corrupt file warp_id 142806 skycell.1162.062 . Increased poll limit to 128
+ * 13:45 found 2 publishRuns that were failing and reverting repeatedly 9GB of log files was the result
+ * 14:06 ran  nightly_science.pl --queue_stacks --date 2011-01-05
 
 === Thursday : 2011.01.06 ===