Changes between Version 43 and Version 44 of PS1_IPP_CzarLog_20120312

Timestamp:

Mar 13, 2012, 5:29:54 PM (14 years ago)

Author:

Mark Huber

Comment:

--

PS1_IPP_CzarLog_20120312

@@ -49 +49 @@
  * 12:40 nightly science cam/warp/stack faults mostly due to processed chip data needed that lives on ipp015. same goes for LAP so LAP is effectively stalled.
  * 12:50 replication pantasks back running, target.on only.
- * 13:00 Gavin setup access to the power control for us on compute3 nodes
+ * 13:00 Gavin setup access to the power control for us on compute3 nodes.
  * 14:10 Serge: Restarted replication on ippdb03. {{{CHANGE MASTER TO MASTER_LOG_FILE='mysqld-bin.020936', MASTER_LOG_POS=73345738;}}}
  * 14:30 registration has been very unhappy. ipp011 often having lockd server not responding for ipp008,ipp014 so set those two to repair and will watch with tonight's data.
- * 15:00 all data downloaded and registered from last night, finally. There will be some data stuck (3 MD06 warps, 3 MD07 stacks, 14 3PI warps, 7 3PI camera, 1 3PI diff) that are waiting for access to intermediate processing products on ipp015.
+ * 15:00 all data downloaded and registered from last night, finally. There will be some data stuck (3 MD06 warps, 3 MD07 stacks, 14 3PI warps, 7 3PI camera, 1 3PI diff) that are waiting for access to intermediate processing products on ipp015. Turning chip/camera/warp.revert.off for now and will toggle manually every so often as remaining science data moves through.
  * 16:00 extra LAP run queued to see if will process without needing any past processed products from ipp015. 
  * 16:30 Chris bumped up the stdscience unwant=10 again since it was reset back to 5 after the pantasks restart. May just remove ipp015 from pantasks (rather than off) tonight if still running slow waiting for targeted ipp015 jobs to be triggered remotely.