== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2011.07.18 ===
  
  * 11:05 heather reverted a few cam files - if they fault again the chips might be corrupt.
  * 13:30 Mark: queued up better set of ~136 MD10 z-band exposures to run through warp stage under MD10.GR0 (20110718) label for refstack
  * 14:46 CZW: ran a month's worth of cleanup commands to ensure no leftover uncleaned data is still on disk.
  * 19:00 Mark: 3PI RINGS.V0 to RINGS.V3 (SweetSpot OSS ESS MSS PI ThreePi microtest) in nightly_science.config. Messed up mergeinfo and will need to fix.
  * 21:09 CZW: restarted czarpoll
  * 21:47 CZW: restarted replication pantasks to increase job execution.
=== Tuesday : 2011.07.19 ===
  * 13:30 Mark: started MD10.refstack.20110715 for MD10.V3 y-band and more chips->warp added to MD10.GRO 
  * 14:52 CZW: restarted stdscience pantasks after pantasks_server crash.
  * 15:33 CZW: enabled detrend pantask to run dark verify against my trial dark.
=== Wednesday : 2011.07.20 ===

 * 10:00 heather restarted summitcopy, it crashed at 2am..
=== Thursday : 2011-07-21 ===
 * 10:55 Serge: After MOPS complain, killed process running for more than one hour on ippc10 (ppSub diff_id=146335)
 * 11:00 Serge: 146335 reverted with difftool {{{difftool -dbname gpc1 -revertdiffskyfile -diff_id 146335}}})
 * 11:05 Serge: reverted failed publishing entries for IPP-MOPS-TEST ({{{pubtool -revert -dbname gpc1 -client_id 5}}})
 * 18:30 Heather: Chris/Mark noticed odd behaviour with warps not getting processed. A restart of stdscience fixed this.
 * 18:30 heather: I fixed one of the MD10.GR0 chips (got corrupted, was blocking camera...), it was chip_id 259564 class_id XY34 It's reverting with bill's runchipimfile.pl script.
 * 18:52 Heather: chip XY54 class_id 299995 was also corrupted, same fix (runchipimfile.pl)
=== Friday : 2011.07.22 ===
  * 11:00 heather restarted publishing, and turned off reverts. This allowed last night's data to go through publish. There are lots of faults, it looks like pubruns involving data from 7/15 got queued (possibly after stdsci restart yesterday?)  They still fault and I need to investigate.  
  * 14:29 Serge {{{perl runchipimfile.pl --chip_id 260958  --class_id XY27}}} warp_id 225964 was failing.
  * 18:00 CZW Dropped/cleaned LAP data from 14hrs.  Queued SAS2.123 data for processing.
=== Saturday : YYYY.MM.DD ===
  * 18:00 Mark: manually reverted some failed failed MD stacks, publishing 
  * 20:00 Mark: set remaining MD10.refstacks for distribution, changed MD10.refstack.20110715,20110718 label to the primary 20110722. 
=== Sunday : YYYY.MM.DD ===
  * 11:00 Mark: manually reverted some failed publishing