== PS1 IPP Czar Logs for the week 2013.06.17 - 2017.06.23 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2016.06.17 ===
 * 16:10 Bill has a pantasks running out of ~bills/m31/test using the compute3 nodes. There are 40 exposures so it should finish in a few hours. If not I will stop it once we start downloading data.
  
=== Tuesday : 2016.06.18 ===
mark is czar
 * 07:30 MEH: appears to be a summit fault, trying to clear, 68 incomplete summitcopy to go -- appears to be 503 errors from summitcopy, email sent to ps-camera, ps-obs if is due to camera stuck problem in night summary.
  * summitcopy stop until fixed -- restarting as well as registration, has been a while
  * Sidik is calling summit to try and get person to reboot pixel server t30 -- unknown how long until nightly data will be available
  * he notes o6461g0487o will also be messed up
 * 07:40 MEH: doing needed restart of stdscience while waiting for downloads. doing restarts on other long running pantasks -- distribution,stack,pstamp,publishing
 * 08:30 MEH: archiving pantask logs (relocate to ippc18.1 and bzip) to try and recover space on home disk ippc18.0.. not much recovered, only 80G free  (9%)..
 * 11:10 MEH: summitcopy back on, downloading remaining nightly data
 * 13:00 Gene taking ipp034 out of processing for few hours of DVO work 
 * 13:40 MEH:
  * o6461g0486o is MIA at summit, asking if recoverable 
  * o6461g0487o is a junk image as Sidik noted, failing chip stage of course so setting chip to drop and exposure to ignored 
 * 15:00 MEH: many misc files and directories in ~ipp, relocated many misc files to /export/ippc18.1/ipp/all_misc_junk_files_should_clean_when_ready.. more should be done as well or just rm'd by users when done with tests etc.
=== Wednesday : 2016.06.19 ===
 * 01:30 MEH: summitcopy seems to be a bit more behind than normal, ~100 exp so by dawn likely ~150 still to download. 
 * 07:10 MEH: yes, clearly slower with 140 left to download still. turning some wave1 hosts back on to try and speed up..
 * 10:45 MEH: nightly data downloaded and running through (all 3PI), will take <1hr to finish
 * people are looking into the download rate problem, to be resolved for tonight

 * 20:00 MEH: can no longer monitor processing as czar
=== Thursday : 2016.06.20 ===

=== Friday : 2016.06.21 ===

=== Saturday : 2016.06.22 ===

=== Sunday : 2016.06.23 ===