== PS1 IPP Czar Logs for the week 2014.01.20 - 2014.01.26 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2014.01.20 ===
 MLK
 * 07:30 MEH: nightly finishing up okay
  
=== Tuesday : 2014.01.21 ===
mark is czar
 * 07:00 MEH: nightly slowly finishing, stdsci needs a regular restart later today, raising poll to 500 to help keep fully loaded
 * 09:00 MEH: regular restart of many pantasks for the week -- registration,summitcopy,pstamp,update,distribution,publishing,stack,stdsci
 * 09:30 appears to be a hanging QUB stamp 330715
   * 11:15 Bill: modified pstamp_checkdependent.pl to detect that input warp had quality error on update and fault the job
   * 14:00 MEH: manually set warp 879292 skycell.0803.024 quality to 0 and tried to update again, sets quality 8006 again 
   * 15:00 MEH: manually set diff_id 509887 skycell.0803.024 to quality 42 then since will never update either
 * 17:00 Haydn fixed mobo for ipp046, neb-host repair but still out of processing
 * 17:05: MEH: update neb-host status
  * ippb01 back to repair
  * stsci00,01,02 repair in prep for move tomorrow
  * stsci03-09 back up since checkout okay
  * ipp031,032 must remain down until all stsci machines moved
=== Wednesday : 2014.01.22 ===
Bill is czar today
 * 05:45 set staticsky and cleanup to stop in preparation for the ippc18 shutdown and move
 * 06:15 shut down all pantasks except staticsky, pstamp, and update
 * 09:00 all processes shut down.
 * 10:00 machines shut down
 * 11:35 ippc18 booted
 * 12:28 started roboczar and czarpoll scripts on ippc11
 * ippc17 and the data store is up c30 is not so the postage stamp server apace is unavailable
 * 14:32 started up the pantasks including ~ippsky/staticsky fullforce jobs are falling over because of stsci00-02 being down for moving. pstamp set to stop waiting for the working directories to be available.
=== Thursday : 2014.01.23 ===
Bill is czar today
 * 08:30 prepared for cab7 move. Down are ippc10, c12-c16, ippc19, db03, and ipp049-053
 * 10:19 restarted pantasks. Moved stdscience to ippc02, and the servers usually on ippc15 to ippc03
 * 10:35 changed czarpoll config file to point to scidbm (db01) instead of scidbs(db03)
   * note ippmonitor is on db03 so is not available. There is old version on ippdb01 mostly works, but doesn't show staticsky
 * 11:18 ippsky/staticsky pantasks restarted
 * 14:20 turned off staticsky tasks in the staticsky pantasks. I want to allow my fullforce test to finish.
 * 14:30 cab 7 machines are back up. Set ipp049-53 to nebulous repair state. (There were a large number of chips failing because both copies of a detrend image were in that cabinet.
 * 14:42 restarted czarpoll with nominal configuration
 * 16:00 fixed about 40 chips whose burtool tables have gone missing. All class_ids were XY60 and XY61
 * 16:15 set pantasks to stop
 * 16:25 restarted distribution, summitcopy, and publishing on their regular system ippc15 
 * 16:30 restarted stdsciene on it's usual system ippc16
=== Friday : 2014.01.24 ===
The cabinet with the nebulous apache servers is being moved today.
 * 07:40 Bill shut down pstamp pantasks and set ippsky/staticsky and stack to stop 
  * removed LAP label from stdscience added a set of compute3 nodes to work with the remaining nightly science processing
 * 07:57 Bill all pantasks stopped except for stdscience, publishing and distribution which each have a bit of work left
 * 08:25 CZW shut down all pantasks.
 * 17:00 CZW brought all pantasks back up.  ippc02 is commented out in the .tcsh for ipp, as it did not come up cleanly after the move.
=== Saturday : 2014.01.25 ===

=== Sunday : 2014.01.26 ===