PS1 IPP Czar Logs for the week 2014.04.21 - 2014.04.27

(Up to PS1 IPP Czar Logs)

Monday : 2014.04.21

Bill is czar today

• 06:20 : EAM : preparing to add ps2_tc3 to the ippdb03 mysql replication (dumped the ippdb01 database & copied to ippdb03):

  mysql> show master status;
  +-------------------+----------+--------------+------------------+
  | File              | Position | Binlog_Do_DB | Binlog_Ignore_DB |
  +-------------------+----------+--------------+------------------+
  | mysqld-bin.030242 | 44889610 |              |                  | 
  +-------------------+----------+--------------+------------------+
  

• 11:48 Bill set pantasks to stop in preparation for periodic restart
  • added ipp055 to the hosts ignore list while the raid is rebuilding. Haydn put it in repair
• 11:55 restarted stdscience, distribution, cleanup, pstamp, registration, stack, and summitcopy. Leaving stack in stopped state because I forgot to wait for jobs started by the previous instance to finish.
  • 12:12 stack set to run
• 13:50 investigated corrupt raw file neb://ipp036.0/gpc1/20120817/o6156g0277o/o6156g0277o.ota55.fits. Turns out that there were 5 instances only one of which was corrupt. Fixed this.

Tuesday : 2014.04.22

Bill is czar today

• 12:52 setting pantasks to stop in preparation for periodic restart
  • with restart ipp055 has been removed from the hosts ignore list since it has successfully finished it's RAID rebuild

Wednesday : 2014.04.23

• 07:35 MEH: clearing two fault 5 diffims for OSS, several still needed for WS

  difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.0853.066 -diff_id 542889  -fault 0
  difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.0938.000 -diff_id 542890  -fault 0
  

• 15:30 CZW: I wanted to push stacks so LAP could finish in a reasonable amount of time. I've changed the host allocation:
  • m0 => continue to be out of processing for stacking tests
  • m1 => into staticsky to work on the galactic center
  • c0x3, c1bx2, c2x3 => taken from staticsky into stack
• 18:15 MEH: remaining fault 5 WS diffim need clearing, warps only online for 5d so need to not ignore

  difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.0852.062 -diff_id 542893  -fault 0
  difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.0853.032 -diff_id 542920  -fault 0
  difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.0854.039 -diff_id 542920  -fault 0
  difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.0853.090 -diff_id 542924  -fault 0
  difftool -dbname gpc1 -updatediffskyfile -set_quality 14006 -skycell_id skycell.0853.091 -diff_id 542924  -fault 0
  

Thursday : 2014.04.24

• 09:37 Bill: load is oscillating. Time to restart stdscience. Set pantasks' to stop except for stack and staticsky.

Friday : 2014.04.25

Saturday : 2014.04.26

Sunday : 2014.04.27