PS1 IPP Czar Logs for the week 2015.08.31 - 2015.09.06

(Up to PS1 IPP Czar Logs)

Monday : 2015.08.31

• 06:30 Bill: set quality = 66 for 8 diffskyfiles for skycell.1486.051 The template stack's (3187917) cmf is missing.
• 06:45 EAM: I've removed all but the w-band LAP labels from ~ipp/src/ippMonitor/czartoo/czartool/CzarDb.pm and restarted czarpoll and roboczar

Tuesday : 2015.09.01

Wednesday : 2015.09.02

Thursday : 2015.09.03

• 22:15 EAM: restarting all ipp pantasks in case we get data

Friday : 2015.09.04

• 15:15 EAM : rebooted ippb06 which had a kernel crash (load of 400)
• 15:50 MEH: clearing WS diffs from x.20150903 -- had turned off cleanup earlier in day (init.day doesn't seem to do cleanup on state new diffs which is good) but not necessary
  • diff.revert.off
  • batch execute per diff_id, skycell_id with fault 2 for %.20150903 -- use quality 66 established by Bill to distinguish from normal manual 42 in case want to know which were this case

    difftool -dbname gpc1 -updatediffskyfile -fault 0 -set_quality 66 -diff_id "$1" -skycell_id "$2
    

  • 571 skycells will be missing, 24 OSS, 34 3PI, 8 SNIa exposures delayed out of 101 exposures total.. -- will need to be manually sent to cleanup later
• 21:25 MEH: will manually queue the new LIGO obs_mode in morning along with clearing the WS diff faults --

Saturday : 2015.09.05

• 07:30 MEH: cleared remaining WSDiff faults
• 09:00 MEH: using nightly processing nodes for label LIGOtest
• 11:15 MEH: LIGOtest finished using nightly processing nodes for now
• 15:10 MEH: restarting at least stdsci pantasks -- with all the diff faults from missing data products, the counter can rapidly increase over night (of if data left uncleared)
• 16:15 MEH: Serge needs 20150905 data to not get cleaned up until he fixes the MOPS servers -- cleanup stop for now

  difftool -dbname gpc1 -updaterun -data_group OSS.20150905 -set_label OSS.20150905.mopshold
  chiptool -dbname gpc1 -updaterun -data_group OSS.20150905 -set_label OSS.20150905.mopshold
  

• 21:50 MEH: warp repeat fault 4 from cannot build growth curve

  warptool -dbname gpc1 -updateskyfile -set_quality 42 -fault 0 -skycell_id skycell.0864.025 -warp_id 1618894  
  warptool -dbname gpc1 -updateskyfile -set_quality 42 -fault 0 -skycell_id skycell.0862.030 -warp_id 1618925 
  warptool -dbname gpc1 -updateskyfile -set_quality 42 -fault 0 -skycell_id skycell.0953.043 -warp_id 1618949  
  

• 23:45 MEH: clearing 1587 skycells unable to make WS diff

Sunday : 2015.09.06

• 00:45 MEH: another diff fault (4, cannot build growth curve)

  difftool -dbname gpc1 -updatediffskyfile -set_quality 42 -fault 0 -skycell_id skycell.1044.069 -diff_id 1197676 
  

• 08:00 MEH: setting quality 66 for skycells unable to make WS diff -- 1843 skycells lost, 328 exposures cleared