PS1 IPP Czar Logs for the week 2017.02.13 - 2017.02.19

(Up to PS1 IPP Czar Logs)

Monday : 2017.02.13

• 11:00 EAM : ippmonitor shows 2 camera runs are not yet done. looking at pantasks.stdout.log, they have been failing since friday night. I do not immediately see the cause in the logs for the jobs. I'll note them here so we can manually re-run, but I will mark these cam_ids as bad quality:

  camera_exp.pl --exp_tag o7795g0123o.1202244 --cam_id 1868106 --camera GPC1 --outroot neb://any/gpc1/OSS.nt/2017/02/11//o7795g0123o.1202244/o7795g0123o.1202244.cm.1868106 --redirect-output --run-state new --reduction SWEETSPOT --dbname gpc1 --verbose
  camera_exp.pl --exp_tag o7795g0124o.1202246 --cam_id 1868107 --camera GPC1 --outroot neb://any/gpc1/OSS.nt/2017/02/11//o7795g0124o.1202246/o7795g0124o.1202246.cm.1868107 --redirect-output --run-state new --reduction SWEETSPOT --dbname gpc1 --verbose
  

  update : actually, since the camRuns were already reverted and since the chips have been cleaned, I've set the state of the camRuns to 'drop' to remove them from the ippMonitor list:

  camtool -dbname gpc1 -updaterun -exp_name o7795g0123o -set_state drop
  camtool -dbname gpc1 -updaterun -exp_name o7795g0124o -set_state drop
  

• 19:30 EAM : stopping & restarting pantasks

Tuesday : YYYY.MM.DD

Wednesday : 2017-02-15

• 15:15 CZW: Haydn is taking ippc65 down to fix a bad drive.
• 16:10 CZW: Restarting IPP pantasks.
• 17:00 CZW: Haydn is taking down ippc69, not ippc65, for the bad drive fix.

Thursday : 2017-02-16

• 9:20 HAF requeued these at the request of Serge. Needs to be debugged why they were not handled by nightlyscience.

  difftool -dbname gpc1 -definewarpwarp -exp_id 1204352 -template_exp_id 1204358 -backwards -set_workdir neb://@HOST@.0/gpc1/ESS.nt/2017/02/16 -set_dist_group SweetSpot -set_label ESS.nightlyscience -set_data_group ESS.20170216 -set_reduction SWEETSPOT -simple -rerun
  difftool -dbname gpc1 -definewarpwarp -exp_id 1204351 -template_exp_id 1204361 -backwards -set_workdir neb://@HOST@.0/gpc1/ESS.nt/2017/02/16 -set_dist_group SweetSpot -set_label ESS.nightlyscience -set_data_group ESS.20170216 -set_reduction SWEETSPOT -simple -rerun
  difftool -dbname gpc1 -definewarpwarp -exp_id 1204355 -template_exp_id 1204362 -backwards -set_workdir neb://@HOST@.0/gpc1/ESS.nt/2017/02/16 -set_dist_group SweetSpot -set_label ESS.nightlyscience -set_data_group ESS.20170216 -set_reduction SWEETSPOT -simple -rerun
  difftool -dbname gpc1 -definewarpwarp -exp_id 1204354 -template_exp_id 1204360 -backwards -set_workdir neb://@HOST@.0/gpc1/ESS.nt/2017/02/16 -set_dist_group SweetSpot -set_label ESS.nightlyscience -set_data_group ESS.20170216 -set_reduction SWEETSPOT -simple -rerun
  

• 12:26 CZW: I'm setting ipp105 to up in nebulous. This should have no effect on processing, but should distribute shuffle load slightly.

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD