PS1 IPP Czar Logs for the week 2014.09.22 - 2014.09.28

(Up to PS1 IPP Czar Logs)

Monday : 2014.09.22

• 09:35 EAM : processing went OK last night, but there are a number of outstanding failures related to glockfile failures. I am stopping processing to clear out these lock issues.
• 10:40 EAM : I needed to reboot ipp058 so it could run glockfile on remote machines. nightly science processing then finished and I re-started stdlocal
• 13:15 EAM : ipp013 crashed. I power cycled it and it came back up with help from Haydn (BIOS setting?)
• 22:33 EAM : ipp013 was overloaded and crashed again. i rebooted it, but needed to take it out of all pantasks. i am stopping and restarting standard pantasks. i'm also restarting the ippdb01 db which has been extra sluggish.

Tuesday : YYYY.MM.DD

• HAF: fun times after gpc1 reboot - restarted czartool
• HAF: requeing manually diff for serge:

  o6923g0113o 	FAIL (Diff1 stage) 	OSSR.R19S3.4.Q.w ps1_20_5599 visit 3
  o6923g0115o 	FAIL (Diff1 stage) 	OSSR.R19S3.4.Q.w ps1_20_2310 visit 3
  o6923g0130o 	FAIL (Diff1 stage) 	OSSR.R19S3.4.Q.w ps1_20_5599 visit 4
  o6923g0132o 	FAIL (Diff1 stage) 	OSSR.R19S3.4.Q.w ps1_20_2310 visit 4
  

used these commands

[heather@ippc18 ~]$ difftool -dbname gpc1 -definewarpwarp -exp_id 798994 -template_exp_id 799008 -backwards -set_workdir neb://@HOST@.0/gpc1/OSS.nt/2014/09/23 -set_dist_group SweetSpot -set_label OSS.nightlyscience -set_data_group OSS.20140923 -set_reduction SWEETSPOT -simple -rerun 
597963 new neb://@HOST@.0/gpc1/OSS.nt/2014/09/23 OSS.nightlyscience OSS.20140923 SweetSpot SWEETSPOT  2014-09-23T21:51:33.763077 RINGS.V3 T T 0  0.000000 nan nan nan nan 1 
597964 new neb://@HOST@.0/gpc1/OSS.nt/2014/09/23 OSS.nightlyscience OSS.20140923 SweetSpot SWEETSPOT  2014-09-23T21:51:33.763077 RINGS.V3 T T 0  0.000000 nan nan nan nan 1 
[heather@ippc18 ~]$ difftool -dbname gpc1 -definewarpwarp -exp_id 798992  -template_exp_id 799009 -backwards -set_workdir neb://@HOST@.0/gpc1/OSS.nt/2014/09/23 -set_dist_group SweetSpot -set_label OSS.nightlyscience -set_data_group OSS.20140923 -set_reduction SWEETSPOT -simple -rerun -pretend
1041931 RINGS.V3 798992 OSS.20140923 1041973
[heather@ippc18 ~]$ difftool -dbname gpc1 -definewarpwarp -exp_id 798992 -template_exp_id 799009 -backwards -set_workdir neb://@HOST@.0/gpc1/OSS.nt/2014/09/23 -set_dist_group SweetSpot -set_label OSS.nightlyscience -set_data_group OSS.20140923 -set_reduction SWEETSPOT -simple -rerun 
597965 new neb://@HOST@.0/gpc1/OSS.nt/2014/09/23 OSS.nightlyscience OSS.20140923 SweetSpot SWEETSPOT  2014-09-23T21:51:59.973567 RINGS.V3 T T 0  0.000000 nan nan nan nan 1 

Wednesday : YYYY.MM.DD

• HAF registration got stuck at like 2am - I kicked it. It's 7:20am, and we still have 60 images to download. Why is it so sloooooooow?

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD