Changes between Version 1 and Version 2 of PS1_IPP_Czarlog_20180723

Timestamp:

Jul 28, 2018, 11:44:27 AM (8 years ago)

Author:

Mark Huber

Comment:

--

PS1_IPP_Czarlog_20180723

@@ -5 +5 @@
 
 === Monday : 2018.07.23 ===
+ * MEH: Haydn wants to reboot ippc19 to see if fixes errors -- 
+ * MEH: start running PS2 WSdiffs during night -- stdscience default set unwant 5... needs to be bumped up to 30 or so
   
-=== Tuesday : YYYY.MM.DD ===
+=== Tuesday : 2018.07.24 ===
 
-=== Wednesday : YYYY.MM.DD ===
 
-=== Thursday : YYYY.MM.DD ===
+=== Wednesday : 2018.07.25 ===
+ * MEH: unhandled camera quality fault again due to no stars -- o8324g0525o -- manually set quality 42 and state full
+ * MEH: PS2 grid angle reduce to +-3 from +-10 degree in gpc2/psastro.config 
+ * MEH: PS2 WSdiffs running during night may need more nodes -- 
+ * MEH: PS1 zEuclid observations during full moon for next 4 nights -- looks like comment string overloaded and missing visit # -- will need to manually queue WWdiffs
 
-=== Friday : YYYY.MM.DD ===
+=== Thursday : 2018.07.26 === 
+ * MEH: manually setup PS1 zEuclid WWdiffs, manually publish single field zEuclid WWdiff to MOPS -- tried to do stage publishing in groups since not able to currently parse comment string, tedious so doing in bulk seemed to work out ok
+ * MEH: non-critical bias faults not cleared
+{{{
+2 summit faults
+  exp_name         registered      fault
+c8325h0018b   2018-07-26 07:51:23   110
+c8325h0019b   2018-07-26 07:59:53   110
+}}}
 
-=== Saturday : YYYY.MM.DD ===
+=== Friday : 2018.07.27 ===
+ * MEH: PS2 OSS pointing wrong for o8326h0033o for target_id, nightly auto-queue still made an empty WWdiff with no overlapping skycells.. have to manually queue v2-3
+ * MEH: manually queue PS1 zEuclid WWdiffs again and manually publish all to MOPS -- after clearing multiple warp faults due to problems with ippx nodes out of disk space...
+ * MEH: the ippx nodes running PS2 nightly processing were a minefield waiting to bork processing -- manually forced log rotation /var/log/messages (>14G..) and Heather helped clear out the excessive mysql data so now they should be more reliable...
+ * MEH: added PS2 masking issues to [[IPP_for_PS2]]
+ * MEH: ippc19 having issues again -- Haydn rebooted but had to power cycle, still another disk problem but binaries working again -- ganglia reporting some ippb nodes down, Gavin fixed 
+
+=== Saturday : 2018.07.28 ===
+ * MEH: Angie will try to modify the comment string for zEuclid moon observations to the format {{{Euclid r0000d30S.8326.10b.z ps1_29_1244 Day 1 visit 1}}} that should help make queuing the WWdiffs easier -- leaving auto-queuing off for Bright.nightlyscience WWdiff regardless
 
 === Sunday : YYYY.MM.DD ===