Changes between Version 7 and Version 8 of PS1_IPP_Czarlog_20190401

Timestamp:

Apr 2, 2019, 12:47:09 PM (7 years ago)

Author:

tdeboer

Comment:

--

PS1_IPP_Czarlog_20190401

@@ -36 +36 @@
  That solved the issue. Unfortunately, some exposures were faulting due to ipp127 not being available yet, at the time. I will follow-up more on these, once the rack is back up.
 
+
+ *TdB: Shortly after observing started, we noticed that processing was not proceeding at all, and an IPP hosed was called. Mark, Roger, John and myself met via zoom to solve the issue, around 21:00.
+  We noticed the issues was related to missing detrend files, which was somewhat strange since nothing changed since last week, when we did open and data went through fine. Initially, we figured it was due to some machines being down, so we turned those back to repair (ipp057, ippb17,18,21,22) and set the exposures to revert. This did not help.
+
+ Further digging into the log files revealed that there was a failed detselect command:
+{{{
+detselect -search -inst GPC1 -det_type VIDEODARK -time 2019-01-02T06:40:20.575099 -dbserver ippdb08 -dbname gpc1 -dbuser XXXXX -dbpassword XXXXX
+}}}
+ which did not return anything. Following this, we looked at:
+{{{
+detselect -show -det_type VIDEODARK -dbname gpc1 -simple
+
+981 0 chip VIDEODARK register 2013-05-25T01:15:22.000000  2011-11-01T00:00:00.000000 2019-04-01T00:00:00.000000
+}}}
+ This revealed an end_time for the videodark validity of 2019-04-01T00:00:00.000000, which was last night. hence, there was no videodark available to use. So, we had to reset the end_time of the file using:
+{{{
+dettool -updatedetrun -dbname gpc1 -set_time_end 2099-04-01T00:00:00.000000 -det_id 981
+}}}
+ This solved most of the issue, but we found chip CY05 kept failing on each of the exposures. We inspected the actual file using ds9 and it looked ok. We inspected its replication and checksum using neb-stat -validate, which also looked ok. But then we realised this was in fact a DARKTEST file, which the processing was now using, which had no end_time at all. We stopped the stdscience pantasks while investigating, since this is not a good file to use in the nightly processing.
+
+The det_id of the file was 866, and we set its end_time to yesterday to prevent it from being grabbed in the future:
+{{{
+dettool -updatedetrun -dbname gpc1 -set_time_end 2099-04-01T00:00:00.000000 -det_id 866
+}}}
+ The problem then shifted to the currently processed file being bad. So, we had to move the offending exposures to a broke label (OSS.20190402.baddet) and requeue it for processing from scratch, using: 
+{{{
+mysql -hippdb08 -uXXXXX -pXXXXX gpc1 -e "SELECT rawExp.exp_name,chipRun.chip_id,chipRun.state,chipRun.label FROM chipRun JOIN rawExp USING (exp_id) WHERE chipRun.data_group = 'OSS.20190402.baddet' " | grep -v chip_id | awk '{print "chiptool -dbname gpc1 -definebyquery -set_label OSS.nightlyscience -set_workdir neb://@HOST@.0/gpc1/OSS.nt/2019/04/02 -set_dist_group SweetSpot -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction SWEETSPOT -set_data_group OSS.20190402 -exp_name "$1}' | tcsh
+}}}
+ Then, we restarted the pantask just in case, and set it to run:
+{{{
+pantask:shutdown now
+
+./start_server.sh stdscience
+pantask:server input input
+pantask:setup 
+pantask:run
+}}}
+ Unfortunately, that re-added the update label for LAP.PV3 processing, so that need to be removed:
+{{{ 
+del.label update.LAP.PV3
+}}}
+
+ Finally, things were running through fine. But, we were well behind, since it was close to midnight. So, we decided to add more nodes to the processing, to try to catch up.
+{{{
+hosts add m0
+hosts add m1
+we do this 4 times for each
+
+set.poll 500
+}}}
+ We checked ganglia and things looked ok. So, we added 4 more sets of each. We kept an eye on:
+{{{
+~ipp/src/ipp-20141024/tools/checkexp --verbose
+}}}
+ to see if this was causing any problems upstream, but things looked good. So, we added even more nodes
+{{{
+hosts add c2
+we did this 4 times
+}}}
+ Finally,we removed the stuck exposures for ps_ud_MOPS (due to ipp127 being offline) to a broke label, to prevent them from taking up cycles:
+{{{
+warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1281048 
+warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1284676 
+warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1284689 
+warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1391868 
+warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1392030 
+warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1392058 
+warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1392250 
+warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1413405 
+warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1413407 
+warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1414496 
+warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1414496 
+warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1414848 
+warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1027567 
+warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1027638 
+warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1027645 
+warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1027695 
+warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1027697 
+warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1027705 
+warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1027711 
+warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1027781 
+warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1027785 
+warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1027828 
+}}}
+
+
 === Tuesday : 2019.04.02 ===