Changes between Version 7 and Version 8 of PS1_IPP_Czarlog_20190510

Timestamp:

May 12, 2019, 5:35:07 PM (7 years ago)

Author:

tdeboer

Comment:

--

PS1_IPP_Czarlog_20190510

@@ -28 +28 @@
 }}}
 
-There are a lot of exps that have corresponding warps in the hold label, so the ones currently in warp update are waiting on their chip updates to finish. Send some of the warps in the hold label back to the regular label if their corresponding chip state is already in full, they can run through:
+ There are a lot of exps that have corresponding warps in the hold label, so the ones currently in warp update are waiting on their chip updates to finish. Send some of the warps in the hold label back to the regular label if their corresponding chip state is already in full, they can run through:
 {{{
 mysql -hscidbm -uX -pX gpc1 -B -e ' select chip_id,warp_id,warpRun.label,warpRun.data_group,warpRun.state,chipRun.label,chipRun.state from warpRun join warpSkyfile using (warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join chipProcessedImfile using (chip_id) where chipRun.label like "update.LAP.PV3" and chipRun.state like "full" and warpRun.label LIKE "update.LAP.PV3.lazy" group by warp_id limit 1000; ' | grep -v data_group | awk '{print "warptool -dbname gpc1 -updaterun -set_label update.LAP.PV3 -warp_id "$2 }' | tcsh
@@ -112 +112 @@
  * TdB: Machine ippb18 is now down on ganglia (though no email from nagios) and the console display log is listing EDAC errors. I put it to neb-host down and powered it off in the console (it was already reported as down anyways). Then, I emailed Haydn about it.
 
+ Fixing some missing .mk files again:
+{{{
+warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1295348
+}}}
+ Didn't work. send to reprocess:
+{{{
+chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o5231g0024o
+
+chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1344272
+warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1295348
+}}}
+
+ Also some corrupted memory blocks that need reprocessing:
+{{{
+chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o6235g0387o
+chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o6284g0320o
+
+chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1364687
+chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1413646
+warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1334220
+warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1363203
+}}}
+
+ *TdB: 18:00 we have finished going through the chip exposure that Roger queued up, but there are still ~6k warps that are lagging behind and are imbalanced. In principle it would be ok to run with just warps updating to remove the imbalance, but might as well feed in a small batch of chip+warps anyway, since warp run at a higher rate than chips. But unsure which data_group we were doing, since I could find no info on the czarlog from last week about which daa_groups Roger queued up. Looking at the query John provided we should be at data_group = LAP.PV3.20140730.20150% (but please correct me if I am wrong). Therefore, setting a batch of 1000 exp to update, using:
+{{{
+mysql -hscidbm -uX -pX gpc1 -B -e ' select exp_id from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id)  where warpRun.data_group like "LAP.PV3.20140730.20150%"  group by warp_id ; ' | grep -v exp_id |  awk '{ print "mysql -hscidbm -u ippuser -pippuser gpc1 -B -e \" select s.exp_name,s.label,s.data_group,s.chip_id,s.warp_id,s.state from ( select chip_id,warp_id,exp_name,camRun.label,camRun.data_group,warpRun.state from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where exp_id="$1" order by warp_id desc ) as s limit 1 ;\" " }' | tcsh | grep -v exp_name | grep LAP.PV3.2014 | grep cleaned | awk '{print "chiptool -dbname gpc1 -updaterun  -set_label update.LAP.PV3 -chip_id  "$4" ; chiptool -dbname gpc1 -setimfiletoupdate  -set_label update.LAP.PV3 -chip_id  "$4" ;warptool -dbname gpc1  -updaterun -set_label update.LAP.PV3 -warp_id "$5"; warptool -dbname gpc1  -setskyfiletoupdate -set_label update.LAP.PV3 -warp_id "$5 }' | head -n 1000 | tcsh
+}}}
+
 === Monday : 2019.05.13 ===