Changes between Version 17 and Version 18 of PS1_IPP_Czarlog_20190311

Timestamp:

Mar 15, 2019, 2:33:38 PM (7 years ago)

Author:

tdeboer

Comment:

--

PS1_IPP_Czarlog_20190311

@@ -6 +6 @@
 === Monday : 2019.03.11 ===
 
- * MEH: simple oneliner to remove the faulted chip/warps in update.OSS to update.OSS.broke until ipp067 back online -- remember to add the user/password
+ * MEH: simple one-liner to remove the faulted chip/warps in update.OSS to update.OSS.broke until ipp067 back online -- remember to add the user/password
 {{{
 mysql -hscidbm -uXXXXX -pXXXXX gpc1 -B -e ' select chip_id,warp_id,warpRun.label,warpRun.data_group,warpRun.state,chipRun.label,chipRun.state from warpRun join warpSkyfile using (warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join chipProcessedImfile using (chip_id) where warpRun.label like "update.OSS" and warpRun.data_group like "OSS%" and warpRun.state like "update" and  ( warpSkyfile.fault>0 or chipProcessedImfile.fault>0 ) group by warp_id limit 500; ' | grep -v data_group | awk '{print "chiptool -dbname gpc1 -updaterun -set_label update.OSS.broke -chip_id  "$1" ;warptool -dbname gpc1  -updaterun -set_label update.OSS.broke -warp_id "$2 }'
@@ -104 +104 @@
 === Friday : 2019.03.15 ===
 
+ * TdB: following EDAC error warning, ippb22 and ippb17 have been put back to down in nebulous, so they can be checked out and not interfere with normal IPP operations.
+ * TdB: all exposures under the ps_ud_MOPS.broke label have been cleared. Almost all went through without issues, with a total of 6 exposures failing due to files being located on ippb22. These have been reprocessed from scratch to allow Rob to get his stamps in a timely manner. Exposures under the update.OSS.broke will now be reprocessed intermixed with the regular update processing. A summary of the process is given below:
+   * the broke labels for ps_ud_MOPS have been re-evaulated according to the following steps:
+     step 1) use an update run command to move label back from broke to regular:
+{{{
+mysql -hscidbm -uXXXXX -pXXXXX gpc1 -B -e ' select chip_id,warp_id,warpRun.label,warpRun.data_group,warpRun.state,chipRun.label,chipRun.state from warpRun join warpSkyfile using (warp_id) join fakeRun using (fake_id) joi camRun using (cam_id) join chipRun using (chip_id) join chipProcessedImfile using (chip_id) where warpRun.label like "ps_ud_MOPS.broke" and warpRun.state like "update" group by warp_id limit 200; ' | grep -v data_group | awk '{print "chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id  "$1" ;warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id "$2 }' | tcsh
+}}}
+     step 2) if the processing fails, use the revertprocessedimfile command to revert the processing and try again
+     step 3) if the processing still fails, do a full reprocessing of the image
+
+     In this instance, faults cleared from IPP monitor. However, it is important to check if that actually means the images needed to make the pstamp are in the full state. To that end, query the list of exposures Rob provided, and see if their most recent chip_ids and warp_ids are in the full state:
+{{{
+mysql -hXXXXX -uXXXXX -pippuser gpc1 -B -e 'SELECT suba.exp_name,suba.exp_id,subb.max_warpid,subc.max_warpstate,subc.max_chipid,subc.max_chipstate FROM (SELECT exp_id,exp_name FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.exp_name IN ("o5093g0045o","o5093g0054o","o5577g0554o","o5590g0733o","o5590g0740o","o5593g0599o","o5593g0622o","o5645g0512o","o5645g0513o","o5645g0529o","o5649g0305o","o5649g0321o","o5649g0322o","o5649g0339o","o5649g0340o","o5649g0356o","o5649g0357o","o5665g0138o","o5665g0156o","o5727g0153o","o5727g0176o","o5727g0200o","o5727g0223o","o6148g0466o","o6148g0489o","o6169g0172o","o6316g0103o","o6316g0114o","o6827g0269o","o6827g0289o","o6827g0309o","o6827g0329o","o6940g0066o","o7444g0312o","o7444g0330o","o7444g0348o","o7444g0366o","o7480g0136o","o7480g0155o","o7480g0174o","o7480g0193o","o7499g0102o","o7499g0120o","o7499g0138o","o7499g0156o","o7506g0103o","o7506g0122o","o7506g0141o","o7506g0160o","o7510g0106o","o7510g0124o","o7510g0142o","o7510g0160o","o7526g0096o","o7526g0118o","o7526g0140o","o7526g0162o") GROUP BY exp_id) as suba INNER JOIN (SELECT MAX(warp_id) as max_warpid,chipRun.exp_id,warpRun.state FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.camera LIKE "%GPC1%" GROUP BY chipRun.exp_id) AS subb ON (subb.exp_id = suba.exp_id) INNER JOIN (SELECT warp_id,chipRun.exp_id,warpRun.state as max_warpstate,chipRun.state as max_chipstate,chipRun.chip_id as max_chipid FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE  rawExp.camera LIKE "%GPC1%") AS subc ON (subc.exp_id = subb.exp_id AND subc.warp_id = subb.max_warpid);'
+}}}
+     This revealed that while most chip_IDs had processed to full state (1 had not), all the warp_IDs were either in update or cleaned state. The appropriate warp_IDs were subsequently processed to full by setting the skyfile to update, ensuring full-state recently processed pixel data is available for pstamps.
+
 === Saturday : 2019.03.16 ===