Changes between Version 15 and Version 16 of PS1_IPP_Czarlog_20190426

Timestamp:

May 1, 2019, 2:38:56 PM (7 years ago)

Author:

fairlamb

Comment:

--

PS1_IPP_Czarlog_20190426

@@ -538 +538 @@
     SELECT * FROM diffInputSkyfile WHERE warp1 = 2085046 AND warp2 = 2085082 LIMIT 100;
 }}}
-and here:
-{{{
-    http://datastore.ipp.ifa.hawaii.edu/IPP-MOPS-TEST/
-}}}
-
-Bedtime! Will add some more details on Wednesday.
+and also with the datastore ipp IPP-MOPS-TEST
+
+UPDATE: A new and improved way of getting all the info:
+{{{
+mysql -hscidbm --uXXXXXX -pXXXXXXX gpc1 -B -e "SELECT 
+selchunk.chunk,selchunk.object,MAX(CASE WHEN selchunk.visit=1 THEN 
+selchunk.warp_id ELSE 0 END) as warp1,MAX(CASE WHEN selchunk.visit=2 
+THEN selchunk.warp_id ELSE 0 END) as warp2,MAX(CASE WHEN 
+selchunk.visit=3 THEN selchunk.warp_id ELSE 0 END) as warp3,MAX(CASE 
+WHEN selchunk.visit=4 THEN selchunk.warp_id ELSE 0 END) as 
+warp4,MAX(CASE WHEN diffchunk.visit=1 THEN diffchunk.diff_id ELSE 0 END) 
+as diff1,MAX(CASE WHEN diffchunk.visit=2 THEN diffchunk.diff_id ELSE 0 
+END) as diff2,MAX(CASE WHEN diffchunk.visit=3 THEN diffchunk.diff_id 
+ELSE 0 END) as diff3,MAX(CASE WHEN diffchunk.visit=4 THEN 
+diffchunk.diff_id ELSE 0 END) as 
+diff4,selchunk.workdir,selchunk.label,selchunk.data_group,selchunk.reduction 
+FROM (SELECT SUBSTRING_INDEX(comment, ' ',1) AS 
+chunk,SUBSTRING_INDEX(comment, ' ',-1) AS 
+visit,object,warp_id,rawExp.workdir,chipRun.label,chipRun.data_group,rawExp.reduction 
+FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) 
+JOIN camProcessedExp USING (cam_id) JOIN chipRun USING (chip_id) JOIN 
+rawExp USING (exp_id) WHERE rawExp.exp_name LIKE 'o%' AND rawExp.dateobs 
+LIKE '`date -u "+%Y-%m-%d"`%' ORDER BY warp_id DESC) as selchunk LEFT 
+JOIN (SELECT SUBSTRING_INDEX(comment, ' ',1) AS 
+chunk,SUBSTRING_INDEX(comment, ' ',-1) AS visit, object,warp_id,diff_id 
+FROM diffRun JOIN diffInputSkyfile USING (diff_id) JOIN warpRun ON 
+(warp1=warp_id) JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) 
+JOIN camProcessedExp USING (cam_id) JOIN chipRun USING (chip_id) JOIN 
+rawExp USING (exp_id) WHERE rawExp.exp_name LIKE 'o%' AND rawExp.dateobs 
+LIKE '`date -u "+%Y-%m-%d"`%' AND stack2 IS NULL GROUP BY warp_id) as 
+diffchunk ON selchunk.warp_id=diffchunk.warp_id GROUP BY 
+selchunk.chunk,selchunk.object;" 
+}}}
+
+A simple query to see the structure of the diffRun table. This is important as it shows the style required when running the difftool command:
+{{{
+SELECT data_group, diff_id, dist_group, label, workdir, reduction FROM  diffRun WHERE data_group LIKE "%20190501";
+
+EXAMPLE:
+| OSS.20190501         | 1758779 | SweetSpot  | OSS.nightlyscience          | neb://@HOST@.0/gpc1/OSS.nt/2019/05/01                      | SWEETSPOT    | 
+| OSS.20190501         | 1758780 | SweetSpot  | OSS.nightlyscience          | neb://@HOST@.0/gpc1/OSS.nt/2019/05/01                      | SWEETSPOT    | 
+| OSS.20190501         | 1758781 | OSS.WS     | OSS.WS.nightlyscience       | neb://@HOST@.0/gpc1/OSS.WS.nightlyscience/2019/05/01       | SWEETSPOT_WS | 
+| OSS.20190501         | 1758782 | OSS.WS     | OSS.WS.nightlyscience       | neb://@HOST@.0/gpc1/OSS.WS.nightlyscience/2019/05/01       | SWEETSPOT_WS | 
+}}}
+So you can see there is a particular format to the workdir, different spellings of sweetspot for the reduction and dist_group etc.
+
+
 
 
 === Wednesday : 2019.05.01 ===
 
+ * EAM? 08:00 : running test processing on ippc128 - ippc150 as user ipptest (label is UNIONS.DR2.20190430).
+
+ * CCL: touch /data/ipp123.0/nebulous/39/3c/10949746515.gpc2:20190117:o8500h0253o:o8500h0253o.ota64.fits time stamp changed
+
+
+JRF: The long night has come to a close, and I have modified last nights notes to be a little more coherent (and a few warnings have been included).
+
+
+''' stuck stamp requests ''' by JRF
+
+Do the usual pstamp check:
+{{{
+pstamptool -dbname ippRequestServer -dbserver ipp113 -pendingdependent -simple
+}}}
+To get a list of the problem warps and skycells.
+
+
+Just need to set them to update since they have been cleaned away
+{{{
+chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 
+warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_MOPS -warp_id 
+}}}
+
+Run a typical query to see how they are doing:
+{{{
+SELECT 
+  rawExp.exp_name,rawExp.exp_id,
+  chipRun.chip_id, chipRun.state chip_state, chipRun.label chip_label,
+  warpRun.warp_id,warpSkyfile.skycell_id,warpRun.state,warpRun.label
+ FROM 
+  warpSkyfile, warpRun, fakeRun, camRun, chipRun, rawExp 
+ WHERE 
+  warpSkyfile.warp_id = warpRun.warp_id AND warpRun.fake_id = fakeRun.fake_id AND fakeRun.cam_id = camRun.cam_id AND camRun.chip_id = chipRun.chip_id AND chipRun.exp_id = rawExp.exp_id 
+ AND warpRun.warp_id IN (1610722,1610657,1610652,1610623,1610130,1610117,1610095,1610077)
+ GROUP BY warpRun.warp_id
+ ;
+ }}}
+
+However, there is an issue that the chips are not updating as their state is stuck as "wait". So I've tried to update the run and set the state:
+{{{
+    chiptool -dbname gpc1 -updaterun -state wait -set_state update -set_label ps_ud_MOPS -chip_id 1663243
+    chiptool -dbname gpc1 -updaterun -state wait -set_state update -set_label ps_ud_MOPS -chip_id 1663225
+    chiptool -dbname gpc1 -updaterun -state wait -set_state update -set_label ps_ud_MOPS -chip_id 1663205
+    chiptool -dbname gpc1 -updaterun -state wait -set_state update -set_label ps_ud_MOPS -chip_id 1663186
+    chiptool -dbname gpc1 -updaterun -state wait -set_state update -set_label ps_ud_MOPS -chip_id 1662696
+    chiptool -dbname gpc1 -updaterun -state wait -set_state update -set_label ps_ud_MOPS -chip_id 1662677
+    chiptool -dbname gpc1 -updaterun -state wait -set_state update -set_label ps_ud_MOPS -chip_id 1662658
+    chiptool -dbname gpc1 -updaterun -state wait -set_state update -set_label ps_ud_MOPS -chip_id 1662639
+}}}
+Note that I search for the ones that say wait, and have updated their label to update.
+
+They are now beginning to error in the chiptool stage:
+{{{
+burntool state vs burntoolStateGoodUpdate : -14 vs 13
+Running [/data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/neb-repair neb://ipp088.0/gpc1/20150801/o7235g0278o/o7235g0278o.ota31.fits]...
+Reading from filehandle failed at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/neb-repair line 78.
+Unable to attempt repair: neb://ipp088.0/gpc1/20150801/o7235g0278o/o7235g0278o.ota31.fits 1280 at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/chip_imfile.pl line 834
+    main::my_die('Unable to attempt repair: neb://ipp088.0/gpc1/20150801/o7235g...', 951756, 1662658, 'XY31', 2) called at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/chip_imfile.pl line 358
+}}}
+I think it might be best to reprocess the lot, then send these broken jobs to clean (be sure to check the reduction)
+{{{
+chiptool -dbname gpc1 -definebyquery -set_label mopsfix  -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenOSS.20190501  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenOSS.20190501 -set_reduction SWEETSPOT -exp_name 
+}}}
+
+
+Check they have reprocessed (you'll need to query the exposure idea because of the reprocess)
+And then clean up the ones stuck under the ps_ud_WEB label:
+{{{
+    chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label ps_ud_MOPS -chip_id 
+    warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label ps_ud_MOPS -warp_id 
+}}}
+
+
+
+
+''' Restoring computer node functionality''' by JRF
+
+The compute nodes require a specific set of files to be on them and some 'tmp' directories to temporarily store data while processing. The general process is copy across all the static files that are needeed, then sort out all the links between the drives and '/local/'
+
+First sure there is an ipp directory to put things in on volume .0, then rsync across the data files required (and set permissions):
+{{{
+  mkdir -p /export/`hostname`.0/ipp
+  rsync -auv /data/ippc64.0/ipp/ /export/`hostname`.0/ipp
+  chown -R ipp:users /export/`hostname`.0/ipp
+  chmod -R 775 /export/`hostname`.0/ipp
+}}}
+
+Then, replace the symlink for 'tmp' in the /local/ directory to point to volume .0:
+{{{
+  rm /local/ipp/tmp
+  ln -s /export/`hostname`.0/ipp /local/ipp
+}}}
+
+Finally, make a 'tmp' directory on volume .1, and have the 'tmp' dir from .0 link to it.
+{{{
+  mkdir -p /export/`hostname`.1/ipp/tmp
+  chown -R ipp:users /export/`hostname`.1/ipp/tmp
+  chmod -R 775 /export/`hostname`.1/ipp/tmp
+  ln -s /export/`hostname`.1/ipp/tmp /export/`hostname`.0/ipp/tmp
+}}}
+'''Note:''' the setup can be slightly different between nodes e.g. not all nodes have volume .1
+
+
+Now the real important part is to actually test the node to see if it works.
+
+Add it back to stdscience:
+{{{
+  ~ippitc/stdscience> pantasks_client
+  pantasks> controller host on ippc91
+}}}
+This will allow one connection to be made to ippc91. Do it multiple times to add more, but before that first let it handle a job or two and keep an eye out for errors (check pantasks.stdout.log).
+
+Once satisfied add the host more times to pantasks_client, and it can be added back to the hosts list file if removed {{{ ~ippitc/ippconfig/pantasks_hosts.input}}}.
+
+
+'''More stamp fun:''' by JRF
+
+Check what the problem warps are:
+{{{
+pstamptool -dbname ippRequestServer -dbserver ipp113 -pendingdependent -simple
+}}}
+
+Get all the details
+{{{
+SELECT 
+  rawExp.exp_name,rawExp.exp_id,
+  chipRun.chip_id, chipRun.state chip_state, chipRun.label chip_label,
+  warpRun.warp_id,warpSkyfile.skycell_id,warpRun.state warp_state, warpRun.label warp_label
+ FROM 
+  warpSkyfile, warpRun, fakeRun, camRun, chipRun, rawExp 
+ WHERE 
+  warpSkyfile.warp_id = warpRun.warp_id AND warpRun.fake_id = fakeRun.fake_id AND fakeRun.cam_id = camRun.cam_id AND camRun.chip_id = chipRun.chip_id AND chipRun.exp_id = rawExp.exp_id 
+ AND 
+  warpRun.warp_id IN (1921752,1921730,1921712,1921694,1206567)
+ GROUP BY warpRun.warp_id
+ ;
+
++-------------+---------+---------+-------+---------+------------------+---------+--------------+
+| exp_name    | exp_id  | chip_id | state | warp_id | skycell_id       | state   | label        |
++-------------+---------+---------+-------+---------+------------------+---------+--------------+
+| o6889g0624o |  781745 | 1250507 | wait  | 1206567 | skycell.1591.080 | cleaned | goto_cleaned | 
+| o7977g0243o | 1283018 | 1977751 | wait  | 1921694 | skycell.1218.090 | cleaned | goto_cleaned | 
+| o7977g0262o | 1283037 | 1977770 | wait  | 1921712 | skycell.1218.090 | cleaned | goto_cleaned | 
+| o7977g0281o | 1283056 | 1977789 | wait  | 1921730 | skycell.1218.090 | cleaned | goto_cleaned | 
+| o7977g0300o | 1283075 | 1977808 | wait  | 1921752 | skycell.1218.090 | cleaned | goto_cleaned | 
++-------------+---------+---------+-------+---------+------------------+---------+--------------+
+}}}
+
+This looks identical to earlier, so try to update and then force the chips that are stuck in 'wait':
+{{{
+  chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id XXXXX
+  warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_MOPS -warp_id XXXXX
+
+  chiptool -dbname gpc1 -updaterun -state wait -set_state update -set_label ps_ud_MOPS -chip_id XXXXXX
+}}}
+
+Query the result:
+{{{
+  SELECT 
+  rawExp.exp_name,rawExp.exp_id,
+  chipRun.chip_id, chipRun.state chip_state, chipRun.label chip_label
+ FROM 
+  chipRun, rawExp 
+ WHERE 
+  chipRun.exp_id = rawExp.exp_id 
+ AND 
+  rawExp.exp_name IN ("o6889g0624o","o7977g0243o","o7977g0262o","o7977g0281o","o7977g0300o")
+  ;
+
+
+  SELECT 
+  rawExp.exp_name,rawExp.exp_id,
+  chipRun.chip_id, chipRun.state chip_state, chipRun.label chip_label,
+  warpRun.warp_id,warpSkyfile.skycell_id,warpRun.state warp_state, warpRun.label warp_label
+ FROM 
+  warpSkyfile, warpRun, fakeRun, camRun, chipRun, rawExp 
+ WHERE 
+  warpSkyfile.warp_id = warpRun.warp_id AND warpRun.fake_id = fakeRun.fake_id AND fakeRun.cam_id = camRun.cam_id AND camRun.chip_id = chipRun.chip_id AND chipRun.exp_id = rawExp.exp_id 
+ AND 
+  rawExp.exp_name IN ("o6889g0624o","o7977g0243o","o7977g0262o","o7977g0281o","o7977g0300o")
+  ;
+}}}
+
+However... even though they are in update it does not seem to be picking them up...
+
+
+
 TODO:
-  - Finish my notes for the desperate diffs stuff...
- - configure ippc91 for processing (after it got wiped due to a disk failure). Then put it back into the hosts list
-
- * 08:00 : running test processing on ippc128 - ippc150 as user ipptest (label is UNIONS.DR2.20190430).
-
- * CCL: touch /data/ipp123.0/nebulous/39/3c/10949746515.gpc2:20190117:o8500h0253o:o8500h0253o.ota64.fits time stamp changed
+ - fix the pstamps 'wait' issue
+ - prepare the other compute nodes
+ - read up on the desperate diffs scripts so far
+ - add ippc76 as an apache node (it has been performing well for mops stdscience)
+ - ipp137-139 to up and use for nightly!