Changes between Version 6 and Version 7 of PS1_IPP_Czarlog_20190426

Timestamp:

Apr 30, 2019, 11:04:13 AM (7 years ago)

Author:

fairlamb

Comment:

--

PS1_IPP_Czarlog_20190426

@@ -219 +219 @@
 
 
-'''Broken update stuff (continued)'''
+'''Broken update stuff (continued)''' by JRF
 
 a broken job in the ps_ud_WEB chip stage:
@@ -239 +239 @@
 
 
-
-
-
-To do: 
- - Once ipp134-139 are back online revert the mopsfix job stuck in the camera stage
- - send everything in update.LAP.PV3 to clean...
- - ...once cleaned, send them all to update. This should fix most of the chip issues
- - For the remaining problems:
-    -- if they complain about missing cm (camera) files then they should be reprocessed.
+'''Hardware stuff update'' by JRF
+
+17:00 - Haydn wishes to replace some failing drives on some of the compute nodes (ippc91, ippc52, ippc53, and ippc85). To allow him to do this the nodes need to be turned off in pantasks; e.g. to turn off ippc91:
+{{{
+  check_system.sh hostoff ippc91
+}}}
+Note that this needs to be done for as many times as the machine appears in the controller list (pantasks> controller status). Once done sufficient times, and allowing some time for jobs to finish if they were in use, then they should all report as 'OFF', for example:
+{{{
+477  ippc91    OFF  0.0.0.bf
+478  ippc91    OFF  0.0.0.f9
+479  ippc91    OFF  0.0.0.133
+480  ippc91    OFF  0.0.0.16d
+481  ippc91    OFF  0.0.0.1a7
+}}}
+At this point maintenance can go ahead on the machine.
+
+To see which nodes are used by which pantasks check the '~ippitc/ippconfig/pantasks_hosts.input' file. None of the main tasks use c52 or c53. MEH has confirmed that they are off for the mops stdscience stuff.
+
+
+''' Geet big cleanup and process ''' by JRF
+
+First, see if I can cleanup all the ps_ud_WEB stuff:
+i) query the faulted jobs to get chip/warp ids
+ii) send them to cleaned (both chip and warp)
+iii) Once cleaned set them to update
+
+Next there are a lot of stuck jobs under the update.LAP.PV3 label. Repeat the process of:
+i) send them all to clean
+ii) once cleaned send them to update (I put them under ps_ud_WEB since there is nothing there now)
+
+This solved the majority of issues, about 437 of them, leaving only 4 problems, which shall be dealt with tomorrow.
 
   
 === Tuesday : 2019.04.30 ===
 
+'''stamp request is stuck''' by JRF
+
+So check it out:
+{{{
+pstamptool -dbname ippRequestServer -dbserver ipp113 -pendingdependent -simple
+
+10977094 new warp 1436300 skycell.0750.010 gpc1 /data/ipp113.0/pstamp/work/2019/04/30/920240 ps_ud_MOPS F 0 0 MOPS 920240 /data/ipp113.0/pstamp/work/2019/04/30/920240/checkdep.10977094.log 500
+}}}
+
+Query it:
+{{{
+SELECT 
+  rawExp.exp_name,rawExp.exp_id,
+  chipRun.chip_id,
+  warpRun.warp_id,warpSkyfile.skycell_id,warpRun.state,warpRun.label,
+  warpSkyfile.path_base 
+ FROM 
+  warpSkyfile, warpRun, fakeRun, camRun, chipRun, rawExp 
+ WHERE 
+  warpSkyfile.warp_id = warpRun.warp_id AND warpRun.fake_id = fakeRun.fake_id AND fakeRun.cam_id = camRun.cam_id AND camRun.chip_id = chipRun.chip_id AND chipRun.exp_id = rawExp.exp_id 
+ AND 
+  warpRun.warp_id = '1436300' 
+ AND 
+  warpSkyfile.skycell_id like 'skycell.0750.010' 
+  ;
+
+
++-------------+--------+---------+---------+------------------+--------+----------------+
+| exp_name    | exp_id | chip_id | warp_id | skycell_id       | state  | label          |
++-------------+--------+---------+---------+------------------+--------+----------------+
+| o6284g0630o | 558924 | 1488053 | 1436300 | skycell.0750.010 | update | update.LAP.PV3 |
++-------------+--------+---------+---------+------------------+--------+----------------+
+}}}
+
+Update the run to a higher priority label, and set it to update:
+{{{
+chiptool -dbname gpc1 -updaterun -chip_id 1488053 -state update -set_label ps_ud_MOPS
+warptool -dbname gpc1 -updaterun -warp_id 1436300 -state update -set_label ps_ud_MOPS
+chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1488053
+warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_MOPS -warp_id 1436300
+}}}
+
+
+'''Broken jobs''' by JRF
+
+Continuing on from yesterday, I sent around 400 unique jobs to clean -> then update. The majority of this proceeded successfully!
+There are a handful (about 60 failures, only 3-4 unique chip/warp combos) that are still failing. Checking their log it complains about a missing file of the form:
+{{{
+    *cm*.XY*.mk.fits
+}}}
+The cm indicates that this is from the camera stage. Because of this updating the chip or warp stage will not help, we have to reprocess instead.
+
+So, fetch the jobs that have the issue:
+{{{
+SELECT 
+ rawExp.exp_name,rawExp.exp_id,rawExp.filter,
+ chipRun.chip_id,warpRun.warp_id,warpSkyfile.skycell_id,warpRun.state,warpRun.label,warpSkyfile.fault 
+ FROM 
+ warpSkyfile, warpRun, fakeRun, camRun, chipRun, rawExp 
+ WHERE 
+ warpSkyfile.warp_id = warpRun.warp_id AND warpRun.fake_id = fakeRun.fake_id AND fakeRun.cam_id = camRun.cam_id AND camRun.chip_id = chipRun.chip_id AND chipRun.exp_id = rawExp.exp_id 
+ AND warpSkyfile.fault != 0 
+ AND warpRun.label like 'ps_ud_WEB'
+ GROUP BY warpRun.warp_id
+;
+
++-------------+--------+---------+---------+------------------+--------+-----------+-------+
+| exp_name    | exp_id | chip_id | warp_id | skycell_id       | state  | label     | fault |
++-------------+--------+---------+---------+------------------+--------+-----------+-------+
+| o5794g0540o | 380413 | 1335129 | 1285254 | skycell.2501.030 | update | ps_ud_WEB |     2 | 
+| o5576g0112o | 280226 | 1335126 | 1285655 | skycell.2457.094 | update | ps_ud_WEB |     2 | 
+| o6575g0385o | 664527 | 1335259 | 1286115 | skycell.2353.026 | update | ps_ud_WEB |     2 | 
+| o6237g0252o | 545358 | 1335251 | 1286194 | skycell.2352.010 | update | ps_ud_WEB |     2 | 
+| o6302g0235o | 565722 | 1335991 | 1288841 | skycell.2169.079 | update | ps_ud_WEB |     4 | 
++-------------+--------+---------+---------+------------------+--------+-----------+-------+
+}}}
+The fault 4 can just be reverted. As for the others, send them to reprocess:
+{{{
+    chiptool -dbname gpc1 -definebyquery -set_label mopsfix  -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190430  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190430 -set_reduction LAP_SCIENCE -exp_name o5794g0540o
+    chiptool -dbname gpc1 -definebyquery -set_label mopsfix  -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190430  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190430 -set_reduction LAP_SCIENCE -exp_name o5576g0112o
+    chiptool -dbname gpc1 -definebyquery -set_label mopsfix  -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190430  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190430 -set_reduction LAP_SCIENCE -exp_name o6575g0385o
+    chiptool -dbname gpc1 -definebyquery -set_label mopsfix  -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190430  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190430 -set_reduction LAP_SCIENCE -exp_name o6237g0252o
+}}}
+
+Wait for them to finish processing...
+
+Looks like problems (in the chip and cam stage):
+They're all waiting on some files that are only on ipp134-136 - I just set these down so that ITC could install the PDU. PENDING
+
+Once good, cleanup remaining files:
+{{{
+    chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 
+    warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 
+}}}
+
+
+'''Morning hardware things''' by JRF
+
+ITC are going to install the PDUs in ipp134-136, so these will need to be set to down:
+neb-host ipp134 down --note "JRF: repair -> down: so that PDU can be installed in the cabinet"
+neb-host ipp135 down --note "JRF: repair -> down: so that PDU can be installed in the cabinet"
+neb-host ipp136 down --note "JRF: repair -> down: so that PDU can be installed in the cabinet"
+set down at about 09:20am
+
+
+
+TODO:
+ - once the PDU is done for ipp134-136:
+   -- check the console
+   -- put them to up to allow updates to continue and check on them
+
+
 === Wednesday : 2019.05.01 ===