Changes between Version 8 and Version 9 of PS1_IPP_Czarlog_20190318

Timestamp:

Mar 20, 2019, 3:00:08 PM (7 years ago)

Author:

fairlamb

Comment:

--

PS1_IPP_Czarlog_20190318

@@ -114 +114 @@
 remove '-simple' to get some more information.
 
+JRF (evening): Jobs approaching 100k, so restarting stdscience pantasks. A bunch of failed jobs are beginning to accumulate too, but I'll save that fun for the morning.
+
+
 === Tuesday : 2019.03.19 ===
 
+
+JRF (morning): Jobs approaching 100k, so restarting stdscience pantasks. Flicked ippb04 and ippb05 to repair, from down, so that some files could be accessed to clear the backlog of tasks.
+
+
+JRF (afternoon) - czar formation ft. MEH:
+
+
+Maybe tomorrow ecliptic.RP
+
+
+Weird warp fault where the state is full:
+{{{
+    o7071g0124o 875375  1530574 1497865 1471180 1477774 skycell.1618.088    full    update.OSS
+}}}
+Query the neb-workdir (note the wild card at the end):
+{{{
+    neb-ls -p -l neb://ipp069.0/gpc1/OSS.nt/2015/02/18//o7071g0124o.875375/o7071g0124o.875375.wrp.1477774.skycell.1618.088%
+}}}
+shows us that one of the mask files is in a funny state where is 0 size:
+{{{
+    -rw-rw-r-- 1 ippitc users 0 Mar  8 09:27 /data/ipp125.0/nebulous/9f/e9/11203362808.gpc1:OSS.nt:2015:02:18:o7071g0124o.875375:o7071g0124o.875375.wrp.1477774.skycell.1618.088.mask.fits
+}}}
+To solve this, we will have to put the warp to clean:
+{{{
+    warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1477774
+}}}
+check that is has cleaned succesfully (query the database). Then set it to update:
+{{{
+warptool -dbname gpc1  -setskyfiletoupdate -set_label update.OSS -warp_id 1477774
+}}}
+
+
+'''Solving warp issues'''
+Move the stuff that is currently broken under the label 'update.OSS' to the 'update.OSS.broke' label.
+To figure out which ones need to be moved have a look at the state, fault, and skyfile
+
+Will need to add this new label to pantasks so it can get priority and actually work on them.
+
+The following shows the query used, and how to pipe it to 'awks' so that it can run the chiptool/warptool command to update the labels on the chip/warp id pairs.
+{{{
+mysql -uXXXXX -pXXXXX -hippdb08 gpc1 -B --skip-column-names -e 
+
+'SELECT chip_id,
+warp_id,
+warpRun.label,
+warpRun.state,
+warpSkyfile.fault,
+chipRun.label,
+chipRun.state
+ FROM warpRun
+ JOIN warpSkyfile USING (warp_id)
+ JOIN fakeRun USING (fake_id)
+ JOIN camRun USING (cam_id)
+ JOIN chipRun USING (chip_id)
+ WHERE warpRun.label LIKE "update.OSS"
+ AND warpSkyfile.fault > 0
+ AND warpRun.state LIKE "update"
+ GROUP BY warp_id;'
+
+| awk '{print"
+ chiptool -dbname gpc1 -updaterun -chip_id "$1" -state update -set_label update.OSS.broke -pretend;
+ warptool -dbname gpc1 -updaterun -warp_id "$2" -state update -set_label update.OSS.broke -pretend
+ "}'
+ | xargs -0 bash -c
+
+
+}}}
+Note: The above its formatted for readability, you'll have to squish it into one line, fill in the XXXXXs, and remove pretend.
+
+
+Next, add the label to stdscience ({{{pantasks> add.label update.OSS.broke}}})
+Then put the b nodes up to repair to help clear things out.
+
+JRF (late evening): Gave pantasks a restart as it was close to 100k jobs
+
+
 === Wednesday : 2019.03.20 ===
 
+
+MEH moved across the remaining update.OSS.broke labels.
+
+The SVN server died!
+
+May want to figure out which exposures might have already been reprocessed too, as these ones do not need to be done.
+we are going to add the LAP PV3 label
+
+
+There are some broken stamps that Rob has requested.
+
+{{{
+count   label   req_id  name                    timestamp                       fault   exp_id  dtime                   utc_timestamp()
+1       MOPS    918829  weryk_5C92B8D3  2019-03-20 22:15:56     25      466358  00:04:05        2019-03-20 22:20:01
+2       MOPS    918829  weryk_5C92B8D3  2019-03-20 22:15:56     25      466379  00:04:05        2019-03-20 22:20:01
+2       MOPS    918829  weryk_5C92B8D3  2019-03-20 22:15:56     25      705006  00:04:05        2019-03-20 22:20:01
+2       MOPS    918829  weryk_5C92B8D3  2019-03-20 22:15:56     25      705027  00:04:05        2019-03-20 22:20:01
+1       MOPS    918829  weryk_5C92B8D3  2019-03-20 22:15:56     25      705045  00:04:05        2019-03-20 22:20:01
+1       MOPS    918829  weryk_5C92B8D3  2019-03-20 22:15:56     25      705068  00:04:05        2019-03-20 22:20:01
+}}}
+
+Time to take the exp_id and have a look:
+{{{
+SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label, chipRun.state, camRun.cam_id, camRun.state, warpRun.warp_id, warpRun.label, warpRun.state  FROM rawExp JOIN chipRun USING (exp_id) JOIN camRun USING (chip_id) JOIN fakeRun USING (cam_id) JOIN warpRun USING (fake_id) WHERE exp_id IN (466358,466379,705006,705027,705045,705068);
+}}}
+
+There are a bunch of cleaned ones but the most recent ones have the label '''update.hold20190318'''
+
+Moved them to the ps_ud_MOPS label e.g.
+{{{
+chiptool -dbname gpc1 -updaterun -chip_id 428036 -state update -set_label ps_ud_MOPS
+warptool -dbname gpc1 -updaterun -warp_id 387740 -state update -set_label ps_ud_MOPS
+}}}
+
+And then set them to update:
+{{{
+chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 428036 
+}}}
+
 === Thursday : 2019.03.21 ===