Changes between Version 1 and Version 2 of PS1_IPP_Czarlog_20190426

Timestamp:

Apr 29, 2019, 9:27:52 AM (7 years ago)

Author:

fairlamb

Comment:

--

PS1_IPP_Czarlog_20190426

@@ -1 @@
-Note the new date!
+== PS1 IPP Czar Logs for the week 2019.04.26 - 2019.MM.DD == 
+[[PageOutline]]
+
+(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])
+
+Check out the new date!
+
+=== Friday : 2019.04.26 ===
+
+JRF: Since we're now switching on Fridays it makes sense to have the czarlogs do the same.
+
+=== Saturday : 2019.04.27 ===
+
+=== Sunday : 2019.04.28 ===
+
+
+'''Missing mdc file + burntool problem''' by JRF
+
+Reporting missing .mdc file, so it fails in the burntool stage, so I reprocessed it (really should have tried updating, this was in the morning)
+Will reprocess:
+{{{ chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190428 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190428 -set_reduction LAP_SCIENCE -exp_name o5444g0110o}}}
+Check they're finished:
+{{{SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state FROM rawExp JOIN chipRun USING (exp_id) WHERE exp_name IN ("o5444g0110o");}}}
+
+Now the mopsfix job is stuck as new in the chip stage, complaining about the burntool:
+{{{o5444g0110o XY10    219612  2142808 new mopsfix mops.fixbrokenLAP.20190428  NULL    2}}}
+
+Perhaps try:
+{{{ipp_apply_burntool_fix.pl --exp_name o5444g0110o --class_id XY10 --verbose --dbname gpc1}}}
+
+Tell it to update?
+{{{chiptool -dbname gpc1 -setimfiletoupdate  -set_label mopsfix -chip_id 2142808}}}
+
+
+'''Missing mk.fits files:''' by JRF
+Leaving jobs stuck in the warp stage
+
+Try to update them first:
+{{{chiptool -dbname gpc1 -setimfiletoupdate  -set_label ps_ud_WEB -chip_id XXXXXX
+warptool -dbname gpc1  -setskyfiletoupdate -set_label ps_ud_WEB -warp_id XXXXXX}}}
+
+One of the ones sent to update came back complaining about a missing file, but it is there, so try again:
+{{{o6224g0254o 537186  1281637 1295352 1270256 1276893 skycell.1060.063    update  ps_ud_WEB
+chiptool -dbname gpc1 -setimfiletoupdate  -set_label ps_ud_WEB -chip_id 1281637
+warptool -dbname gpc1  -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1276893}}}
+
+Failing that, reprocess them:
+{{{chiptool -dbname gpc1 -definebyquery -set_label mopsfix  -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190428  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190428 -set_reduction LAP_SCIENCE -exp_name}}}
+have sent them to reprocess, check status soon!
+
+The o6624g0189o exposure is done now, so clean up the remaining errors under the original label:
+o6624g0189o
+{{{
+chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1269880
+warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1222762
+}}}
+
+As for o6224g0254o, there is still an issue:
+It still complains about being unable to find a file, but nebulous says it is there, and I have checked that the file is there and it does exists (and looks a reasonable size).
+Cheeky tip -> set to clean, then set it back to update to clean up the stuck ones.
+{{{chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1281637
+warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1276893}}}
+Then, set to update again:
+{{{chiptool -dbname gpc1 -setimfiletoupdate  -set_label ps_ud_WEB -chip_id 1281637
+warptool -dbname gpc1  -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1276893}}}
+
+
+
+
+
+
+'''Evening'''
+
+There are now loads of broken jobs (must have hit a bad patch).
+
+Greater than 600 jobs...but only around 50 unique warp ids
+
+Best action is probably... set to clean, then update?
+{{{
+chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id xxxxxx
+warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id xxxxxx
+chiptool -dbname gpc1 -setimfiletoupdate  -set_label ps_ud_WEB -chip_id xxxxxx
+warptool -dbname gpc1  -setskyfiletoupdate -set_label ps_ud_WEB -warp_id xxxxxx
+}}}
+
+
+
+
+
+
+'''missing 2 darks''' by JRF
+
+Two darks failed to be copied over. It could be an issue with summitcopy. So I'll try restarting it
+
+Now, this is where I made a mistake. '''DO NOT DO THIS'''
+{{{check_server.sh start.server summitcopy
+}}}
+Turns out it does not accept an argument and actually just starts all the servers...and I already had all the servers running...
+
+
+So now I'm going through all the machines that have the pantasks_servers on them and clearing them out. In general just log into pantasks for each service and:
+{{{stop
+shutdown now}}}
+make sure the jobs have actually cleared before shutting down. Do this until all the servers are gone. Then go to the correct machine, and start it up:
+{{{
+pantasks_server & pantasks_client
+server input input
+setup
+run}}}
+Note that you need to be in the correct directory, and logged in as the correct user.
+
+The problem was not solved...
+
+Questions?
+How come there is no cleanup/stack stage for ps2?
+
+
+=== Monday : 2019.04.29 ===
+
+Still have the dark issue.
+
+
+Still a bunch of broken jobs to work through, so here we go!
+
+  
+=== Tuesday : 2019.04.30 ===
+
+=== Wednesday : 2019.05.01 ===
+
+=== Thursday : 2019.05.02 ===