Changes between Version 10 and Version 11 of PS1_IPP_Czarlog_20190401

Timestamp:

Apr 2, 2019, 1:21:50 PM (7 years ago)

Author:

tdeboer

Comment:

--

PS1_IPP_Czarlog_20190401

@@ -120 +120 @@
 }}}
 
-
 === Tuesday : 2019.04.02 ===
+ * TdB: We (Roger, Mark, John and myself) met up after coffee to go through some issues. First off, due to rack i.13 being down for repairs, we set chip and warp reverts to off, to prevent error pile-ups:
+{{{
+chip.revert.off
+warp.revert.off
+}}}
+ That rack also contains other machines, namely ippx077-084, which are used by ipppss stdscience pantasks.So, set them to off for now, even though they will come back on after the daily restart. Go the ~ippp2/stdscience pantask and do:
+{{{
+controller host off ippx077
+controller host off ippx078
+controller host off ippx079
+controller host off ippx080
+controller host off ippx081
+controller host off ippx082
+controller host off ippx083
+controller host off ippx084
+}}}
+ Next, we went into the regular gpc1 stdscience pantask and removed all the extra machines that were added by doing hosts off m0, etc and until all mahcines are off. We also set the poll back to 300:
+{{{
+hosts off m0 (8 times)
+hosts off m1 (8 times)
+hosts off c2 (4 times)
+set.poll 300
+}}}
+
+ On IPPmonitor, two exposures were stuck for ps_ud_MOPS. We had a look at them to find out what the problem was:
+{{{
+mysql -hippdb08 -uXXXXX -pXXXXX gpc1 -e "SELECT chipRun.chip_id,chipRun.state,camRun.cam_id,fakeRun.fake_id,warpRun.warp_id,warpRun.state,warpRun.label,warpRun.data_group FROM warpRun JOIN fakeRun using (fake_id) JOIN camRun using (cam_id) JOIN chipRun  using (chip_id) WHERE warpRun.label like 'ps_ud_MOPS' AND warpRun.state = 'update'"
+
++---------+--------+---------+---------+---------+--------+------------+---------------------------+
+| chip_id | state  | cam_id  | fake_id | warp_id | state  | label      | data_group                |
++---------+--------+---------+---------+---------+--------+------------+---------------------------+
+| 1192618 | update | 1160583 | 1135857 | 1142465 | update | ps_ud_MOPS | LAP.PV3.20140730.20141029 | 
+| 1359800 | update | 1323930 | 1300281 | 1306944 | update | ps_ud_MOPS | LAP.PV3.20140730.20141224 | 
+| 1359803 | update | 1323939 | 1300286 | 1306953 | update | ps_ud_MOPS | LAP.PV3.20140730.20141224 | 
+| 1829092 | update | 1796609 | 1769133 | 1775748 | update | ps_ud_MOPS | OSS.20161016              | 
+| 1829109 | update | 1796626 | 1769150 | 1775765 | update | ps_ud_MOPS | OSS.20161016              | 
+| 1829126 | update | 1796643 | 1769167 | 1775782 | update | ps_ud_MOPS | OSS.20161016              | 
+| 1829143 | update | 1796660 | 1769188 | 1775803 | update | ps_ud_MOPS | OSS.20161016              | 
+| 1884134 | update | 1851648 | 1823771 | 1830386 | update | ps_ud_MOPS | ThreePi.20170111          | 
+| 1902599 | update | 1870113 | 1842164 | 1848779 | update | ps_ud_MOPS | OSS.20170216              | 
++---------+--------+---------+---------+---------+--------+------------+---------------------------+
+}}}
+ They were stuck in update. We then looked at their data_states using:
+{{{
+mysql -hippdb08 -uXXXXX -pXXXXX gpc1 -e "SELECT chipRun.chip_id,chipRun.state,camRun.cam_id,fakeRun.fake_id,warpRun.warp_id,warpRun.state,warpRun.label,warpRun.data_group,chipProcessedImfile.data_state,warpSkyfile.data_state FROM warpRun join warpSkyfile using (warp_id) JOIN fakeRun using (fake_id) JOIN camRun using (cam_id) JOIN chipRun  using (chip_id) JOIN chipProcessedImfile using (chip_id) WHERE warpRun.label like 'ps_ud_MOPS' AND warpRun.state = 'update'"
+}}}
+ This showed that the data_state for the warps were partially full and partially update. So, the full warp_id was set to update, but the chips are not set to update, and stuck in clean.
+ To solve that, set the chips to update again:
+{{{
+chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1192618
+chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1359800
+chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1359803
+chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1829092
+chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1829109
+chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1829126
+chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1829143
+chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1884134
+chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1902599
+}}}
+ Finally, there was one more detrend file that has the same issue as last night, namely a VIDEMASK file with end_time set to 2021. It's det_id is found through:
+{{{
+detselect -show -det_type VIDEOMASK -dbname gpc1 -simple
+
+956 0 chip VIDEOMASK register 2012-05-26T00:01:23.000000  2011-08-02T00:00:00.000000 2021-05-01T00:00:00.000000
+955 0 chip VIDEOMASK register 2012-05-26T00:01:13.000000  2011-03-09T00:00:00.000000 2011-03-10T23:59:59.000000
+954 0 chip VIDEOMASK register 2012-05-26T00:01:04.000000  2011-01-06T00:00:00.000000 NULL
+953 0 chip VIDEOMASK register 2012-05-26T00:00:55.000000  2010-01-01T00:00:00.000000 NULL
+952 0 chip VIDEOMASK register 2012-05-26T00:00:47.000000  2009-12-09T00:00:00.000000 NULL
+951 0 chip VIDEOMASK register 2012-05-26T00:00:39.000000  2009-01-01T00:00:00.000000 NULL
+}}}
+ and the fixed using:
+{{{
+dettool -updatedetrun -dbname gpc1 -set_time_end 2099-04-01T00:00:00.000000 -det_id 956
+}}}
 
 === Wednesday : 2019.04.03 ===