Changes between Version 4 and Version 5 of PS1_IPP_Czarlog_20190325

Timestamp:

Mar 25, 2019, 4:25:47 PM (7 years ago)

Author:

cclin33

Comment:

--

PS1_IPP_Czarlog_20190325

@@ -172 +172 @@
 which contains the config from earlier in the month before we were targeting some new nodes for updates. It also includes 4 of the new nodes, ipp127-ipp130, as these seem ok to be used (the ones after this cannot be accessed from the console yet).
   
+
+CCL: Some pending jobs on pstamp were sent in title of [ps-ipp-ops] Cron <ippitc@ippc30> /bin/bash /data/ippc64.1/ippitc/check_pstamp_stalled.bat
+Following previous steps [wiki:PS1_IPP_Czarlog_20190318 Czar Logs 2019-03-18] to solve the problem.
+{{{
+email contents:
+count(*) label    req_id            name            timestamp fault        exp_id    dtime            utc_timestamp()
+176       MOPS    919001  weryk_5C994224  2019-03-25 21:10:32     0       1330887 02:15:39        2019-03-25 23:20:01
+
+Under terminal, check the warp id with “pstamptool”, warp id is the 4th column.
+pstamptool -dbname ippRequestServer -dbserver ipp113 -pendingdependent -simple
+10961873 new warp 1018709 skycell.0925.068 gpc1 /data/ipp113.0/pstamp/work/2019/03/25/919001 ps_ud_MOPS F 0 0 MOPS 919001 /data/ipp113.0/pstamp/work/2019/03/25/919001/checkdep.10961873.log 500
+10961874 new warp 1018709 skycell.0925.079 gpc1 /data/ipp113.0/pstamp/work/2019/03/25/919001 ps_ud_MOPS F 0 0 MOPS 919001 /data/ipp113.0/pstamp/work/2019/03/25/919001/checkdep.10961874.log 500
+10961875 new warp 1018709 skycell.0925.069 gpc1 /data/ipp113.0/pstamp/work/2019/03/25/919001 ps_ud_MOPS F 0 0 MOPS 919001 /data/ipp113.0/pstamp/work/2019/03/25/919001/checkdep.10961875.log 500
+10961876 new warp 1018709 skycell.0925.078 gpc1 /data/ipp113.0/pstamp/work/2019/03/25/919001 ps_ud_MOPS F 0 0 MOPS 919001 /data/ipp113.0/pstamp/work/2019/03/25/919001/checkdep.10961876.log 500
+10961880 new warp 1018708 skycell.0925.068 gpc1 /data/ipp113.0/pstamp/work/2019/03/25/919001 ps_ud_MOPS F 0 0 MOPS 919001 /data/ipp113.0/pstamp/work/2019/03/25/919001/checkdep.10961880.log 500
+10961881 new warp 1018708 skycell.0925.079 gpc1 /data/ipp113.0/pstamp/work/2019/03/25/919001 ps_ud_MOPS F 0 0 MOPS 919001 /data/ipp113.0/pstamp/work/2019/03/25/919001/checkdep.10961881.log 500
+10961882 new warp 1018708 skycell.0925.069 gpc1 /data/ipp113.0/pstamp/work/2019/03/25/919001 ps_ud_MOPS F 0 0 MOPS 919001 /data/ipp113.0/pstamp/work/2019/03/25/919001/checkdep.10961882.log 500
+10961883 new warp 1018708 skycell.0925.078 gpc1 /data/ipp113.0/pstamp/work/2019/03/25/919001 ps_ud_MOPS F 0 0 MOPS 919001 /data/ipp113.0/pstamp/work/2019/03/25/919001/checkdep.10961883.log 500
+
+Under mysql, check their labels.
+SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chipLabel, chipRun.state, warpRun.warp_id, warpRun.label warpLabel, warpRun.state FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE warp_id IN (1018709, 1018708);
++-------------+--------+---------+----------------+--------+---------+----------------+--------+
+| exp_name    | exp_id | chip_id | chipLabel      | state  | warp_id | warpLabel      | state  |
++-------------+--------+---------+----------------+--------+---------+----------------+--------+
+| o6864g0069o | 770608 | 1070075 | update.LAP.PV3 | update | 1018708 | update.LAP.PV3 | update |
+| o6864g0070o | 770607 | 1070076 | update.LAP.PV3 | update | 1018709 | update.LAP.PV3 | update |
++-------------+--------+---------+----------------+--------+---------+----------------+--------+
+2 rows in set (0.00 sec)
+
+Under terminal, set their label to ps_ud_MOPS (right?)
+chiptool -dbname gpc1 -updaterun -chip_id 1070075 -set_label ps_ud_MOPS
+chiptool -dbname gpc1 -updaterun -chip_id 1070076 -set_label ps_ud_MOPS
+warptool -dbname gpc1 -updaterun -warp_id 1018708 -set_label ps_ud_MOPS
+warptool -dbname gpc1 -updaterun -warp_id 1018709 -set_label ps_ud_MOPS
+
+Under mysql, check their label
+SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chipLabel, chipRun.state, warpRun.warp_id, warpRun.label warpLabel, warpRun.state FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE warp_id IN (1018709, 1018708);
++-------------+--------+---------+------------+--------+---------+------------+--------+
+| exp_name    | exp_id | chip_id | chipLabel  | state  | warp_id | warpLabel  | state  |
++-------------+--------+---------+------------+--------+---------+------------+--------+
+| o6864g0069o | 770608 | 1070075 | ps_ud_MOPS | update | 1018708 | ps_ud_MOPS | update |
+| o6864g0070o | 770607 | 1070076 | ps_ud_MOPS | update | 1018709 | ps_ud_MOPS | update |
++-------------+--------+---------+------------+--------+---------+------------+--------+
+2 rows in set (0.00 sec)
+
+Few minutes later, the state changed to full and problem solved!
+SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chipLabel, chipRun.state, warpRun.warp_id, warpRun.label warpLabel, warpRun.state FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE warp_id IN (1018709, 1018708);
++-------------+--------+---------+------------+-------+---------+------------+-------+
+| exp_name    | exp_id | chip_id | chipLabel  | state | warp_id | warpLabel  | state |
++-------------+--------+---------+------------+-------+---------+------------+-------+
+| o6864g0069o | 770608 | 1070075 | ps_ud_MOPS | full  | 1018708 | ps_ud_MOPS | full  |
+| o6864g0070o | 770607 | 1070076 | ps_ud_MOPS | full  | 1018709 | ps_ud_MOPS | full  |
++-------------+--------+---------+------------+-------+---------+------------+-------+
+2 rows in set (0.01 sec)
+}}}
+
 === Tuesday : 2019.03.26 ===