IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Version 13 and Version 14 of PS1_IPP_Czarlog_20190318


Ignore:
Timestamp:
Mar 25, 2019, 10:24:42 AM (7 years ago)
Author:
fairlamb
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • PS1_IPP_Czarlog_20190318

    v13 v14  
    218218}}}
    219219
    220 Time to take the exp_id and have a look:
     220
     221Time to take the exp_id and have a look at them:
    221222{{{
    222223SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label, chipRun.state, camRun.cam_id, camRun.state, warpRun.warp_id, warpRun.label, warpRun.state  FROM rawExp JOIN chipRun USING (exp_id) JOIN camRun USING (chip_id) JOIN fakeRun USING (cam_id) JOIN warpRun USING (fake_id) WHERE exp_id IN (466358,466379,705006,705027,705045,705068);
    223 }}}
    224 
     224
     225+-------------+--------+---------+---------------------+---------+---------+---------+---------+---------------------+---------+
     226| exp_name    | exp_id | chip_id | label               | state   | cam_id  | state   | warp_id | label               | state   |
     227+-------------+--------+---------+---------------------+---------+---------+---------+---------+---------------------+---------+
     228| o6002g0321o | 466358 | 1067421 | update.hold20190318 | update  | 1036073 | full    | 1018412 | update.hold20190318 | update  |
     229| o6002g0340o | 466379 | 1067424 | update.hold20190318 | update  | 1036086 | full    | 1018431 | update.hold20190318 | update  |
     230| o6698g0496o | 705045 | 1078654 | update.hold20190318 | update  | 1040692 | full    | 1022614 | update.hold20190318 | update  |
     231| o6698g0517o | 705068 | 1078658 | update.hold20190318 | update  | 1040696 | full    | 1022879 | update.hold20190318 | update  |
     232+-------------+--------+---------+---------------------+---------+---------+---------+---------+---------------------+---------+
     233}}}
    225234There are a bunch of cleaned ones but the most recent ones have the label '''update.hold20190318'''
    226235
    227 Moved them to the ps_ud_MOPS label e.g.
    228 {{{
    229 chiptool -dbname gpc1 -updaterun -chip_id 428036 -state update -set_label ps_ud_MOPS
    230 warptool -dbname gpc1 -updaterun -warp_id 387740 -state update -set_label ps_ud_MOPS
    231 }}}
    232 
    233 And then set them to update:
    234 {{{
    235 chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 428036
    236 }}}
     236set them to update:
     237{{{
     238chiptool -dbname gpc1 -updaterun -chip_id XXXXX -state update -set_label ps_ud_MOPS
     239warptool -dbname gpc1 -updaterun -warp_id XXXXX -state update -set_label ps_ud_MOPS
     240}}}
     241
     242None of them seem to be moving through...
     243Mark suggested trying it will -setimfiletoupdate:
     244{{{
     245chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id XXXXX
     246}}}
     247
     248Two of these failed...
     249{{{
     250o6002g0321o XY37    466358  542901  update  ps_ud_MOPS  LAP.ThreePi.20120706.20120813   LAP.ThreePi 2
     251o6002g0340o XY46    466379  542904  update  ps_ud_MOPS  LAP.ThreePi.20120706.20120813   LAP.ThreePi 2
     252}}}
     253..., so I'll revert them with the suggested command
     254{{{
     255chiptool -revertprocessedimfile -fault 2 -label ps_ud_MOPS -dbname gpc1
     256}}}
     257
     258
     259Going back to the stuff that had hung pstamps, on the pstamp server we can see that nothing has moved for about 10 hours.
     260
     261
     262If you come across a stamp that you suspect maybe be stuck, have a look at it using:
     263{{{
     264pstamptool -dbname ippRequestServer -dbserver ipp113 -pendingdependent -simple
     265}}}
     266remove '-simple' to get some more information.
     267
     268Going back to checking out Rob's stuff:
     269{{{
     270SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label, chipRun.state, camRun.cam_id, camRun.state, warpRun.warp_id, warpRun.label, warpRun.state  FROM rawExp JOIN chipRun USING (exp_id) JOIN camRun USING (chip_id) JOIN fakeRun USING (cam_id) JOIN warpRun USING (fake_id) WHERE exp_id IN (466358,466379,705006,705027,705045,705068);
     271+-------------+--------+---------+------------+-------+---------+---------+---------+--------------+---------+
     272| exp_name    | exp_id | chip_id | label      | state | cam_id  | state   | warp_id | label        | state   |
     273+-------------+--------+---------+------------+-------+---------+---------+---------+--------------+---------+
     274| o6002g0321o | 466358 | 1067421 | ps_ud_MOPS | full  | 1036073 | full    | 1018412 | ps_ud_MOPS   | update  |
     275| o6002g0340o | 466379 | 1067424 | ps_ud_MOPS | full  | 1036086 | full    | 1018431 | ps_ud_MOPS   | update  |
     276| o6698g0496o | 705045 | 1078654 | ps_ud_MOPS | full  | 1040692 | full    | 1022614 | ps_ud_MOPS   | update  |
     277| o6698g0517o | 705068 | 1078658 | ps_ud_MOPS | full  | 1040696 | full    | 1022879 | ps_ud_MOPS   | update  |
     278+-------------+--------+---------+------------+-------+---------+---------+---------+--------------+---------+
     279}}}
     280
     281There are a still a bunch that have warps in the update state.
     282I'm going to try cleaning...
     283{{{
     284chiptool -dbname gpc1 -updaterun -chip_id 1067421 -set_state cleaned
     285chiptool -dbname gpc1 -updaterun -chip_id 1067424 -set_state cleaned
     286chiptool -dbname gpc1 -updaterun -chip_id 1078654 -set_state cleaned
     287chiptool -dbname gpc1 -updaterun -chip_id 1078658 -set_state cleaned
     288}}}
     289...then updating:
     290{{{
     291chiptool -dbname gpc1 -setimfiletoupdate -chip_id 1067421 -set_label ps_ud_MOPS
     292chiptool -dbname gpc1 -setimfiletoupdate -chip_id 1067424 -set_label ps_ud_MOPS
     293chiptool -dbname gpc1 -setimfiletoupdate -chip_id 1078654 -set_label ps_ud_MOPS
     294chiptool -dbname gpc1 -setimfiletoupdate -chip_id 1078658 -set_label ps_ud_MOPS
     295}}}
     296
     297'''More pstamp investigations:'''
     298
     299{{{
     3002       MOPS    918835  weryk_5C92D8E1  2019-03-21 00:40:58     26      289960  00:14:03        2019-03-21 00:55:01
     3012       MOPS    918835  weryk_5C92D8E1  2019-03-21 00:40:58     26      289981  00:14:03        2019-03-21 00:55:01
     302}}}
     303
     304Time to take the exp_id and have a look at them:
     305{{{
     306SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label, chipRun.state, camRun.cam_id, camRun.state, warpRun.warp_id, warpRun.label, warpRun.state  FROM rawExp JOIN chipRun USING (exp_id) JOIN camRun USING (chip_id) JOIN fakeRun USING (cam_id) JOIN warpRun USING (fake_id) WHERE exp_id IN (289960, 289981);
     307
     308+-------------+--------+---------+-------------------+---------+--------+-------+---------+-----------------------------------+---------+
     309| exp_name    | exp_id | chip_id | label             | state   | cam_id | state | warp_id | label                             | state   |
     310+-------------+--------+---------+-------------------+---------+--------+-------+---------+-----------------------------------+---------+
     311| o5591g0062o | 289960 |  830503 | ps_ud_MOPS        | update  | 806578 | full  |  786578 | PI_20110130.reprocessing.20130426 | drop    |
     312| o5591g0062o | 289960 |  830503 | ps_ud_MOPS        | update  | 806578 | full  |  788140 | ps_ud_MOPS                        | update  |
     313| o5591g0083o | 289981 |  830524 | ps_ud_MOPS        | update  | 806604 | full  |  786618 | PI_20110130.reprocessing.20130426 | drop    |
     314| o5591g0083o | 289981 |  830524 | ps_ud_MOPS        | update  | 806604 | full  |  788180 | ps_ud_MOPS                        | update  |
     315+-------------+--------+---------+-------------------+---------+--------+-------+---------+-----------------------------------+---------+
     316}}}
     317set them to update:
     318{{{
     319chiptool -dbname gpc1 -updaterun -chip_id 830503 -state update -set_label ps_ud_MOPS
     320chiptool -dbname gpc1 -updaterun -chip_id 830524 -state update -set_label ps_ud_MOPS
     321warptool -dbname gpc1 -updaterun -warp_id 788140 -state update -set_label ps_ud_MOPS
     322warptool -dbname gpc1 -updaterun -warp_id 788180 -state update -set_label ps_ud_MOPS
     323}}}
     324
     325Then run this to force them to update...
     326{{{
     327chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 830503
     328chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 830524
     329}}}
     330
     331And for the warps:
     332{{{
     333warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_MOPS -warp_id 788140
     334warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_MOPS -warp_id 788180
     335}}}
     336
     337
     338'''Without a mask'''
     339
     340Rob has requested some things to be processed without a mask:
     341o7255g0537o 04:22:26.871 +21:06:31.62 K17W15C
     342o7255g0531o 04:22:26.688 +21:06:27.34 K17W15C
     343o7255g0525o 04:22:26.506 +21:06:23.05 K17W15C
     344o7255g0519o 04:22:26.323 +21:06:18.76 K17W15C
     345
     346So I've queued them up
     347{{{
     348chiptool -definebyquery -set_label mops.nomasktest  -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o7255g0537o
     349chiptool -definebyquery -set_label mops.nomasktest  -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o7255g0531o
     350chiptool -definebyquery -set_label mops.nomasktest  -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o7255g0525o
     351chiptool -definebyquery -set_label mops.nomasktest  -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o7255g0519o
     352}}}
     353They go through fine
     354
     355'''Broken IPP jobs'''
     356
     357There are two warp_ids that have problems: '''1640466, 1640460'''
     358
     359Find out the sate of these:
     360{{{
     361SELECT chipRun.chip_id, chipRun.label, chipRun.state, camRun.cam_id, camRun.state, warpRun.warp_id, warpRun.label, warpRun.state FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) WHERE warpRun.warp_id IN (1640466, 1640460);
     362
     363+---------+------------------+--------+---------+-------+---------+------------------+--------+
     364| chip_id | label            | state  | cam_id  | state | warp_id | label            | state  |
     365+---------+------------------+--------+---------+-------+---------+------------------+--------+
     366| 1693054 | update.X.ThreePi | update | 1660571 | full  | 1640460 | update.X.ThreePi | update |
     367| 1693060 | update.X.ThreePi | update | 1660577 | full  | 1640466 | update.X.ThreePi | update |
     368+---------+------------------+--------+---------+-------+---------+------------------+--------+
     369}}}
     370
     371For exercises sake, we're going to change the label for the first line's chip and warp id to ps_ud_MOPS:
     372{{{
     373chiptool -dbname gpc1 -updaterun -chip_id 1693054 -state update -set_label ps_ud_MOPS
     374warptool -dbname gpc1 -updaterun -warp_id 1640460 -state update -set_label ps_ud_MOPS
     375}}}
     376
     377Alternatively, you could pipe the results from pstamptool to do the update like so:
     378{{{
     379pstamptool -dbname ippRequestServer -dbserver ipp113 -pendingdependent -simple  | grep "new warp" | cut -f4 -d" " | awk '{ print " warptool -dbname gpc1 -listrun -simple -warp_id "$1}' | tcsh
     380
     381pstamptool -dbname ippRequestServer -dbserver ipp113 -pendingdependent -simple  | grep "new warp" | cut -f4 -d" " | awk '{ print " warptool -dbname gpc1 -listrun -simple -warp_id "$1}' | tcsh |  awk '{ print "chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id "$23" ; warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id "$1 }'
     382}}}
     383
     384'''More IPP monitor errors (solving missing log)'''
     385
     386Checking out some of the errors, one of them complains about a file that it could not use:
     387{{{
     388Unable to perform warp.
     389 -> pmConfigFileRead (pmConfig.c:317): Problem in configure files
     390     Unable to read user configuration from /data/ipp069.0/nebulous/2c/d5/7195324599.gpc1:OSS.nt:2013:10:08:o6573g0097o.663490:o6573g0097o.663490.wrp.846597.skycell.1501.036.pswarp.mdc
     391}}}
     392
     393Having a look at the file it is empty...It's probably not supposed to be, so we will have a look at the nebulous entry:
     394{{{
     395neb-stat --validate neb://ipp124.1/gpc1/OSS.nt/2013/10/08//o6573g0097o.663490/o6573g0097o.663490.wrp.846597.skycell.1501.036.pswarp.mdc
     396}}}
     397
     398This shows that the other two copies of this file are not empty (see the checksum: d41d8cd98f00b204e9800998ecf8427e)
     399{{{
     400    36.pswarp.mdc
     401object id:             3133998992
     402key:                   gpc1/OSS.nt/2013/10/08/o6573g0097o.663490/o6573g0097o.663490.wrp.846597.skycell.1501.036.pswarp.mdc
     403read lock:             0
     404write lock:           
     405epoch:                 2013-10-08 00:08:21
     406mtime:                 0000-00-00 00:00:00
     407md5sum count:          2
     408requested instances:   2
     409available instances:   3
     410existing instances:    3
     411total instances:       3
     412instance location:
     413      1 d41d8cd98f00b204e9800998ecf8427e file:///data/ipp069.0/nebulous/2c/d5/7195324599.gpc1:OSS.nt:2013:10:08:o6573g0097o.663490:o6573g0097o.663490.wrp.846597.skycell.1501.036.pswarp.mdc
     414      1 7e5f777b2a1658fb976957e5c5c9639b file:///data/ipp091.0/nebulous/2c/d5/7195346311.gpc1:OSS.nt:2013:10:08:o6573g0097o.663490:o6573g0097o.663490.wrp.846597.skycell.1501.036.pswarp.mdc
     415      1 7e5f777b2a1658fb976957e5c5c9639b file:///data/ipp072.0/nebulous/2c/d5/7296944772.gpc1:OSS.nt:2013:10:08:o6573g0097o.663490:o6573g0097o.663490.wrp.846597.skycell.1501.036.pswarp.mdc
     416
     417}}}
     418
     419To solve this there are two options:
     420    1) delete the empty file
     421    2) replace the empty file with one of the correct ones.
     422We're chose option 1.
     423
     424
     425'''Stalled IPP monitor jobs'''
     426
     427Checking out some stalled update.OSS.broke things that are not faulted, so do not appear in IPP monitor:
     428
     429SELECT chip_id, chipProcessedImfile.fault, warpRun.warp_id,warpRun.label,warpRun.state,warpSkyfile.fault,chipRun.label chiplabel,chipRun.state FROM warpRun JOIN warpSkyfile USING (warp_id) JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN chipProcessedImfile USING (chip_id) WHERE warpRun.label LIKE "update.OSS.broke" AND warpRun.state LIKE "update" GROUP BY warpRun.warp_id;
     430{{{
     431+---------+-------+---------+------------------+--------+-------+--------------+--------------+
     432| chip_id | fault | warp_id | label            | state  | fault | label        | state        |
     433+---------+-------+---------+------------------+--------+-------+--------------+--------------+
     434| 1257054 |     0 | 1200666 | update.OSS.broke | update |     0 | update.OSS   | goto_cleaned |
     435| 1268153 |     0 | 1214242 | update.OSS.broke | update |     0 | goto_cleaned | cleaned      |
     436| 1280537 |     0 | 1225941 | update.OSS.broke | update |     0 | goto_cleaned | cleaned      |
     437.
     438.
     439.
     440| 1730278 |     0 | 1677448 | update.OSS.broke | update |     0 | goto_cleaned | cleaned      |
     441| 1749642 |     0 | 1696720 | update.OSS.broke | update |     0 | update.OSS   | full         |
     442.
     443.
     444.
     445| 1762720 |     0 | 1709795 | update.OSS.broke | update |     0 | update.OSS   | full         |
     446+---------+-------+---------+------------------+--------+-------+--------------+--------------+
     447}}}
     448
     449The first one is kinda suspicious, as it's state is goto_cleaned. So let's set it to update:
     450{{{
     451chiptool -dbname gpc1 -setimfiletoupdate -chip_id 1257054 -set_label update.OSS.broke
     452}}}
     453That didn't quite work,
     454Sending to clean instead:
     455{{{
     456chiptool -dbname gpc1 -updaterun -chip_id 1257054 -set_label goto_cleaned
     457}}}
     458
     459In the meantime, try the second on the list:
     460{{{
     461chiptool -dbname gpc1 -setimfiletoupdate -chip_id 1268153 -set_label update.OSS.broke
     462}}}
     463that one was fine...
     464{{{
     465+---------+-------+---------+------------------+--------+-------+------------------+--------------+
     466| chip_id | fault | warp_id | label            | state  | fault | chiplabel        | state        |
     467+---------+-------+---------+------------------+--------+-------+------------------+--------------+
     468| 1257054 |     0 | 1200666 | update.OSS.broke | update |     0 | goto_cleaned     | goto_cleaned |
     469| 1268153 |     0 | 1214242 | update.OSS.broke | update |     0 | update.OSS.broke | update       |
     470| 1280537 |     0 | 1225941 | update.OSS.broke | update |     0 | goto_cleaned     | cleaned      |
     471}}}
     472
     473So what happened to the first chip 1257054?
     474Check out the cleanup stdout logs (pantasks logs), and we see some problems:
     475It mostly complains about missing log files that were expected on old stsci machines.
     476
     477So then we went to have a look at the chipProcssedImfile to see if there is anything weird:
     478{{{
     479SELECT chip_id, chipProcessedImfile.fault, chipProcessedImfile.class_id,chipProcessedImfile.data_state,chipProcessedImfile.quality FROM chipProcessedImfile WHERE chip_id=1257054;
     480}}}
     481Running the above query you can see that the chips are in a mix of full/cleaned states...which is not the way it should be
     482
     483We'll set it all to cleaned:
     484chiptool -dbname gpc1 -updaterun -chip_id 1257054 -set_state cleaned
     485
     486Now, we can finally try again with chiptool stuff:
     487chiptool -dbname gpc1 -setimfiletoupdate -chip_id 1257054 -set_label update.OSS.broke
     488
     489And so the chip state updated to clean...but the warp tool is still in update.
     490
     491Most things just seem stuck on the old missing files from ippb05 problem (so I put it into repair for the time being).
     492
     493chiptool -dbname gpc1 -setimfiletoupdate -chip_id 1268153 -set_label update.OSS.broke
     494chiptool -dbname gpc1 -setimfiletoupdate -chip_id 1280537 -set_label update.OSS.broke
     495chiptool -dbname gpc1 -setimfiletoupdate -chip_id 1530574 -set_label update.OSS.broke
     496chiptool -dbname gpc1 -setimfiletoupdate -chip_id 1591282 -set_label update.OSS.broke
     497chiptool -dbname gpc1 -setimfiletoupdate -chip_id 1599586 -set_label update.OSS.broke
     498chiptool -dbname gpc1 -setimfiletoupdate -chip_id 1617899 -set_label update.OSS.broke
     499chiptool -dbname gpc1 -setimfiletoupdate -chip_id 1685737 -set_label update.OSS.broke
     500chiptool -dbname gpc1 -setimfiletoupdate -chip_id 1726939 -set_label update.OSS.broke
     501chiptool -dbname gpc1 -setimfiletoupdate -chip_id 1730278 -set_label update.OSS.broke
     502
    237503
    238504=== Thursday : 2019.03.21 ===
    239505
     506'''There are a couple of broken stamp requests'''
     507
     508
     509Tasks that are pending, as found by:
     510{{{
     511    pstamptool -dbname ippRequestServer -dbserver ipp113 -pendingdependent -simple
     512}}}
     51310818461 new warp 475995 skycell.1432.053 gpc1 /data/ipp113.0/pstamp/work/2019/01/10/914957 ps_ud_MOPS.2 F 0 1 MOPS 918857 /data/ipp113.0/pstamp/work/2019/01/10/914957/checkdep.10818461.log 500
     51410818468 new warp 475962 skycell.1432.053 gpc1 /data/ipp113.0/pstamp/work/2019/01/10/914957 ps_ud_MOPS.2 F 0 1 MOPS 918857 /data/ipp113.0/pstamp/work/2019/01/10/914957/checkdep.10818468.log 500
     51510959532 new warp 475995 skycell.1432.054 gpc1 /data/ipp113.0/pstamp/work/2019/03/21/918857 ps_ud_MOPS F 0 0 MOPS 918857 /data/ipp113.0/pstamp/work/2019/03/21/918857/checkdep.10959532.log 500
     51610959534 new warp 475962 skycell.1432.054 gpc1 /data/ipp113.0/pstamp/work/2019/03/21/918857 ps_ud_MOPS F 0 0 MOPS 918857 /data/ipp113.0/pstamp/work/2019/03/21/918857/checkdep.10959534.log 500
     51710959535 new warp 475957 skycell.1432.054 gpc1 /data/ipp113.0/pstamp/work/2019/03/21/918857 ps_ud_MOPS F 0 0 MOPS 918857 /data/ipp113.0/pstamp/work/2019/03/21/918857/checkdep.10959535.log 500
     518
     519Let's query these:
     520{{{
     521    SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chipLabel, chipRun.state, warpRun.warp_id, warpRun.label warpLabel, warpRun.state FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE warp_id IN (475995,475962,475995,475962,475957);
     522
     523+---------+----------------------------+--------+---------+----------------------------+--------+
     524| chip_id | chipLabel                  | state  | warp_id | warpLabel                  | state  |
     525+---------+----------------------------+--------+---------+----------------------------+--------+
     526|  511806 | update.X.ecliptic.rp.broke | update |  475957 | update.X.ecliptic.rp.broke | update |
     527|  511811 | update.X.ecliptic.rp.broke | update |  475962 | update.X.ecliptic.rp.broke | update |
     528|  511845 | update.X.ecliptic.rp.broke | update |  475995 | update.X.ecliptic.rp.broke | update |
     529+---------+----------------------------+--------+---------+----------------------------+--------+
     530}}}
     531These one are in update, and under the broke label.
     532So we'll change their labels to ps_ud_MOPS so that they get processed and can diagnose the errors from there:
     533{{{
     534chiptool -dbname gpc1 -updaterun -chip_id 511806 -set_label ps_ud_MOPS
     535chiptool -dbname gpc1 -updaterun -chip_id 511811 -set_label ps_ud_MOPS
     536chiptool -dbname gpc1 -updaterun -chip_id 511845 -set_label ps_ud_MOPS
     537warptool -dbname gpc1 -updaterun -warp_id 475957 -set_label ps_ud_MOPS
     538warptool -dbname gpc1 -updaterun -warp_id 475962 -set_label ps_ud_MOPS
     539warptool -dbname gpc1 -updaterun -warp_id 475995 -set_label ps_ud_MOPS
     540}}}
     541
     542Some of these jobs now fail. Three of them all fail in the same way, complaining about:
     543{{{
     544    ppImage: psphotSourceSize.c:122: psphotSourceSizeReadout: Assertion `status' failed.
     545}}}
     546This implies that there is failure in the software, likelt because of code changes, so it cannot reprodue previous results.
     547
     548Get the exposure names for these, then reprocess them (since they are w-band we should use the following, see the PSNSC_MOPS page for details):
     549{{{
     550chiptool -dbname gpc1 -definebyquery -set_label PSNSC.wfix.20190321  -set_workdir neb://@HOST@.0/gpc1/PSNSC.wfix.20190321  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction SWEETSPOT -set_data_group PSNSC.wfix.20190321.x.date -exp_name o5535g0266o
     551chiptool -dbname gpc1 -definebyquery -set_label PSNSC.wfix.20190321  -set_workdir neb://@HOST@.0/gpc1/PSNSC.wfix.20190321  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction SWEETSPOT -set_data_group PSNSC.wfix.20190321.x.date -exp_name o5535g0271o
     552chiptool -dbname gpc1 -definebyquery -set_label PSNSC.wfix.20190321  -set_workdir neb://@HOST@.0/gpc1/PSNSC.wfix.20190321  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction SWEETSPOT -set_data_group PSNSC.wfix.20190321.x.date -exp_name o5535g0313o
     553}}}
     554and don't forget to add the label!
     555
     556Secret email from ippc18 gives the following:
     557{{{
     558count(*)    label   req_id  name    timestamp   fault   exp_id  dtime   utc_timestamp()
     5591   MOPS    918857  weryk_5C941E46  2019-03-21 23:30:40 25  263293  00:15:07    2019-03-21 23:45:01
     560}}}
     561Let's check out this exposure, 263293:
     562{{{
     563SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state, warpRun.warp_id, warpRun.label warp_label, warpRun.state warp_state FROM rawExp JOIN chipRun USING (exp_id) JOIN camRun USING (chip_id) JOIN fakeRun USING (cam_id) JOIN warpRun USING (fake_id) WHERE exp_id=263293;
     564}}}
     565{{{
     566+-------------+--------+---------+----------------------------+------------+---------+----------------------------+------------+
     567| exp_name    | exp_id | chip_id | chip_label                 | chip_state | warp_id | warp_label                 | warp_state |
     568+-------------+--------+---------+----------------------------+------------+---------+----------------------------+------------+
     569| o5535g0308o | 263293 |  167699 | goto_cleaned_redo          | cleaned    |  135866 | goto_cleaned               | cleaned    |
     570| o5535g0308o | 263293 |  511840 | update.X.ecliptic.rp.broke | update     |  475985 | update.X.ecliptic.rp.broke | update     |
     571+-------------+--------+---------+----------------------------+------------+---------+----------------------------+------------+
     572}}}
     573Might as well move this across too:
     574{{{
     575    chiptool -dbname gpc1 -updaterun -chip_id 511840 -set_label ps_ud_MOPS
     576    warptool -dbname gpc1 -updaterun -warp_id 475985 -set_label ps_ud_MOPS
     577}}}
     578
     579When ps_ud_MOPS processes this one, it faults. It complains about missing config files...so just reprocess it:
     580
     581But how should we reprocess it you ask? Well check the commment/filter etc from the rawExp table.
     582This too is w-band, so the following will do:
     583{{{
     584    chiptool -dbname gpc1 -definebyquery -set_label PSNSC.wfix.20190321  -set_workdir neb://@HOST@.0/gpc1/PSNSC.wfix.20190321  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction SWEETSPOT -set_data_group PSNSC.wfix.20190321.x.date -exp_name o5535g0308o
     585}}}
     586
     587Now cleanup all of the old stuff:
     588{{{
     589chiptool -dbname gpc1 -updaterun -chip_id 511806 -set_state goto_cleaned -set_label goto_cleaned
     590chiptool -dbname gpc1 -updaterun -chip_id 511811 -set_state goto_cleaned -set_label goto_cleaned
     591chiptool -dbname gpc1 -updaterun -chip_id 511845 -set_state goto_cleaned -set_label goto_cleaned
     592warptool -dbname gpc1 -updaterun -warp_id 475957 -set_state goto_cleaned -set_label goto_cleaned
     593warptool -dbname gpc1 -updaterun -warp_id 475962 -set_state goto_cleaned -set_label goto_cleaned
     594warptool -dbname gpc1 -updaterun -warp_id 475995 -set_state goto_cleaned -set_label goto_cleaned
     595chiptool -dbname gpc1 -updaterun -chip_id 511840 -set_state goto_cleaned -set_label goto_cleaned
     596warptool -dbname gpc1 -updaterun -warp_id 475985 -set_state goto_cleaned -set_label goto_cleaned
     597}}}
     598
     599
     600
     601'''We need to add a stamp for the next set of tasks after update.OSS'''
     602
     603The LAP.PV3 stuff needs to updated next, and this will take a while. So it makes sense to add a new label that will be loaded automatically when pantasks starts up.
     604
     605To do this simply add the following like to the 'input' file in the stdscience directory:
     606{{
     607    add.label update.LAP.PV3
     608}}
     609put it in a sensible place too, like next to all the other labels (best take a copy of input too).
     610
     611
     612This new label currently has the highest priority, as it defaults to highest if there is no entry... this is not what we want. To add the label to the list of labels, and set the priority, use the label tool!
     613To check if it is present first:
     614{{{
     615labeltool -dbname gpc1 -listlabel -simple | grep update
     616}}}
     617
     618Add label and set priority:
     619{{{
     620labeltool -dbname gpc1 -definelabel -set_label update.LAP.PV3 -set_priority 301
     621}}}
     622In this case we want it to be a lower priority than MOPS.2 which was 310, so we chose 301 (and 1 higher that update.OSS hehehe)
     623
     624Give it a few minutes and ippmonitor should reflect these changes by showing it in the correct place.
     625
     626
     627'''There are some broken jobs on IPP monitor'''
     628
     629Most had memory problems and needed reverting.
     630
     631
    240632=== Friday : 2019.03.22 ===
     633
     634'''Broken jobs breakfast'''
     635
     636What better way to start the day than a cup of coffee and some red stalled jobs on IPP monitor.
     637
     638A ps_ud_MOPS job stuck in the chip stage:
     639{{{
     640exp_name    chip    exp_id  chip_id state   label       data_group                      dist_group
     641o5102g0194o XY65    101412  1280264 update  ps_ud_MOPS  LAP.PV3.20140730.ipp.20141126   LAP.ThreePi
     642
     643SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state FROM rawExp JOIN chipRun USING (exp_id) WHERE exp_id=101412;
     644
     645+-------------+--------+---------+---------------------+------------+
     646| exp_name    | exp_id | chip_id | chip_label          | chip_state |
     647+-------------+--------+---------+---------------------+------------+
     648| o5102g0194o | 101412 | 1280264 | ps_ud_MOPS          | update     |
     649+-------------+--------+---------+---------------------+------------+
     650}}}
     651However, the log appears to be missing for this file...
     652{{{
     653    log file (LOG.IMFILE.UPDATE, 'neb://@HOST@.0/gpc1/LAP.PV3.20140730.ipp/2014/11/26/o5102g0194o.101412/o5102g0194o.101412.ch.1280264') not found
     654}}}
     655
     656I tried to revert the job:
     657{{{
     658    chiptool -revertprocessedimfile -fault 2 -label ps_ud_MOPS -dbname gpc1
     659}}}
     660Which did not work...
     661
     662Perhaps it should be reprocessed? It's LAP.PV3 in the r-band, so should use:
     663{{{
     664    chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190322 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190322 -set_reduction LAP_SCIENCE -exp_name o5102g0194o
     665}}}
     666I have submitted this:
     667
     668Query the exposure:
     669{{{
     670SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state, warpRun.warp_id, warpRun.label warp_label, warpRun.state warp_state FROM rawExp JOIN chipRun USING (exp_id) JOIN camRun USING (chip_id) JOIN fakeRun USING (cam_id) JOIN warpRun USING (fake_id) WHERE rawExp.exp_id=101412;
     671
     672+-------------+--------+---------+--------------------+------------+---------+--------------------+------------+
     673| exp_name    | exp_id | chip_id | chip_label         | chip_state | warp_id | warp_label         | warp_state |
     674+-------------+--------+---------+--------------------+------------+---------+--------------------+------------+
     675| o5102g0194o | 101412 |   29631 | SVS.20091005       | cleaned    |   13660 | SVS.20091005       | cleaned    |
     676| o5102g0194o | 101412 |  223654 | goto_cleaned.rerun | cleaned    |  191413 | goto_cleaned.rerun | cleaned    |
     677| o5102g0194o | 101412 |  693009 | goto_cleaned_redo  | cleaned    |  650278 | goto_cleaned       | cleaned    |
     678| o5102g0194o | 101412 | 1280264 | ps_ud_MOPS         | update     | 1260815 | ps_ud_MOPS         | update     |
     679| o5102g0194o | 101412 | 2129470 | mopsfix            | full       | 2071601 | mopsfix            | full       |
     680+-------------+--------+---------+--------------------+------------+---------+--------------------+------------+
     681}}}
     682All done, send the old stuff to clean:
     683{{{
     684    chiptool -dbname gpc1 -updaterun -chip_id 1280264 -set_state goto_cleaned -set_label goto_cleaned
     685    warptool -dbname gpc1 -updaterun -warp_id ??????? -set_state goto_cleaned -set_label goto_cleaned
     686}}}
     687
     688Exp Name    Class   Exp ID  Chip ID State   Label       data grp                            dist grp    Fault
     689o5426g0329o XY26    208836  1345211 update  ps_ud_MOPS  LAP.PV3.20140730.bigmem.20141219    LAP.ThreePi 2
     690Trying to revert this failed...so reprocess it (it's i-band):
     691{{{
     692    chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190322 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190322 -set_reduction LAP_SCIENCE -exp_name o5426g0329o
     693}}}
     694Reprocessed ok, time to cleanup the old stuff:
     695{{{
     696    chiptool -dbname gpc1 -updaterun -chip_id 1345211 -set_state goto_cleaned -set_label goto_cleaned
     697    warptool -dbname gpc1 -updaterun -warp_id 1301704 -set_state goto_cleaned -set_label goto_cleaned
     698}}}
     699
     700
     701'''warp'''
     702
     703There are two ps_ud_MOPS stuck in the warp stage:
     704{{{
     705Exp Name     Exp ID  Chip ID Cam ID  Fake ID Warp ID Skycell ID          State   Label       Fault
     706o6055g0196o  484983  1093091 1059583 1035273 1041865 skycell.2257.095    update  ps_ud_MOPS  2
     707o6055g0220o  485006  1093094 1059586 1035276 1041868 skycell.2257.095    update  ps_ud_MOPS  2
     708}}}
     709These both have the same error as the previous chip problem; no log file can be found...so reprocess. They are LAP.PV3 y-band:
     710{{{
     711    chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190322 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190322 -set_reduction LAP_SCIENCE -exp_name o6055g0196o
     712    chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190322 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190322 -set_reduction LAP_SCIENCE -exp_name o6055g0220o
     713}}}
     714I've sent them to reprocess... so check that they are done:
     715{{{
     716SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state, warpRun.warp_id, warpRun.label warp_label, warpRun.state warp_state FROM rawExp JOIN chipRun USING (exp_id) JOIN camRun USING (chip_id) JOIN fakeRun USING (cam_id) JOIN warpRun USING (fake_id) WHERE rawExp.exp_id IN (484983,485006);
     717
     718+-------------+--------+---------+--------------------+------------+---------+--------------------+------------+
     719| exp_name    | exp_id | chip_id | chip_label         | chip_state | warp_id | warp_label         | warp_state |
     720+-------------+--------+---------+--------------------+------------+---------+--------------------+------------+
     721| o5102g0194o | 101412 |   29631 | SVS.20091005       | cleaned    |   13660 | SVS.20091005       | cleaned    |
     722| o5102g0194o | 101412 |  223654 | goto_cleaned.rerun | cleaned    |  191413 | goto_cleaned.rerun | cleaned    |
     723| o5102g0194o | 101412 |  693009 | goto_cleaned_redo  | cleaned    |  650278 | goto_cleaned       | cleaned    |
     724| o5102g0194o | 101412 | 1280264 | ps_ud_MOPS         | update     | 1260815 | ps_ud_MOPS         | update     |
     725| o5102g0194o | 101412 | 2129470 | mopsfix            | full       | 2071601 | mopsfix            | full       |
     726+-------------+--------+---------+--------------------+------------+---------+--------------------+------------+
     727}}}
     728
     729They're now full, so clean 'em up
     730{{{
     731    warptool -dbname gpc1 -updaterun -warp_id 1041865 -set_state goto_cleaned -set_label goto_cleaned
     732    warptool -dbname gpc1 -updaterun -warp_id 1041868 -set_state goto_cleaned -set_label goto_cleaned
     733}}}
     734
     735
     736
     737'''Lots of update.LAP.PV3 have missing logs'''
     738
     739All of this show a warp state of full, but still have a fault of 2, and they complain about missing logs file along the lines of:
     740{{{
     741    neb://stsci05.1/gpc1/LAP.PV3.20140730.ipp/2014/10/04/o5358g0257o.179786/o5358g0257o.179786.wrp.1069329.skycell.0938.038
     742}}}
     743i.e. they all start neb://stsciXX.X
     744
     745This looks different to yesterday's issue... they are
     746
     747
     748 SELECT rawExp.exp_name,rawExp.exp_id,chipRun.chip_id,warpRun.warp_id,warpSkyfile.skycell_id,warpRun.state warp_state,warpRun.label warp_label,warpRun.data_group,warpSkyfile.fault,warpSkyfile.path_base FROM warpSkyfile, warpRun, fakeRun, camRun, chipRun, rawExp WHERE warpSkyfile.warp_id = warpRun.warp_id AND warpRun.fake_id = fakeRun.fake_id AND fakeRun.cam_id = camRun.cam_id AND camRun.chip_id = chipRun.chip_id AND chipRun.exp_id = rawExp.exp_id AND warpSkyfile.fault != 0 AND warpRun.warp_id = '1302416' AND warpSkyfile.skycell_id like 'skycell.0788.077' LIMIT 80;
     749
     750    Maybe do this from earlier in the week:
     751    Weird warp fault where the state is full:
     752    {{{
     753        o5410g0454o 201542  1345623 1321690 1295801 1302416 skycell.0788.077    full    update.LAP.PV3
     754    }}}
     755    Query the neb-workdir (note the wild card at the end):
     756    {{{
     757        neb-ls -p -l neb://stsci08.1/gpc1/LAP.PV3.20140730.bigmem/2014/12/19/o5410g0454o.201542/o5410g0454o.201542.wrp.1302416.skycell.0788.077%
     758    }}}
     759   
     760    We get a warning about the stats file being missing:
     761    {{{
     762        2019/03/22 13:53:48 | ipp113 | FATAL | Nebulous::Client::find_instances - unhandled fault - database error: no instances available for key: neb:///gpc1/LAP.PV3.20140730.bigmem/2014/12/19/o5410g0454o.201542/o5410g0454o.201542.wrp.1302416.skycell.0788.077.stats at /usr/lib64/perl5/site_perl/5.8.8/Nebulous/Server.pm line 1991, <DATA> line 14.
     763        unhandled fault - database error: no instances available for key: neb:///gpc1/LAP.PV3.20140730.bigmem/2014/12/19/o5410g0454o.201542/o5410g0454o.201542.wrp.1302416.skycell.0788.077.stats at /usr/lib64/perl5/site_perl/5.8.8/Nebulous/Server.pm line 1991, <DATA> line 14.
     764    }}}
     765
     766    To solve this, we will have to put the warp to clean:
     767    {{{
     768        warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1477774
     769    }}}
     770    check that is has cleaned succesfully (query the database).
     771    {{{
     772        SELECT warpRun.warp_id, warpRun.state FROM warpRun WHERE warpRun.warp_id IN (1069329,1073367,1073369,1073428,1074105,1075041,1075048,1075067,1075098,1075975,1075977,1076071,1078051,1078064,1078068,1078071,1078077,1078082,1078083,1078084,1078085,1078087,1078088,1078090,1078091,1078092,1078093,1078094,1078095,1078096,1078098,1078099,1078110,1078112,1078133,1078136,1078138,1078139,1078143,1078144,1078145,1078147,1078173,1078748,1078753,1078760,1078768,1082284,1082971,1083016,1083027,1083255,1083912,1302416);
     773    }}}
     774
     775    Then set it to update:
     776    {{{
     777        warptool -dbname gpc1  -setskyfiletoupdate -set_label update.OSS -warp_id 1477774
     778    }}}
     779
     780That solved the majority, 139/141, issues. Two issues remain:
     7811) is a fault 4 so just revert it
     7822) the other is a fault 3, but has the same issues as the above, where the state is full but it is failing to find a specific log file. So -> set to clean -> set to update?
     783{{{
     784warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1079367
     785warptool -dbname gpc1  -setskyfiletoupdate -set_label update.OSS -warp_id 1079367
     786}}}
     787Might require more investigation...
     788
     789
     790
     791'''Broken chip in update.OSS'''
     792
     793'''Broken warp in update.OSS'''
     794
     795The broken warp in update.OSS wants ippb04...so will need to put that up into repair and revert the warptool job.
     796
     797
     798
     799
     800
     801'''Maybe Rob job not all good'''
     802
     803Maybe Rob job not all good
     804
     8051       MOPS    918892  weryk_5C95386D  2019-03-22 20:19:52     25      217487  00:10:09        2019-03-22 20:30:01
     8061       MOPS    918892  weryk_5C95386D  2019-03-22 20:19:52     25      217488  00:10:09        2019-03-22 20:30:01
     8071       MOPS    918892  weryk_5C95386D  2019-03-22 20:19:52     25      217490  00:10:09        2019-03-22 20:30:01
     8082       MOPS    918892  weryk_5C95386D  2019-03-22 20:19:52     25      217508  00:10:09        2019-03-22 20:30:01
     809Check 'em oot:
     810{{{
     811    SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state, warpRun.warp_id, warpRun.label warp_label, warpRun.state warp_state FROM rawExp JOIN chipRun USING (exp_id) JOIN camRun USING (chip_id) JOIN fakeRun USING (cam_id) JOIN warpRun USING (fake_id) WHERE exp_id IN (217487,217488,217490,217508);
     812
     813+-------------+--------+---------+--------------+------------+---------+-------------------+------------+
     814| exp_name    | exp_id | chip_id | chip_label   | chip_state | warp_id | warp_label        | warp_state |
     815+-------------+--------+---------+--------------+------------+---------+-------------------+------------+
     816| o5440g0432o | 217487 | 1267031 | goto_cleaned | cleaned    | 1214718 | MD09.PV3.20141104 | full       |
     817| o5440g0433o | 217488 | 1267032 | goto_cleaned | cleaned    | 1214735 | MD09.PV3.20141104 | full       |
     818| o5440g0436o | 217490 | 1267034 | goto_cleaned | cleaned    | 1214706 | MD09.PV3.20141104 | full       |
     819| o5440g0453o | 217508 | 1262954 | goto_cleaned | cleaned    | 1213228 | MD09.PV3.20141104 | full       |
     820+-------------+--------+---------+--------------+------------+---------+-------------------+------------+
     821}}}
     822The warps look good..., so maybe no problem here
     823
     824Still a bunch of pending things:
     825{{{
     826pstamptool -dbname ippRequestServer -dbserver ipp113 -pendingdependent -simple
     827
     828pstamptool -dbname ippRequestServer -dbserver ipp113 -pendingdependent -simple  | grep "new warp" | cut -f4 -d" " | awk '{ print " warptool -dbname gpc1 -listrun -simple -warp_id "$1}' | tcsh
     829
     830pstamptool -dbname ippRequestServer -dbserver ipp113 -pendingdependent -simple  | grep "new warp" | cut -f4 -d" " | awk '{ print " warptool -dbname gpc1 -listrun -simple -warp_id "$1}' | tcsh |  awk '{ print "chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id "$23" ; warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id "$1 }'
     831}}}
     832
     833warptool -dbname gpc1 -listrun -simple -warp_id 1196545
     834warptool -dbname gpc1 -listrun -simple -warp_id 1195189
     835warptool -dbname gpc1 -listrun -simple -warp_id 1195185
     836warptool -dbname gpc1 -listrun -simple -warp_id 1289563
     837warptool -dbname gpc1 -listrun -simple -warp_id 1289536
     838warptool -dbname gpc1 -listrun -simple -warp_id 1289497
     839warptool -dbname gpc1 -listrun -simple -warp_id 1289466
     840warptool -dbname gpc1 -listrun -simple -warp_id 1979742
     841warptool -dbname gpc1 -listrun -simple -warp_id 1979741
     842warptool -dbname gpc1 -listrun -simple -warp_id 1979739
     843warptool -dbname gpc1 -listrun -simple -warp_id 1979740
     844
     845warp    chip    exp
     8461196545 1247225 536246 o6222g0265o
     8471195189 1245021 661797 o6568g0237o
     8481195185 1245018 661777 o6568g0218o
     8491289563 1347972 841210 o7011g0400o
     8501289536 1347954 841190 o7011g0382o
     8511289497 1347936 841174 o7011g0364o
     8521289466 1347918 841156 o7011g0346o
     8531979742 2036560 1354013 o8120g0081o
     8541979741 2036559 1354012 o8120g0080o
     8551979739 2036558 1354011 o8120g0079o
     8561979740 2036557 1354010 o8120g0078o
     857
     858chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id
     859warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id
     860
     861chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id  1247225
     862chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id  1245021
     863chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id  1245018
     864chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id  1347972
     865chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id  1347954
     866chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id  1347936
     867chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id  1347918
     868chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id  2036560
     869chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id  2036559
     870chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id  2036558
     871chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id  2036557
     872
     873warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id 1196545
     874warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id 1195189
     875warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id 1195185
     876warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id 1289563
     877warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id 1289536
     878warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id 1289497
     879warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id 1289466
     880warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id 1979742
     881warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id 1979741
     882warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id 1979739
     883warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id 1979740
     884
     885
     886
     887More work from Weryk
     888
     8891       MOPS    918925  weryk_5C9579F7  2019-03-23 00:16:53     25      306270  00:13:08        2019-03-23 00:30:01
     8901       MOPS    918925  weryk_5C9579F7  2019-03-23 00:16:53     25      306271  00:13:08        2019-03-23 00:30:01
     8911       MOPS    918925  weryk_5C9579F7  2019-03-23 00:16:53     25      306274  00:13:08        2019-03-23 00:30:01
     8921       MOPS    918925  weryk_5C9579F7  2019-03-23 00:16:53     25      306275  00:13:08        2019-03-23 00:30:01
     8931       MOPS    918925  weryk_5C9579F7  2019-03-23 00:16:53     25      308291  00:13:08        2019-03-23 00:30:01
     8941       MOPS    918925  weryk_5C9579F7  2019-03-23 00:16:53     25      308294  00:13:08        2019-03-23 00:30:01
     8951       MOPS    918925  weryk_5C9579F7  2019-03-23 00:16:53     25      308298  00:13:08        2019-03-23 00:30:01
     8961       MOPS    918925  weryk_5C9579F7  2019-03-23 00:16:53     25      308994  00:13:08        2019-03-23 00:30:01
     8972       MOPS    918909  weryk_5C95761A  2019-03-23 00:17:15     25      606306  00:12:46        2019-03-23 00:30:01
     898
     899Check these out... they might be good since I've just updated the run on some other jobs
     900
     901
     902
     903
     904'''We need to prepare for nightly processing again'''
     905
     906
    241907
    242908=== Saturday : 2019.03.23 ===
    … …  
    247913
    248914
     915''' ganglia demon failed across the cluster '''
     916
     917JRF: Had to restart it on a whole load of machines... Add the command to fix it here (note that the ubuntu ones look a little strange)
     918
     919
     920''ps_ud_MOPS error'''
     921
     922o6473g0463o     XY44    627000  1347469 update  ps_ud_MOPS      LAP.PV3.20140730.bigmem.20141220        LAP.ThreePi     3       PS1     GPC1
     923Tried to revert, it failed. Then tried to update, it also failed.
     924
     925
     926So I'm just going to rerun the exposure:
     927chiptool -dbname gpc1 -definebyquery -set_label mopsfix  -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190323  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190323 -set_reduction LAP_SCIENCE -exp_name 627000
     928
     929Check it's done
     930SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state, warpRun.warp_id, warpRun.label warp_label, warpRun.state warp_state FROM rawExp JOIN chipRun USING (exp_id) JOIN camRun USING (chip_id) JOIN fakeRun USING (cam_id) JOIN warpRun USING (fake_id) WHERE exp_name="o6473g0463o";
     931
     932Cleanup the old stuff:
     933
     934
     935''' pstamps '''
     936
     93710960392 new warp 1999761 skycell.0751.033 gpc1 /data/ipp113.0/pstamp/work/2019/03/24/918963 ps_ud_MOPS F 0 0 MOPS 918963 /data/ipp113.0/pstamp/work/2019/03/24/918963/checkdep.10960392.log 500
     93810960394 new warp 1999761 skycell.0751.034 gpc1 /data/ipp113.0/pstamp/work/2019/03/24/918963 ps_ud_MOPS F 0 0 MOPS 918963 /data/ipp113.0/pstamp/work/2019/03/24/918963/checkdep.10960394.log 500
     93910960485 new warp 1875836 skycell.1195.025 gpc1 /data/ipp113.0/pstamp/work/2019/03/24/918965 ps_ud_MOPS F 0 0 MOPS 918965 /data/ipp113.0/pstamp/work/2019/03/24/918965/checkdep.10960485.log 500
     94010960487 new warp 1875812 skycell.1195.025 gpc1 /data/ipp113.0/pstamp/work/2019/03/24/918965 ps_ud_MOPS F 0 0 MOPS 918965 /data/ipp113.0/pstamp/work/2019/03/24/918965/checkdep.10960487.log 500
     94110960522 new warp 1712458 skycell.1547.065 gpc1 /data/ipp113.0/pstamp/work/2019/03/24/918965 ps_ud_MOPS F 0 0 MOPS 918965 /data/ipp113.0/pstamp/work/2019/03/24/918965/checkdep.10960522.log 500
     942
     943SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state, warpRun.warp_id, warpRun.label warp_label, warpRun.state warp_state FROM rawExp JOIN chipRun USING (exp_id) JOIN camRun USING (chip_id) JOIN fakeRun USING (cam_id) JOIN warpRun USING (fake_id) WHERE exp_id IN (1999761,1875836,1875812,1712458);
     944
     945+-------------+---------+---------+------------+------------+---------+-----------------+------------+
     946| exp_name    | exp_id  | chip_id | chip_label | chip_state | warp_id | warp_label      | warp_state |
     947+-------------+---------+---------+------------+------------+---------+-----------------+------------+
     948| o7487g0126o | 1073157 | 1765399 | ps_ud_MOPS | update     | 1712458 | ps_ud_MOPS      | update     |
     949| o7869g0337o | 1236447 | 1929751 | ps_ud_MOPS | update     | 1875812 | ps_ud_MOPS      | update     |
     950| o7869g0356o | 1236465 | 1929770 | ps_ud_MOPS | update     | 1875836 | update_recovery | update     |
     951| o8185g0309o | 1379681 | 2057059 | ps_ud_MOPS | update     | 1999761 | ps_ud_MOPS      | update     |
     952+-------------+---------+---------+------------+------------+---------+-----------------+------------+
     953
     954I'm just gonna try and update them, see what happens.
     955chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id 1765399
     956chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id 1929751
     957chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id 1929770
     958chiptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -chip_id 2057059
     959warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id 1712458
     960warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id 1875812
     961warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id 1875836
     962warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS -warp_id 1999761
     963
    249964=== Sunday : 2019.03.24 ===
     965
     966There are a bunch of chips and warps stuck in the full state...
     967
     968SELECT rawExp.exp_name,rawExp.exp_id,chipRun.chip_id,chipRun.state,chipRun.label,chipProcessedImfile.fault,rawExp.camera FROM chipRun, rawExp, chipProcessedImfile WHERE chipRun.exp_id = rawExp.exp_id AND chipProcessedImfile.exp_id = chipRun.exp_id AND chipProcessedImfile.chip_id = chipRun.chip_id AND chipProcessedImfile.fault != 0 AND chipRun.label like 'update.LAP.PV3' AND chipRun.state LIKE "full" LIMIT 80 OFFSET 0;
     969
     970+-------------+--------+---------+-------+----------------+-------+--------+
     971| exp_name    | exp_id | chip_id | state | label          | fault | camera |
     972+-------------+--------+---------+-------+----------------+-------+--------+
     973| o5252g0508o | 141142 | 1047387 | full  | update.LAP.PV3 |     2 | GPC1   |
     974| o5653g0315o | 320147 | 1047394 | full  | update.LAP.PV3 |     2 | GPC1   |
     975| o6396g0082o | 599426 | 1047433 | full  | update.LAP.PV3 |     2 | GPC1   |
     976| o6412g0186o | 607383 | 1047440 | full  | update.LAP.PV3 |     2 | GPC1   |
     977| o6741g0472o | 720298 | 1047444 | full  | update.LAP.PV3 |     2 | GPC1   |
     978| o6741g0491o | 720316 | 1047445 | full  | update.LAP.PV3 |     2 | GPC1   |
     979| o6782g0167o | 732865 | 1047446 | full  | update.LAP.PV3 |     2 | GPC1   |
     980| o6782g0200o | 732898 | 1047451 | full  | update.LAP.PV3 |     2 | GPC1   |
     981| o6782g0200o | 732898 | 1047451 | full  | update.LAP.PV3 |     2 | GPC1   |
     982| o5297g0292o | 156364 | 1047462 | full  | update.LAP.PV3 |     2 | GPC1   |
     983| o5297g0292o | 156364 | 1047462 | full  | update.LAP.PV3 |     2 | GPC1   |
     984| o5306g0280o | 159686 | 1047465 | full  | update.LAP.PV3 |     2 | GPC1   |
     985| o5653g0302o | 320134 | 1047468 | full  | update.LAP.PV3 |     2 | GPC1   |
     986| o6001g1268o | 465334 | 1047470 | full  | update.LAP.PV3 |     2 | GPC1   |
     987| o6001g1268o | 465334 | 1047470 | full  | update.LAP.PV3 |     2 | GPC1   |
     988| o6001g1269o | 465335 | 1047471 | full  | update.LAP.PV3 |     2 | GPC1   |
     989| o6001g1269o | 465335 | 1047471 | full  | update.LAP.PV3 |     2 | GPC1   |
     990| o6001g1269o | 465335 | 1047471 | full  | update.LAP.PV3 |     2 | GPC1   |
     991| o6001g1286o | 465347 | 1047472 | full  | update.LAP.PV3 |     2 | GPC1   |
     992| o6001g1287o | 465350 | 1047473 | full  | update.LAP.PV3 |     2 | GPC1   |
     993| o6013g0292o | 469066 | 1047474 | full  | update.LAP.PV3 |     2 | GPC1   |
     994| o6013g0294o | 469068 | 1047475 | full  | update.LAP.PV3 |     2 | GPC1   |
     995| o6013g0294o | 469068 | 1047475 | full  | update.LAP.PV3 |     2 | GPC1   |
     996| o6013g0294o | 469068 | 1047475 | full  | update.LAP.PV3 |     2 | GPC1   |
     997| o6013g0294o | 469068 | 1047475 | full  | update.LAP.PV3 |     2 | GPC1   |
     998| o6013g0294o | 469068 | 1047475 | full  | update.LAP.PV3 |     2 | GPC1   |
     999+-------------+--------+---------+-------+----------------+-------+--------+
     1000
     1001chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1047387
     1002chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1047394
     1003chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1047433
     1004chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1047440
     1005chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1047444
     1006chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1047445
     1007chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1047446
     1008chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1047451
     1009chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1047462
     1010chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1047465
     1011chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1047468
     1012chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1047470
     1013chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1047471
     1014chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1047472
     1015chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1047473
     1016chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1047474
     1017chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1047475
     1018
     1019chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1047387
     1020chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1047394
     1021chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1047433
     1022chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1047440
     1023chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1047444
     1024chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1047445
     1025chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1047446
     1026chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1047451
     1027chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1047462
     1028chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1047465
     1029chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1047468
     1030chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1047470
     1031chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1047471
     1032chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1047472
     1033chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1047473
     1034chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1047474
     1035chiptool -dbname gpc1  -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1047475
     1036
     1037HOWEVER, this failed and I'm not sure why...
     1038So there are now some fault 2 entries with goto_cleaned...
     1039FIX ON MONDAY!
     1040
     1041{{{
     1042SELECT rawExp.exp_name,rawExp.exp_id,chipRun.chip_id,chipRun.state,chipRun.label,chipProcessedImfile.fault,rawExp.camera FROM chipRun, rawExp, chipProcessedImfile WHERE chipRun.exp_id = rawExp.exp_id AND chipProcessedImfile.exp_id = chipRun.exp_id AND chipProcessedImfile.chip_id = chipRun.chip_id AND chipRun.chip_id IN (1047387,1047394,1047433,1047440,1047444,1047445,1047446,1047451,1047462,1047465,1047468,1047470,1047471,1047472,1047473,1047474,1047475);
     1043}}}
     1044
     1045Remaining chip issues:
     1046
     1047All 4 of them:
     1048o5280g0301o     XY01    151113  1059611 update  update.LAP.PV3
     1049o5280g0317o     XY01    151129  1059616 update  update.LAP.PV3
     1050o5280g0316o     XY01    151128  1059817 update  update.LAP.PV3
     1051o6091g0094o     XY22    498248  1060186 update  update.LAP.PV3 
     1052Complain about a missing mdc config file:
     1053{{{
     1054WARNING: Config dump file neb://@HOST@.0/gpc1/LAP.PV3.20140730/2014/09/01/o5280g0317o.151129/o5280g0317o.151129.ch.1059616.XY01.ppImage.mdc is missing. Using current recipes and file rules.
     1055}}}
     1056For the first 3 it also complains about burntool state not matching that in the chip_imfile.pl
     1057
     1058Sent them to revert...perhaps they will clear themselves up.
     1059TODO TOMORROW MORNING:
     1060Perhaps revert these to sort them out, or reprocess.
     1061
     1062warps:
     1063
     1064SELECT rawExp.exp_name,rawExp.exp_id,chipRun.chip_id,warpRun.warp_id,warpSkyfile.skycell_id,warpRun.state,warpRun.label,warpSkyfile.fault,rawExp.camera FROM warpSkyfile, warpRun, fakeRun, camRun, chipRun, rawExp WHERE warpSkyfile.warp_id = warpRun.warp_id AND warpRun.fake_id = fakeRun.fake_id AND fakeRun.cam_id = camRun.cam_id AND camRun.chip_id = chipRun.chip_id AND chipRun.exp_id = rawExp.exp_id AND warpSkyfile.fault != 0 AND warpRun.label like 'update.LAP.PV3' LIMIT 80 OFFSET 0;
     1065
     1066+-------------+--------+---------+---------+------------------+--------+----------------+-------+--------+
     1067| exp_name    | exp_id | chip_id | warp_id | skycell_id       | state  | label          | fault | camera |
     1068+-------------+--------+---------+---------+------------------+--------+----------------+-------+--------+
     1069| o6013g0286o | 469059 | 1047397 | 1004746 | skycell.0840.097 | full   | update.LAP.PV3 |     2 | GPC1   |
     1070| o6782g0167o | 732865 | 1047446 | 1004795 | skycell.0841.016 | full   | update.LAP.PV3 |     2 | GPC1   |
     1071| o6782g0200o | 732898 | 1047451 | 1004800 | skycell.0840.017 | full   | update.LAP.PV3 |     2 | GPC1   |
     1072| o5297g0307o | 156380 | 1047463 | 1004812 | skycell.0759.058 | full   | update.LAP.PV3 |     2 | GPC1   |
     1073| o5649g0216o | 318039 | 1048195 | 1003988 | skycell.0839.048 | update | update.LAP.PV3 |     4 | GPC1   |
     1074+-------------+--------+---------+---------+------------------+--------+----------------+-------+--------+
     1075
     1076
     1077
     1078To solve this, we will have to put the warps to clean:
     1079{{{
     1080    warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id XXXXX
     1081}}}
     1082check that is has cleaned succesfully (query the database). Then set it to update:
     1083{{{
     1084warptool -dbname gpc1  -setskyfiletoupdate -set_label update.OSS -warp_id XXXXX
     1085}}}
     1086
     1087warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1004746
     1088warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1004795
     1089warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1004800
     1090warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1004812
     1091warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1003988
     1092
     1093SELECT rawExp.exp_name,rawExp.exp_id,chipRun.chip_id,warpRun.warp_id,warpSkyfile.skycell_id,warpRun.state,warpRun.label,warpSkyfile.fault,rawExp.camera FROM warpSkyfile, warpRun, fakeRun, camRun, chipRun, rawExp WHERE warpSkyfile.warp_id = warpRun.warp_id AND warpRun.fake_id = fakeRun.fake_id AND fakeRun.cam_id = camRun.cam_id AND camRun.chip_id = chipRun.chip_id AND chipRun.exp_id = rawExp.exp_id AND warpSkyfile.warp_id IN (1004746,1004795,1004800,1004812,1003988);
     1094
     1095warptool -dbname gpc1  -setskyfiletoupdate -set_label update.OSS -warp_id 1004746
     1096warptool -dbname gpc1  -setskyfiletoupdate -set_label update.OSS -warp_id 1004795
     1097warptool -dbname gpc1  -setskyfiletoupdate -set_label update.OSS -warp_id 1004800
     1098warptool -dbname gpc1  -setskyfiletoupdate -set_label update.OSS -warp_id 1004812
     1099warptool -dbname gpc1  -setskyfiletoupdate -set_label update.OSS -warp_id 1003988
     1100