IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Version 1 and Version 2 of PS1_IPP_Czarlog_20190325


Ignore:
Timestamp:
Mar 25, 2019, 2:55:58 PM (7 years ago)
Author:
fairlamb
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • PS1_IPP_Czarlog_20190325

    v1 v2  
    44(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])
    55
     6Czar of the week: CCL, with EAM as czar-lord (previously: JRF + MEH)
     7
    68=== Monday : 2019.03.25 ===
    79
    810'''LAP.PV3 faulted:''' by JRF
    911
    10 We have some faults:
     12The following are broken:
    1113{{{
    12 o5280g0301o     XY01    151113  1059611 update  update.LAP.PV3  LAP.PV3.20140730.20140901       LAP.ThreePi     2
    13 o5280g0317o     XY01    151129  1059616 update  update.LAP.PV3  LAP.PV3.20140730.20140901       LAP.ThreePi     2
    14 o5280g0316o     XY01    151128  1059817 update  update.LAP.PV3  LAP.PV3.20140730.20140901       LAP.ThreePi     2
    15 o6091g0094o     XY22    498248  1060186 update  update.LAP.PV3  LAP.PV3.20140730.20140901       LAP.ThreePi     2
    16 o5582g0644o     XY44    283730  1062952 update  update.LAP.PV3  LAP.PV3.20140730.20140905       LAP.ThreePi     2
    17 o5582g0647o     XY45    283733  1062954 update  update.LAP.PV3  LAP.PV3.20140730.20140905       LAP.ThreePi     2
    18 o5582g0665o     XY44    283751  1062956 update  update.LAP.PV3  LAP.PV3.20140730.20140905       LAP.ThreePi     2
    19 o5582g0649o     XY45    283735  1062992 update  update.LAP.PV3  LAP.PV3.20140730.20140905       LAP.ThreePi     2
    20 o5719g0065o     XY73    344295  1063341 update  update.LAP.PV3  LAP.PV3.20140730.20140905       LAP.ThreePi     2
    21 o5282g0287o     XY01    152148  1063529 update  update.LAP.PV3  LAP.PV3.20140730.20140905       LAP.ThreePi     2
     14o5280g0301o XY01    151113  1059611 update  update.LAP.PV3  LAP.PV3.20140730.20140901   LAP.ThreePi 2
     15o5280g0317o XY01    151129  1059616 update  update.LAP.PV3  LAP.PV3.20140730.20140901   LAP.ThreePi 2
     16o5280g0316o XY01    151128  1059817 update  update.LAP.PV3  LAP.PV3.20140730.20140901   LAP.ThreePi 2
     17o6091g0094o XY22    498248  1060186 update  update.LAP.PV3  LAP.PV3.20140730.20140901   LAP.ThreePi 2
     18o5582g0644o XY44    283730  1062952 update  update.LAP.PV3  LAP.PV3.20140730.20140905   LAP.ThreePi 2
     19o5582g0647o XY45    283733  1062954 update  update.LAP.PV3  LAP.PV3.20140730.20140905   LAP.ThreePi 2
     20o5582g0665o XY44    283751  1062956 update  update.LAP.PV3  LAP.PV3.20140730.20140905   LAP.ThreePi 2
     21o5582g0649o XY45    283735  1062992 update  update.LAP.PV3  LAP.PV3.20140730.20140905   LAP.ThreePi 2
     22o5719g0065o XY73    344295  1063341 update  update.LAP.PV3  LAP.PV3.20140730.20140905   LAP.ThreePi 2
     23o5282g0287o XY01    152148  1063529 update  update.LAP.PV3  LAP.PV3.20140730.20140905   LAP.ThreePi 2
    2224}}}
    2325
    2426All of these are complaining about some missing mdc files, so it fails with a poor burntool state. I will try to reprocess them:
    2527{{{
    26 chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190325 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190325 -set_reduction LAP_SCIENCE -exp_name o5280g0301o
    27 chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190325 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190325 -set_reduction LAP_SCIENCE -exp_name o5280g0317o
    28 chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190325 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190325 -set_reduction LAP_SCIENCE -exp_name o5280g0316o
    29 chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190325 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190325 -set_reduction LAP_SCIENCE -exp_name o6091g0094o
    30 chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190325 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190325 -set_reduction LAP_SCIENCE -exp_name o5582g0644o
    31 chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190325 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190325 -set_reduction LAP_SCIENCE -exp_name o5582g0647o
    32 chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190325 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190325 -set_reduction LAP_SCIENCE -exp_name o5582g0665o
    33 chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190325 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190325 -set_reduction LAP_SCIENCE -exp_name o5582g0649o
    34 chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190325 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190325 -set_reduction LAP_SCIENCE -exp_name o5719g0065o
    35 chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190325 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190325 -set_reduction LAP_SCIENCE -exp_name o5282g0287o
     28chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190325 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190325 -set_reduction LAP_SCIENCE -exp_name XXXXXX
    3629}}}
    3730
    … …  
    4134SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state FROM rawExp JOIN chipRun USING (exp_id) WHERE exp_name IN ("o5280g0301o","o5280g0317o","o5280g0316o","o6091g0094o","o5582g0644o","o5582g0647o","o5582g0665o","o5582g0649o","o5719g0065o","o5282g0287o");
    4235}}}
    43 They are not done...
    44 
     36They are not done, and they contiue to fail...
    4537However, two of them made it through!
    4638o6091g0094o
    … …  
    4941SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state, warpRun.warp_id, warpRun.state warp_state FROM rawExp JOIN chipRun USING (exp_id) JOIN camRun USING (chip_id) JOIN fakeRun USING (cam_id) JOIN warpRun USING (fake_id) WHERE exp_name IN ("o6091g0094o","o5719g0065o");
    5042}}}
    51 So we can clean these two? MAYBE! TO BE DONE LATER THIS MORNING
     43
     44These two had slightly different errors. They did not complain about the burntool, so reprocessing worked for them.
     45
     46Instead, they are complaining about a missing psf file. Take a look at the missing file using:
    5247{{{
    53 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1060186
    54 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1063341
     48neb-stat -validate
     49}}}
     50we can see that is does not exist:
     51{{{
     52instance location:
     53      0                     NON-EXISTANT file:///data/ipp120.1/nebulous/32/5c/9229835926.gpc1:LAP.PV3.20140730:2014:09:01:o6091g0094o.498248:o6091g0094o.498248.ch.1060186.XY22.psf
     54}}}
     55Indeed, doing 'ls' on the file location tells us that it does not exist.
     56
     57The best way to fix this is to move that neb location to bad.
     58{{{
     59neb-mv neb://@HOST@.0/gpc1/LAP.PV3.20140730/2014/09/01/o6091g0094o.498248/o6091g0094o.498248.ch.1060186.XY22.psf neb://@HOST@.0/gpc1/LAP.PV3.20140730/2014/09/01/o6091g0094o.498248/o6091g0094o.498248.ch.1060186.XY22.psf.bad
    5560}}}
    5661
    57 WILL SORT IT ALL OUT AFTER MEETING
     62Then we can reprocess the file and it should hopefully make a new psf file:
     63{{{
     64    chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 -chip_id 1060186
     65}}}
     66
     67'''burntool issues'''
     68
     69The remaining issues complain about the burntool state. We have little information, it just says that:
     70{{{
     71    burntool state vs burntoolStateGood : 0 vs 14
     72Image burntool version does not match current accepted version. at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/chip_imfile.pl line 834
     73    main::my_die('Image burntool version does not match current accepted version.', 151113, 2131394, 'XY01', 2) called at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/chip_imfile.pl line 328
     74}}}
     75
     76Check out the uri:
     77{{{
     78neb-stat -validate neb://ipp047.0/gpc1/20100325/o5280g0301o/o5280g0301o.ota01.burn.tbl
     79}}}
     80Note that the neb file it the same as the fits file in the error log, but with '''.burn.tbl''' (I didn't include the fits part above)
     81
     82We find that this burn.tbl file has 0 size, and it's located on an available node ipp070.0
     83
     84Check out: https://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/PS1_IPP_Czarlog_20180521
     85for info on how to fix this:
     86
     87Ok, so we are going to fix this as '''ippitc''', as it's probably safer. Run the burntool fix:
     88{{{
     89ipp_apply_burntool_fix.pl --exp_name o5280g0301o --class_id XY01 --verbose --dbname gpc1
     90}}}
     91Make sure to give it the correct class_id.
     92
     93It will spew out some useful information on what's happening e.g. that it has removed the old stuff, and made some new ones. It's best to check that it created some new burn tables, which can be done by looking up the table again:
     94{{{
     95    neb-stat -validate neb://ipp047.0/gpc1/20100325/o5280g0301o/o5280g0301o.ota01.burn.tbl
     96}}}
     97Running this we see two new copies, in new places, they are the same size, and that size is non-zero!
     98
     99Repeat for the remaining:
     100{{{
     101    ipp_apply_burntool_fix.pl --exp_name o5280g0316o --class_id XY01 --verbose --dbname gpc1
     102    ipp_apply_burntool_fix.pl --exp_name o5582g0644o --class_id XY44 --verbose --dbname gpc1
     103    ipp_apply_burntool_fix.pl --exp_name o5582g0647o --class_id XY45 --verbose --dbname gpc1
     104    ipp_apply_burntool_fix.pl --exp_name o5582g0665o --class_id XY44 --verbose --dbname gpc1
     105    ipp_apply_burntool_fix.pl --exp_name o5582g0649o --class_id XY45 --verbose --dbname gpc1
     106    ipp_apply_burntool_fix.pl --exp_name o5719g0065o --class_id XY73 --verbose --dbname gpc1
     107    ipp_apply_burntool_fix.pl --exp_name o5282g0287o --class_id XY01 --verbose --dbname gpc1
     108    ipp_apply_burntool_fix.pl --exp_name o5582g0608o --class_id XY44 --verbose --dbname gpc1
     109}}}
    58110
    59111
    60 '''MOPS No Masks''' by JRF
     112''' Missing PSF files ''' by JRF
     113
     114This was originally solved by some reprocessing (see the two that worked above). However, the correct way to fix it is as follows:
     115
     116The errors complain about a config file, psf file, and burn tool. So look at all of them:
     117{{{
     118    $> neb-locate neb://@HOST@.0/gpc1/LAP.PV3.20140730/2014/09/01/o5280g0317o.151129/o5280g0317o.151129.ch.1059616.XY01.ppImage.mdc
     119    no instances
     120
     121    $> neb-locate neb://@HOST@.0/gpc1/LAP.PV3.20140730/2014/09/01/o5280g0317o.151129/o5280g0317o.151129.ch.1059616.XY01.psf
     122    no instances
     123
     124    $> neb-stat -validate neb://ipp047.0/gpc1/20100325/o5280g0317o/o5280g0317o.ota01.burn.tbl
     125    object id:             3915331061
     126    key:                   gpc1/20100325/o5280g0317o/o5280g0317o.ota01.burn.tbl
     127    read lock:             0
     128    write lock:           
     129    epoch:                 2014-08-31 18:02:51
     130    mtime:                 0000-00-00 00:00:00
     131    md5sum count:          1
     132    requested instances:   1
     133    available instances:   1
     134    existing instances:    1
     135    total instances:       1
     136    instances location: 
     137      1 d41d8cd98f00b204e9800998ecf8427e file:///data/ipp069.0/nebulous/b3/3f/5048738853.gpc1:20100325:o5280g0317o:o5280g0317o.ota01.burn.tbl
     138
     139}}}
     140No instances for the first two, and the burn.tbl has the magic 0 size MD5 sum '''d41d8cd98f00b204e9800998ecf8427e'''
     141
     142So, first off let's make new burn.tbls and then revert:
     143{{{
     144    ipp_apply_burntool_fix.pl --exp_name o5280g0317o --class_id XY01 --verbose --dbname gpc1
     145    neb-stat -validate neb://ipp047.0/gpc1/20100325/o5280g0317o/o5280g0317o.ota01.burn.tbl
     146    chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 -chip_id 1059616
     147}}}
     148
     149
     150'''MOPS No Masks'''
    61151
    62152MOPS have requested some exposures to be reprocessed with no mask:
    … …  
    71161SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state, warpRun.warp_id, warpRun.state warp_state FROM rawExp JOIN chipRun USING (exp_id) JOIN camRun USING (chip_id) JOIN fakeRun USING (cam_id) JOIN warpRun USING (fake_id) WHERE exp_name IN ("o8347g0145o","o8347g0126o","o8347g0107o","o8347g0088o");
    72162}}}
     163They're done
    73164
     165
     166''' Prep for nightly ''' by JRF
     167
     168Have created a config for targeting:
     169{{{
     170    ipphosts.mhpcc.config.190325.ready_for_nightly_and_more
     171}}}
     172which contains the config from earlier in the month before we were targeting some new nodes for updates. It also includes 4 of the new nodes, ipp127-ipp130, as these seem ok to be used (the ones after this cannot be accessed from the console yet).
    74173 
    75174=== Tuesday : 2019.03.26 ===