== PS1 IPP Czar Logs for the week 2019.03.25 - 2019.03.31 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

Czar of the week: CCL, with EAM as czar-lord (previously: JRF + MEH)

=== Monday : 2019.03.25 ===

'''LAP.PV3 faulted:''' by JRF

The following are broken:
{{{
o5280g0301o XY01    151113  1059611 update  update.LAP.PV3  LAP.PV3.20140730.20140901   LAP.ThreePi 2 
o5280g0317o XY01    151129  1059616 update  update.LAP.PV3  LAP.PV3.20140730.20140901   LAP.ThreePi 2 
o5280g0316o XY01    151128  1059817 update  update.LAP.PV3  LAP.PV3.20140730.20140901   LAP.ThreePi 2 
o6091g0094o XY22    498248  1060186 update  update.LAP.PV3  LAP.PV3.20140730.20140901   LAP.ThreePi 2 
o5582g0644o XY44    283730  1062952 update  update.LAP.PV3  LAP.PV3.20140730.20140905   LAP.ThreePi 2 
o5582g0647o XY45    283733  1062954 update  update.LAP.PV3  LAP.PV3.20140730.20140905   LAP.ThreePi 2 
o5582g0665o XY44    283751  1062956 update  update.LAP.PV3  LAP.PV3.20140730.20140905   LAP.ThreePi 2 
o5582g0649o XY45    283735  1062992 update  update.LAP.PV3  LAP.PV3.20140730.20140905   LAP.ThreePi 2 
o5719g0065o XY73    344295  1063341 update  update.LAP.PV3  LAP.PV3.20140730.20140905   LAP.ThreePi 2 
o5282g0287o XY01    152148  1063529 update  update.LAP.PV3  LAP.PV3.20140730.20140905   LAP.ThreePi 2 
}}}

All of these are complaining about some missing mdc files, so it fails with a poor burntool state. I will try to reprocess them:
{{{
chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190325 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190325 -set_reduction LAP_SCIENCE -exp_name XXXXXX
}}}


Check they are done:
{{{
SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state FROM rawExp JOIN chipRun USING (exp_id) WHERE exp_name IN ("o5280g0301o","o5280g0317o","o5280g0316o","o6091g0094o","o5582g0644o","o5582g0647o","o5582g0665o","o5582g0649o","o5719g0065o","o5282g0287o");
}}}
They are not done, and they contiue to fail...
However, two of them made it through!
o6091g0094o
o5719g0065o
{{{
SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state, warpRun.warp_id, warpRun.state warp_state FROM rawExp JOIN chipRun USING (exp_id) JOIN camRun USING (chip_id) JOIN fakeRun USING (cam_id) JOIN warpRun USING (fake_id) WHERE exp_name IN ("o6091g0094o","o5719g0065o");
}}}

These two had slightly different errors. They did not complain about the burntool, so reprocessing worked for them.

Instead, they are complaining about a missing psf file. Take a look at the missing file using:
{{{
neb-stat -validate
}}}
we can see that is does not exist:
{{{
instance location:
      0                     NON-EXISTANT file:///data/ipp120.1/nebulous/32/5c/9229835926.gpc1:LAP.PV3.20140730:2014:09:01:o6091g0094o.498248:o6091g0094o.498248.ch.1060186.XY22.psf
}}}
Indeed, doing 'ls' on the file location tells us that it does not exist.

The best way to fix this is to move that neb location to bad.
{{{
neb-mv neb://@HOST@.0/gpc1/LAP.PV3.20140730/2014/09/01/o6091g0094o.498248/o6091g0094o.498248.ch.1060186.XY22.psf neb://@HOST@.0/gpc1/LAP.PV3.20140730/2014/09/01/o6091g0094o.498248/o6091g0094o.498248.ch.1060186.XY22.psf.bad
}}}

Then we can reprocess the file and it should hopefully make a new psf file:
{{{
    chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 -chip_id 1060186
}}}

'''burntool issues''' by JRF

The remaining issues complain about the burntool state. We have little information, it just says that:
{{{
    burntool state vs burntoolStateGood : 0 vs 14
Image burntool version does not match current accepted version. at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/chip_imfile.pl line 834
    main::my_die('Image burntool version does not match current accepted version.', 151113, 2131394, 'XY01', 2) called at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/chip_imfile.pl line 328
}}}

Check out the uri:
{{{
neb-stat -validate neb://ipp047.0/gpc1/20100325/o5280g0301o/o5280g0301o.ota01.burn.tbl
}}}
Note that the neb file it the same as the fits file in the error log, but with '''.burn.tbl''' (I didn't include the fits part above)

We find that this burn.tbl file has 0 size, and it's located on an available node ipp070.0

Check out: https://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/PS1_IPP_Czarlog_20180521
for info on how to fix this:

Ok, so we are going to fix this as '''ippitc''', as it's probably safer. Run the burntool fix:
{{{
ipp_apply_burntool_fix.pl --exp_name o5280g0301o --class_id XY01 --verbose --dbname gpc1
}}}
Make sure to give it the correct class_id.

It will spew out some useful information on what's happening e.g. that it has removed the old stuff, and made some new ones. It's best to check that it created some new burn tables, which can be done by looking up the table again:
{{{
    neb-stat -validate neb://ipp047.0/gpc1/20100325/o5280g0301o/o5280g0301o.ota01.burn.tbl
}}}
Running this we see two new copies, in new places, they are the same size, and that size is non-zero!

Repeat for the remaining:
{{{
    ipp_apply_burntool_fix.pl --exp_name o5280g0316o --class_id XY01 --verbose --dbname gpc1
    ipp_apply_burntool_fix.pl --exp_name o5582g0644o --class_id XY44 --verbose --dbname gpc1
    ipp_apply_burntool_fix.pl --exp_name o5582g0647o --class_id XY45 --verbose --dbname gpc1
    ipp_apply_burntool_fix.pl --exp_name o5582g0665o --class_id XY44 --verbose --dbname gpc1
    ipp_apply_burntool_fix.pl --exp_name o5582g0649o --class_id XY45 --verbose --dbname gpc1
    ipp_apply_burntool_fix.pl --exp_name o5719g0065o --class_id XY73 --verbose --dbname gpc1
    ipp_apply_burntool_fix.pl --exp_name o5282g0287o --class_id XY01 --verbose --dbname gpc1
    ipp_apply_burntool_fix.pl --exp_name o5582g0608o --class_id XY44 --verbose --dbname gpc1
}}}


''' Missing PSF files ''' by JRF

This was originally solved by some reprocessing (see the two that worked above). However, the correct way to fix it is as follows:

The errors complain about a config file, psf file, and burn tool. So look at all of them:
{{{
    $> neb-locate neb://@HOST@.0/gpc1/LAP.PV3.20140730/2014/09/01/o5280g0317o.151129/o5280g0317o.151129.ch.1059616.XY01.ppImage.mdc
    no instances

    $> neb-locate neb://@HOST@.0/gpc1/LAP.PV3.20140730/2014/09/01/o5280g0317o.151129/o5280g0317o.151129.ch.1059616.XY01.psf
    no instances

    $> neb-stat -validate neb://ipp047.0/gpc1/20100325/o5280g0317o/o5280g0317o.ota01.burn.tbl
    object id:             3915331061
    key:                   gpc1/20100325/o5280g0317o/o5280g0317o.ota01.burn.tbl
    read lock:             0
    write lock:            
    epoch:                 2014-08-31 18:02:51
    mtime:                 0000-00-00 00:00:00
    md5sum count:          1
    requested instances:   1
    available instances:   1
    existing instances:    1
    total instances:       1
    instances location:  
      1 d41d8cd98f00b204e9800998ecf8427e file:///data/ipp069.0/nebulous/b3/3f/5048738853.gpc1:20100325:o5280g0317o:o5280g0317o.ota01.burn.tbl

}}}
No instances for the first two, and the burn.tbl has the magic 0 size MD5 sum '''d41d8cd98f00b204e9800998ecf8427e'''

So, first off let's make new burn.tbls and then revert:
{{{
    ipp_apply_burntool_fix.pl --exp_name o5280g0317o --class_id XY01 --verbose --dbname gpc1
    neb-stat -validate neb://ipp047.0/gpc1/20100325/o5280g0317o/o5280g0317o.ota01.burn.tbl
    chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 -chip_id 1059616
}}}


'''MOPS No Masks''' by JRF

MOPS have requested some exposures to be reprocessed with no mask:
{{{
chiptool -definebyquery -set_label mops.nomasktest  -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o8347g0145o
chiptool -definebyquery -set_label mops.nomasktest  -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o8347g0126o
chiptool -definebyquery -set_label mops.nomasktest  -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o8347g0107o
chiptool -definebyquery -set_label mops.nomasktest  -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o8347g0088o
}}}
Check they are done:
{{{
SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state, warpRun.warp_id, warpRun.state warp_state FROM rawExp JOIN chipRun USING (exp_id) JOIN camRun USING (chip_id) JOIN fakeRun USING (cam_id) JOIN warpRun USING (fake_id) WHERE exp_name IN ("o8347g0145o","o8347g0126o","o8347g0107o","o8347g0088o");
}}}
They're done


''' Prep for nightly ''' by JRF

Have created a config for targeting:
{{{
    ipphosts.mhpcc.config.190325.ready_for_nightly_and_more
}}}
which contains the config from earlier in the month before we were targeting some new nodes for updates. It also includes 4 of the new nodes, ipp127-ipp130, as these seem ok to be used (the ones after this cannot be accessed from the console yet).
  
=== Tuesday : 2019.03.26 ===

JRF: Don't forget about these that I couldn't get to update again from last Sunday:
{{{
    SELECT rawExp.exp_name,rawExp.exp_id,chipRun.chip_id,chipRun.state,chipRun.label,chipProcessedImfile.fault,rawExp.camera FROM chipRun, rawExp, chipProcessedImfile WHERE chipRun.exp_id = rawExp.exp_id AND chipProcessedImfile.exp_id = chipRun.exp_id AND chipProcessedImfile.chip_id = chipRun.chip_id AND chipRun.chip_id IN (1047387,1047394,1047433,1047440,1047444,1047445,1047446,1047451,1047462,1047465,1047468,1047470,1047471,1047472,1047473,1047474,1047475);
}}}

=== Wednesday : 2019.03.27 ===

=== Thursday : 2019.03.28 ===

=== Friday : 2019.03.29 ===

=== Saturday : 2019.03.30 ===

=== Sunday : 2019.03.31 ===