| Version 2 (modified by , 7 years ago) ( diff ) |
|---|
PS1 IPP Czar Logs for the week 2019.03.25 - 2019.03.31
(Up to PS1 IPP Czar Logs)
Czar of the week: CCL, with EAM as czar-lord (previously: JRF + MEH)
Monday : 2019.03.25
LAP.PV3 faulted: by JRF
The following are broken:
o5280g0301o XY01 151113 1059611 update update.LAP.PV3 LAP.PV3.20140730.20140901 LAP.ThreePi 2 o5280g0317o XY01 151129 1059616 update update.LAP.PV3 LAP.PV3.20140730.20140901 LAP.ThreePi 2 o5280g0316o XY01 151128 1059817 update update.LAP.PV3 LAP.PV3.20140730.20140901 LAP.ThreePi 2 o6091g0094o XY22 498248 1060186 update update.LAP.PV3 LAP.PV3.20140730.20140901 LAP.ThreePi 2 o5582g0644o XY44 283730 1062952 update update.LAP.PV3 LAP.PV3.20140730.20140905 LAP.ThreePi 2 o5582g0647o XY45 283733 1062954 update update.LAP.PV3 LAP.PV3.20140730.20140905 LAP.ThreePi 2 o5582g0665o XY44 283751 1062956 update update.LAP.PV3 LAP.PV3.20140730.20140905 LAP.ThreePi 2 o5582g0649o XY45 283735 1062992 update update.LAP.PV3 LAP.PV3.20140730.20140905 LAP.ThreePi 2 o5719g0065o XY73 344295 1063341 update update.LAP.PV3 LAP.PV3.20140730.20140905 LAP.ThreePi 2 o5282g0287o XY01 152148 1063529 update update.LAP.PV3 LAP.PV3.20140730.20140905 LAP.ThreePi 2
All of these are complaining about some missing mdc files, so it fails with a poor burntool state. I will try to reprocess them:
chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190325 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190325 -set_reduction LAP_SCIENCE -exp_name XXXXXX
Check they are done:
SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state FROM rawExp JOIN chipRun USING (exp_id) WHERE exp_name IN ("o5280g0301o","o5280g0317o","o5280g0316o","o6091g0094o","o5582g0644o","o5582g0647o","o5582g0665o","o5582g0649o","o5719g0065o","o5282g0287o");
They are not done, and they contiue to fail... However, two of them made it through! o6091g0094o o5719g0065o
SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state, warpRun.warp_id, warpRun.state warp_state FROM rawExp JOIN chipRun USING (exp_id) JOIN camRun USING (chip_id) JOIN fakeRun USING (cam_id) JOIN warpRun USING (fake_id) WHERE exp_name IN ("o6091g0094o","o5719g0065o");
These two had slightly different errors. They did not complain about the burntool, so reprocessing worked for them.
Instead, they are complaining about a missing psf file. Take a look at the missing file using:
neb-stat -validate
we can see that is does not exist:
instance location:
0 NON-EXISTANT file:///data/ipp120.1/nebulous/32/5c/9229835926.gpc1:LAP.PV3.20140730:2014:09:01:o6091g0094o.498248:o6091g0094o.498248.ch.1060186.XY22.psf
Indeed, doing 'ls' on the file location tells us that it does not exist.
The best way to fix this is to move that neb location to bad.
neb-mv neb://@HOST@.0/gpc1/LAP.PV3.20140730/2014/09/01/o6091g0094o.498248/o6091g0094o.498248.ch.1060186.XY22.psf neb://@HOST@.0/gpc1/LAP.PV3.20140730/2014/09/01/o6091g0094o.498248/o6091g0094o.498248.ch.1060186.XY22.psf.bad
Then we can reprocess the file and it should hopefully make a new psf file:
chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 -chip_id 1060186
burntool issues
The remaining issues complain about the burntool state. We have little information, it just says that:
burntool state vs burntoolStateGood : 0 vs 14
Image burntool version does not match current accepted version. at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/chip_imfile.pl line 834
main::my_die('Image burntool version does not match current accepted version.', 151113, 2131394, 'XY01', 2) called at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/chip_imfile.pl line 328
Check out the uri:
neb-stat -validate neb://ipp047.0/gpc1/20100325/o5280g0301o/o5280g0301o.ota01.burn.tbl
Note that the neb file it the same as the fits file in the error log, but with .burn.tbl (I didn't include the fits part above)
We find that this burn.tbl file has 0 size, and it's located on an available node ipp070.0
Check out: https://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/PS1_IPP_Czarlog_20180521 for info on how to fix this:
Ok, so we are going to fix this as ippitc, as it's probably safer. Run the burntool fix:
ipp_apply_burntool_fix.pl --exp_name o5280g0301o --class_id XY01 --verbose --dbname gpc1
Make sure to give it the correct class_id.
It will spew out some useful information on what's happening e.g. that it has removed the old stuff, and made some new ones. It's best to check that it created some new burn tables, which can be done by looking up the table again:
neb-stat -validate neb://ipp047.0/gpc1/20100325/o5280g0301o/o5280g0301o.ota01.burn.tbl
Running this we see two new copies, in new places, they are the same size, and that size is non-zero!
Repeat for the remaining:
ipp_apply_burntool_fix.pl --exp_name o5280g0316o --class_id XY01 --verbose --dbname gpc1
ipp_apply_burntool_fix.pl --exp_name o5582g0644o --class_id XY44 --verbose --dbname gpc1
ipp_apply_burntool_fix.pl --exp_name o5582g0647o --class_id XY45 --verbose --dbname gpc1
ipp_apply_burntool_fix.pl --exp_name o5582g0665o --class_id XY44 --verbose --dbname gpc1
ipp_apply_burntool_fix.pl --exp_name o5582g0649o --class_id XY45 --verbose --dbname gpc1
ipp_apply_burntool_fix.pl --exp_name o5719g0065o --class_id XY73 --verbose --dbname gpc1
ipp_apply_burntool_fix.pl --exp_name o5282g0287o --class_id XY01 --verbose --dbname gpc1
ipp_apply_burntool_fix.pl --exp_name o5582g0608o --class_id XY44 --verbose --dbname gpc1
Missing PSF files by JRF
This was originally solved by some reprocessing (see the two that worked above). However, the correct way to fix it is as follows:
The errors complain about a config file, psf file, and burn tool. So look at all of them:
$> neb-locate neb://@HOST@.0/gpc1/LAP.PV3.20140730/2014/09/01/o5280g0317o.151129/o5280g0317o.151129.ch.1059616.XY01.ppImage.mdc
no instances
$> neb-locate neb://@HOST@.0/gpc1/LAP.PV3.20140730/2014/09/01/o5280g0317o.151129/o5280g0317o.151129.ch.1059616.XY01.psf
no instances
$> neb-stat -validate neb://ipp047.0/gpc1/20100325/o5280g0317o/o5280g0317o.ota01.burn.tbl
object id: 3915331061
key: gpc1/20100325/o5280g0317o/o5280g0317o.ota01.burn.tbl
read lock: 0
write lock:
epoch: 2014-08-31 18:02:51
mtime: 0000-00-00 00:00:00
md5sum count: 1
requested instances: 1
available instances: 1
existing instances: 1
total instances: 1
instances location:
1 d41d8cd98f00b204e9800998ecf8427e file:///data/ipp069.0/nebulous/b3/3f/5048738853.gpc1:20100325:o5280g0317o:o5280g0317o.ota01.burn.tbl
No instances for the first two, and the burn.tbl has the magic 0 size MD5 sum d41d8cd98f00b204e9800998ecf8427e
So, first off let's make new burn.tbls and then revert:
ipp_apply_burntool_fix.pl --exp_name o5280g0317o --class_id XY01 --verbose --dbname gpc1
neb-stat -validate neb://ipp047.0/gpc1/20100325/o5280g0317o/o5280g0317o.ota01.burn.tbl
chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 -chip_id 1059616
MOPS No Masks
MOPS have requested some exposures to be reprocessed with no mask:
chiptool -definebyquery -set_label mops.nomasktest -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o8347g0145o chiptool -definebyquery -set_label mops.nomasktest -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o8347g0126o chiptool -definebyquery -set_label mops.nomasktest -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o8347g0107o chiptool -definebyquery -set_label mops.nomasktest -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o8347g0088o
Check they are done:
SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state, warpRun.warp_id, warpRun.state warp_state FROM rawExp JOIN chipRun USING (exp_id) JOIN camRun USING (chip_id) JOIN fakeRun USING (cam_id) JOIN warpRun USING (fake_id) WHERE exp_name IN ("o8347g0145o","o8347g0126o","o8347g0107o","o8347g0088o");
They're done
Prep for nightly by JRF
Have created a config for targeting:
ipphosts.mhpcc.config.190325.ready_for_nightly_and_more
which contains the config from earlier in the month before we were targeting some new nodes for updates. It also includes 4 of the new nodes, ipp127-ipp130, as these seem ok to be used (the ones after this cannot be accessed from the console yet).
