PS1 IPP Czar Logs for the week 2019.04.26 - 2019.05.02

(Up to PS1 IPP Czar Logs)

Check out the new date!

Friday : 2019.04.26

JRF: Since we're now switching on Fridays it makes sense to have the czarlogs do the same.

Saturday : 2019.04.27

• MEH: queuing night stack and SSdiff once normal nightly processing finished for (OSS.nightlyscience, OSS.SS.nightlyscience)

Sunday : 2019.04.28

• MEH: since pstamp removed from daily restarts because auto-restart script still does not respect parsing jobs and this corrupting them and pantasks still needs to be restarted when Njobs~100k, have added a crontab on ippitc@ippc26 (where pstamp pantasks runs) that checks every hour if nearling ~100k and needs to be restarted (also with note if parsing jobs running)

Missing mdc file + burntool problem by JRF

Reporting missing .mdc file, so it fails in the burntool stage, so I reprocessed it (really should have tried updating, this was in the morning) Will reprocess:

chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190428 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190428 -set_reduction LAP_SCIENCE -exp_name o5444g0110o

Check they're finished:

SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state FROM rawExp JOIN chipRun USING (exp_id) WHERE exp_name IN ("o5444g0110o");

Now the mopsfix job is stuck as new in the chip stage, complaining about the burntool:

o5444g0110o XY10    219612  2142808 new mopsfix mops.fixbrokenLAP.20190428  NULL    2

Perhaps try:

ipp_apply_burntool_fix.pl --exp_name o5444g0110o --class_id XY10 --verbose --dbname gpc1

Tell it to update?

chiptool -dbname gpc1 -setimfiletoupdate  -set_label mopsfix -chip_id 2142808

Missing mk.fits files: by JRF Leaving jobs stuck in the warp stage

Try to update them first:

chiptool -dbname gpc1 -setimfiletoupdate  -set_label ps_ud_WEB -chip_id XXXXXX
warptool -dbname gpc1  -setskyfiletoupdate -set_label ps_ud_WEB -warp_id XXXXXX

One of the ones sent to update came back complaining about a missing file, but it is there, so try again:

o6224g0254o 537186  1281637 1295352 1270256 1276893 skycell.1060.063    update  ps_ud_WEB
chiptool -dbname gpc1 -setimfiletoupdate  -set_label ps_ud_WEB -chip_id 1281637
warptool -dbname gpc1  -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1276893

Failing that, reprocess them:

chiptool -dbname gpc1 -definebyquery -set_label mopsfix  -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190428  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190428 -set_reduction LAP_SCIENCE -exp_name

have sent them to reprocess, check status soon!

The o6624g0189o exposure is done now, so clean up the remaining errors under the original label: o6624g0189o

chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1269880
warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1222762

As for o6224g0254o, there is still an issue: It still complains about being unable to find a file, but nebulous says it is there, and I have checked that the file is there and it does exists (and looks a reasonable size). Cheeky tip -> set to clean, then set it back to update to clean up the stuck ones.

chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1281637
warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1276893

Then, set to update again:

chiptool -dbname gpc1 -setimfiletoupdate  -set_label ps_ud_WEB -chip_id 1281637
warptool -dbname gpc1  -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1276893

Evening

There are now loads of broken jobs (must have hit a bad patch).

Greater than 600 jobs...but only around 50 unique warp ids

Best action is probably... set to clean, then update?

chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id xxxxxx
warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id xxxxxx
chiptool -dbname gpc1 -setimfiletoupdate  -set_label ps_ud_WEB -chip_id xxxxxx
warptool -dbname gpc1  -setskyfiletoupdate -set_label ps_ud_WEB -warp_id xxxxxx

missing 2 darks by JRF

Two darks failed to be copied over. It could be an issue with summitcopy. So I'll try restarting it

Now, this is where I made a mistake. DO NOT DO THIS

check_server.sh start.server summitcopy

Turns out it does not accept an argument and actually just starts all the servers...and I already had all the servers running...

So now I'm going through all the machines that have the pantasks_servers on them and clearing them out. In general just log into pantasks for each service and: {{{stop shutdown now }}} make sure the jobs have actually cleared before shutting down. Do this until all the servers are gone. Then go to the correct machine, and start it up:

pantasks_server & pantasks_client
server input input
setup
run

Note that you need to be in the correct directory, and logged in as the correct user.

The problem was not solved...

Questions? How come there is no cleanup/stack stage for ps2?

Monday : 2019.04.29

Prep for Haydn by JRF

I've set the following to repair to prepare for Haydn doing some maintenance:

ipp107, ipp108, ipp117
ipp068
ipp127-ipp142 (at least all the ones that are tracked by neb-host)

There are a bunch of compute nodes that Haydn might be messing with soon and they are currently in use. So these should be removed from current processing (c91, c85):

check_system.sh host off

12:20pm : Haydn is now installing the PDUs in M.03, so I've set ipp137-139 to down, along with ipp134-ipp136 as these are affected by the switch in that cabinet (THIS IS NOT DOCUMENTED ANYWHERE).

Dark issue from yesterday by JRF

The two exposures o8602g0001d and o8602g0001d are being reported as not copied to MHPCC.

Had a look at the logs for the summit copy:

logs/201904/pantasks_logs.20190428.153056/summit.copy.log.bz2

nothing about these exposures, starts at the one after them (o8602g0003d).

Having a look at some of the IPPmonitor tables, we can see it in the summitExp table, and there's also some stuff in the pzPending table (this is really the pzDownload table), but it is not seen in the newExp table, and the state in the summitExp table is 'run', even though we have all 60 expected parts of the image.

Check out the status:

~/src/ipp-20170121/tools/checkexp --verbose

It reports that these two exposures should be moved to the next stage of processing.

Going to try and revert the copy, to do it again:

pztool -revertcopied -exp_name o8602g0001d -inst gpc1 -telescope ps1 -dbname gpc1

get an error saying it should have affected atleast 1 row, but it did not

mysql> SELECT * FROM pzDownloadExp WHERE exp_name IN ("o8602g0001d","o8602g0002d");
+-----------+-------------+--------+-----------+-------+---------------------+
| summit_id | exp_name    | camera | telescope | state | epoch               |
+-----------+-------------+--------+-----------+-------+---------------------+
|   1460891 | o8602g0001d | gpc1   | ps1       | run   | 2019-04-28 15:28:28 | 
|   1460892 | o8602g0002d | gpc1   | ps1       | run   | 2019-04-28 15:28:57 | 
+-----------+-------------+--------+-----------+-------+---------------------+

We updated the first one to the stop state

pztool -updatepzexp -exp_name o8602g0001d -inst gpc1 -telescope ps1 -summit_id 1460891 -set_state stop -dbname gpc1

However, it does not appear in the newExp table (SELECT * FROM newExp WHERE summit_id = 1460891;)

Using 'checkexp' again we see that the exp_id is 'null' for o8602g0001d

What we need to do is properly advance it.

pztool -advance -summit_id 1460891 -exp_name o8602g0001d -inst gpc1 -telescope ps1 -end_stage reg -workdir neb://@HOST@.0/gpc1/20190429 -dbname gpc1

Might have done this twice... so there are two copies in the rawExp table.

Ok, let's fix the 2nd dark! :

pztool -updatepzexp -exp_name o8602g0002d -inst gpc1 -telescope ps1 -summit_id 1460892 -set_state stop -dbname gpc1
pztool -advance -summit_id 1460892 -exp_name o8602g0002d -inst gpc1 -telescope ps1 -end_stage reg -workdir neb://@HOST@.0/gpc1/20190429 -dbname gpc1

Done!

Broken update stuff (continued)

a broken job in the ps_ud_WEB chip stage:

o6224g0254o XY02  537186  1281637 update  ps_ud_WEB LAP.PV3.20140730.ipp.20141126 LAP.ThreePi 2

complains about:{{{ neb://@HOST@.0/gpc1/LAP.PV3.20140730.ipp/2014/11/26/o6224g0254o.537186/o6224g0254o.537186.ch.1281637.XY02.ppImage.mdc is missing }}} try to update: chiptool -dbname gpc1 -setimfiletoupdate -set_label mopsfix -chip_id 1281637 Note that it is now under a different label. It still complains, and the files it is missing are on ippb04, so we turned on the nodes that it complained about. It still fails. The main complain now is that the burntool version does not match. So we will attempt a burntool fix:

ipp_apply_burntool_fix.pl --exp_name o6224g0254o --class_id XY02 --verbose --dbname gpc1

This fails in the camera stage now, but this is because the file it is after is on a node that is temporarily down (Haydn is doing work on it). So we should wait until it is back up.

To do:

• Once ipp134-139 are back online revert the mopsfix job stuck in the camera stage
• send everything in update.LAP.PV3 to clean...
• ...once cleaned, send them all to update. This should fix most of the chip issues
• For the remaining problems:

-- if they complain about missing cm (camera) files then they should be reprocessed.

Tuesday : 2019.04.30

Wednesday : 2019.05.01

Thursday : 2019.05.02