== PS1 IPP Czar Logs for the week 2019.04.26 - 2019.05.02 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

Check out the new date!

=== Friday : 2019.04.26 ===

JRF: Since we're now switching on Fridays it makes sense to have the czarlogs do the same.

=== Saturday : 2019.04.27 ===

=== Sunday : 2019.04.28 ===


'''Missing mdc file + burntool problem''' by JRF

Reporting missing .mdc file, so it fails in the burntool stage, so I reprocessed it (really should have tried updating, this was in the morning)
Will reprocess:
{{{ 
chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190428 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190428 -set_reduction LAP_SCIENCE -exp_name o5444g0110o
}}}
Check they're finished:
{{{
SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state FROM rawExp JOIN chipRun USING (exp_id) WHERE exp_name IN ("o5444g0110o");
}}}

Now the mopsfix job is stuck as new in the chip stage, complaining about the burntool:
{{{
o5444g0110o XY10    219612  2142808 new mopsfix mops.fixbrokenLAP.20190428  NULL    2
}}}

Perhaps try:
{{{
ipp_apply_burntool_fix.pl --exp_name o5444g0110o --class_id XY10 --verbose --dbname gpc1
}}}

Tell it to update?
{{{
chiptool -dbname gpc1 -setimfiletoupdate  -set_label mopsfix -chip_id 2142808
}}}


'''Missing mk.fits files:''' by JRF
Leaving jobs stuck in the warp stage

Try to update them first:
{{{
chiptool -dbname gpc1 -setimfiletoupdate  -set_label ps_ud_WEB -chip_id XXXXXX
warptool -dbname gpc1  -setskyfiletoupdate -set_label ps_ud_WEB -warp_id XXXXXX
}}}

One of the ones sent to update came back complaining about a missing file, but it is there, so try again:
{{{
o6224g0254o 537186  1281637 1295352 1270256 1276893 skycell.1060.063    update  ps_ud_WEB
chiptool -dbname gpc1 -setimfiletoupdate  -set_label ps_ud_WEB -chip_id 1281637
warptool -dbname gpc1  -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1276893
}}}

Failing that, reprocess them:
{{{
chiptool -dbname gpc1 -definebyquery -set_label mopsfix  -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190428  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190428 -set_reduction LAP_SCIENCE -exp_name
}}}
have sent them to reprocess, check status soon!

The o6624g0189o exposure is done now, so clean up the remaining errors under the original label:
o6624g0189o
{{{
chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1269880
warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1222762
}}}

As for o6224g0254o, there is still an issue:
It still complains about being unable to find a file, but nebulous says it is there, and I have checked that the file is there and it does exists (and looks a reasonable size).
Cheeky tip -> set to clean, then set it back to update to clean up the stuck ones.
{{{
chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1281637
warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1276893
}}}
Then, set to update again:
{{{
chiptool -dbname gpc1 -setimfiletoupdate  -set_label ps_ud_WEB -chip_id 1281637
warptool -dbname gpc1  -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1276893
}}}






'''Evening'''

There are now loads of broken jobs (must have hit a bad patch).

Greater than 600 jobs...but only around 50 unique warp ids

Best action is probably... set to clean, then update?
{{{
chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id xxxxxx
warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id xxxxxx
chiptool -dbname gpc1 -setimfiletoupdate  -set_label ps_ud_WEB -chip_id xxxxxx
warptool -dbname gpc1  -setskyfiletoupdate -set_label ps_ud_WEB -warp_id xxxxxx
}}}



'''missing 2 darks''' by JRF

Two darks failed to be copied over. It could be an issue with summitcopy. So I'll try restarting it

Now, this is where I made a mistake. '''DO NOT DO THIS'''
{{{
check_server.sh start.server summitcopy
}}}
Turns out it does not accept an argument and actually just starts all the servers...and I already had all the servers running...


So now I'm going through all the machines that have the pantasks_servers on them and clearing them out. In general just log into pantasks for each service and:
{{{stop
shutdown now
}}}
make sure the jobs have actually cleared before shutting down. Do this until all the servers are gone. Then go to the correct machine, and start it up:
{{{
pantasks_server & pantasks_client
server input input
setup
run
}}}
Note that you need to be in the correct directory, and logged in as the correct user.

The problem was not solved...

Questions?
How come there is no cleanup/stack stage for ps2?


=== Monday : 2019.04.29 ===

Still have the dark issue.


Still a bunch of broken jobs to work through, so here we go!

  
=== Tuesday : 2019.04.30 ===

=== Wednesday : 2019.05.01 ===

=== Thursday : 2019.05.02 ===