== PS1 IPP Czar Logs for the week 2019.04.26 - 2019.05.02 == [[PageOutline]] (Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs]) Check out the new date! === Friday : 2019.04.26 === JRF: Since we're now switching on Fridays it makes sense to have the czarlogs do the same. === Saturday : 2019.04.27 === * MEH: queuing night stack and SSdiff once normal nightly processing finished for (OSS.nightlyscience, OSS.SS.nightlyscience) === Sunday : 2019.04.28 === * MEH: since pstamp removed from daily restarts because auto-restart script still does not respect parsing jobs and this corrupting them and pantasks still needs to be restarted when Njobs~100k, have added a crontab on ippitc@ippc26 (where pstamp pantasks runs) that checks every hour if nearling ~100k and needs to be restarted (also with note if parsing jobs running) '''Missing mdc file + burntool problem''' by JRF Reporting missing .mdc file, so it fails in the burntool stage, so I reprocessed it (really should have tried updating, this was in the morning) Will reprocess: {{{ chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190428 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190428 -set_reduction LAP_SCIENCE -exp_name o5444g0110o }}} Check they're finished: {{{ SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state FROM rawExp JOIN chipRun USING (exp_id) WHERE exp_name IN ("o5444g0110o"); }}} Now the mopsfix job is stuck as new in the chip stage, complaining about the burntool: {{{ o5444g0110o XY10 219612 2142808 new mopsfix mops.fixbrokenLAP.20190428 NULL 2 }}} Perhaps try: {{{ ipp_apply_burntool_fix.pl --exp_name o5444g0110o --class_id XY10 --verbose --dbname gpc1 }}} Tell it to update? {{{ chiptool -dbname gpc1 -setimfiletoupdate -set_label mopsfix -chip_id 2142808 }}} '''Missing mk.fits files:''' by JRF Leaving jobs stuck in the warp stage Try to update them first: {{{ chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_WEB -chip_id XXXXXX warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_WEB -warp_id XXXXXX }}} One of the ones sent to update came back complaining about a missing file, but it is there, so try again: {{{ o6224g0254o 537186 1281637 1295352 1270256 1276893 skycell.1060.063 update ps_ud_WEB chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_WEB -chip_id 1281637 warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1276893 }}} Failing that, reprocess them: {{{ chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190428 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190428 -set_reduction LAP_SCIENCE -exp_name }}} have sent them to reprocess, check status soon! The o6624g0189o exposure is done now, so clean up the remaining errors under the original label: o6624g0189o {{{ chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1269880 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1222762 }}} As for o6224g0254o, there is still an issue: It still complains about being unable to find a file, but nebulous says it is there, and I have checked that the file is there and it does exists (and looks a reasonable size). Cheeky tip -> set to clean, then set it back to update to clean up the stuck ones. {{{ chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1281637 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1276893 }}} Then, set to update again: {{{ chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_WEB -chip_id 1281637 warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1276893 }}} '''Evening''' There are now loads of broken jobs (must have hit a bad patch). Greater than 600 jobs...but only around 50 unique warp ids Best action is probably... set to clean, then update? {{{ chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id xxxxxx warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id xxxxxx chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_WEB -chip_id xxxxxx warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_WEB -warp_id xxxxxx }}} '''missing 2 darks''' by JRF Two darks failed to be copied over. It could be an issue with summitcopy. So I'll try restarting it Now, this is where I made a mistake. '''DO NOT DO THIS''' {{{ check_server.sh start.server summitcopy }}} Turns out it does not accept an argument and actually just starts all the servers...and I already had all the servers running... So now I'm going through all the machines that have the pantasks_servers on them and clearing them out. In general just log into pantasks for each service and: {{{stop shutdown now }}} make sure the jobs have actually cleared before shutting down. Do this until all the servers are gone. Then go to the correct machine, and start it up: {{{ pantasks_server & pantasks_client server input input setup run }}} Note that you need to be in the correct directory, and logged in as the correct user. The problem was not solved... Questions? How come there is no cleanup/stack stage for ps2? === Monday : 2019.04.29 === '''Prep for Haydn''' by JRF I've set the following to repair to prepare for Haydn doing some maintenance: {{{ ipp107, ipp108, ipp117 ipp068 ipp127-ipp142 (at least all the ones that are tracked by neb-host) }}} There are a bunch of compute nodes that Haydn might be messing with soon and they are currently in use. So these should be removed from current processing (c91, c85): {{{ check_system.sh host off }}} 12:20pm : Haydn is now installing the PDUs in M.03, so I've set ipp137-139 to down, along with ipp134-ipp136 as these are affected by the switch in that cabinet (THIS IS NOT DOCUMENTED ANYWHERE). '''Dark issue from yesterday''' by JRF The two exposures {{{o8602g0001d}}} and {{{o8602g0001d}}} are being reported as not copied to MHPCC. Had a look at the logs for the summit copy: {{{ logs/201904/pantasks_logs.20190428.153056/summit.copy.log.bz2 }}} nothing about these exposures, starts at the one after them (o8602g0003d). Having a look at some of the IPPmonitor tables, we can see it in the summitExp table, and there's also some stuff in the pzPending table (this is really the pzDownload table), but it is not seen in the newExp table, and the state in the summitExp table is 'run', even though we have all 60 expected parts of the image. Check out the status: {{{ ~/src/ipp-20170121/tools/checkexp --verbose }}} It reports that these two exposures should be moved to the next stage of processing. Going to try and revert the copy, to do it again: {{{ pztool -revertcopied -exp_name o8602g0001d -inst gpc1 -telescope ps1 -dbname gpc1 }}} get an error saying it should have affected atleast 1 row, but it did not {{{ mysql> SELECT * FROM pzDownloadExp WHERE exp_name IN ("o8602g0001d","o8602g0002d"); +-----------+-------------+--------+-----------+-------+---------------------+ | summit_id | exp_name | camera | telescope | state | epoch | +-----------+-------------+--------+-----------+-------+---------------------+ | 1460891 | o8602g0001d | gpc1 | ps1 | run | 2019-04-28 15:28:28 | | 1460892 | o8602g0002d | gpc1 | ps1 | run | 2019-04-28 15:28:57 | +-----------+-------------+--------+-----------+-------+---------------------+ }}} We updated the first one to the stop state {{{ pztool -updatepzexp -exp_name o8602g0001d -inst gpc1 -telescope ps1 -summit_id 1460891 -set_state stop -dbname gpc1 }}} However, it does not appear in the newExp table (SELECT * FROM newExp WHERE summit_id = 1460891;) Using 'checkexp' again we see that the exp_id is 'null' for o8602g0001d What we need to do is properly advance it. {{{ pztool -advance -summit_id 1460891 -exp_name o8602g0001d -inst gpc1 -telescope ps1 -end_stage reg -workdir neb://@HOST@.0/gpc1/20190429 -dbname gpc1 }}} Might have done this twice... so there are two copies in the rawExp table. Ok, let's fix the 2nd dark! : {{{ pztool -updatepzexp -exp_name o8602g0002d -inst gpc1 -telescope ps1 -summit_id 1460892 -set_state stop -dbname gpc1 pztool -advance -summit_id 1460892 -exp_name o8602g0002d -inst gpc1 -telescope ps1 -end_stage reg -workdir neb://@HOST@.0/gpc1/20190429 -dbname gpc1 }}} Done! '''Broken update stuff (continued)''' by JRF a broken job in the ps_ud_WEB chip stage: {{{ o6224g0254o XY02 537186 1281637 update ps_ud_WEB LAP.PV3.20140730.ipp.20141126 LAP.ThreePi 2 }}} complains about:{{{ neb://@HOST@.0/gpc1/LAP.PV3.20140730.ipp/2014/11/26/o6224g0254o.537186/o6224g0254o.537186.ch.1281637.XY02.ppImage.mdc is missing }}} try to update: chiptool -dbname gpc1 -setimfiletoupdate -set_label mopsfix -chip_id 1281637 Note that it is now under a different label. It still complains, and the files it is missing are on ippb04, so we turned on the nodes that it complained about. It still fails. The main complain now is that the burntool version does not match. So we will attempt a burntool fix: {{{ ipp_apply_burntool_fix.pl --exp_name o6224g0254o --class_id XY02 --verbose --dbname gpc1 }}} This fails in the camera stage now, but this is because the file it is after is on a node that is temporarily down (Haydn is doing work on it). So we should wait until it is back up. '''Hardware stuff update'' by JRF 17:00 - Haydn wishes to replace some failing drives on some of the compute nodes (ippc91, ippc52, ippc53, and ippc85). To allow him to do this the nodes need to be turned off in pantasks; e.g. to turn off ippc91: {{{ check_system.sh hostoff ippc91 }}} Note that this needs to be done for as many times as the machine appears in the controller list (pantasks> controller status). Once done sufficient times, and allowing some time for jobs to finish if they were in use, then they should all report as 'OFF', for example: {{{ 477 ippc91 OFF 0.0.0.bf 478 ippc91 OFF 0.0.0.f9 479 ippc91 OFF 0.0.0.133 480 ippc91 OFF 0.0.0.16d 481 ippc91 OFF 0.0.0.1a7 }}} At this point maintenance can go ahead on the machine. To see which nodes are used by which pantasks check the '~ippitc/ippconfig/pantasks_hosts.input' file. None of the main tasks use c52 or c53. MEH has confirmed that they are off for the mops stdscience stuff. ''' Geet big cleanup and process ''' by JRF First, see if I can cleanup all the ps_ud_WEB stuff: i) query the faulted jobs to get chip/warp ids ii) send them to cleaned (both chip and warp) iii) Once cleaned set them to update Next there are a lot of stuck jobs under the update.LAP.PV3 label. Repeat the process of: i) send them all to clean ii) once cleaned send them to update (I put them under ps_ud_WEB since there is nothing there now) This solved the majority of issues, about 437 of them, leaving only 4 problems, which shall be dealt with tomorrow. === Tuesday : 2019.04.30 === '''stamp request is stuck''' by JRF So check it out: {{{ pstamptool -dbname ippRequestServer -dbserver ipp113 -pendingdependent -simple 10977094 new warp 1436300 skycell.0750.010 gpc1 /data/ipp113.0/pstamp/work/2019/04/30/920240 ps_ud_MOPS F 0 0 MOPS 920240 /data/ipp113.0/pstamp/work/2019/04/30/920240/checkdep.10977094.log 500 }}} Query it: {{{ SELECT rawExp.exp_name,rawExp.exp_id, chipRun.chip_id, warpRun.warp_id,warpSkyfile.skycell_id,warpRun.state,warpRun.label, warpSkyfile.path_base FROM warpSkyfile, warpRun, fakeRun, camRun, chipRun, rawExp WHERE warpSkyfile.warp_id = warpRun.warp_id AND warpRun.fake_id = fakeRun.fake_id AND fakeRun.cam_id = camRun.cam_id AND camRun.chip_id = chipRun.chip_id AND chipRun.exp_id = rawExp.exp_id AND warpRun.warp_id = '1436300' AND warpSkyfile.skycell_id like 'skycell.0750.010' ; +-------------+--------+---------+---------+------------------+--------+----------------+ | exp_name | exp_id | chip_id | warp_id | skycell_id | state | label | +-------------+--------+---------+---------+------------------+--------+----------------+ | o6284g0630o | 558924 | 1488053 | 1436300 | skycell.0750.010 | update | update.LAP.PV3 | +-------------+--------+---------+---------+------------------+--------+----------------+ }}} Update the run to a higher priority label, and set it to update: {{{ chiptool -dbname gpc1 -updaterun -chip_id 1488053 -state update -set_label ps_ud_MOPS warptool -dbname gpc1 -updaterun -warp_id 1436300 -state update -set_label ps_ud_MOPS chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1488053 warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_MOPS -warp_id 1436300 }}} '''Broken jobs''' by JRF Continuing on from yesterday, I sent around 400 unique jobs to clean -> then update. The majority of this proceeded successfully! There are a handful (about 60 failures, only 3-4 unique chip/warp combos) that are still failing. Checking their log it complains about a missing file of the form: {{{ *cm*.XY*.mk.fits }}} The cm indicates that this is from the camera stage. Because of this updating the chip or warp stage will not help, we have to reprocess instead. So, fetch the jobs that have the issue: {{{ SELECT rawExp.exp_name,rawExp.exp_id,rawExp.filter, chipRun.chip_id,warpRun.warp_id,warpSkyfile.skycell_id,warpRun.state,warpRun.label,warpSkyfile.fault FROM warpSkyfile, warpRun, fakeRun, camRun, chipRun, rawExp WHERE warpSkyfile.warp_id = warpRun.warp_id AND warpRun.fake_id = fakeRun.fake_id AND fakeRun.cam_id = camRun.cam_id AND camRun.chip_id = chipRun.chip_id AND chipRun.exp_id = rawExp.exp_id AND warpSkyfile.fault != 0 AND warpRun.label like 'ps_ud_WEB' GROUP BY warpRun.warp_id ; +-------------+--------+---------+---------+------------------+--------+-----------+-------+ | exp_name | exp_id | chip_id | warp_id | skycell_id | state | label | fault | +-------------+--------+---------+---------+------------------+--------+-----------+-------+ | o5794g0540o | 380413 | 1335129 | 1285254 | skycell.2501.030 | update | ps_ud_WEB | 2 | | o5576g0112o | 280226 | 1335126 | 1285655 | skycell.2457.094 | update | ps_ud_WEB | 2 | | o6575g0385o | 664527 | 1335259 | 1286115 | skycell.2353.026 | update | ps_ud_WEB | 2 | | o6237g0252o | 545358 | 1335251 | 1286194 | skycell.2352.010 | update | ps_ud_WEB | 2 | | o6302g0235o | 565722 | 1335991 | 1288841 | skycell.2169.079 | update | ps_ud_WEB | 4 | +-------------+--------+---------+---------+------------------+--------+-----------+-------+ }}} The fault 4 can just be reverted. As for the others, send them to reprocess: {{{ chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190430 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190430 -set_reduction LAP_SCIENCE -exp_name o5794g0540o chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190430 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190430 -set_reduction LAP_SCIENCE -exp_name o5576g0112o chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190430 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190430 -set_reduction LAP_SCIENCE -exp_name o6575g0385o chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190430 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190430 -set_reduction LAP_SCIENCE -exp_name o6237g0252o }}} Wait for them to finish processing... Looks like problems (in the chip and cam stage): They're all waiting on some files that are only on ipp134-136 - I just set these down so that ITC could install the PDU. PENDING Once good, cleanup remaining files: {{{ chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id }}} '''Morning hardware things''' by JRF ITC are going to install the PDUs in ipp134-136, so these will need to be set to down: neb-host ipp134 down --note "JRF: repair -> down: so that PDU can be installed in the cabinet" neb-host ipp135 down --note "JRF: repair -> down: so that PDU can be installed in the cabinet" neb-host ipp136 down --note "JRF: repair -> down: so that PDU can be installed in the cabinet" set down at about 09:20am ''' Making another apache server for nebulous''' by JRF We want to add ippc76 as a nebulous apache server Check out: {{{ /etc/apache2/httpd.conf /etc/apache2/modules.d/apache2-mod_perl-startup.pl /etc/apache2/modules.d/00_mpm.conf }}} They contain information on the configuration of the nebulous apache server. Have a look at the current servers to see how they are setup There is a config for the nebulous servers: {{{ ~ippitc/ippconfig/nebservers.csh }}} This is referenced only by the .tcshrc file of ippitc - '''which is not tracked!''' In here we can set which nodes should be used for nebulous, so make a copy of the file and add the new server. {{{ set nebservers = ($nebservers http://ippc76/nebulous); }}} We will not do this step just yet. NOTE: That the total connections from the apache nodes should be less than the max connections allowed in the nebulous database, you can check this with: {{{ mysql -uXXXXXX -pXXXXXXX -h ippdb09 nebulous mysql> select @@max_connections; }}} At present the mas connections are 2048. If we are using ippc71-76, which have 256 connections each, then fortunately: 6*256 < 2048 Right, we need to make sure that ippc76 is actually ready to be used as a nebulous server: One of these steps it to make sure it has a nebulous log file that it can use, which should be located under /tmp/ : {{{ ls -l /tmp/ -rw-rw-r-- 1 apache nebulous 0 Apr 22 03:10 nebulous_server.log }}} It should belong to apache and be part of the nebulous group. We also need to have the apache server running! Check the status with: {{{ /etc/init/d/apache2 status }}} The apache server also needs to be pointing to the correct database (this is in the perl script). Once you're happy with all of the config, then go ahead and add: {{{ set nebservers = ($nebservers http://ippc76/nebulous); }}} to the nebservers.csh file After this the server started getting used, but the nebulous_server.log file disappeared. Simply restarting the apache server recreated it. However, if it does not you may need to touch the file (and make sure the permissions are correct). NEW PRBOLEMS! The log file is just not updating at all, even when we are using ippc76 to connect to the nebulous database (you can forcably change to use a specfic apache node by {{{set NEB_SERVER = http://ippc76}}}). Not sure what the problem is so we'll remove ippc76 from the nebservers list for the time being... ''' Tidying up labels ''' Under ~ippitc/stdscience/input are all the labels that are added when pantasks starts up. We've commented out some lesser used ones. To do: TODO: - once the PDU is done for ipp134-136: -- check the console -- put them to up to allow updates to continue and check on them (but put to repair for nightly...maybe) - check that the commented out labels are gone after reset. - For remaining update problems: -- if they complain about missing cm (camera) files then they should be reprocessed. -- revert the stuck jobs once ipp134-139 are available again === Wednesday : 2019.05.01 === === Thursday : 2019.05.02 ===