PS1 IPP Czar Logs for the week 2019.04.26 - 2019.05.02
(Up to PS1 IPP Czar Logs)
Check out the new date!
Friday : 2019.04.26
JRF: Since we're now switching on Fridays it makes sense to have the czarlogs do the same.
Saturday : 2019.04.27
- MEH: queuing night stack and SSdiff once normal nightly processing finished for (OSS.nightlyscience, OSS.SS.nightlyscience)
Sunday : 2019.04.28
- MEH: since pstamp removed from daily restarts because auto-restart script still does not respect parsing jobs and this corrupting them and pantasks still needs to be restarted when Njobs~100k, have added a crontab on ippitc@ippc26 (where pstamp pantasks runs) that checks every hour if nearling ~100k and needs to be restarted (also with note if parsing jobs running)
Missing mdc file + burntool problem by JRF
Reporting missing .mdc file, so it fails in the burntool stage, so I reprocessed it (really should have tried updating, this was in the morning) Will reprocess:
chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190428 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190428 -set_reduction LAP_SCIENCE -exp_name o5444g0110o
Check they're finished:
SELECT rawExp.exp_name, rawExp.exp_id, chipRun.chip_id, chipRun.label chip_label, chipRun.state chip_state FROM rawExp JOIN chipRun USING (exp_id) WHERE exp_name IN ("o5444g0110o");
Now the mopsfix job is stuck as new in the chip stage, complaining about the burntool:
o5444g0110o XY10 219612 2142808 new mopsfix mops.fixbrokenLAP.20190428 NULL 2
Perhaps try:
ipp_apply_burntool_fix.pl --exp_name o5444g0110o --class_id XY10 --verbose --dbname gpc1
Tell it to update?
chiptool -dbname gpc1 -setimfiletoupdate -set_label mopsfix -chip_id 2142808
Missing mk.fits files: by JRF Leaving jobs stuck in the warp stage
Try to update them first:
chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_WEB -chip_id XXXXXX warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_WEB -warp_id XXXXXX
One of the ones sent to update came back complaining about a missing file, but it is there, so try again:
o6224g0254o 537186 1281637 1295352 1270256 1276893 skycell.1060.063 update ps_ud_WEB chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_WEB -chip_id 1281637 warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1276893
Failing that, reprocess them:
chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190428 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190428 -set_reduction LAP_SCIENCE -exp_name
have sent them to reprocess, check status soon!
The o6624g0189o exposure is done now, so clean up the remaining errors under the original label: o6624g0189o
chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1269880 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1222762
As for o6224g0254o, there is still an issue: It still complains about being unable to find a file, but nebulous says it is there, and I have checked that the file is there and it does exists (and looks a reasonable size). Cheeky tip -> set to clean, then set it back to update to clean up the stuck ones.
chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1281637 warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1276893
Then, set to update again:
chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_WEB -chip_id 1281637 warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_WEB -warp_id 1276893
Evening
There are now loads of broken jobs (must have hit a bad patch).
Greater than 600 jobs...but only around 50 unique warp ids
Best action is probably... set to clean, then update?
chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id xxxxxx warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id xxxxxx chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_WEB -chip_id xxxxxx warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_WEB -warp_id xxxxxx
missing 2 darks by JRF
Two darks failed to be copied over. It could be an issue with summitcopy. So I'll try restarting it
Now, this is where I made a mistake. DO NOT DO THIS
check_server.sh start.server summitcopy
Turns out it does not accept an argument and actually just starts all the servers...and I already had all the servers running...
So now I'm going through all the machines that have the pantasks_servers on them and clearing them out. In general just log into pantasks for each service and: {{{stop shutdown now }}} make sure the jobs have actually cleared before shutting down. Do this until all the servers are gone. Then go to the correct machine, and start it up:
pantasks_server & pantasks_client server input input setup run
Note that you need to be in the correct directory, and logged in as the correct user.
The problem was not solved...
Questions? How come there is no cleanup/stack stage for ps2?
Monday : 2019.04.29
Prep for Haydn by JRF
I've set the following to repair to prepare for Haydn doing some maintenance:
ipp107, ipp108, ipp117 ipp068 ipp127-ipp142 (at least all the ones that are tracked by neb-host)
There are a bunch of compute nodes that Haydn might be messing with soon and they are currently in use. So these should be removed from current processing (c91, c85):
check_system.sh host off
12:20pm : Haydn is now installing the PDUs in M.03, so I've set ipp137-139 to down, along with ipp134-ipp136 as these are affected by the switch in that cabinet (THIS IS NOT DOCUMENTED ANYWHERE).
Dark issue from yesterday by JRF
The two exposures o8602g0001d and o8602g0001d are being reported as not copied to MHPCC.
Had a look at the logs for the summit copy:
logs/201904/pantasks_logs.20190428.153056/summit.copy.log.bz2
nothing about these exposures, starts at the one after them (o8602g0003d).
Having a look at some of the IPPmonitor tables, we can see it in the summitExp table, and there's also some stuff in the pzPending table (this is really the pzDownload table), but it is not seen in the newExp table, and the state in the summitExp table is 'run', even though we have all 60 expected parts of the image.
Check out the status:
~/src/ipp-20170121/tools/checkexp --verbose
It reports that these two exposures should be moved to the next stage of processing.
Going to try and revert the copy, to do it again:
pztool -revertcopied -exp_name o8602g0001d -inst gpc1 -telescope ps1 -dbname gpc1
get an error saying it should have affected atleast 1 row, but it did not
mysql> SELECT * FROM pzDownloadExp WHERE exp_name IN ("o8602g0001d","o8602g0002d");
+-----------+-------------+--------+-----------+-------+---------------------+
| summit_id | exp_name | camera | telescope | state | epoch |
+-----------+-------------+--------+-----------+-------+---------------------+
| 1460891 | o8602g0001d | gpc1 | ps1 | run | 2019-04-28 15:28:28 |
| 1460892 | o8602g0002d | gpc1 | ps1 | run | 2019-04-28 15:28:57 |
+-----------+-------------+--------+-----------+-------+---------------------+
We updated the first one to the stop state
pztool -updatepzexp -exp_name o8602g0001d -inst gpc1 -telescope ps1 -summit_id 1460891 -set_state stop -dbname gpc1
However, it does not appear in the newExp table (SELECT * FROM newExp WHERE summit_id = 1460891;)
Using 'checkexp' again we see that the exp_id is 'null' for o8602g0001d
What we need to do is properly advance it.
pztool -advance -summit_id 1460891 -exp_name o8602g0001d -inst gpc1 -telescope ps1 -end_stage reg -workdir neb://@HOST@.0/gpc1/20190429 -dbname gpc1
Might have done this twice... so there are two copies in the rawExp table.
Ok, let's fix the 2nd dark! :
pztool -updatepzexp -exp_name o8602g0002d -inst gpc1 -telescope ps1 -summit_id 1460892 -set_state stop -dbname gpc1 pztool -advance -summit_id 1460892 -exp_name o8602g0002d -inst gpc1 -telescope ps1 -end_stage reg -workdir neb://@HOST@.0/gpc1/20190429 -dbname gpc1
Done!
Broken update stuff (continued) by JRF
a broken job in the ps_ud_WEB chip stage:
o6224g0254o XY02 537186 1281637 update ps_ud_WEB LAP.PV3.20140730.ipp.20141126 LAP.ThreePi 2
complains about:{{{ neb://@HOST@.0/gpc1/LAP.PV3.20140730.ipp/2014/11/26/o6224g0254o.537186/o6224g0254o.537186.ch.1281637.XY02.ppImage.mdc is missing }}} try to update: chiptool -dbname gpc1 -setimfiletoupdate -set_label mopsfix -chip_id 1281637 Note that it is now under a different label. It still complains, and the files it is missing are on ippb04, so we turned on the nodes that it complained about. It still fails. The main complain now is that the burntool version does not match. So we will attempt a burntool fix:
ipp_apply_burntool_fix.pl --exp_name o6224g0254o --class_id XY02 --verbose --dbname gpc1
This fails in the camera stage now, but this is because the file it is after is on a node that is temporarily down (Haydn is doing work on it). So we should wait until it is back up.
Hardware stuff update by JRF
17:00 - Haydn wishes to replace some failing drives on some of the compute nodes (ippc91, ippc52, ippc53, and ippc85). To allow him to do this the nodes need to be turned off in pantasks; e.g. to turn off ippc91:
check_system.sh hostoff ippc91
Note that this needs to be done for as many times as the machine appears in the controller list (pantasks> controller status). Once done sufficient times, and allowing some time for jobs to finish if they were in use, then they should all report as 'OFF', for example:
477 ippc91 OFF 0.0.0.bf 478 ippc91 OFF 0.0.0.f9 479 ippc91 OFF 0.0.0.133 480 ippc91 OFF 0.0.0.16d 481 ippc91 OFF 0.0.0.1a7
At this point maintenance can go ahead on the machine.
To see which nodes are used by which pantasks check the '~ippitc/ippconfig/pantasks_hosts.input' file. None of the main tasks use c52 or c53. MEH has confirmed that they are off for the mops stdscience stuff.
Geet big cleanup and process by JRF
First, see if I can cleanup all the ps_ud_WEB stuff: i) query the faulted jobs to get chip/warp ids ii) send them to cleaned (both chip and warp) iii) Once cleaned set them to update
Next there are a lot of stuck jobs under the update.LAP.PV3 label. Repeat the process of: i) send them all to clean ii) once cleaned send them to update (I put them under ps_ud_WEB since there is nothing there now)
This solved the majority of issues, about 437 of them, leaving only 4 problems, which shall be dealt with tomorrow.
Tuesday : 2019.04.30
stamp request is stuck by JRF
So check it out:
pstamptool -dbname ippRequestServer -dbserver ipp113 -pendingdependent -simple 10977094 new warp 1436300 skycell.0750.010 gpc1 /data/ipp113.0/pstamp/work/2019/04/30/920240 ps_ud_MOPS F 0 0 MOPS 920240 /data/ipp113.0/pstamp/work/2019/04/30/920240/checkdep.10977094.log 500
Query it:
SELECT rawExp.exp_name,rawExp.exp_id, chipRun.chip_id, warpRun.warp_id,warpSkyfile.skycell_id,warpRun.state,warpRun.label, warpSkyfile.path_base FROM warpSkyfile, warpRun, fakeRun, camRun, chipRun, rawExp WHERE warpSkyfile.warp_id = warpRun.warp_id AND warpRun.fake_id = fakeRun.fake_id AND fakeRun.cam_id = camRun.cam_id AND camRun.chip_id = chipRun.chip_id AND chipRun.exp_id = rawExp.exp_id AND warpRun.warp_id = '1436300' AND warpSkyfile.skycell_id like 'skycell.0750.010' ; +-------------+--------+---------+---------+------------------+--------+----------------+ | exp_name | exp_id | chip_id | warp_id | skycell_id | state | label | +-------------+--------+---------+---------+------------------+--------+----------------+ | o6284g0630o | 558924 | 1488053 | 1436300 | skycell.0750.010 | update | update.LAP.PV3 | +-------------+--------+---------+---------+------------------+--------+----------------+
Update the run to a higher priority label, and set it to update:
chiptool -dbname gpc1 -updaterun -chip_id 1488053 -state update -set_label ps_ud_MOPS warptool -dbname gpc1 -updaterun -warp_id 1436300 -state update -set_label ps_ud_MOPS chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1488053 warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_MOPS -warp_id 1436300
Broken jobs by JRF
Continuing on from yesterday, I sent around 400 unique jobs to clean -> then update. The majority of this proceeded successfully! There are a handful (about 60 failures, only 3-4 unique chip/warp combos) that are still failing. Checking their log it complains about a missing file of the form:
*cm*.XY*.mk.fits
The cm indicates that this is from the camera stage. Because of this updating the chip or warp stage will not help, we have to reprocess instead.
So, fetch the jobs that have the issue:
SELECT rawExp.exp_name,rawExp.exp_id,rawExp.filter, chipRun.chip_id,warpRun.warp_id,warpSkyfile.skycell_id,warpRun.state,warpRun.label,warpSkyfile.fault FROM warpSkyfile, warpRun, fakeRun, camRun, chipRun, rawExp WHERE warpSkyfile.warp_id = warpRun.warp_id AND warpRun.fake_id = fakeRun.fake_id AND fakeRun.cam_id = camRun.cam_id AND camRun.chip_id = chipRun.chip_id AND chipRun.exp_id = rawExp.exp_id AND warpSkyfile.fault != 0 AND warpRun.label like 'ps_ud_WEB' GROUP BY warpRun.warp_id ; +-------------+--------+---------+---------+------------------+--------+-----------+-------+ | exp_name | exp_id | chip_id | warp_id | skycell_id | state | label | fault | +-------------+--------+---------+---------+------------------+--------+-----------+-------+ | o5794g0540o | 380413 | 1335129 | 1285254 | skycell.2501.030 | update | ps_ud_WEB | 2 | | o5576g0112o | 280226 | 1335126 | 1285655 | skycell.2457.094 | update | ps_ud_WEB | 2 | | o6575g0385o | 664527 | 1335259 | 1286115 | skycell.2353.026 | update | ps_ud_WEB | 2 | | o6237g0252o | 545358 | 1335251 | 1286194 | skycell.2352.010 | update | ps_ud_WEB | 2 | | o6302g0235o | 565722 | 1335991 | 1288841 | skycell.2169.079 | update | ps_ud_WEB | 4 | +-------------+--------+---------+---------+------------------+--------+-----------+-------+
The fault 4 can just be reverted. As for the others, send them to reprocess:
chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190430 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190430 -set_reduction LAP_SCIENCE -exp_name o5794g0540o
chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190430 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190430 -set_reduction LAP_SCIENCE -exp_name o5576g0112o
chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190430 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190430 -set_reduction LAP_SCIENCE -exp_name o6575g0385o
chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190430 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190430 -set_reduction LAP_SCIENCE -exp_name o6237g0252o
Wait for them to finish processing...
Looks like problems (in the chip and cam stage): They're all waiting on some files that are only on ipp134-136 - I just set these down so that ITC could install the PDU. PENDING
Once good, cleanup remaining files:
chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id
warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id
Morning hardware things by JRF
ITC are going to install the PDUs in ipp134-136, so these will need to be set to down: neb-host ipp134 down --note "JRF: repair -> down: so that PDU can be installed in the cabinet" neb-host ipp135 down --note "JRF: repair -> down: so that PDU can be installed in the cabinet" neb-host ipp136 down --note "JRF: repair -> down: so that PDU can be installed in the cabinet" set down at about 09:20am
Making another apache server for nebulous by JRF
We want to add ippc76 as a nebulous apache server
Check out:
/etc/apache2/httpd.conf /etc/apache2/modules.d/apache2-mod_perl-startup.pl /etc/apache2/modules.d/00_mpm.conf
They contain information on the configuration of the nebulous apache server. Have a look at the current servers to see how they are setup
There is a config for the nebulous servers:
~ippitc/ippconfig/nebservers.csh
This is referenced only by the .tcshrc file of ippitc - which is not tracked''' In here we can set which nodes should be used for nebulous, so make a copy of the file and add the new server.
set nebservers = ($nebservers http://ippc76/nebulous);
We will not do this step just yet.
NOTE: That the total connections from the apache nodes should be less than the max connections allowed in the nebulous database, you can check this with:
mysql -uXXXXXX -pXXXXXXX -h ippdb09 nebulous mysql> select @@max_connections;
At present the mas connections are 2048. If we are using ippc71-76, which have 256 connections each, then fortunately: 6*256 < 2048
Right, we need to make sure that ippc76 is actually ready to be used as a nebulous server: One of these steps it to make sure it has a nebulous log file that it can use, which should be located under /tmp/ :
ls -l /tmp/ -rw-rw-r-- 1 apache nebulous 0 Apr 22 03:10 nebulous_server.log
It should belong to apache and be part of the nebulous group.
We also need to have the apache server running! Check the status with:
/etc/init/d/apache2 status
The apache server also needs to be pointing to the correct database (this is in the perl script). Once you're happy with all of the config, then go ahead and add:
set nebservers = ($nebservers http://ippc76/nebulous);
to the nebservers.csh file
After this the server started getting used, but the nebulous_server.log file disappeared. Simply restarting the apache server recreated it. However, if it does not you may need to touch the file (and make sure the permissions are correct).
NEW PROBLEMS!
The log file is just not updating at all, even when we are using ippc76 to connect to the nebulous database (you can forcibly change to use a specific apache node by set NEB_SERVER = http://ippc76).
Not sure what the problem is so we'll remove ippc76 from the nebservers list for the time being...
Tidying up labels
Under ~ippitc/stdscience/input are all the labels that are added when pantasks starts up. We've commented out some lesser used ones.
Hardware update b JRF
13:30 - ITS has finished installing PDUs in cabinet M.02, for machines ipp134-136. At the same time Haydn has installed the fan extensions. I have checked that I can log into the console for each machine and view the power status. I'm going to put this machines back into repair so that updates can proceed:
neb-host ipp134 repair --note "JRF: down -> repair: install of cabinet PDU and fan extension complete, should be set to up when confident" neb-host ipp135 repair --note "JRF: down -> repair: install of cabinet PDU and fan extension complete, should be set to up when confident" neb-host ipp136 repair --note "JRF: down -> repair: install of cabinet PDU and fan extension complete, should be set to up when confident"
Now revert all the stalled jobs, and they should pick up that the files only on ipp134-136 are now available. SUCCESS, they all cleared! 16:20 - ippc91 has had a rough time and we completely lost the .0 volume. This means that it will need to be reconfigured before it can be used for processing. Therefore, it should be removed from the hosts config:
~ippitc/ippconfig/pantasks_hosts.input
I've commented it out with the reason that it needs to be configured after the disk failure.
Nightly Science: Featuring Desperate Diff Dan by JRF
Here we go, desperate diff time boys an girls!
It appears that despite a full first chunk, there are some issues. Running the night report, I see:
Current status of SS and Euclid chunks:
chunk | Nquad_firstvisit | Nquad_fourvisits | Ndiffs done |
OSSR.R10S1.14.Q.w | 20 | 20 | 76/ 80 | chunk not finished
OSSR.R11S2.14.Q.w | 20 | 20 | 80/ 80 | chunk done
(it wasn't quite like this at the time, but close enough). So there are some missing diffs.
Doing the following query shows us that indeed there are 2 problem diffs, where the FWHM is very high.
mysql -hscidbm -uXXXXXX -pXXXXXX gpc1 -B -e " select exp_id,exp_name,camProcessedExp.path_base,filter,camProcessedExp.fault,fwhm_major,camProcessedExp.quality,camProcessedExp.deteff,sigma_ra,sigma_dec from camRun join camProcessedExp using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where dateobs like '`date -u "+%Y-%m-%d"`%' and quality>0; " exp_id exp_name path_base filter fault fwhm_major quality deteff sigma_ra sigma_dec 1465386 o8604g0101o neb://any/gpc1/OSS.nt/2019/05/01//o8604g0101o.1465386/o8604g0101o.1465386.cm.2109315 w.00000 0 15.3435 4007 -4.02871 NULL NULL 1465402 o8604g0117o neb://any/gpc1/OSS.nt/2019/05/01//o8604g0117o.1465402/o8604g0117o.1465402.cm.2109350 w.00000 0 13.0833 4007 -4.04619 NULL NULL
This is supported by the observers log, https://ps1wiki.ifa.hawaii.edu/trac/wiki/ObsLog20190501 where they mentions issues. And you can see how poor the data is if you look at the exposure on the datastore. In this particular case you see nice pairs for every object.
The problem chunk is OSSR.R10S1.14.Q.w
Find out which visits are the issue:
mysql -hscidbm -uXXXXXX -pXXXXXX gpc1 -B -e "SELECT selchunk.chunk,selchunk.object,MAX(CASE WHEN selchunk.visit=1 THEN selchunk.warp_id ELSE 0 END) as warp1,MAX(CASE WHEN selchunk.visit=2 THEN selchunk.warp_id ELSE 0 END) as warp2,MAX(CASE WHEN selchunk.visit=3 THEN selchunk.warp_id ELSE 0 END) as warp3,MAX(CASE WHEN selchunk.visit=4 THEN selchunk.warp_id ELSE 0 END) as warp4,selchunk.workdir,selchunk.label,selchunk.data_group,selchunk.reduction FROM (SELECT SUBSTRING_INDEX(comment, ' ',1) AS chunk,SUBSTRING_INDEX(comment, ' ',-1) AS visit,object,warp_id,rawExp.workdir,chipRun.label,chipRun.data_group,rawExp.reduction FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN camProcessedExp USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.exp_name LIKE 'o%' AND substr(comment, 1, position(' ' in comment)) LIKE 'OSSR.R10S1.14.Q.w%' AND rawExp.dateobs LIKE '`date -u "+%Y-%m-%d"`%' ORDER BY warp_id DESC) as selchunk GROUP BY selchunk.object;"
chunk object warp1 warp2 warp3 warp4 workdir label data_group reduction
OSSR.R10S1.14.Q.w ps1_30_1462 2084925 2084995 2085050 2085086 neb://@HOST@.0/gpc1/20190501 OSS.nightlyscience OSS.20190501 SWEETSPOT
OSSR.R10S1.14.Q.w ps1_30_2554 2084884 2084972 2085007 2085062 neb://@HOST@.0/gpc1/20190501 OSS.nightlyscience OSS.20190501 SWEETSPOT
OSSR.R10S1.14.Q.w ps1_30_2555 2084887 2084962 2085012 2085059 neb://@HOST@.0/gpc1/20190501 OSS.nightlyscience OSS.20190501 SWEETSPOT
OSSR.R10S1.14.Q.w ps1_30_2711 0 2084924 2085000 2085055 neb://@HOST@.0/gpc1/20190501 OSS.nightlyscience OSS.20190501 SWEETSPOT
OSSR.R10S1.14.Q.w ps1_30_2734 2084881 2084956 2085003 2085057 neb://@HOST@.0/gpc1/20190501 OSS.nightlyscience OSS.20190501 SWEETSPOT
OSSR.R10S1.14.Q.w ps1_30_2735 2084891 2084970 2085008 2085066 neb://@HOST@.0/gpc1/20190501 OSS.nightlyscience OSS.20190501 SWEETSPOT
OSSR.R10S1.14.Q.w ps1_30_2736 2084890 2084965 2085011 2085064 neb://@HOST@.0/gpc1/20190501 OSS.nightlyscience OSS.20190501 SWEETSPOT
OSSR.R10S1.14.Q.w ps1_30_2737 2084899 2084969 2085017 2085073 neb://@HOST@.0/gpc1/20190501 OSS.nightlyscience OSS.20190501 SWEETSPOT
OSSR.R10S1.14.Q.w ps1_30_2750 2084897 2084979 2085026 2085067 neb://@HOST@.0/gpc1/20190501 OSS.nightlyscience OSS.20190501 SWEETSPOT
OSSR.R10S1.14.Q.w ps1_30_2752 2084909 2084990 2085034 2085079 neb://@HOST@.0/gpc1/20190501 OSS.nightlyscience OSS.20190501 SWEETSPOT
OSSR.R10S1.14.Q.w ps1_30_2830 2084907 2084980 2085023 2085080 neb://@HOST@.0/gpc1/20190501 OSS.nightlyscience OSS.20190501 SWEETSPOT
OSSR.R10S1.14.Q.w ps1_30_2831 2084904 2084988 2085030 2085077 neb://@HOST@.0/gpc1/20190501 OSS.nightlyscience OSS.20190501 SWEETSPOT
OSSR.R10S1.14.Q.w ps1_30_2832 2084923 2084992 2085042 2085084 neb://@HOST@.0/gpc1/20190501 OSS.nightlyscience OSS.20190501 SWEETSPOT
OSSR.R10S1.14.Q.w ps1_30_2833 0 2084996 2085046 2085082 neb://@HOST@.0/gpc1/20190501 OSS.nightlyscience OSS.20190501 SWEETSPOT
OSSR.R10S1.14.Q.w ps1_30_2834 2084902 2084983 2085031 2085078 neb://@HOST@.0/gpc1/20190501 OSS.nightlyscience OSS.20190501 SWEETSPOT
OSSR.R10S1.14.Q.w ps1_30_2835 2084917 2084985 2085041 2085083 neb://@HOST@.0/gpc1/20190501 OSS.nightlyscience OSS.20190501 SWEETSPOT
OSSR.R10S1.14.Q.w ps1_30_2838 2084912 2084993 2085043 2085081 neb://@HOST@.0/gpc1/20190501 OSS.nightlyscience OSS.20190501 SWEETSPOT
OSSR.R10S1.14.Q.w ps1_30_5622 2084920 2085004 2085051 2085085 neb://@HOST@.0/gpc1/20190501 OSS.nightlyscience OSS.20190501 SWEETSPOT
OSSR.R10S1.14.Q.w ps1_30_5635 2084908 2084977 2085029 2085074 neb://@HOST@.0/gpc1/20190501 OSS.nightlyscience OSS.20190501 SWEETSPOT
OSSR.R10S1.14.Q.w ps1_30_5636 2084901 2084978 2085020 2085070 neb://@HOST@.0/gpc1/20190501 OSS.nightlyscience OSS.20190501 SWEETSPOT
Since visit 1 is the issue, we still need to get our diffs from visit 3 and 4. The diff pairs we need are:
2085000 & 2085055
2085046 & 2085082
Take these warp_ids and run the difftool command for creating these diffs:
difftool -dbname gpc1 -definewarpwarp -warp_id 2085000 -template_warp_id 2085055 -backwards -set_workdir neb://@HOST@.0/gpc1/OSS.nt/2019/05/01 -set_dist_group SweetSpot -set_label OSS.nightlyscience -set_data_group OSS.20190501 -set_reduction SWEETSPOT -simple -rerun -pretend difftool -dbname gpc1 -definewarpwarp -warp_id 2085046 -template_warp_id 2085082 -backwards -set_workdir neb://@HOST@.0/gpc1/OSS.nt/2019/05/01 -set_dist_group SweetSpot -set_label OSS.nightlyscience -set_data_group OSS.20190501 -set_reduction SWEETSPOT -simple -rerun -pretend
IMPORTANT: the warp_id, template_warp_id should be in time series e.g. visit 3 then visit 4, so be carefully as the warp_ids themselves might not be sequential (so tie back to the visits). Also, be very careful with the arguments used in difftool, I had used '-warp_id' twice which still runs and causes funkiness. SO DOUBLE CHECK EVERYTHING
You'll get a response along the lines of:
10:18:46.128181 RINGS.V3 T T 0 0.000000 nan nan nan nan 1
But more importantly you should see them in the diffInputSkyfile table:
SELECT * FROM diffInputSkyfile WHERE warp1 = 2085046 AND warp2 = 2085082 LIMIT 100;
and also with the datastore ipp IPP-MOPS-TEST
UPDATE: A new and improved way of getting all the info:
mysql -hscidbm --uXXXXXX -pXXXXXXX gpc1 -B -e "SELECT selchunk.chunk,selchunk.object,MAX(CASE WHEN selchunk.visit=1 THEN selchunk.warp_id ELSE 0 END) as warp1,MAX(CASE WHEN selchunk.visit=2 THEN selchunk.warp_id ELSE 0 END) as warp2,MAX(CASE WHEN selchunk.visit=3 THEN selchunk.warp_id ELSE 0 END) as warp3,MAX(CASE WHEN selchunk.visit=4 THEN selchunk.warp_id ELSE 0 END) as warp4,MAX(CASE WHEN diffchunk.visit=1 THEN diffchunk.diff_id ELSE 0 END) as diff1,MAX(CASE WHEN diffchunk.visit=2 THEN diffchunk.diff_id ELSE 0 END) as diff2,MAX(CASE WHEN diffchunk.visit=3 THEN diffchunk.diff_id ELSE 0 END) as diff3,MAX(CASE WHEN diffchunk.visit=4 THEN diffchunk.diff_id ELSE 0 END) as diff4,selchunk.workdir,selchunk.label,selchunk.data_group,selchunk.reduction FROM (SELECT SUBSTRING_INDEX(comment, ' ',1) AS chunk,SUBSTRING_INDEX(comment, ' ',-1) AS visit,object,warp_id,rawExp.workdir,chipRun.label,chipRun.data_group,rawExp.reduction FROM warpRun JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN camProcessedExp USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.exp_name LIKE 'o%' AND rawExp.dateobs LIKE '`date -u "+%Y-%m-%d"`%' ORDER BY warp_id DESC) as selchunk LEFT JOIN (SELECT SUBSTRING_INDEX(comment, ' ',1) AS chunk,SUBSTRING_INDEX(comment, ' ',-1) AS visit, object,warp_id,diff_id FROM diffRun JOIN diffInputSkyfile USING (diff_id) JOIN warpRun ON (warp1=warp_id) JOIN fakeRun USING (fake_id) JOIN camRun USING (cam_id) JOIN camProcessedExp USING (cam_id) JOIN chipRun USING (chip_id) JOIN rawExp USING (exp_id) WHERE rawExp.exp_name LIKE 'o%' AND rawExp.dateobs LIKE '`date -u "+%Y-%m-%d"`%' AND stack2 IS NULL GROUP BY warp_id) as diffchunk ON selchunk.warp_id=diffchunk.warp_id GROUP BY selchunk.chunk,selchunk.object;"
A simple query to see the structure of the diffRun table. This is important as it shows the style required when running the difftool command:
SELECT data_group, diff_id, dist_group, label, workdir, reduction FROM diffRun WHERE data_group LIKE "%20190501"; EXAMPLE: | OSS.20190501 | 1758779 | SweetSpot | OSS.nightlyscience | neb://@HOST@.0/gpc1/OSS.nt/2019/05/01 | SWEETSPOT | | OSS.20190501 | 1758780 | SweetSpot | OSS.nightlyscience | neb://@HOST@.0/gpc1/OSS.nt/2019/05/01 | SWEETSPOT | | OSS.20190501 | 1758781 | OSS.WS | OSS.WS.nightlyscience | neb://@HOST@.0/gpc1/OSS.WS.nightlyscience/2019/05/01 | SWEETSPOT_WS | | OSS.20190501 | 1758782 | OSS.WS | OSS.WS.nightlyscience | neb://@HOST@.0/gpc1/OSS.WS.nightlyscience/2019/05/01 | SWEETSPOT_WS |
So you can see there is a particular format to the workdir, different spellings of sweetspot for the reduction and dist_group etc.
Wednesday : 2019.05.01
- EAM? 08:00 : running test processing on ippc128 - ippc150 as user ipptest (label is UNIONS.DR2.20190430).
- CCL: touch /data/ipp123.0/nebulous/39/3c/10949746515.gpc2:20190117:o8500h0253o:o8500h0253o.ota64.fits time stamp changed
JRF: The long night has come to a close, and I have modified last nights notes to be a little more coherent (and a few warnings have been included).
stuck stamp requests by JRF
Do the usual pstamp check:
pstamptool -dbname ippRequestServer -dbserver ipp113 -pendingdependent -simple
To get a list of the problem warps and skycells.
Just need to set them to update since they have been cleaned away
chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_MOPS -warp_id
Run a typical query to see how they are doing:
SELECT rawExp.exp_name,rawExp.exp_id, chipRun.chip_id, chipRun.state chip_state, chipRun.label chip_label, warpRun.warp_id,warpSkyfile.skycell_id,warpRun.state,warpRun.label FROM warpSkyfile, warpRun, fakeRun, camRun, chipRun, rawExp WHERE warpSkyfile.warp_id = warpRun.warp_id AND warpRun.fake_id = fakeRun.fake_id AND fakeRun.cam_id = camRun.cam_id AND camRun.chip_id = chipRun.chip_id AND chipRun.exp_id = rawExp.exp_id AND warpRun.warp_id IN (1610722,1610657,1610652,1610623,1610130,1610117,1610095,1610077) GROUP BY warpRun.warp_id ;
However, there is an issue that the chips are not updating as their state is stuck as "wait". So I've tried to update the run and set the state:
chiptool -dbname gpc1 -updaterun -state wait -set_state update -set_label ps_ud_MOPS -chip_id 1663243
chiptool -dbname gpc1 -updaterun -state wait -set_state update -set_label ps_ud_MOPS -chip_id 1663225
chiptool -dbname gpc1 -updaterun -state wait -set_state update -set_label ps_ud_MOPS -chip_id 1663205
chiptool -dbname gpc1 -updaterun -state wait -set_state update -set_label ps_ud_MOPS -chip_id 1663186
chiptool -dbname gpc1 -updaterun -state wait -set_state update -set_label ps_ud_MOPS -chip_id 1662696
chiptool -dbname gpc1 -updaterun -state wait -set_state update -set_label ps_ud_MOPS -chip_id 1662677
chiptool -dbname gpc1 -updaterun -state wait -set_state update -set_label ps_ud_MOPS -chip_id 1662658
chiptool -dbname gpc1 -updaterun -state wait -set_state update -set_label ps_ud_MOPS -chip_id 1662639
Note that I search for the ones that say wait, and have updated their label to update.
They are now beginning to error in the chiptool stage:
burntool state vs burntoolStateGoodUpdate : -14 vs 13
Running [/data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/neb-repair neb://ipp088.0/gpc1/20150801/o7235g0278o/o7235g0278o.ota31.fits]...
Reading from filehandle failed at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/neb-repair line 78.
Unable to attempt repair: neb://ipp088.0/gpc1/20150801/o7235g0278o/o7235g0278o.ota31.fits 1280 at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/chip_imfile.pl line 834
main::my_die('Unable to attempt repair: neb://ipp088.0/gpc1/20150801/o7235g...', 951756, 1662658, 'XY31', 2) called at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/chip_imfile.pl line 358
I think it might be best to reprocess the lot, then send these broken jobs to clean (be sure to check the reduction)
chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenOSS.20190501 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenOSS.20190501 -set_reduction SWEETSPOT -exp_name
Check they have reprocessed (you'll need to query the exposure idea because of the reprocess) And then clean up the ones stuck under the ps_ud_WEB label:
chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label ps_ud_MOPS -chip_id
warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label ps_ud_MOPS -warp_id
Restoring computer node functionality by JRF
The compute nodes require a specific set of files to be on them and some 'tmp' directories to temporarily store data while processing. The general process is copy across all the static files that are needeed, then sort out all the links between the drives and '/local/'
Make sure there is an ipp directory to put things in on volume .0, then rsync the data across to it(and set permissions):
mkdir -p /export/`hostname`.0/ipp rsync -auv /data/ippc64.0/ipp/ /export/`hostname`.0/ipp
Then make sure the permissions are set for the ipp dir on vol 0
chown -R ipp:users /export/`hostname`.0/ipp chmod -R 775 /export/`hostname`.0/ipp
Then, replace the symlink for 'ipp' in the /local/ directory to point to volume .0:
rm /local/ipp ln -s /export/`hostname`.0/ipp /local/ipp
Finally, make the tmp directory on .1, and have the and have the 'tmp' dir from .0 link to it.
mkdir -p /export/`hostname`.1/ipp/tmp chown -R ipp:users /export/`hostname`.1/ipp chmod -R 775 /export/`hostname`.1/ipp ln -s /export/`hostname`.1/ipp/tmp /export/`hostname`.0/ipp/tmp
Note: the setup can be different, not all nodes have volume .1
Now the real important part is to actually test the node to see if it works.
Add it back to stdscience:
~ippitc/stdscience> pantasks_client pantasks> controller host on ippc91
This will allow one connection to be made to ippc91. Do it multiple times to add more, but before that first let it handle a job or two and keep an eye out for errors (check pantasks.stdout.log).
Once satisfied add the host more times to pantasks_client, and it can be added back to the hosts list file if removed ~ippitc/ippconfig/pantasks_hosts.input.
More stamp fun: by JRF
Check what the problem warps are:
pstamptool -dbname ippRequestServer -dbserver ipp113 -pendingdependent -simple
Get all the details
SELECT rawExp.exp_name,rawExp.exp_id, chipRun.chip_id, chipRun.state chip_state, chipRun.label chip_label, warpRun.warp_id,warpSkyfile.skycell_id,warpRun.state warp_state, warpRun.label warp_label FROM warpSkyfile, warpRun, fakeRun, camRun, chipRun, rawExp WHERE warpSkyfile.warp_id = warpRun.warp_id AND warpRun.fake_id = fakeRun.fake_id AND fakeRun.cam_id = camRun.cam_id AND camRun.chip_id = chipRun.chip_id AND chipRun.exp_id = rawExp.exp_id AND warpRun.warp_id IN (1921752,1921730,1921712,1921694,1206567) GROUP BY warpRun.warp_id ; +-------------+---------+---------+-------+---------+------------------+---------+--------------+ | exp_name | exp_id | chip_id | state | warp_id | skycell_id | state | label | +-------------+---------+---------+-------+---------+------------------+---------+--------------+ | o6889g0624o | 781745 | 1250507 | wait | 1206567 | skycell.1591.080 | cleaned | goto_cleaned | | o7977g0243o | 1283018 | 1977751 | wait | 1921694 | skycell.1218.090 | cleaned | goto_cleaned | | o7977g0262o | 1283037 | 1977770 | wait | 1921712 | skycell.1218.090 | cleaned | goto_cleaned | | o7977g0281o | 1283056 | 1977789 | wait | 1921730 | skycell.1218.090 | cleaned | goto_cleaned | | o7977g0300o | 1283075 | 1977808 | wait | 1921752 | skycell.1218.090 | cleaned | goto_cleaned | +-------------+---------+---------+-------+---------+------------------+---------+--------------+
This looks identical to earlier, so try to update and then force the chips that are stuck in 'wait':
chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id XXXXX warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_MOPS -warp_id XXXXX chiptool -dbname gpc1 -updaterun -state wait -set_state update -set_label ps_ud_MOPS -chip_id XXXXXX
Query the result:
SELECT
rawExp.exp_name,rawExp.exp_id,
chipRun.chip_id, chipRun.state chip_state, chipRun.label chip_label
FROM
chipRun, rawExp
WHERE
chipRun.exp_id = rawExp.exp_id
AND
rawExp.exp_name IN ("o6889g0624o","o7977g0243o","o7977g0262o","o7977g0281o","o7977g0300o")
;
SELECT
rawExp.exp_name,rawExp.exp_id,
chipRun.chip_id, chipRun.state chip_state, chipRun.label chip_label,
warpRun.warp_id,warpSkyfile.skycell_id,warpRun.state warp_state, warpRun.label warp_label
FROM
warpSkyfile, warpRun, fakeRun, camRun, chipRun, rawExp
WHERE
warpSkyfile.warp_id = warpRun.warp_id AND warpRun.fake_id = fakeRun.fake_id AND fakeRun.cam_id = camRun.cam_id AND camRun.chip_id = chipRun.chip_id AND chipRun.exp_id = rawExp.exp_id
AND
rawExp.exp_name IN ("o6889g0624o","o7977g0243o","o7977g0262o","o7977g0281o","o7977g0300o")
;
However... even though they are in update it does not seem to be picking them up... So 'setimfiletoupdate' again:
chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id XXXXX
Success!
Haydn Hardware (h)update by JRF
ipp137-ipp139 had had their fan extensions installed. Since we also have console access to them they are ready for taking some nightly data! So I've put them up on neb-host.
ippc91 has had it's drive replaced, has been reconfigured as per the guide above, and tested on updates. So it's good to go in nightly again.
Drives have been replaced on: ippc35, ippc52, ippc53, and ippc85 So I shall reconfigure these too following the guide above. The majority of them needed no modifications (as they were successfully rebuilt)
Random faulted jobs by JRF
Some chips reporting errors in the full state: Do the usual attempt to clean...
chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id
and once cleaned, update...
chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id
Thursday : 2019.05.02
JRF: a few broken files:
o6601g0088o XY61 673156 1200555 update update.LAP.PV3 LAP.PV3.20140730.20141101 LAP.ThreePi 2 o5391g0113o XY10 192159 1202176 update update.LAP.PV3 LAP.PV3.20140730.20141101 LAP.ThreePi 2 o5391g0123o XY10 192168 1202408 update update.LAP.PV3 LAP.PV3.20140730.20141102 LAP.ThreePi 2
send to clean then update:
chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1200555 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1202176 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1202408 chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1200555 chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1202176 chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1202408
Hardware fallout by JRF
Following a lot of nodes yo-yo-ing up and down while hardware was being sorted out this week it's worth checking what needs sorting out I missed putting these back up, so done:
neb-host --host ipp127 --state up --note "JRF: repair -> up: fan extensions installed, look good!" neb-host --host ipp128 --state up --note "JRF: repair -> up: fan extensions installed, look good!" neb-host --host ipp129 --state up --note "JRF: repair -> up: fan extensions installed, look good!" neb-host --host ipp130 --state up --note "JRF: repair -> up: fan extensions installed, look good!"
"Get three 480GB SSD's for ipp107, ipp108, and ipp117. SAMSUNG MZ7GE480HMHP or use 1TB 2.5" drives" - unclear if this was done, or what the next steps are exactly... I don't think this was done, so putting them back up: neb-host --host ipp107 --state up --note "JRF: repair -> up: I don't think SSDs were installed, so putting back up" neb-host --host ipp108 --state up --note "JRF: repair -> up: I don't think SSDs were installed, so putting back up" neb-host --host ipp117 --state up --note "JRF: repair -> up: I don't think SSDs were installed, so putting back up" ask Haydn about this.
put a few more up too (ipp081 and maybe others...neb-host holds the details)
Asked Haydn to keep the done items in the list in future, so that we can cross-reference afterwards.
ippx005 going down suspiciously by JRF
To see who's been sneaking around on a machine
last | less
We see that there has been periods of the machine rebooting (which is not great).
Mentions of previous rebooting of ippx005 (and other x nodes in the same cabinet) in October's czarlog:
http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/PS1_IPP_Czarlog_20181001
The page above mentioned that the boot time for x005 was much slower than the rest.
So, let's see what ganglia says about the boot times. Indeed, multiple x nodes rebooted this morning:
ippx001-4, 5-8, 9-12, 25-28, 29-32
They all rebooted about the same time, except for ippx005 which took a further hour and half to reboot..
This is suspiciously similar to the notes from back in October, and indicates a possible problem.
All of the affected nodes were in the same cabinet (13-16, 21-24, 37-40, and 41-44 are also in the cabinet but were unaffected)
Had PS2 been running this could cause a real problem where we would have to bring other nodes online to handle tasks, and many jobs way get into a funny state.
Feeding the update beast (how to queue more update jobs) by JRF
For the update.LAP.PV3 the format of the data_group is of the form:
LAP.PV3.20140730.20141% = 105k -- 201411% 7.4k 5/1 left, then 201412% LAP.PV3.20140730.20150% = 66k -- to run, split by month
So they all have the original start date of 20140730, followed by the sub-processing date. So to queue up some more simply change the sub-date in this upcoming massive query...
MEH's query for updating stuff:
mysql -hscidbm -uippuser -pippuser gpc1 -B -e ' select exp_id from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) where warpRun.data_group like "LAP.PV3.20140730.201412%" group by warp_id ; ' | grep -v exp_id | awk '{ print "mysql -hscidbm -u ippuser -pippuser gpc1 -B -e \" select s.exp_name,s.label,s.data_group,s.chip_id,s.warp_id,s.state from ( select chip_id,warp_id,exp_name,camRun.label,camRun.data_group,warpRun.state from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where exp_id="$1" order by warp_id desc ) as s limit 1 ;\" " }' | tcsh | grep -v exp_name | grep LAP.PV3.2014 | grep cleaned | awk '{print "chiptool -dbname gpc1 -updaterun -set_label update.LAP.PV3 -chip_id "$4" ; chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id "$4" ;warptool -dbname gpc1 -updaterun -set_label update.LAP.PV3 -warp_id "$5"; warptool -dbname gpc1 -setskyfiletoupdate -set_label update.LAP.PV3 -warp_id "$5 }' | head -n 1000 | tcsh
The gist of this is to:
- get exp_ids related to the data_group specified
- get the latest state (given by whichever warp is latest for that exposure)
- if the state is cleaned then queue it for update!
- and the updates are limited to 1000 at a time.
To see how far along we are you use this query. As of today we have ~260k warps done (nice).
select count(*),state,label,data_group from chipRun where label like "update%" group by label,state; select count(*),state,label,data_group from warpRun where label like "update%" group by label,state ;
So, the next one for us to do is 201412, so change that in the big-ass query and send it. I've run it twice to set 2000 or so jobs to go. I'll queue some more tomorrow too
Added ippc76 as neb apache server by JRF
modified the nebservers.csh file to include it. We had check the setup and functionality earlier this week.
neb-size issue by JRF
There is a chip fault mentioning a missing psf file:
PSF file neb://ipp037.0/gpc1/LAP.PV3.20140730.ipp/2014/11/22/o6917g0416o.795959/o6917g0416o.795959.ch.1270824.XY56.psf is missing at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/chip_imfile.pl line 653 ipp067.0 available ipp078.0 available ipp107.1 available PSPHOT.PSF.SAVE is missing but we cannot regenerate it at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/chip_imfile.pl line 834
The file is available so check it out:
neb-stat --validate neb://ipp037.0/gpc1/LAP.PV3.20140730.ipp/2014/11/22/o6917g0416o.795959/o6917g0416o.795959.ch.1270824.XY56.psf
1 d41d8cd98f00b204e9800998ecf8427e file:///data/ipp067.0/nebulous/3e/00/7202381844.gpc1:LAP.PV3.20140730.ipp:2014:11:22:o6917g0416o.795959:o6917g0416o.795959.ch.1270824.XY56.psf
1 88c2915120a480fb5ea907c88576d6e4 file:///data/ipp078.0/nebulous/3e/00/7407183092.gpc1:LAP.PV3.20140730.ipp:2014:11:22:o6917g0416o.795959:o6917g0416o.795959.ch.1270824.XY56.psf
1 d41d8cd98f00b204e9800998ecf8427e file:///data/ipp107.1/nebulous/3e/00/9361845752.gpc1:LAP.PV3.20140730.ipp:2014:11:22:o6917g0416o.795959:o6917g0416o.795959.ch.1270824.XY56.psf
We can see that 2 of the 3 files have a 0-size file. This could indicate that zero-size files are getting replicated!
Let's simply replace these zero size files with the good one! Then revert the job to proceed.
chip wait by JRF
Two chip jobs with state of 'wait', so do the following:
- Force a change of the state, then setimfiletoupdate (so that is actually proceeds).
chiptool -dbname gpc1 -updaterun -state wait -set_state update -set_label update.LAP.PV3 -chip_id 1305081 chiptool -dbname gpc1 -updaterun -state wait -set_state update -set_label update.LAP.PV3 -chip_id 1305187 chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1305081 chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1305187
