| Version 9 (modified by , 7 years ago) ( diff ) |
|---|
PS1 IPP Czar Logs for the week 2019.04.01 - 2019.04.07
(Up to PS1 IPP Czar Logs)
Czar of the week: TdB, with MEH as czar-lord (previously: CCL + EAM)
Monday : 2019.04.01
- EAM 09:00: powered-off ipp001, ipp003, ipp022
- CCL 09:20: restart stdscience manually
- CCL: not fixed chip or warp ids, they are related to stsci servers, mask files, psf files, or memory block, etc.
* chip mysql -hippdb08 -uippuser -p gpc1 -e "SELECT chipRun.chip_id,chipRun.state,chipRun.label,chipRun.data_group,chipRun.dist_group FROM chipRun WHERE chipRun.label like 'update.LAP.PV3.broke'" * warp mysql -hippdb08 -uippuser -pippuser gpc1 -e \ "SELECT chipRun.chip_id,camRun.cam_id,fakeRun.fake_id,warpRun.warp_id,warpRun.state,warpRun.label,warpRun.data_group \ FROM warpRun, fakeRun, camRun, chipRun \ WHERE warpRun.fake_id = fakeRun.fake_id \ AND fakeRun.cam_id = camRun.cam_id \ AND camRun.chip_id = chipRun.chip_id \ AND warpRun.label LIKE 'ps_ud_MOPS' \ AND warpRun.warp_id IN (1027567 , 1027638 , 1027645 , 1027695 , 1027697 , 1027705 , 1027711 , 1027781 , 1027785 , 1027828 , 1050735 , 1142465 , 1281048 , 1284676 , 1284689 , 1391868 , 1392030 , 1392058 , 1392250 , 1413405 , 1413407 , 1414496 , 1414848 , 1848779)"
- EAM 11:00 : Gavin has switch ippops1 & ippops2 to use ippops3 for homedirs and ippc18 for NIS
- TdB: around lunch time, Haydn asked for ipp091 and ipp127 to be put to down, in order for him to look at it, following a bad BBU and subsequent high load. They were both put to neb-host down and ipp091 was subsequently put back to up after repairs. During his look at ipp127, Haydn found a blown fuse on the PDU. He will look into the full rack tomorrow, and as a precaution we put all the nodes in that rack to repair, to prevent this ngihts data from going there and not being available to MOPS tomorrow.
- TdB: John, Roger, Mark and myself met to look at some of the errors on IPPmonitor. There were some errors left over from the weekend, in particular errors due to missing weight files on Sunday. We sent the chip_IDs for those exposures to cleanup and then set their imfiles back to update, followed by skyfiles back to update:
chiptool -dbname gpc1 -updaterun -chip_id XXXXX -set_state goto_cleaned -set_label goto_cleaned chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id XXXXX warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_MOPS -warp_id XXXXX
That solved the issue. Unfortunately, some exposures were faulting due to ipp127 not being available yet, at the time. I will follow-up more on these, once the rack is back up.
*TdB: Shortly after observing started, we noticed that processing was not proceeding at all, and an IPP hosed was called. Mark, Roger, John and myself met via zoom to solve the issue, around 21:00.
We noticed the issues was related to missing detrend files, which was somewhat strange since nothing changed since last week, when we did open and data went through fine. Initially, we figured it was due to some machines being down, so we turned those back to repair (ipp057, ippb17,18,21,22) and set the exposures to revert. This did not help.
Further digging into the log files revealed that there was a failed detselect command:
detselect -search -inst GPC1 -det_type VIDEODARK -time 2019-01-02T06:40:20.575099 -dbserver ippdb08 -dbname gpc1 -dbuser XXXXX -dbpassword XXXXXwhich did not return anything. Following this, we looked at:
detselect -show -det_type VIDEODARK -dbname gpc1 -simple 981 0 chip VIDEODARK register 2013-05-25T01:15:22.000000 2011-11-01T00:00:00.000000 2019-04-01T00:00:00.000000This revealed an end_time for the videodark validity of 2019-04-01T00:00:00.000000, which was last night. hence, there was no videodark available to use. So, we had to reset the end_time of the file using:
dettool -updatedetrun -dbname gpc1 -set_time_end 2099-04-01T00:00:00.000000 -det_id 981This solved most of the issue, but we found chip CY05 kept failing on each of the exposures. We inspected the actual file using ds9 and it looked ok. We inspected its replication and checksum using neb-stat -validate, which also looked ok. But then we realised this was in fact a DARKTEST file, which the processing was now using, which had no end_time at all. We stopped the stdscience pantasks while investigating, since this is not a good file to use in the nightly processing.
The det_id of the file was 866, and we set its end_time to yesterday to prevent it from being grabbed in the future:
dettool -updatedetrun -dbname gpc1 -set_time_end 2099-04-01T00:00:00.000000 -det_id 866The problem then shifted to the currently processed file being bad. So, we had to move the offending exposures to a broke label (OSS.20190402.baddet) and requeue it for processing from scratch, using:
mysql -hippdb08 -uXXXXX -pXXXXX gpc1 -e "SELECT rawExp.exp_name,chipRun.chip_id,chipRun.state,chipRun.label FROM chipRun JOIN rawExp USING (exp_id) WHERE chipRun.data_group = 'OSS.20190402.baddet' " | grep -v chip_id | awk '{print "chiptool -dbname gpc1 -definebyquery -set_label OSS.nightlyscience -set_workdir neb://@HOST@.0/gpc1/OSS.nt/2019/04/02 -set_dist_group SweetSpot -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction SWEETSPOT -set_data_group OSS.20190402 -exp_name "$1}' | tcshThen, we restarted the pantask just in case, and set it to run:
pantask:shutdown now ./start_server.sh stdscience pantask:server input input pantask:setup pantask:runUnfortunately, that re-added the update label for LAP.PV3 processing, so that need to be removed:
del.label update.LAP.PV3
Finally, things were running through fine. But, we were well behind, since it was close to midnight. So, we decided to add more nodes to the processing, to try to catch up.
hosts add m0 hosts add m1 we do this 4 times for each set.poll 500We checked ganglia and things looked ok. So, we added 4 more sets of each. We kept an eye on:
~ipp/src/ipp-20141024/tools/checkexp --verboseto see if this was causing any problems upstream, but things looked good. So, we added even more nodes
hosts add c2 we did this 4 timesFinally,we removed the stuck exposures for ps_ud_MOPS (due to ipp127 being offline) to a broke label, to prevent them from taking up cycles:
warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1281048 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1284676 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1284689 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1391868 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1392030 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1392058 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1392250 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1413405 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1413407 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1414496 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1414496 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1414848 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1027567 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1027638 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1027645 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1027695 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1027697 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1027705 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1027711 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1027781 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1027785 warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1027828
