IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links
wiki:PS1_IPP_Czarlog_20190401

PS1 IPP Czar Logs for the week 2019.04.01 - 2019.04.07

    1. PS1 IPP Czar Logs for the week 2019.04.01 - 2019.04.07
      1. Monday : 2019.04.01
      2. Tuesday : 2019.04.02
      3. Wednesday : 2019.04.03
      4. Thursday : 2019.04.04
      5. Friday : 2019.04.05
      6. Saturday : 2019.04.06
      7. Sunday : 2019.04.07

(Up to PS1 IPP Czar Logs)

Czar of the week: TdB, with MEH as czar-lord (previously: CCL + EAM)

Monday : 2019.04.01

  • EAM 09:00: powered-off ipp001, ipp003, ipp022
  • CCL 09:20: restart stdscience manually
  • CCL: not fixed chip or warp ids, they are related to stsci servers, mask files, psf files, or memory block, etc. 
    * chip
mysql -hippdb08 -uXXXXX -pXXXXX gpc1 -e "SELECT chipRun.chip_id,chipRun.state,chipRun.label,chipRun.data_group,chipRun.dist_group FROM chipRun WHERE chipRun.label like 'update.LAP.PV3.broke'"

* warp
mysql -hippdb08 -uXXXXX -pXXXXX gpc1 -e \
"SELECT chipRun.chip_id,camRun.cam_id,fakeRun.fake_id,warpRun.warp_id,warpRun.state,warpRun.label,warpRun.data_group \
FROM warpRun, fakeRun, camRun, chipRun \
WHERE warpRun.fake_id = fakeRun.fake_id \
AND fakeRun.cam_id = camRun.cam_id \
AND camRun.chip_id = chipRun.chip_id \
AND warpRun.label LIKE 'ps_ud_MOPS' \
AND warpRun.warp_id IN (1027567 , 1027638 , 1027645 , 1027695 , 1027697 , 1027705 , 1027711 , 1027781 , 1027785 , 1027828 , 1050735 , 1142465 , 1281048 , 1284676 , 1284689 , 1391868 , 1392030 , 1392058 , 1392250 , 1413405 , 1413407 , 1414496 , 1414848 , 1848779)"
  • EAM 11:00 : Gavin has switch ippops1 & ippops2 to use ippops3 for homedirs and ippc18 for NIS
  • TdB: around lunch time, Haydn asked for ipp091 and ipp127 to be put to down, in order for him to look at it, following a bad BBU and subsequent high load. They were both put to neb-host down and ipp091 was subsequently put back to up after repairs. During his look at ipp127, Haydn found a blown fuse on the PDU. He will look into the full rack tomorrow, and as a precaution we put all the nodes in that rack to repair, to prevent this ngihts data from going there and not being available to MOPS tomorrow.
  • TdB: John, Roger, Mark and myself met to look at some of the errors on IPPmonitor. There were some errors left over from the weekend, in particular errors due to missing weight files on Sunday. We sent the chip_IDs for those exposures to cleanup and then set their imfiles back to update, followed by skyfiles back to update: 
    chiptool -dbname gpc1 -updaterun -chip_id XXXXX -set_state goto_cleaned -set_label goto_cleaned
chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id XXXXX
warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_MOPS -warp_id XXXXX
    That solved the issue. Unfortunately, some exposures were faulting due to ipp127 not being available yet, at the time. I will follow-up more on these, once the rack is back up.
  • TdB: Shortly after observing started, we noticed that processing was not proceeding at all, and an IPP hosed was called. Mark, Roger, John and myself met via zoom to solve the issue, around 21:00. We noticed the issues was related to missing detrend files, which was somewhat strange since nothing changed since last week, when we did open and data went through fine. Initially, we figured it was due to some machines being down, so we turned those back to repair (ipp057, ippb17,18,21,22) and set the exposures to revert. This did not help.

Further digging into the log files revealed that there was a failed detselect command: 

detselect -search -inst GPC1 -det_type VIDEODARK -time 2019-01-02T06:40:20.575099 -dbserver ippdb08 -dbname gpc1 -dbuser XXXXX -dbpassword XXXXX

which did not return anything. Following this, we looked at: 

detselect -show -det_type VIDEODARK -dbname gpc1 -simple

981 0 chip VIDEODARK register 2013-05-25T01:15:22.000000  2011-11-01T00:00:00.000000 2019-04-01T00:00:00.000000

This revealed an end_time for the videodark validity of 2019-04-01T00:00:00.000000, which was last night. hence, there was no videodark available to use. So, we had to reset the end_time of the file using: 

dettool -updatedetrun -dbname gpc1 -set_time_end 2099-04-01T00:00:00.000000 -det_id 981

This solved most of the issue, but we found chip CY05 kept failing on each of the exposures. We inspected the actual file using ds9 and it looked ok. We inspected its replication and checksum using neb-stat -validate, which also looked ok. But then we realised this was in fact a DARKTEST file, which the processing was now using, which had no end_time at all. We stopped the stdscience pantasks while investigating, since this is not a good file to use in the nightly processing.

The det_id of the file was 866, and we set its end_time to yesterday to prevent it from being grabbed in the future: 

dettool -updatedetrun -dbname gpc1 -set_time_end 2099-04-01T00:00:00.000000 -det_id 866

The problem then shifted to the currently processed file being bad. So, we had to move the offending exposures to a broke label (OSS.20190402.baddet) and requeue it for processing from scratch, using: 

mysql -hippdb08 -uXXXXX -pXXXXX gpc1 -e "SELECT rawExp.exp_name,chipRun.chip_id,chipRun.state,chipRun.label FROM chipRun JOIN rawExp USING (exp_id) WHERE chipRun.data_group = 'OSS.20190402.baddet' " | grep -v chip_id | awk '{print "chiptool -dbname gpc1 -definebyquery -set_label OSS.nightlyscience -set_workdir neb://@HOST@.0/gpc1/OSS.nt/2019/04/02 -set_dist_group SweetSpot -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction SWEETSPOT -set_data_group OSS.20190402 -exp_name "$1}' | tcsh

Then, we restarted the pantask just in case, and set it to run: 

pantask:shutdown now

./start_server.sh stdscience
pantask:server input input
pantask:setup 
pantask:run

Unfortunately, that re-added the update label for LAP.PV3 processing, so that need to be removed: 

del.label update.LAP.PV3

Finally, things were running through fine. But, we were well behind, since it was close to midnight. So, we decided to add more nodes to the processing, to try to catch up. 

hosts add m0
hosts add m1
we do this 4 times for each

set.poll 500

We checked ganglia and things looked ok. So, we added 4 more sets of each. We kept an eye on: 

~ipp/src/ipp-20141024/tools/checkexp --verbose

to see if this was causing any problems upstream, but things looked good. So, we added even more nodes 

hosts add c2
we did this 4 times

Finally,we removed the stuck exposures for ps_ud_MOPS (due to ipp127 being offline) to a broke label, to prevent them from taking up cycles: 

warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1281048 
warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1284676 
warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1284689 
warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1391868 
warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1392030 
warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1392058 
warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1392250 
warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1413405 
warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1413407 
warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1414496 
warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1414496 
warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1414848 
warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1027567 
warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1027638 
warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1027645 
warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1027695 
warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1027697 
warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1027705 
warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1027711 
warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1027781 
warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1027785 
warptool -dbname gpc1 -updaterun -set_label ps_ud_MOPS.broken -warp_id 1027828

Tuesday : 2019.04.02

  • TdB: We (Roger, Mark, John and myself) met up after coffee to go through some issues. First off, due to rack i.13 being down for repairs, we set chip and warp reverts to off, to prevent error pile-ups: 
    chip.revert.off
warp.revert.off
    That rack also contains other machines, namely ippx077-084, which are used by ipppss stdscience pantasks.So, set them to off for now, even though they will come back on after the daily restart. Go the ~ippp2/stdscience pantask and do: 
    controller host off ippx077
controller host off ippx078
controller host off ippx079
controller host off ippx080
controller host off ippx081
controller host off ippx082
controller host off ippx083
controller host off ippx084
    Next, we went into the regular gpc1 stdscience pantask and removed all the extra machines that were added by doing hosts off m0, etc and until all mahcines are off. We also set the poll back to 300: 
    hosts off m0 (8 times)
hosts off m1 (8 times)
hosts off c2 (4 times)
set.poll 300

On IPPmonitor, two exposures were stuck for ps_ud_MOPS. We had a look at them to find out what the problem was: 

mysql -hippdb08 -uXXXXX -pXXXXX gpc1 -e "SELECT chipRun.chip_id,chipRun.state,camRun.cam_id,fakeRun.fake_id,warpRun.warp_id,warpRun.state,warpRun.label,warpRun.data_group FROM warpRun JOIN fakeRun using (fake_id) JOIN camRun using (cam_id) JOIN chipRun  using (chip_id) WHERE warpRun.label like 'ps_ud_MOPS' AND warpRun.state = 'update'"

+---------+--------+---------+---------+---------+--------+------------+---------------------------+
| chip_id | state  | cam_id  | fake_id | warp_id | state  | label      | data_group                |
+---------+--------+---------+---------+---------+--------+------------+---------------------------+
| 1192618 | update | 1160583 | 1135857 | 1142465 | update | ps_ud_MOPS | LAP.PV3.20140730.20141029 | 
| 1359800 | update | 1323930 | 1300281 | 1306944 | update | ps_ud_MOPS | LAP.PV3.20140730.20141224 | 
| 1359803 | update | 1323939 | 1300286 | 1306953 | update | ps_ud_MOPS | LAP.PV3.20140730.20141224 | 
| 1829092 | update | 1796609 | 1769133 | 1775748 | update | ps_ud_MOPS | OSS.20161016              | 
| 1829109 | update | 1796626 | 1769150 | 1775765 | update | ps_ud_MOPS | OSS.20161016              | 
| 1829126 | update | 1796643 | 1769167 | 1775782 | update | ps_ud_MOPS | OSS.20161016              | 
| 1829143 | update | 1796660 | 1769188 | 1775803 | update | ps_ud_MOPS | OSS.20161016              | 
| 1884134 | update | 1851648 | 1823771 | 1830386 | update | ps_ud_MOPS | ThreePi.20170111          | 
| 1902599 | update | 1870113 | 1842164 | 1848779 | update | ps_ud_MOPS | OSS.20170216              | 
+---------+--------+---------+---------+---------+--------+------------+---------------------------+

They were stuck in update. We then looked at their data_states using: 

mysql -hippdb08 -uXXXXX -pXXXXX gpc1 -e "SELECT chipRun.chip_id,chipRun.state,camRun.cam_id,fakeRun.fake_id,warpRun.warp_id,warpRun.state,warpRun.label,warpRun.data_group,chipProcessedImfile.data_state,warpSkyfile.data_state FROM warpRun join warpSkyfile using (warp_id) JOIN fakeRun using (fake_id) JOIN camRun using (cam_id) JOIN chipRun  using (chip_id) JOIN chipProcessedImfile using (chip_id) WHERE warpRun.label like 'ps_ud_MOPS' AND warpRun.state = 'update'"

This showed that the data_state for the warps were partially full and partially update. So, the full warp_id was set to update, but the chips are not set to update, and stuck in clean. To solve that, set the chips to update again: 

chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1192618
chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1359800
chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1359803
chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1829092
chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1829109
chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1829126
chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1829143
chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1884134
chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_MOPS -chip_id 1902599

Finally, there was one more detrend file that has the same issue as last night, namely a VIDEMASK file with end_time set to 2021. It's det_id is found through: 

detselect -show -det_type VIDEOMASK -dbname gpc1 -simple

956 0 chip VIDEOMASK register 2012-05-26T00:01:23.000000  2011-08-02T00:00:00.000000 2021-05-01T00:00:00.000000
955 0 chip VIDEOMASK register 2012-05-26T00:01:13.000000  2011-03-09T00:00:00.000000 2011-03-10T23:59:59.000000
954 0 chip VIDEOMASK register 2012-05-26T00:01:04.000000  2011-01-06T00:00:00.000000 NULL
953 0 chip VIDEOMASK register 2012-05-26T00:00:55.000000  2010-01-01T00:00:00.000000 NULL
952 0 chip VIDEOMASK register 2012-05-26T00:00:47.000000  2009-12-09T00:00:00.000000 NULL
951 0 chip VIDEOMASK register 2012-05-26T00:00:39.000000  2009-01-01T00:00:00.000000 NULL

and the fixed using: 

dettool -updatedetrun -dbname gpc1 -set_time_end 2099-04-01T00:00:00.000000 -det_id 956
  • TdB: The IPPmonitor errors for ps_ud_MOPS flagged by Roger over the weekend have all been resolved. Some were resolved by machines coming back up, while most were related to files that were simply gone. Those have been reprocessed using: 
    chiptool -dbname gpc1 -definebyquery -set_label mopsfix  -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name XXXXX
    and the original IDs are sent to clean.

Wednesday : 2019.04.03

  • TdB: ipp070 seems to be very desirable this morning, with a lot of LAP.PV3 updates using data that is on there. To take the load off, I put it in repair mode for the time being, so no new data gets written there.
  • TdB: Following Haydn's repair of the fuse on rack i.13 and PDu set-up across the machines on the rack, ipp123-130 have now been put back to up. Machine ippb22 has also ben put back to repair to test it, after haydn replaced the complete RAM on it.
  • TdB: Following testing during the last two nights, the updates made to the night_report.pl script are now committed to the svn repository. It is found under trunk/ippScripts/scripts/night_report.pl or alternatively can also be found under my own homedir /home/panstarrs/tdeboer/night_report.pl just in case. The new addition (provided under the --verbose or the --chunks flag) provides information about the %SS% chunk processing and allows the czar to determine if the first chunk of the night has finished processing completely.
  • TdB: There will be maintenance at the ITC tomorrow morning around 07:00. Therefore, it is important to get nightly processing done before that time. To speed up processing, I increased the poll size to 500 and added some s nodes to the stdscience pantask, using: 
    set.poll 500
hosts add m0 (4 times)
hosts add m1 (4 times)

Thursday : 2019.04.04

  • TdB: Nightly processing finished on time, around 06:30. I set the pol back to 300 and remove the extra s nodes using: 
    set.poll 300
hosts delete m0 (4 times)
hosts delete m1 (4 times)
    Following this, all pantasks were stopped at 06:30 just in case, since no word on actual start time of maintenance at the ITC was received. This was done using: 
    ./check_system.sh stop
    After we finally received word of maintenance completing around 09:45 I set the system to LAP.PV3 updating by adding the appropriate label and putting machines ipp134-139 back to up and setting the pantasks back to run.
  • tdB: 11:30 Mark has finished fixing the config files for 3Pi and Twi processing. So, add the labels back to stdscience pantask, and make sure it is persistent by adding it to the input file under stdscience directory, under add.surveys. 
    ## TdB 20190404 -- add back label for BrightTwi processing
add.label BrightTwi.nightlyscience
add.label Bright3Pi.nightlyscience
    Mark will re-queue the exposures that did not go through before, during the day. 
mysql -hscidbm -uX -pX gpc1 -B -e ' select exp_name,date_format(dateobs,"%Y%m%d"),date_format(dateobs,"%m"),date_format(dateobs,"%d"),date_format(dateobs,"%Y"),exp_time,obs_mode,filter,comment from rawExp where comment like "%3PI%" and ( dateobs like "2019-%" or dateobs like "2018-12-%" ) and filter like "%" and obs_mode="BRIGHT"  and exp_id not in ( select exp_id from chipRun join rawExp using (exp_id) where filter like "%" and comment like "%3PI%" and ( dateobs like "2018-12-%" or dateobs like "2019-%" ) and obs_mode="BRIGHT" ) ; ' | grep -v exp_name | awk '{print "chiptool -definebyquery -set_label Bright3Pi.nightlyscience  -set_workdir neb://@HOST@.0/gpc1/Bright3Pi.nt/"$5"/"$3"/"$4" -set_data_group Bright3Pi."$2" -set_dist_group BRIGHT.WS -set_tess_id RINGS.V3 -set_end_stage warp -dbname gpc1 -exp_name "$1}'

Friday : 2019.04.05

  • TdB:

clearing some red on IPPmonitor. Some warps need to be redone from scratch: 

chiptool -dbname gpc1 -definebyquery -set_label mopsfix  -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o5967g0536o
chiptool -dbname gpc1 -definebyquery -set_label mopsfix  -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o5292g0385o
chiptool -dbname gpc1 -definebyquery -set_label mopsfix  -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o5694g0114o
chiptool -dbname gpc1 -definebyquery -set_label mopsfix  -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o6045g0242o
chiptool -dbname gpc1 -definebyquery -set_label mopsfix  -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o6457g0105o
chiptool -dbname gpc1 -definebyquery -set_label mopsfix  -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o6457g0107o
chiptool -dbname gpc1 -definebyquery -set_label mopsfix  -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o5583g0638o

chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1108817
chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1112071
chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1112081
chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1112088
chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1112095
chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1112096
chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1061918

warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1059342
warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1063016
warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1063202
warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1063221
warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1063237
warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1063239
warptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -warp_id 1012820

There are also some burntool errors: 

ipp_apply_burntool_fix.pl --exp_name o5664g0147o --class_id XY20 --verbose --dbname gpc1
ipp_apply_burntool_fix.pl --exp_name o5583g0674o --class_id XY22 --verbose --dbname gpc1
ipp_apply_burntool_fix.pl --exp_name o5399g0105o --class_id XY22 --verbose --dbname gpc1


chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3  -dbname gpc1 -chip_id 1112487
chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3  -dbname gpc1 -chip_id 1112892
chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3  -dbname gpc1 -chip_id 1113150

And also some b04/b05 related I/O errors, which we can revert: 

chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 -chip_id 1113987
chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1

There is also a ps_ud_MOPS file failing due to needing really old detrend file that is reprocessed from scratch: 

chiptool -dbname gpc1 -definebyquery -set_label mopsfix  -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20180323  -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20180323 -set_reduction LAP_SCIENCE -exp_name o6584g0058o 

chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1386558
  • TdB: Around 16:30 a large number of errors started appearing on IPPmonitor. As it turns out, these were due to the same label appearing in multiple pantasks, and this was causing a variety of errors to pop up. All of these have been moved to a broke label while the labels are removed.

Saturday : 2019.04.06

  • TdB: Some of the broke label exposures have been cleared. These were exposures that were in full state but were complaining about I/O errors on output files that were actually available. They have been sent to clean and subsequently reverted

Sunday : 2019.04.07

  • TdB: Rob has requested several exposures to be reprocessed without the CTE masking applied. 
    chiptool -definebyquery -set_label mops.nomasktest -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o8578g0251o
chiptool -definebyquery -set_label mops.nomasktest -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o8578g0234o
chiptool -definebyquery -set_label mops.nomasktest -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o8578g0217o
chiptool -definebyquery -set_label mops.nomasktest -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o8578g0200o

chiptool -definebyquery -set_label mops.nomasktest -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o8576g0439o
chiptool -definebyquery -set_label mops.nomasktest -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o8576g0421o
chiptool -definebyquery -set_label mops.nomasktest -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o8576g0403o
chiptool -definebyquery -set_label mops.nomasktest -set_workdir neb://@HOST@.0/gpc1/mops.nomasktest -set_data_group mops.nomasktest -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction NOMASKTEST -dbname gpc1 -exp_name o8576g0385o
Last modified 7 years ago Last modified on Apr 8, 2019, 10:53:25 AM
Note: See TracWiki for help on using the wiki.