PS1 IPP Czar Logs for the week 2019.02.04 - 2019.02.11
(Up to PS1 IPP Czar Logs)
Monday : 2019.02.04
JRF: Took over as czar, from the previous night there was a hiccup in the PS2 summitcopy logs, see below. Likely just a disc error (funpack) -- seemed to revert okay and processed
*** stderr *** stderr 13980 funpack returned exit status 26624 Unable to perform dsget: 29 at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/summit_copy. failure for: summit_copy.pl --uri http://ipp113.ifa.hawaii.edu/ds-gpc1/o8518g0235o/o8518g0235o34.fits --filename neb://ipp093.0/gpc1/20190204/o8518g0235o/o8518g0235o.ota34.fits --summit_id 1445517 --exp_name o8518g0235o --inst gpc1 --telescope ps1 --class chip --class_id ota34 --bytes 49432320 --md5 a86555c5072ea6e317764896c0a0b02c --dbname gpc1 --timeout 600 --verbose --copies 2 --compress --nebulous job exit status: 29 job host: ippc99 job dtime: 33.97878 job exit date: Sun Feb 3 22:27:45 2019
Tuesday : 2019.02.05
JRF: Ran the night report and saw that there was one exposure that had not gone to the chipRun stage (o8519g0755o). It was intentionally not sent to the next stage, as there is no config for processing this type of exposure yet. It's end stage was registration so all was good. JRF: Will be meeting with Mark later to go through some errors that were reported in the logs (TODO: update this)
JRF: stdscience error log reported too many connections to the database. This had come up previously (in the past month):
-> psDBAlloc (psDB.c:166): Database error generated by the server
Failed to connect to database. Error: Too many connections
-> faketoolConfig (faketoolConfig.c:373): unknown psLib error
Can't configure database
-> main (faketool.c:70): (null)
failed to configure
Unable to perform faketool -addproces
config error for: fake_imfile.pl --exp_id 1450315 --fake_id 2062988 --class_id XY31 --chiproot=neb://ipp090.0/gpc1/OSS.nt/2019/02/05//o8519g0140o.1450315/o8519g0140o.1450315.ch.2127506 --camroot=neb://any/gpc1/OSS.nt/2019/02/05//o8519g0140o.1450315/o8519g0140o.1450315.cm.2093978 --camera GPC1 --outroot neb://ipp090.0/gpc1/OSS.nt/2019/02/05//o8519g0140o.1450315/o8519g0140o.1450315.fk.2062988 --reduction SWEETSPOT --dbname gpc1 --verbose
job exit status: 3
job host: ippc69
job dtime: 3.755007
job exit date: Mon Feb 4 20:22:07 2019
*** stdout ***
stdout 13038
and another:
-> psDBAlloc (psDB.c:166): Database error generated by the server
Failed to connect to database. Error: Too many connections
-> faketoolConfig (faketoolConfig.c:373): unknown psLib error
Can't configure database
-> main (faketool.c:70): (null)
failed to configure
Unable to perform faketool -addproces
config error for: fake_imfile.pl --exp_id 1450315 --fake_id 2062988 --class_id XY54 --chiproot=neb://ipp112.0/gpc1/OSS.nt/2019/02/05//o8519g0140o.1450315/o8519g0140o.1450315.ch.2127506 --camroot=neb://any/gpc1/OSS.nt/2019/02/05//o8519g0140o.1450315/o8519g0140o.1450315.cm.2093978 --camera GPC1 --outroot neb://ipp112.0/gpc1/OSS.nt/2019/02/05//o8519g0140o.1450315/o8519g0140o.1450315.fk.2062988 --reduction SWEETSPOT --dbname gpc1 --verbose
job exit status: 3
job host: ippc79
job dtime: 3.872324
job exit date: Mon Feb 4 20:22:07 2019
*** stdout ***
stdout 11212
JRF: The stdscience log last night on PS1 showed a bunch of errors reported around 20:30. The majority of these were on ippc118. Look at the ganglia logs there was a spike in network activity, up to about 17Mb, but not too large that it should cause an issue. Either way the tasks that failed reverted successfully and continued on after this.
JRF: MEH will look into the reported nightly_science.pl errors. Some are reported for gpc2 in the gpc1 logs!! e.g.
failure for: nightly_science.pl --queue_diffs --date 2019-02-05 --dbname gpc2 --camera GPC2 job exit status: 2 job host: localhost job dtime: 8.072818 job exit date: Mon Feb 4 21:04:02 2019
Commented out the use of gpc2 dates in the 'stdscience/input' file. Believe this is the reason that there are gpc2 errors being reported in the gpc1 logs above.
MEH: the logs have useful info for tracking down problems and while the ~ippitc/start_server.sh script archives the logs, it might also be good to keep a note of running tasks and taskstats before doing a shutdown -- one way can be like this
echo "status;quit" | pantasks_client -c ~ippitc/stdscience/ptolemy.rc >& stdsci_status.`date +%y%m%d_%H%M%S` echo "status -taskstats;quit" | pantasks_client -c ~ippitc/stdscience/ptolemy.rc >& stdsci_tasksstats.`date +%y%m%d_%H%M%S` echo "controller status;quit" | pantasks_client -c ~ippitc/stdscience/ptolemy.rc >& stdsci_nodes.`date +%y%m%d_%H%M%S`
MEH: increased ippdb08 mysql from 256 to 512 for tonight with set global (ephemeral) and not in my.cnf to monitor how things go
JRF: killed a 4 day old job in the database that was run as ippuser to query something. WARNING: very dangerous to go around killing jobs.
Wednesday : 2019.02.06
MEH: recent [ps-ipp-ops] email
- PROBLEM alert - ippc75/Root Partition is CRITICAL -- nagios on 20190123 -- / nearly full, getting dangerous -- need to plan on resetting the nebulous_server.log (log rotation already in place)
- ippc75 apache nebulous disk warning -- cronjob check on 20190130 -- 98% left, now dangerous since rotated logs can reach this size before rotation, need to deal with ASAP
- RECOVERY alert - ippc75/Root Partition is OK -- nagios on 20190206 -- good
- Fatal | Event occured on: ipp097.ifa.hawaii.edu -- 20190202 -- BBU failed, could consider putting neb-host repair particularly if many of the ipp067-122 nodes start filling up (close)
- Fatal | Event occured on: ipp087.ifa.hawaii.edu -- 20190206 -- BBU failed, already in neb-host repair for this reason
- FATAL Event occurred on: ipp054 -- 20190205 -- BBU failed, already in neb-host repair, nothing to do and just reminder (applies for ipp054-066)
- PROBLEM alert - ippdb05/Root Partition is WARNING -- 20190112 -- / nearly full and very dangerous, log rotate on syslog will help
- RECOVERY alert - ippdb05/Root Partition is OK -- 20190206 -- good
- MySQL replication problem on ippc17 -- 20190206 -- reminder replication for ippRequestServer (pstamp+datastore) DB on ipp113 doesn't have replication setup still
- Cron <ippps2@ippc23> /bin/tcsh /data/ippc18.0/home/ippps2/cron_fix_fault_c23/ipp_ps2_rev4.bat --
- PROBLEM alert - ipp112dev/sda1 is WARNING -- nagios on 20190203 --
JRF: MEH, TdB and myself are going to archive the nebulous logs as they are getting very large, hence nagios complaining about lack of disk space on root. Log sizes:
- ippc71 - 20GB
- ippc72 - 46GB
- ippc73 - 78GB
- ippc74 - 74GB
- ippc75 - 78GB
There are also other logs in the /var/log/ folder that already have log rotation setup and are archived periodically like messages and apache2 that can be useful to find info on problems
- archive the neb log (in this case this is for ippc75, change the location appropriately)
user@ippc75$ rsync -avP /tmp/nebulous_server.log /export/ippc75.0/ cd /export/ippc75.0/ mv nebulous_server.log nebulous_server.log.180206 bzip2 nebulous_server.log.180206
- MEH: nebulous apache server (ippc70-c75) log reset --
stop all pantasks -- email ipp-dev to let know offline for X time sudo /etc/init.d/apache2 stop sudo /etc/init.d/apache2 status sudo rm /tmp/nebulous_server.log ; sudo touch /tmp/nebulous_server.log ; sudo chown apache /tmp/nebulous_server.log ; sudo chmod g+w /tmp/nebulous_server.log ; ls -l /tmp/*log sudo /etc/init.d/apache2 start sudo /etc/init.d/apache2 status
The above makes sure to check that apache is no longer writing to the log before moving it. After moving a new file needs to be created, with the correct permissions, for apache to write to in the future. Then the service can be restarted.
- do a test exposure
After setting everything back up it's a good idea to do a test exposure. So turn on stdscience from the pantasks_client and run the following chiptool command (change the date).
chiptool -dbname gpc1 -definebyquery -set_label MOPS.dailytestset -set_workdir neb://@HOST@.0/gpc1/MOPS.dailytestset.20190206 -set_data_group MOPS.dailytestset.20190206 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction SWEETSPOT -simple -exp_name o6370g0351o -pretend
JRF: When checking that we could use the console commands on ipp113, it has some troublesome status. MEH will report this to Hayden.
Outlet Name Status Post-on Delay(s) i05_pdu0[15] i48A_15_BAD OFF(locked) 0.5
N.B. To get out of the console, do: Shift + ~, . That is press shift and tilde together (possibly twice), then release and press period.
MEH: running LAP.PV3 update tests in ~ippmops:stdscience using ippc31-c63, ipps00-14
Thursday : 2019.02.07
JRF: No data last night, but there were a bunch of errors from some reprocessing that MEH started.
Checking error logs: A typical error from the ippitc/stdscience log:
Starting script /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/warp_skycell.pl on ipp109 at Wed Feb 6 18:40:41 *** stderr *** system failure for: warp_skycell.pl --threads @MAX_THREADS@ --warp_id 440532 --warp_skyfile_id 37161875 --skycell_id skycell.0868.085 --tess_dir RINGS.V3 --camera GPC1 --outroot neb:///gpc1/ecliptic/20:00hr/o5773g0307o.370390/o5773g0307o.370390.wrp.440532.skycell.0868.085 --redirect-output --run-state update --dbname gpc1 --verbose job exit status: 2 job host: ipp109 job dtime: 3.714778 job exit date: Wed Feb 6 18:40:45 2019 *** stdout *** stdout 936
The majority of the errors all occurred around 18:40. Check out the error in more detail by following the outroot provided in the error message using neb-ls
Then neb-locate the log and have a read (in this case it is the update log as the state is update). Curiously the log file was b-zipped, which they should not be normally and might have perhaps caused the problem.
Looking at the bottom of the file we get the error:
neb entry neb:///gpc1/ecliptic/20:00hr/o5773g0307o.370390/o5773g0307o.370390.wrp.440532.skycell.0868.085.skycell not found, not created at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/warp_skycell.pl line 210 Running [/data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/dvoImageExtract -D CATDIR /local/ipp/gpc1/tess/RINGS.V3 skycell.0868.085 -o /data/ipp123.0/nebulous/e6/f8/10999657730.gpc1:ecliptic:20:00hr:o5773g0307o.370390:o5773g0307o.370390.wrp.440532.skycell.0868.085.skycell]... matched chips for 0 images, 0 failed (264400 total) 2019/02/06 18:40:44 | ipp109 | FATAL | Nebulous::Client::find_instances - unhandled fault - database error: no instances available for key: neb:///gpc1/ecliptic/20:00hr/o5773g0307o.370390/o5773g0307o.370390.cm.458814.XY76.mk.fits at /usr/lib64/perl5/site_perl/5.8.8/Nebulous/Server.pm line 1991, <DATA> line 14. Unable to find instance of Nebulous handle neb:///gpc1/ecliptic/20:00hr/o5773g0307o.370390/o5773g0307o.370390.cm.458814.XY76.mk.fits at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/warp_skycell.pl line 237 Couldn't find input file: neb:///gpc1/ecliptic/20:00hr/o5773g0307o.370390/o5773g0307o.370390.cm.458814.XY76.mk.fits at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/warp_skycell.pl line 563. Running [/data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/warptool -updateskyfile -warp_id 440532 -skycell_id skycell.0868.085 -fault 2 -dbname gpc1]...
It appears that it couldn't find the fits file it wanted, so we should check the current state of the exposure.
Query the chipRun and rawExp table:
SELECT exp_name, exp_id, label, data_group, chipRun.state FROM rawExp LEFT JOIN chipRun USING (exp_id) WHERE exp_name="o5773g0307o";
The chip state is still stuck in update, so we should sent it to be fixed (details of this are given on the page for PSNSC_MOPS:http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/PSNSC_MOPS):
chiptool -dbname gpc1 -definebyquery -set_label PSNSC.wfix.20180525 -set_workdir neb://@HOST@.0/gpc1/PSNSC.wfix.20180525 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction SWEETSPOT -set_data_group PSNSC.wfix.20180525.x.date -exp_name o5773g0307o
For the above to be actually processed the label needs to be added to the stdscience pantasks. Use 'add.label PSNSC.wfix.20180525' in the client.
There is labeltool for viewing all the priorities of the labels. Higher number -> higher priority.
labeltool -dbname gpc1 -listlabel -simple
Clearing IPPmonitor errors: Next, there were some errors on IPP monitor for ps_ud_MOPS. It turns out these are mostly red herrings. The problems were only in parentheses on IPP monitor, so there isn't a proper problem but some errors reported that should be looked at.
Clicking the links there is a problem with a skycell warp? There is an error log reported on the page, so we can find that from nebulous and look at the issue.
For each chip_id do the following to update it:
chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_WEB -chip_id 1059958
Once the chip stage is done set the warp_ids to update too:
warptool -dbname gpc1 -setskyfiletoupdate -set_label ps_ud_WEB -warp_id XXXXXX
This should clear the errors.
Crontabs: These are important for keeping nightly processing flowing smoothly as they correct some faults that crop up.
Under ippuser home there is a folder called 'cron_tab_c30', the name informs us that the cron_tab is on ippc30. Going to this machine and viewing the crontabs (crontab -l), there are few in operation. Notably one of them is to correct the fault 4 memory issue.
Taking nodes down: There were two nodes that had pretty full disks >97%. It's only one particular volume on each so we used the neb-host command to alter the status and specified that it should just be for that volume:
neb-host --host ipp114.0 --state repair --volume --note "Disk is getting very full, so taking down"
Evening update: Loads of errors reported in the chip stage from some ps_ud_MOPS.2 processing. Checking the stdout there are a bunch of programming errors and system failures (in the warp stage as well).
I couldn't find the log files using nebulous for some of the exposures...
At the top of the list is exposure o5367g0555o. Checking this in the table it's status is still 'update' in the chipRun table (full in expRun). So it looks like it's stuck, therefore I ran the wfix as per PSNSC instructions:
chiptool -dbname gpc1 -definebyquery -set_label PSNSC.wfix.20180525 -set_workdir neb://@HOST@.0/gpc1/PSNSC.wfix.20180525 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction SWEETSPOT -set_data_group PSNSC.wfix.20180525.x.date -exp_name o5367g0555o
I will check back shortly to see if this has cleared and run the remaining process on this exposure.
Then I'll check all the other exposures.
Friday : 2019.02.08
Morning clean up JRF: I didn't finish clearing up last night as it got late and the telescope opened. So continuing on:
I'll query the state of the exposures still listed as problems in the chiptool table: SELECT exp_name, exp_id, label, data_group, chipRun.state FROM rawExp LEFT JOIN chipRun USING (exp_id) WHERE exp_name IN ("o5563g0048o", "o5563g0068o", "o6081g0423o", "o6089g0580o", "o6215g0082o", "o6215g0062o", "o6783g0595o", "o6036g0249o", "o5368g0203o", "o5561g0743o", "o5522g0420o", "o6072g0424o", "o6215g0072o", "o5918g0460o", "o5218g0041o", "o5218g0057o", "o5362g0570o", "o5362g0583o", "o5476g0223o", "o5476g0289o", "o5855g0127o", "o5328g0034o", "o5328g0050o")
From these, the following are done, indicated by the state="full":
+-------------+--------+---------------------+----------------------------+-------+---------+ | exp_name | exp_id | label | data_group | state | chip_id | +-------------+--------+---------------------+----------------------------+-------+---------+ | o5218g0041o | 125344 | PSNSC.wfix.20180525 | PSNSC.wfix.20180525.x.date | full | 2128159 | | o5218g0057o | 125358 | PSNSC.wfix.20180525 | PSNSC.wfix.20180525.x.date | full | 2128160 | | o5362g0570o | 181829 | PSNSC.wfix.20180525 | PSNSC.wfix.20180525.x.date | full | 2128161 | | o5362g0583o | 181842 | PSNSC.wfix.20180525 | PSNSC.wfix.20180525.x.date | full | 2128162 | | o5476g0223o | 235917 | PSNSC.wfix.20180525 | PSNSC.wfix.20180525.x.date | full | 2128164 | | o5476g0289o | 235984 | PSNSC.wfix.20180525 | PSNSC.wfix.20180525.x.date | full | 2128163 | +-------------+--------+---------------------+----------------------------+-------+---------+
Therefore, set their old chips, the ones stuck in update, to clean: SELECT exp_name, exp_id, label, data_group, chipRun.state, chipRun.chip_id FROM rawExp LEFT JOIN chipRun USING (exp_id) WHERE exp_name IN ("o5218g0041o", "o5218g0057o", "o5362g0570o", "o5362g0583o", "o5476g0223o", "o5476g0289o") AND chipRun.state="update";
+-------------+--------+--------------+------------------+--------+---------+ | exp_name | exp_id | label | data_group | state | chip_id | +-------------+--------+--------------+------------------+--------+---------+ | o5218g0041o | 125344 | ps_ud_MOPS.2 | ecliptic.rp.23hr | update | 486920 | | o5218g0057o | 125358 | ps_ud_MOPS.2 | ecliptic.rp.23hr | update | 486927 | | o5362g0570o | 181829 | ps_ud_MOPS.2 | ecliptic.rp.1hr | update | 489051 | | o5362g0583o | 181842 | ps_ud_MOPS.2 | ecliptic.rp.1hr | update | 489062 | | o5476g0223o | 235917 | ps_ud_MOPS.2 | ecliptic.rp.1hr | update | 489311 | | o5476g0289o | 235984 | ps_ud_MOPS.2 | ecliptic.rp.1hr | update | 489341 | +-------------+--------+--------------+------------------+--------+---------+
use:
chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id XXXXXX
Query the remaining with: SELECT exp_name, exp_id, label, data_group, chipRun.state, chipRun.chip_id FROM rawExp LEFT JOIN chipRun USING (exp_id) WHERE exp_name="XXXXXXX";
All of these are stuck in the update step with the MOPS.2 label. I guess I never send this to be reprocessed. So I will do that using the wfix label, then I'll send the old to cleaned
- HOWEVER... reading the wiki the LAP.PV3 may need to be processed without SWEETSPOT reduction, and instead use default...MEH will explain all later.
MEH:
labeltool -dbname gpc1 -listlabel -simple labeltool -dbname gpc1 -definelabel -set_label BrightTri -set_priority 300 labeltool -dbname gpc1 -definelabel -set_label Bright3Pi -set_priority 300 labeltool -dbname gpc1 -updatelabel -label BrightTwi -set_priority 290
- 16:00 EAM : I started a set of rsyncs on ippb04 to copy nebulous directories to ippb16, ippb18, ippb19 (for b04.0, .1, .2 respectively). I needed to add an id_rsa-style key to the ipp user since ippb04 refuses to use dsa keys. (copy of existing authorized_keys saved as authorized_keys.20190207)
JRF: Have been checking through the list of cluster nodes using neb-host to find nodes that have been left in repair. So that they can be brought back into production if they are ok
There are config files located in the modules folder that dictate which nodes to save results do. e.g. processing of files relating to GPC1 XY01 get saved to ipp067.0. The current config is found here:
/data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/share/pantasks/modules
Best practice is to make a backup in the folder. MEH handled the changes to this file.
To use the new config need to stop and shutdown stdscience pantasks. Make sure that there are no jobs still running after stopping before you shutdown.
Then to restart with the new config go to the machine that was running pantasks and run:
./start_server.sh stdscience
This will start pantasks back up, AND it archives the log (which does not happen if you start it normally).
With pantasks running we need to actually get the scheduler and controller going by using the following commands:
server input input setup run
JRF: Going back to the ippmonitor issues: There were a lot of LAP chip errors, which look like an improper state. These just need to have the chip updated e.g.
chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_WEB -chip_id XXXXXX
Saturday : 2019.02.09
JRF: MEH took TdB and myself through checking out ganlia. Monitoring the general trends on CPU, memory and network loads on various machines. How to spot problems e.g. If the CPU report displays mostly orange/red then it indicates that the machine is often waiting before it can do something, which suggests a more thorough investigation.
Can check individual machines in pantasks (or add/delete them too). pantasks_client controller host check ippc118
Configuration failed on processing some files. Used the following:
chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_WEB -chip_id
This was to solve some warps that got stuck in the warp stage when reprocessing, the chip im files just needed to be updated. Found out by querying: SELECT warp_id, state, label FROM warpRun WHERE label LIKE "ps_ud_WEB"; then: select chip_id,chipRun.state,chipRun.label,chipRun.data_group from warpRun join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) where warp_id=1043753; Repeated for the the remainder stuck in this state.
There are a bunch of wfix things stuck in the cam stage, so queried those to get the chip chiptool -dbname gpc1 -setimfiletoupdate -set_label PSNSC.wfix.20180525 -chip_id
+---------+---------+---------+---------+--------+ | warp_id | fake_id | cam_id | chip_id | state | +---------+---------+---------+---------+--------+ | 1043758 | 1037107 | 1061440 | 1093979 | update | | 1168814 | 1161011 | 1185484 | 1210911 | update | | 1168816 | 1161013 | 1185486 | 1210913 | update | | 1169004 | 1161150 | 1185635 | 1211006 | update | | 1169008 | 1161154 | 1185639 | 1211010 | update | | 1169202 | 1161493 | 1185817 | 1211220 | update | | 1169204 | 1161495 | 1185820 | 1211223 | update | | 1175590 | 1168954 | 1193676 | 1228151 | update | | 1181423 | 1173791 | 1198632 | 1233292 | update | | 1187981 | 1179229 | 1203026 | 1238323 | update | | 1265032 | 1258067 | 1284307 | 1321028 | update | | 1265037 | 1258072 | 1284316 | 1321033 | update | | 1842166 | 1835551 | 1863463 | 1895949 | update | +---------+---------+---------+---------+--------+ chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_WEB -chip_id 1093979 chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_WEB -chip_id 1210911 chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_WEB -chip_id 1210913 chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_WEB -chip_id 1211006 chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_WEB -chip_id 1211010 chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_WEB -chip_id 1211220 chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_WEB -chip_id 1211223 chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_WEB -chip_id 1228151 chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_WEB -chip_id 1233292 chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_WEB -chip_id 1238323 chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_WEB -chip_id 1321028 chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_WEB -chip_id 1321033 chiptool -dbname gpc1 -setimfiletoupdate -set_label ps_ud_WEB -chip_id 1895949
Do this on the MOPS.2 stuff to get them going again: chiptool -dbname gpc1 -definebyquery -set_label PSNSC.wfix.20180525 -set_workdir neb://@HOST@.0/gpc1/PSNSC.wfix.20180525 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction SWEETSPOT -set_data_group PSNSC.wfix.20180525.x.date -exp_name o5327g0047o o5675g0136o o5675g0143o o5829g0438o o5328g0050o Once confirmed that the data has been processed, clean up the old stuff...
Sunday : 2019.02.10
- MEH: power at ATRC down, Haydn powered down ippb06-b23 -- set neb-host down for all machines so any primary copies there will switch to secondary on the system
- Gene has ippb04,b05 in repair to rsync and retire -- there are now no backup/archive nodes for replication -- NO processing can proceed -- this is not an acceptable situation as discussed before and a backup node set and plan is needed for future ATRC down event.
JRF: ME, MEH, and TdB never got round to going though reprocessing some data for Rob without some pixels removed (as an NEO is suspected to be falling into a masked area). TODO at a later date.
