PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2019.03.11

• MEH: simple oneliner to remove the faulted chip/warps in update.OSS to update.OSS.broke until ipp067 back online -- remember to add the user/password

  mysql -hscidbm -uXXXXX -pXXXXX gpc1 -B -e ' select chip_id,warp_id,warpRun.label,warpRun.data_group,warpRun.state,chipRun.label,chipRun.state from warpRun join warpSkyfile using (warp_id) join fakeRun using (fake_id) join camRun using (cam_id) join chipRun using (chip_id) join chipProcessedImfile using (chip_id) where warpRun.label like "update.OSS" and warpRun.data_group like "OSS%" and warpRun.state like "update" and  ( warpSkyfile.fault>0 or chipProcessedImfile.fault>0 ) group by warp_id limit 500; ' | grep -v data_group | awk '{print "chiptool -dbname gpc1 -updaterun -set_label update.OSS.broke -chip_id  "$1" ;warptool -dbname gpc1  -updaterun -set_label update.OSS.broke -warp_id "$2 }'
  

• TdB: Following upgrades to Ubuntu and a series of EDAC errors, machines ippb16-23 are now behaving well again. They have been tested rigorously by Gene using rsyncs, and now put back to neb-host up.

• CCL: set ipp134 down -> repair for testing then -> up, there are ipp134_bck.0, set it up only ATRC is offline
• CCL: set ipp135 down -> repair for testing (ipp135_bck.o, same as ipp134_bck.0)
• CCL: restart stdscience at 14:00

• TdB: The ipp067 related faults on IPP monitor are being cleared out on a regular basis, when the total number of faulted exposures becomes too high (>300). In that case, stop the stdscience pantask and run the batch SQL query/chiptool command Mark gives above. Then, set the pantask back to run.

• MEH: adding other updates to run in ~ippmops/stdscience targeting ipp123-129 for warps -- stop pantasks if needed

Tuesday : 2019.03.12

• MEH: ipp130 looks okay, setting neb-host down->repair->up and will just target in ippmops:stdscience for now
  • if it works fine, it could be kept neb-host up when nightly processing starts since it has serial console access in i12 (needs to be added to Consolerc)

• TdB: Fixed a bunch of red on IPPmonitor by changing the labels to update.OSS.broke. However, a few exposures failed with fault=2 and state=full, which was somewhat unusual. An example is exposure o6982g0381o, XY41 which went through a chip update. The bottom of the update log reads:

        Complete ppImage run: 20.814865 sec
        Memory leaks: 0
  Number of leaks to display: 500
  Running [/home/panstarrs/ippmops/psconfig/ipp-20141024.lin64/bin/ppStatsFromMetadata /data/ipp018.0/nebulous/ee/8d/8954414172.gpc1:OSS.nt:2014:11:21:o6982g0390o.821305:o6982g0390o.821305.ch.1262404.XY41.stats - CHIP_IMFILE]...
  -bg 314.070892 -bg_mean_stdev 1.714188 -bg_stdev 19.923737 -dtime_detrend 9.678738 -dtime_photom 0.000003 -dtime_total 20.751261 -quality 0 
  Running [/home/panstarrs/ippmops/psconfig/ipp-20141024.lin64/bin/funpack -S /data/ipp018.0/nebulous/e2/46/8954413865.gpc1:OSS.nt:2014:11:21:o6982g0390o.821305:o6982g0390o.821305.ch.1262404.XY41.ch.fits > /dev/null]...
  Running [/home/panstarrs/ippmops/psconfig/ipp-20141024.lin64/bin/funpack -S /data/ipp018.0/nebulous/58/d7/8954414001.gpc1:OSS.nt:2014:11:21:o6982g0390o.821305:o6982g0390o.821305.ch.1262404.XY41.ch.mk.fits > /dev/null]...
  Running [/home/panstarrs/ippmops/psconfig/ipp-20141024.lin64/bin/funpack -S /data/ipp018.0/nebulous/3e/19/8954414035.gpc1:OSS.nt:2014:11:21:o6982g0390o.821305:o6982g0390o.821305.ch.1262404.XY41.ch.wt.fits > /dev/null]...
  Running [/home/panstarrs/ippmops/psconfig/ipp-20141024.lin64/bin/funpack -S /data/ipp018.0/nebulous/fc/05/8954414163.gpc1:OSS.nt:2014:11:21:o6982g0390o.821305:o6982g0390o.821305.ch.1262404.XY41.mdl.fits > /dev/null]...
  Running [/home/panstarrs/ippmops/psconfig/ipp-20141024.lin64/bin/chiptool -tofullimfile -chip_id 1262404 -class_id XY41 -dbname gpc1]... 
  

  The files in question cannot be accessed since ipp018 is down on nebulous.

• TdB: ipp136 and ipp137 put from down to up, to test for files going there. it is currently not targeted for specific products, but will be tomorrow if all works out.

Wednesday : 2019.03.13

Thursday : 2019.03.14

Friday : 2019.03.15

Saturday : 2019.03.16

Sunday : 2019.03.17