IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links
wiki:PS1_IPP_Czarlog_20190415

PS1 IPP Czar Logs for the week 2019.04.15 - 2019.04.21

    1. PS1 IPP Czar Logs for the week 2019.04.15 - 2019.04.21
      1. Monday : 2019.04.15
      2. Tuesday : 2019.04.16
      3. Wednesday : 2019.04.17
      4. Thursday : 2019.04.18
      5. Friday : 2019.04.19
      6. Saturday : 2019.04.20
      7. Sunday : 2019.04.21

Czar of the week: CCL + MEH (czar-lord), previously JRF

(Up to PS1 IPP Czar Logs)

Monday : 2019.04.15

  • CCL: Haydn replaced OS drive on ippb21 
    neb-host --state down --host ippb04 --note 'CCL: repair --> down update faults finished'
neb-host --state repair --host ippb21 --note 'CCL: down->repair OS drive replaced'
  • MEH: update.% chip cleanup queued overnight again in normal ~ippitc/cleanup -- regularly has to be queued every few days to clean up space while update processing continues in ~ippitc/stdscience and ~ippmops/stdscience
  • MEH: ippb21 not exporting disks after reboot -- neb-host down (was an unstable node as well so should be kept down for a while) and will work with czar in morning to fix exports
  • CCL: did not remove update.LAP.PV3 label due to bright time for testing processing speed and it looks okay even with massive stamp requests.

Tuesday : 2019.04.16

  • CCL: disk warning 
    neb-host --state repair --host ipp106.1 -volume --note 'CCL: disk getting full so moving to repair: up->repair'
  • CCL: turn ippb16-ippb20, ippb22, ippb23 down for Haydn fixing fans.
  • CCL: ippb21 (uname -a to see OS --> ubuntu) seems not to export properly so Mark and I tried to figure it out. we first to check "systemctl satus nfs-server", nothing is strange try to restart "systemctl restart nfs-server", but it does not help after comparing with ippb20, ippb22 we found /etc/exports file has not following inform. 
    # ATRC IPP Storage offsite
/export/ippb21.0 166.122.172.0/255.255.255.128(nohide,rw,insecure,sync,no_subtree_check,root_squash)
/export/ippb21.1 166.122.172.0/255.255.255.128(nohide,rw,insecure,sync,no_subtree_check,root_squash)
# ITC IPP Prod cluster
/export/ippb21.0 128.171.123.224/255.255.255.224(nohide,rw,insecure,sync,no_subtree_check,root_squash)
/export/ippb21.1 128.171.123.224/255.255.255.224(nohide,rw,insecure,sync,no_subtree_check,root_squash)
    we add these into /etc/exports and "sudo exportfs -rav", then the disk is exported properly.
  • CCL: Rob asks for not restart pstamp server at 3:30pm 
    crontab uses "check_system.sh", so modified it, comment the regular one and uncomment the one without pstamp
#set dirs = ( registration stdscience summitcopy pstamp stack cleanup distribution ~ippqub/stdscience_ws ~ippps2/summitcopy ~ippps2/stdscience ~ippps2/registration ~ippps2/distribution )
## no pstamp restart --
set dirs = ( registration stdscience summitcopy stack cleanup distribution ~ippqub/stdscience_ws ~ippps2/summitcopy ~ippps2/stdscience ~ippps2/registration ~ippps2/distribution )
  • CCL: try to manually restart pantasks via check_system.sh, but somehow it did not work so I have to login to each server to restart them. 
    ==== these are for ippitc
pantasks server registration is NOT running (host: ippc25)
pantasks server stdscience is NOT running (host: ippc30)
pantasks server summitcopy is NOT running (host: ippc26)
pantasks server pstamp is NOT running (host: ippc26)
pantasks server stack is NOT running (host: ippc28)
pantasks server cleanup is NOT running (host: ippc23)
pantasks server distribution is NOT running (host: ippc24)
==== this is for ippqub
pantasks server /home/panstarrs/ippqub/stdscience_ws is NOT running (host: ippc28)
==== these are for ippps2
pantasks server /home/panstarrs/ippps2/summitcopy is NOT running (host: ippc24)
pantasks server /home/panstarrs/ippps2/stdscience is NOT running (host: ippc23)
pantasks server /home/panstarrs/ippps2/registration is NOT running (host: ippc25)
pantasks server /home/panstarrs/ippps2/distribution is NOT running (host: ippc30)

Wednesday : 2019.04.17

  • CCL: fix some burntool file 0 size issues 
    o5335g0235o	XY22	170074	1153065	update	update.LAP.PV3	LAP.PV3.20140730.20141014	LAP.ThreePi	2	(chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1
o5335g0235o	XY23	170074	1153065	update	update.LAP.PV3	LAP.PV3.20140730.20141014	LAP.ThreePi	2	(chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1
o5335g0236o	XY22	170076	1153066	update	update.LAP.PV3	LAP.PV3.20140730.20141014	LAP.ThreePi	2	(chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1
o5335g0236o	XY23	170076	1153066	update	update.LAP.PV3	LAP.PV3.20140730.20141014	LAP.ThreePi	2	(chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1
o5335g0240o	XY22	170080	1153067	update	update.LAP.PV3	LAP.PV3.20140730.20141014	LAP.ThreePi	2	(chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1

ipp_apply_burntool_fix.pl --exp_name o5335g0235o --class_id XY22 --verbose --dbname gpc1
ipp_apply_burntool_fix.pl --exp_name o5335g0235o --class_id XY23 --verbose --dbname gpc1
chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1153065
chiptool -dbname gpc1 -updaterun -set_label update.LAP.PV3 -chip_id 1153065

ipp_apply_burntool_fix.pl --exp_name o5335g0236o --class_id XY22 --verbose --dbname gpc1
ipp_apply_burntool_fix.pl --exp_name o5335g0236o --class_id XY23 --verbose --dbname gpc1
chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1153066
chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1153066

ipp_apply_burntool_fix.pl --exp_name o5335g0240o --class_id XY22 --dbname gpc1
chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1153067
chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1153067
  • CCL: at 11:30 Am turn ippb16 and ippb21 from down to repair 
    neb-host --state repair --host ippb16 --note 'CCL: down->repair monitoring its behavior after fans reinstalled'
neb-host --state repair --host ippb21 --note 'CCL: down->repair monitoring its behavior after fans reinstalled'

Thursday : 2019.04.18

  • CCL: reminds ipp134-ipp139 up in the morning repair before observing 
    before observing
 neb-host --state repair --host ipp134 --note 'CCL: up->repair, avoid new data'
 neb-host --state repair --host ipp135 --note 'CCL: up->repair, avoid new data'
 neb-host --state repair --host ipp136 --note 'CCL: up->repair, avoid new data'
 neb-host --state repair --host ipp137 --note 'CCL: up->repair, avoid new data'
 neb-host --state repair --host ipp138 --note 'CCL: up->repair, avoid new data'
 neb-host --state repair --host ipp139 --note 'CCL: up->repair, avoid new data'
after observing
 neb-host --state up --host ipp134 --note 'CCL: repair->up, for updating'
 neb-host --state up --host ipp135 --note 'CCL: repair->up, for updating'
 neb-host --state up --host ipp136 --note 'CCL: repair->up, for updating'
 neb-host --state up --host ipp137 --note 'CCL: repair->up, for updating'
 neb-host --state up --host ipp138 --note 'CCL: repair->up, for updating'
 neb-host --state up --host ipp139 --note 'CCL: repair->up, for updating'
  • CCL: nightly_science.pl is different using ipp "ipp-20170121" and "trunk" version
  • CCL: Haydn's tasks update 
    ippb16     fans fixed --> repair
ippb17     still down
ippb18     still down
ippb19     fans fixed --> repair
ippb20     still down
ippb21     fans fixed --> repair
ippb22     still down
ippb23     still down

Friday : 2019.04.19

  • MEH: sending some WSdiffs to distribution
  • CCL: fix red flags on warp stage which related to the chip stage couldn't find on ipp014 
    e.g.,
Couldn't find input file: neb://ipp014.0/gpc1/LAP.PV3.20140730.bigmem/2014/12/09/o5918g0124o.432165/o5918g0124o.432165.ch.1317544.XY21.ch.fits at 
/data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/warp_skycell.pl line 563.
chiptool -dbname gpc1 -updaterun -chip_id 1317544 -set_state goto_cleaned -set_label goto_cleaned
chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1317544
or
Couldn't find input file: neb://any/gpc1/LAP.PV3.20140730.bigmem/2014/12/16/o5576g0133o.280246/o5576g0133o.280246.cm.1305383.XY56.mk.fits

to fix camera stage fill missing I reprocess the o5492g0066o exp_name
chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190325 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190325 -set_reduction LAP_SCIENCE -exp_name o5492g0066o

but the dist grp and label are different from before, not sure if it needs to be changed?

Saturday : 2019.04.20

  • CCL: the cooling in the ATRC IT room is broken. so we put ippb06, ippb08, ippb16-23 down. temperature status ​https://ps1wiki.ifa.hawaii.edu/cgi-bin/ps1opsDashboard.py 
    ippb07-b16 doesn't seem to work
ippb06>> sensors
coretemp-isa-0000
Adapter: ISA adapter
Physical id 0:
           +45 C  (high =   +84 C)                   

coretemp-isa-0001
Adapter: ISA adapter
Physical id 1:
           +46 C  (high =   +84 C)
  • CCL: remove update.LAP.PV3 label and put ippb04-05, ippb12-b15 repair to see how it impacts processing in case we need to put them down tomorrow.

Sunday : 2019.04.21

  • CCL: shutdown ippb10-11 in the morning, turn on in the afternoon before obseving.
  • CCL: remove update.LAP.PV3 label for easily tracking processing behavior at ATRC.
Last modified 7 years ago Last modified on Apr 21, 2019, 4:46:30 PM
Note: See TracWiki for help on using the wiki.