== PS1 IPP Czar Logs for the week 2019.04.15 - 2019.04.21 == [[PageOutline]] Czar of the week: CCL + MEH (czar-lord), previously JRF (Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs]) === Monday : 2019.04.15 === * CCL: Haydn replaced OS drive on ippb21 {{{ neb-host --state down --host ippb04 --note 'CCL: repair --> down update faults finished' neb-host --state repair --host ippb21 --note 'CCL: down->repair OS drive replaced' }}} * MEH: update.% chip cleanup queued overnight again in normal ~ippitc/cleanup -- regularly has to be queued every few days to clean up space while update processing continues in ~ippitc/stdscience and ~ippmops/stdscience * MEH: ippb21 not exporting disks after reboot -- neb-host down (was an unstable node as well so should be kept down for a while) and will work with czar in morning to fix exports * CCL: did not remove update.LAP.PV3 label due to bright time for testing processing speed and it looks okay even with massive stamp requests. === Tuesday : 2019.04.16 === * CCL: disk warning {{{ neb-host --state repair --host ipp106.1 -volume --note 'CCL: disk getting full so moving to repair: up->repair' }}} * CCL: turn ippb16-ippb20, ippb22, ippb23 down for Haydn fixing fans. * CCL: ippb21 (uname -a to see OS --> ubuntu) seems not to export properly so Mark and I tried to figure it out. we first to check "systemctl satus nfs-server", nothing is strange try to restart "systemctl restart nfs-server", but it does not help after comparing with ippb20, ippb22 we found /etc/exports file has not following inform. {{{ # ATRC IPP Storage offsite /export/ippb21.0 166.122.172.0/255.255.255.128(nohide,rw,insecure,sync,no_subtree_check,root_squash) /export/ippb21.1 166.122.172.0/255.255.255.128(nohide,rw,insecure,sync,no_subtree_check,root_squash) # ITC IPP Prod cluster /export/ippb21.0 128.171.123.224/255.255.255.224(nohide,rw,insecure,sync,no_subtree_check,root_squash) /export/ippb21.1 128.171.123.224/255.255.255.224(nohide,rw,insecure,sync,no_subtree_check,root_squash) }}} we add these into /etc/exports and "sudo exportfs -rav", then the disk is exported properly. * CCL: Rob asks for not restart pstamp server at 3:30pm {{{ crontab uses "check_system.sh", so modified it, comment the regular one and uncomment the one without pstamp #set dirs = ( registration stdscience summitcopy pstamp stack cleanup distribution ~ippqub/stdscience_ws ~ippps2/summitcopy ~ippps2/stdscience ~ippps2/registration ~ippps2/distribution ) ## no pstamp restart -- set dirs = ( registration stdscience summitcopy stack cleanup distribution ~ippqub/stdscience_ws ~ippps2/summitcopy ~ippps2/stdscience ~ippps2/registration ~ippps2/distribution ) }}} * CCL: try to manually restart pantasks via check_system.sh, but somehow it did not work so I have to login to each server to restart them. {{{ ==== these are for ippitc pantasks server registration is NOT running (host: ippc25) pantasks server stdscience is NOT running (host: ippc30) pantasks server summitcopy is NOT running (host: ippc26) pantasks server pstamp is NOT running (host: ippc26) pantasks server stack is NOT running (host: ippc28) pantasks server cleanup is NOT running (host: ippc23) pantasks server distribution is NOT running (host: ippc24) ==== this is for ippqub pantasks server /home/panstarrs/ippqub/stdscience_ws is NOT running (host: ippc28) ==== these are for ippps2 pantasks server /home/panstarrs/ippps2/summitcopy is NOT running (host: ippc24) pantasks server /home/panstarrs/ippps2/stdscience is NOT running (host: ippc23) pantasks server /home/panstarrs/ippps2/registration is NOT running (host: ippc25) pantasks server /home/panstarrs/ippps2/distribution is NOT running (host: ippc30) }}} === Wednesday : 2019.04.17 === * CCL: fix some burntool file 0 size issues {{{ o5335g0235o XY22 170074 1153065 update update.LAP.PV3 LAP.PV3.20140730.20141014 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 o5335g0235o XY23 170074 1153065 update update.LAP.PV3 LAP.PV3.20140730.20141014 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 o5335g0236o XY22 170076 1153066 update update.LAP.PV3 LAP.PV3.20140730.20141014 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 o5335g0236o XY23 170076 1153066 update update.LAP.PV3 LAP.PV3.20140730.20141014 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 o5335g0240o XY22 170080 1153067 update update.LAP.PV3 LAP.PV3.20140730.20141014 LAP.ThreePi 2 (chip.revert.on and chip.revert.off) or chiptool -revertprocessedimfile -fault 2 -label update.LAP.PV3 -dbname gpc1 ipp_apply_burntool_fix.pl --exp_name o5335g0235o --class_id XY22 --verbose --dbname gpc1 ipp_apply_burntool_fix.pl --exp_name o5335g0235o --class_id XY23 --verbose --dbname gpc1 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1153065 chiptool -dbname gpc1 -updaterun -set_label update.LAP.PV3 -chip_id 1153065 ipp_apply_burntool_fix.pl --exp_name o5335g0236o --class_id XY22 --verbose --dbname gpc1 ipp_apply_burntool_fix.pl --exp_name o5335g0236o --class_id XY23 --verbose --dbname gpc1 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1153066 chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1153066 ipp_apply_burntool_fix.pl --exp_name o5335g0240o --class_id XY22 --dbname gpc1 chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id 1153067 chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1153067 }}} * CCL: at 11:30 Am turn ippb16 and ippb21 from down to repair {{{ neb-host --state repair --host ippb16 --note 'CCL: down->repair monitoring its behavior after fans reinstalled' neb-host --state repair --host ippb21 --note 'CCL: down->repair monitoring its behavior after fans reinstalled' }}} === Thursday : 2019.04.18 === * CCL: reminds ipp134-ipp139 up in the morning repair before observing {{{ before observing neb-host --state repair --host ipp134 --note 'CCL: up->repair, avoid new data' neb-host --state repair --host ipp135 --note 'CCL: up->repair, avoid new data' neb-host --state repair --host ipp136 --note 'CCL: up->repair, avoid new data' neb-host --state repair --host ipp137 --note 'CCL: up->repair, avoid new data' neb-host --state repair --host ipp138 --note 'CCL: up->repair, avoid new data' neb-host --state repair --host ipp139 --note 'CCL: up->repair, avoid new data' after observing neb-host --state up --host ipp134 --note 'CCL: repair->up, for updating' neb-host --state up --host ipp135 --note 'CCL: repair->up, for updating' neb-host --state up --host ipp136 --note 'CCL: repair->up, for updating' neb-host --state up --host ipp137 --note 'CCL: repair->up, for updating' neb-host --state up --host ipp138 --note 'CCL: repair->up, for updating' neb-host --state up --host ipp139 --note 'CCL: repair->up, for updating' }}} * CCL: nightly_science.pl is different using ipp "ipp-20170121" and "trunk" version * CCL: Haydn's tasks update {{{ ippb16 fans fixed --> repair ippb17 still down ippb18 still down ippb19 fans fixed --> repair ippb20 still down ippb21 fans fixed --> repair ippb22 still down ippb23 still down }}} === Friday : 2019.04.19 === * MEH: sending some WSdiffs to distribution * CCL: fix red flags on warp stage which related to the chip stage couldn't find on ipp014 {{{ e.g., Couldn't find input file: neb://ipp014.0/gpc1/LAP.PV3.20140730.bigmem/2014/12/09/o5918g0124o.432165/o5918g0124o.432165.ch.1317544.XY21.ch.fits at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/warp_skycell.pl line 563. chiptool -dbname gpc1 -updaterun -chip_id 1317544 -set_state goto_cleaned -set_label goto_cleaned chiptool -dbname gpc1 -setimfiletoupdate -set_label update.LAP.PV3 -chip_id 1317544 or Couldn't find input file: neb://any/gpc1/LAP.PV3.20140730.bigmem/2014/12/16/o5576g0133o.280246/o5576g0133o.280246.cm.1305383.XY56.mk.fits to fix camera stage fill missing I reprocess the o5492g0066o exp_name chiptool -dbname gpc1 -definebyquery -set_label mopsfix -set_workdir neb://@HOST@.0/gpc1/mops.fixbrokenLAP.20190325 -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_data_group mops.fixbrokenLAP.20190325 -set_reduction LAP_SCIENCE -exp_name o5492g0066o but the dist grp and label are different from before, not sure if it needs to be changed? }}} === Saturday : 2019.04.20 === * CCL: the cooling in the ATRC IT room is broken. so we put ippb06, ippb08, ippb16-23 down. temperature status https://ps1wiki.ifa.hawaii.edu/cgi-bin/ps1opsDashboard.py {{{ ippb07-b16 doesn't seem to work ippb06>> sensors coretemp-isa-0000 Adapter: ISA adapter Physical id 0: +45 C (high = +84 C) coretemp-isa-0001 Adapter: ISA adapter Physical id 1: +46 C (high = +84 C) }}} * CCL: remove update.LAP.PV3 label and put ippb04-05, ippb12-b15 repair to see how it impacts processing in case we need to put them down tomorrow. === Sunday : 2019.04.21 === * CCL: shutdown ippb10-11 in the morning, turn on in the afternoon before obseving. * CCL: remove update.LAP.PV3 label for easily tracking processing behavior at ATRC.