PS1 IPP Czar Logs for the week 2019.04.15 - 2019.04.21

Czar of the week: CCL + MEH (czar-lord), previously JRF

(Up to PS1 IPP Czar Logs)

Monday : 2019.04.15

• CCL: Haydn replaced OS drive on ippb21

  neb-host --state down --host ippb04 --note 'CCL: repair --> down update faults finished'
  neb-host --state repair --host ippb21 --note 'CCL: down->repair OS drive replaced'
  

• MEH: update.% chip cleanup queued overnight again in normal ~ippitc/cleanup -- regularly has to be queued every few days to clean up space while update processing continues in ~ippitc/stdscience and ~ippmops/stdscience
• MEH: ippb21 not exporting disks after reboot -- neb-host down (was an unstable node as well so should be kept down for a while) and will work with czar in morning to fix exports
• CCL: did not remove update.LAP.PV3 label due to bright time for testing processing speed and it looks okay even with massive stamp requests.

Tuesday : 2019.04.16

• CCL: disk warning

  neb-host --state repair --host ipp106.1 -volume --note 'CCL: disk getting full so moving to repair: up->repair'
  

• CCL: turn ippb16-ippb20, ippb22, ippb23 down for Haydn fixing fans.
• CCL: ippb21 (uname -a to see OS --> ubuntu) seems not to export properly so Mark and I tried to figure it out. we first to check "systemctl satus nfs-server", nothing is strange try to restart "systemctl restart nfs-server", but it does not help after comparing with ippb20, ippb22 we found /etc/exports file has not following inform.

  # ATRC IPP Storage offsite
  /export/ippb21.0 166.122.172.0/255.255.255.128(nohide,rw,insecure,sync,no_subtree_check,root_squash)
  /export/ippb21.1 166.122.172.0/255.255.255.128(nohide,rw,insecure,sync,no_subtree_check,root_squash)
  # ITC IPP Prod cluster
  /export/ippb21.0 128.171.123.224/255.255.255.224(nohide,rw,insecure,sync,no_subtree_check,root_squash)
  /export/ippb21.1 128.171.123.224/255.255.255.224(nohide,rw,insecure,sync,no_subtree_check,root_squash)
  

  we add these into /etc/exports and "sudo exportfs -rav", then the disk is exported properly.

• CCL: Rob asks for not restart pstamp server at 3:30pm

  crontab uses "check_system.sh", so modified it, comment the regular one and uncomment the one without pstamp
  #set dirs = ( registration stdscience summitcopy pstamp stack cleanup distribution ~ippqub/stdscience_ws ~ippps2/summitcopy ~ippps2/stdscience ~ippps2/registration ~ippps2/distribution )
  ## no pstamp restart --
  set dirs = ( registration stdscience summitcopy stack cleanup distribution ~ippqub/stdscience_ws ~ippps2/summitcopy ~ippps2/stdscience ~ippps2/registration ~ippps2/distribution )
  

*CCL: try to manually restart pantasks via check_system.sh, but somehow it did not work so I have to login to each server to restart them.

==== these are for ippitc
pantasks server registration is NOT running (host: ippc25)
pantasks server stdscience is NOT running (host: ippc30)
pantasks server summitcopy is NOT running (host: ippc26)
pantasks server pstamp is NOT running (host: ippc26)
pantasks server stack is NOT running (host: ippc28)
pantasks server cleanup is NOT running (host: ippc23)
pantasks server distribution is NOT running (host: ippc24)
==== this is for ippqub
pantasks server /home/panstarrs/ippqub/stdscience_ws is NOT running (host: ippc28)
==== these are for ippps2
pantasks server /home/panstarrs/ippps2/summitcopy is NOT running (host: ippc24)
pantasks server /home/panstarrs/ippps2/stdscience is NOT running (host: ippc23)
pantasks server /home/panstarrs/ippps2/registration is NOT running (host: ippc25)
pantasks server /home/panstarrs/ippps2/distribution is NOT running (host: ippc30)

Wednesday : 2019.04.17

Thursday : 2019.04.18

Friday : 2019.04.19

Saturday : 2019.04.20

Sunday : 2019.04.21