PS1 IPP Czar Logs for the week 2019.04.15 - 2019.04.21

Czar of the week: CCL + MEH (czar-lord), previously JRF

(Up to PS1 IPP Czar Logs)

Monday : 2019.04.15

• CCL: Haydn replaced OS drive on ippb21

  neb-host --state down --host ippb04 --note 'CCL: repair --> down update faults finished'
  neb-host --state repair --host ippb21 --note 'CCL: down->repair OS drive replaced'
  

• MEH: update.% chip cleanup queued overnight again in normal ~ippitc/cleanup -- regularly has to be queued every few days to clean up space while update processing continues in ~ippitc/stdscience and ~ippmops/stdscience
• MEH: ippb21 not exporting disks after reboot -- neb-host down (was an unstable node as well so should be kept down for a while) and will work with czar in morning to fix exports
• CCL: did not remove update.LAP.PV3 label due to bright time for testing processing speed and it looks okay even with massive stamp requests.

Tuesday : 2019.04.16

• CCL: disk warning

  neb-host --state repair --host ipp106.1 -volume --note 'CCL: disk getting full so moving to repair: up->repair'
  

• CCL: turn ippb16-ippb20, ippb22, ippb23 down for Haydn fixing fans.
• CCL: ippb21 (uname -a to see OS --> ubuntu) seems not to export properly so Mark and I tried to figure it out. we first to check "systemctl satus nfs-server", nothing is strange try to restart "systemctl restart nfs-server", but it does not help after comparing with ippb20, ippb22 we found /etc/exports file has not following inform.

  # ATRC IPP Storage offsite
  /export/ippb21.0 166.122.172.0/255.255.255.128(nohide,rw,insecure,sync,no_subtree_check,root_squash)
  /export/ippb21.1 166.122.172.0/255.255.255.128(nohide,rw,insecure,sync,no_subtree_check,root_squash)
  # ITC IPP Prod cluster
  /export/ippb21.0 128.171.123.224/255.255.255.224(nohide,rw,insecure,sync,no_subtree_check,root_squash)
  /export/ippb21.1 128.171.123.224/255.255.255.224(nohide,rw,insecure,sync,no_subtree_check,root_squash)
  

  we add these into /etc/exports and "sudo exportfs -rav", then the disk is exported properly.

Wednesday : 2019.04.17

Thursday : 2019.04.18

Friday : 2019.04.19

Saturday : 2019.04.20

Sunday : 2019.04.21