== PS1 IPP Czar Logs for the week 2019.04.15 - 2019.04.21 == 
[[PageOutline]]

Czar of the week: CCL + MEH (czar-lord), previously JRF

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2019.04.15 ===
 * CCL: Haydn replaced OS drive on ippb21
{{{
neb-host --state down --host ippb04 --note 'CCL: repair --> down update faults finished'
neb-host --state repair --host ippb21 --note 'CCL: down->repair OS drive replaced'
}}}
 * MEH: update.% chip cleanup queued overnight again in normal ~ippitc/cleanup -- regularly has to be queued every few days to clean up space while update processing continues in ~ippitc/stdscience and ~ippmops/stdscience
 * MEH: ippb21 not exporting disks after reboot -- neb-host down (was an unstable node as well so should be kept down for a while) and will work with czar in morning to fix exports
 * CCL: did not remove update.LAP.PV3 label due to bright time for testing processing speed and it looks okay even with massive stamp requests.

=== Tuesday : 2019.04.16 ===
 * CCL: disk warning
{{{
neb-host --state repair --host ipp106.1 -volume --note 'CCL: disk getting full so moving to repair: up->repair'
}}}

 * CCL: turn ippb16-ippb20, ippb22, ippb23 down for Haydn fixing fans.
 * CCL: ippb21 (uname -a to see OS --> ubuntu) seems not to export properly so Mark and I tried to figure it out.
 we first to check "systemctl satus nfs-server", nothing is strange
 try to restart "systemctl restart nfs-server", but it does not help 
 after comparing with ippb20, ippb22 we found  /etc/exports file has not following inform.
{{{
# ATRC IPP Storage offsite
/export/ippb21.0 166.122.172.0/255.255.255.128(nohide,rw,insecure,sync,no_subtree_check,root_squash)
/export/ippb21.1 166.122.172.0/255.255.255.128(nohide,rw,insecure,sync,no_subtree_check,root_squash)
# ITC IPP Prod cluster
/export/ippb21.0 128.171.123.224/255.255.255.224(nohide,rw,insecure,sync,no_subtree_check,root_squash)
/export/ippb21.1 128.171.123.224/255.255.255.224(nohide,rw,insecure,sync,no_subtree_check,root_squash)
}}}
 we add these into /etc/exports and "sudo exportfs -rav", then the disk is exported properly.

* CCL: Rob asks for not restart pstamp server at 3:30pm
{{{
crontab uses "check_system.sh", so modified it, comment the regular one and uncomment the one without pstamp
#set dirs = ( registration stdscience summitcopy pstamp stack cleanup distribution ~ippqub/stdscience_ws ~ippps2/summitcopy ~ippps2/stdscience ~ippps2/registration ~ippps2/distribution )
## no pstamp restart --
set dirs = ( registration stdscience summitcopy stack cleanup distribution ~ippqub/stdscience_ws ~ippps2/summitcopy ~ippps2/stdscience ~ippps2/registration ~ippps2/distribution )
}}}

* CCL: try to manually restart pantasks via check_system.sh, but somehow it did not work so I have to login to each server to restart them.
{{{
==== these are for ippitc
pantasks server registration is NOT running (host: ippc25)
pantasks server stdscience is NOT running (host: ippc30)
pantasks server summitcopy is NOT running (host: ippc26)
pantasks server pstamp is NOT running (host: ippc26)
pantasks server stack is NOT running (host: ippc28)
pantasks server cleanup is NOT running (host: ippc23)
pantasks server distribution is NOT running (host: ippc24)
==== this is for ippqub
pantasks server /home/panstarrs/ippqub/stdscience_ws is NOT running (host: ippc28)
==== these are for ippps2
pantasks server /home/panstarrs/ippps2/summitcopy is NOT running (host: ippc24)
pantasks server /home/panstarrs/ippps2/stdscience is NOT running (host: ippc23)
pantasks server /home/panstarrs/ippps2/registration is NOT running (host: ippc25)
pantasks server /home/panstarrs/ippps2/distribution is NOT running (host: ippc30)
}}}
=== Wednesday : 2019.04.17 ===

=== Thursday : 2019.04.18 ===

=== Friday : 2019.04.19 ===

=== Saturday : 2019.04.20 ===

=== Sunday : 2019.04.21 ===