== PS1 IPP Czar Logs for the week 2019.01.28 - 2019.02.03 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2019.01.28 ===
 * MEH: manually sent old update chips to cleanup to free up space where possible since some nodes were going red -- also included all WSdiffs up through past few days and WWdiffs only through end of 2018 since the WWdiff update bug still exists
 * TdB: IPP backup machines ippb16,ippb17,ipp19 taken down for Haydn to convert to Ubuntu
=== Tuesday : 2019.01.29 ===
 * TdB: Summitcopy for gpc2 has been updated to include compression.

=== Wednesday : 2019.01.30 ===
 * TdB: IPP backup machines ippb16,ippb17,ipp19 set back to up and SSH keys reset for users ippitc,ipp,ipps2

=== Thursday : 2019.01.31 ===
 * TdB: IPP backup machines ippb16-20 had automount problems. Gavin fixed those, and machines are set back to up.

=== Friday : 2019.02.01 ===
 * TdB: IPP processing failed due to an issue related to the permissions set on the recently updated backup machines ippb16-20. Owner of nebulous disks was set to "ipp users" instead of "apache nebulous". Machines set to repair while the issue is being fixed and processing has now continued.
 * TdB: ippb16-20 permissions for nebulous drives fixed to apache, and Gene is running chown on all files on these machines. Will be set back to up on monday, due to Friday being a bad day for system-wise changes.
 * TdB: machine ipp70 removed from /home/panstarrs/ipp/local/bin/apachedisk_chk.sh since it is down (and had already been removed from nebulous), but still spawning warning emails every 4 hours.
 * MEH: something caused >250k warps in various update states saved for MOPS and QUB (ps_ud_MOPS%/QUB%) to get send to cleanup what looks to be ~noon, but didn't notice until nagios sent load warning for ippdb09 ~1523 -- all warps are being retained by default, only broken or test ones should be cleaned
  * stop cleanup pantasks and manually try to salvage remaining possible warp_id set state update and label update_recovery for all state goto_cleaned with data_state full

=== Saturday : 2019.02.02 ===
 * MEH: salvage of warps finished, cleanup back to run
=== Sunday : YYYY.MM.DD ===