== PS1 IPP Czar Logs for the week 2017-03-20 - 2017-03-26 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : YYYY.MM.DD ===
  
=== Tuesday : 2017-03-21 ===
 * 16:18 CZW: Setting ipp004-ipp066 to down to check for problems prior to their shutdown next week.
 * 16:20 CZW: Restarting ippitc pantasks servers.
 * 21:45 EAM: summit copy does not work at night with the proxy; returned to conductor for the rest of the night.
 * 22:50 EAM: khcorrect astrom detrend was not on any non-repair node : turned ipp104 to repair and grabbed one there; replicated it 5x at ITC
 
=== Wednesday : 2017.03.22 ===
 * MEH: special PSNSC/QUB test processing using ITC nodes with data to ipp105,117 only in the afternoon
 * 16:45 CZW: Restarted ippitc pantasks.
=== Thursday : 2017.03.23 ===
 * MEH: ippc106 was out of processing since logins broken -- works now but using for QUB processing
=== Friday : YYYY.MM.DD ===

=== Saturday : 2017.03.25 ===
 * MEH: %.20170322 cleanup was missed due to stdscience being down during initday -- manually sending to cleanup
  * ipp100-104.0 nodes also filing up faster -- w/ ipp106-116 most full, re-target data to the other datanodes up at the ITC now
 * MEH: sent some broken IASC/MOPS stamp request to cleanup since not needed
=== Sunday : 2017.03.26 ===
 * MEH: ippMonitor fixes..
  * ipp115 ippMonitor user not have REPLICATION CLIENT grant -- added and now can see ipp115 neb replication status, hack out other repl_ db in  /var/www/localhost/htdocs/ippMonitor/site.php to remove the connection stall until those are back up
  * fault logs broken -- ~ippitc/ippconfig/ippmonitor.config was pointing to ippc64.0 not ippc64.1... ipp_filename.pl hardcoded uses nebserver.ipp.ifa.hawaii.edu... locally hacked to use ippc70...