== PS1 IPP Czar Logs for the week 2017-02-06 - 2017-02-13 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2017-02-06 ===
 * 13:28 CZW: Restarted ipptest/replication due to laggy pantasks server.  Set npend at 400, which hopefully won't overload any particular host.  300 seemed to be working fine over the weekend.  
 * 17:15 CZW: Started script running on stare04 to replicate a copy of the detrend data to the ITC cluster (randomly selected host/volume) and to the new ATRC B nodes (randomly selected from ippb07-ippb15).  This is running single threaded, and is not pushing very hard, but the number of files included in this transfer is fairly small (34 individual det_id values, ~2000 total files) so this shouldn't be an issue.
 * 19:00 EAM : PS1 is down for mirror cleaning

=== Tuesday : 2017.02.07 ===

 * 21:00 EAM : PS1 is down for dome (and weather)
 * 21:00 EAM : I rsynced both the gpc1 and nebulous mysqls from the ipp116 & ipp117 replicants to ipp114 & ipp115 today.  gpc1 (ipp114) finished in the afternoon, so I restarted ipp116 & started mysql for ipp114 with replication running on both.  nebulous just finished.  I have restarted ipp117 and started replication there, but I need to build the 5.6 version of mysql before starting ipp115.

=== Wednesday : 2017.02.08 ===
 * MEH: ipp001 disk full per the regular email warning -- running the gpc1 db dump cleanup script
 * MEH: ipp089 overloaded from excessive remote_md5sum.pl jobs since ~1700 yesterday -- if expected, then probably needed to take out of processing and put neb-host repair to isolate and notice sent to group, if not expected then should be check in on and not fireoff and forget 
  * after ~hour of no response, have stopped ~ipptest/replication in order to get other work done
 * MEH: a possible source of free disk space leaking may be due to error_cleaned on the large numbers of LAP.PV3 warp updates recently -- ~few TB freed up, but many still fail due to missing .mdc files leaving large .fits pixel files still on disk... 
 * MEH: ippc19/home disk warning @97% -- archive pantasks logs as described for the czars on [wiki:Processing] page -- ~7GB freed
 * MEH: Haydn notes the ipp056, 058 non-optimal config emails are not correct and have been just ignored for a while now -- put ipp056,058 neb-host up
{{{
Controller ID: 0 Reminder: Potential non-optimal configuration due, PD commissioned as Emergency spare: --:--:16
Generated on:Wed Feb  8 11:31:59 2017
}}}
 * MEH: ipp121 seems ok past few days and in use today -- neb-host up 


=== Thursday : 2017.02.09 ===
 * MEH: MOPS test chunk updates running today in normal stdscience
=== Friday : YYYY.MM.DD ===
 * 02:20 MEH: ipp121 non-responsive -- another kernel panic -- power cycle -- back to neb-host repair since not reliable (was not back in normal processing)
=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===