== PS1 IPP Czar Logs for the week 2011-12-12 - 2011-12-18 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2011-12-12 ===
 * No observation last night
 * 11:37 Restarted pstamp
 * 11:55 Something weird happened to ippc18. All connections have been closed. I'm investigating.
 * 13:45 I quote Gavin: "In the network switch logs, I see that the uplink that provides network connectivity to the cluster switch went down and immediately came back up. I don't have any info why it went down. I'm wondering if this has anything to do with Hawaiian Telcom network outage yesterday in Kihei."
 * 13:50 14 diffs are repeatedly failing because of files on ipp064. I diff.revert.off'd them. We'll talk about them during the IPP meeting.
 * 13:55 Same for the 118 destreak failures.
 * 14:28 I quote Gavin again (see 13:45 entry) "It appears that our switch (ippcore) is reporting that it was a neighboring UH ITS switch that was flapping."
 
=== Tuesday : 2011-12-13 ===
 * No observation last night
 * 09:35: {{{neb-host ippb02 down}}} (Gavin's request: ethernet bonding deployment)
 * 09:49: We (HF & SC) turned off dist.revert.  There are a lot of dist failures (related to ippb02?).  Anyways, there are a lot of dist failures that keep getting reverted, so we turned it off for now.
 * 09:56: {{{neb-host ippb02 up}}}
 * 10:40: Bill shut distribution down while debugging/fixing the nomagic distribution.
=== Wednesday : 2011-12-14 ===
No observation last night. Distribution and replication were restarted.
 * 09:06 Bill: removed all stages from distribution except for stack and raw. This is to allow the M31 raw data to run quicker. 
 * 15:38 Bill: restarted distribution with the nominal set up
=== Thursday : 2011-12-15 ===

 * 9:30: CZW: nebulous seemed to be having problems. Restarted apache on the nebulous apache servers.  It looked like this might be caused by issues with ipp013/nfs, which has since been rebooted.
=== Friday : 2011-12-16 ===

=== Saturday : 2011-12-17 ===

=== Sunday : 2011-12-18 ===