• 2008-07-15T16:20:00 ethernet became unresponsive in manner consistent with forcedeth. problem power cycled by bills
• 2008-07-15T17:15-30 (approx) system became unresponsive. power cycled by gavin
• 2008-07-16T15:00 system became unresponsive power cycled by bills
• 2008-07-17T20:45:32 IDE Channel 9 Device Removed
• 2008-07-17T21:08:59 IDE Channel 9 Device Failed
• 2008-07-17T23:31:08 IDE Channel 9 Device Removed
• 2008-07-28T10:10:08 system became unresponsive. power cycled by gavin
• 2008-07-28T11:02:00 system became unresponsive. (alerted by nagios) power cycled by gavin
• 2008-08-18T17:15:00 system lost network connection. power cycled by EAM
• 2008-08-31T06:00:00 (approx) system became unresponsive, no console log. (alerted by nagios) power cycled by eam
• 2008-09-12T16:55:00 total deadlock, power cycled
• 2008-09-15T07:04 system unresponsive all night. console unresponsive. power cycled by bills
• 2008-09-16T09:19:05 system unresponsive; no msg on console. notified by nagios. power cycled by gavin
• 2008-09-16T16:40 system deadlocked during fsck, power cycled
• 2008-09-30T11:28 kernel BUG in dmesg
• 2008-10-14T06:00 system hung, no response on console, power cycled.
• 2008-10-14T15:46 system hung, no response on console, power cycled.
• 2008-10-15T09:30 system hung, no response on console, power cycled.
• 2008-10-19T10:46 system hung, no response on console, no output power cycled by bills
• 2008-12-11T12:17 system hung, no response on console, power cycled by gavin
• 2009-04-16T14:08 system hung, no response on console, power cycled by bills
• 2009-05-21T06:50 system hung, no response on console, power cycled by bills
• 2009-07-23T08:40 system hung, no response on console, power cycled by bills
• 2009-07-28T16:39:54 system hung, no response on console, power cycled by gavin
• 2009-07-31T?? system hung, no response on console, power cycled by bills
• 2009-08-03 03:13:34 system hung, no response on console, power cycled by cindy
• 2009-08-04T12:30 system hung, no response on console, power cycled by bills
• 2009-08-17T16:33:57 system hung, no response on console, power cycled by gavin
• 2009-08-28T09:41:15 system hung, no response on console, power cycled by gavin
• 2009-08-28T11:33 IDE Channel 13 Device replaced
• 2009-09-03T09:40:51 system hung, no response on console, power cycled by gavin
• 2009-09-06T14:28 system became unresponsive power cycled by bills
• 2009-09-10 17:03:47 system hung, no response on console, power cycled by cindy
• 2009-09-11T16:52:13 system hung, no response on console, power cycled by gavin
• 2009-09-16 13:44:04 HST system hung, no response on console, power cycled by cindy. ganglia shows high memory usage
• 2009-09-17T11:26:34 system hung, no response on console, memory & CPU spike before crash ; power cycled by gavin
• 2009-09-21T19:46:40 system hung, no response on console, ganglia reported high memory usage before crash ; power cycled by gavin
• 2009-09-22T09:35:20 IDE Channel 3 Device replaced
• 2009-09-23T16:26:51 system hung, no response on console, ganglia memory usage rising before crash ; power cycled by gavin
• 2009-09-30T1948 system hung, no response on console, ganglia memory usage rising before crash ; power cycled by bills at 0405
• 2009-10-30T11:00 replaced memory modules w/ 4 x 2GB DIMMS.
• 2009-10-30T14:29:15 IDE Channel 16 Deviced failed (replaced 2009-10-30T16:26:16)
• 2009-11-02T09:12 system I/O error msg. RAID/volume set in rebuild state from 10/30/2009 (95.5%)
• 2009-11-05T12:28:04 system crash, no response on console, cpu​, doesn't appear to be memory​ issue ; power cycled by gavin
• 2009-11-10T13:00:04 unresponsive, nothing on console, power cycled by gavin
• 2009-11-10T15:34:54 unresponsive, nothing on console, slight network​ / memory​ change, power cycled by gavin
• 2009-11-12T17:24:46 unresponsive, nothing on console, ganglia memory​, /var/log/messages: Nov 12 15:14:49 ipp005 [169533.514255] ppStack[32254]: segfault at 11 ip 000000000041179c sp 00007fff14990a40 error 4 in ppStack[400000+17000] ; power cycled by gavin
• 2009-11-17T01:34:43 unresponsive, nothing on console, power cycled by gavin (2009-11-17T10:00)
• 2009-11-17T11:09:53 unresponsive, nothing on console, power cycled by gavin
• 2009-11-17T18:01:22 unresponsive, nothing on console, power cycled by gavin
• 2009-11-19T18:40:24 unresponsive, nothing on console, power cycled by gavin
• 2009-11-20T23:20 unresponsive, nothing on console, power cycled by blls at 2009-11-21T730
• Nov 23 21:59:36 HST 2009 unresponsive, nothing on console, power cycled by cindy at Nov 24 08:12:46 HST 2009
• Dec 2 20:51:27 HST 2009 unresponsive, nothing on console or logs, power cycled by cindy
• 2009-12-04T06:53:35 unresponsive, nothing on console, power cycled by gavin (2009-12-04T08:40)
• 2009-12-08T09:31 unresponsive, [281585.924355] journal commit I/O error [348308.075629] Kernel panic - not syncing: Attempted to kill init! ; power cycled by gavin
• 2009-12-09T14:45:14 system unresponsive, no response on console, power cycled by gavin
• 2009-12-12T22:04:11 system unresponsive, no response on console, power cycled by gavin (2009-12-13T11:09)
• 2009-12-14T19:40:52 system unresponsive, no response on console, power cycled by gavin (2009-12-15T06:44)
• 2009-12-15T14:20:52 system unresponsive, no response on console, ganglia memory​/load​, power cycled by gavin
• 2009-12-15T14:58:12 system unresponsive, no response on console, ganglia load​, power cycled by gavin
• 2009-12-22T21:31:16 system unresponsive, nothing on console, ganglia load​, power cycled by gavin (2009-12-23T10:19)
• 2009-12-29T11:38:33 system unresponsive, nothing on console, power cycled by gavin (2009-12-29T12:15)