• 2008-07-10T00:42:03 IDE Channel 10 Device Failed(SMART)
• 2008-07-10T02:23:17 IDE Channel 10 Device Removed
• 2008-10-06T05:31 Machine became unresponsive. No message on console. Power cycled by bills 06:07
• 2008-10-16T05:51 kernel panic ipp014_panic_1016
• 2008-10-15T22:30 machine became unresponsive nothing on console power cyled by bills 8:52 10/16
• 2008-10-20T20:49 kernel panic unable to handle kernel paging request ipp014_panic_1020
• 2008-10-22T05:45 machine became unresponsive nothing on console power cycled by bills around 07:20
• 2008-10-25T11:55:39 machine hung (spinlock) ipp014-crash-20081025
• 2008-10-25T11:55:39 machine hung (spinlock) ipp014-crash-20081026
• 2008-10-31T00:00:00 machine hung, no stack trace (a few segfault warnings only)
• 2008-11-05T14:10:00 machine hung (spinlock)
• 2008-11-26T11:50 (approx) console unresponsive; general protection fault on console; power cycled by pap
• 2008-11-26T11:00 (approx) console unresponsive; ipp014-crash-20081127; power cycled by eam
• 2008-12-05 crashed overnight, multiple traces produced; power cycled by pap
• 2008-12-11T01:54 unresponsive no messages power cycled by bills around 08:55
• 2009-03-02T17:00 (approx) console unresponsive; ipp014-crash-20090302; power cycled by eam
• 2009-03-05T18:23 kernel panic ipp014_crash-20090305; power cycled by bills
• 2009-03-11T06:00 kernel panic ipp014_crash-20090311; power cycled by eam
• 2009-03-11T16:00 kernel panic ipp014_crash-20090311b; power cycled by eam
• 2009-04-07T10:29 kernel panic ipp014_crash-20090407; power cycled by bills
• 2009-08-25T07:54 kernel panic ipp014_crash-20090825; power cycled by bills
• 2009-08-25T13:42 root reported down, unresponsive, no messages, power cycled by cindy
• 2009-09-15T11:03:23 kernel panic ipp014_crash-20090915; power cycled gavin
• 2009-09-25T23:14 kernel panic general protection fault: 0000 #1 SMP ipp_crash-20090925 power cycled by bills 2009009-26T04:12
• Sept 26 07:31:24 HST 2009 host down not booting after repower
• 2009-10-06T04:43:12 kernel panic ipp014_crash-20091006; power cycled gavin
• 2009-10-12T15:35:24 kernel panic ipp014_crash-20091012; power cycled gavin
• 2009-10-16 ipp014 remains down since Mon. 2009-10-12. Cycled power, reseated power & serial cables, attached vga monitor/keyboard no response. Steph to go down to MHPCC on Fri. 2009-10-16 to investigate. Plan to export ipp014 raid set to ipp037 chassis (last resort).
• 2009-10-19T10:38:11 IDE Channel 16 Device Degraded (replace 2009-10-19T18:00:15)
• 2009-10-22T18:15:10 high load (134.20), unresponsive, no messages power cycled by gavin (2009-10-22T18:28)
• 2009-10-29T09:46 unresponsive, <Oct/28 06:56 pm>ipp014 login: [527978.637814] journal commit I/O error ; power cycled by gavin
• 2009-11-03T18:44 unresponsive, <Nov/03 03:38 pm>ipp014 login: [461020.846420] journal commit I/O error ; power cycled by gavin
• 2009-11-03T18:51:13 IDE Channel 16 Device Failed (replace 2009-11-04T17:09:23)
• 2009-11-06T09:25 unresponsive, console displays: INIT: Id "s0" respawning too fast: disabled for 5 minutes, Areca web interface unresponsive, /var/log/messages: Nov 6 03:29:04 ipp014 archttp64 WARNING ArecaRaidCtrl Event(WARNING) 2009-11-6 3:29:6 Controller#1(PCI) IDE Channel 16 Reading Error. power cycled by gavin
• 2009-11-06T03:29:04 IDE Channel 16 Reading Error, requested to replace again 2009-11-06T09:42. (replaced 2009-11-06T10:49:31)
• 2009-11-12T12:23 IDE Channel 16 Reading Error (2009-11-13T09:42:33)
• 2009-11-17T13:35:51 ipp014_io_error , resuming scheduled array verification, power cycled by gavin
• 2009-11-18T09:03 Receiving Input/Output error, power cycled by cindy
• 2009-11-24T17:05 I/O error ; sd 6:0:0:0: rejecting I/O to offline device ipp014_crash-20091124, power cycled by gavin
• 2009-11-25T10:50:15 IDE Channel 16 Media Error (replaced 2009-11-25T11:16:28)
• 2010-01-06T09:14 [3449365.470724] Kernel panic - not syncing: Attempted to kill init!
• 2010-02-16T16:04:43 IDE Channel 16 Media Error (replaced 2010-02-16T)
• 2010-02-23T19:48:32 IDE Channel 16 Device Failed (replaced 2010-02-24T13:08:58)
• 2010-02-28T11:20 HST kernel panic ipp014_crash-20100228, power cycled by eam
• 2010-07-16T13:00:05 IDE Channel 9 Device Failed (replaced 2010-07-16T15:03:00)
• 2010-07-25T10:22:38 IDE Channel 13 Device Failed (replaced 2010-07-26T08:55:08)
• 2010-08-15Sun Aug 15 09:05:57 HST 2010 root reported down, unresponsive, no messages, power cycled by cindy
• 2010-08-17T11:04:07 system unresponsive, no messages on console, power cycled by gavin
• 2010-08-19T11:10 or so system unresponsive, no messages on console, power cycled by bills
• 2010-08-27T10:02:07 system unresponsive, console: [114650.520531] journal commit I/O error, power cycled by gavin
• 2010-09-04T08:12 [199375.501727] Kernel panic - not syncing: Attempted to kill init! power cycled by bills previous error on console was <Sep/04 05:34 am> [198304.533766] journal commit I/O error
• 2010-9-13 0:21:21 Channel # 3 Device Failed replaced
• 2010-9-13 replaced cpu's and cpu fan.
• 2010-11-3 16:47:3 Channel # 4 Device Failed replaced and rebuilt
• 2010-11-7 8:38:0 IDE Channel # 19 Device Failed replaced rebuilt
• 2010-11-19 2:1:53 Channel # 14 Device Failed replaced rebuilt
• 2010-12-4 15:9:28 Channel # 15 Device Failed
• 2011-01-09T23:51:6 IDE Channel 23 Device Failed (replaced 2011-01-10T09:12:22)
• 2011-03-15T14:05:03 system unresponsive, [4225756.582647] sd 5:0:0:0: rejecting I/O to offline device, Kernel panic - not syncing: Attempted to kill init! ; power cycled by gavin
• 2011-04-15T11:08:00 system unresponsive, [4308988.009862] Kernel panic - not syncing: Attempted to kill init! ; power cycled by czw
• 2011-06-13T15:42:37 system unresponsive, [3472678.330042] Kernel panic - not syncing: Attempted to kill init! ; power cycled by gavin
• 2011-07-27T10:17:00 system unresponsive, kernel panic journal commit I/O error; power cycled by eam
• 2011-08-09T22:06:00 kernel panic, journal commit I/O error, power cycled by eam
• 2011-8-16 21:27:28 Controller#1(PCI) IDE Channel #16 Reading Error replaced
• 2011-8-22 13:2:42 Controller#1(PCI) IDE Channel #16 Reading Error disk replaced
• 2011-10-4 4:23:41 Controller#1(PCI) IDE Channel #20 Reading Error
• 2011-10-20 9:17:14 Controller#1(PCI) IDE Channel #20 Reading Error
• 2011-11-18 18:45:29 Controller # 1(PCI) IDE Channel # 20 Reading Error
• 2012-02012 system unresponsive to logins ipp014-2012-02-12-crash
• 2012-1-1 11:8:17 Controller#1(PCI) IDE Channel #20 Reading Error REPLACED 2/23/12
• 2012-03-07 09:58 Kernel panic'd: BUG: unable to handle kernel paging request at ffffffff800073d4 more info here ipp014-2012-03-07-crash
• 2012-03-19 13:00 system unresponsive (some communications, but cannot ssh in). crash log here ipp014-2012-03-19-crash
• 2012-10-09 18:45:29 Controller # 1(PCI) IDE Channel # 15 Reading Error - disk replaced and rebuild started
• 2012-11-21 12:46 kernel panic

  <Nov/21 12:46 pm>[73383.009487] BUG: unable to handle kernel paging request at ffffffff800073d4
  <Nov/21 12:46 pm>[73383.012667] IP: [<ffffffff80589612>] xprt_autoclose+0x19/0x4c
  <Nov/21 12:46 pm>[73383.012667] PGD 203067 PUD 207063 PMD 0 
  <Nov/21 12:46 pm>[73383.026527] Oops: 0000 [#1] SMP 
  <Nov/21 12:46 pm>[73383.026527] last sysfs file: /sys/class/i2c-adapter/i2c-0/0-002f/temp6_alarm
  <Nov/21 12:46 pm>[73383.033201] CPU 0 
  

• 2013-02-16 08:00 kernel panic around 03:30, power cycled ipp014_crash-20130216