• 2008-02-26T10:35:50 IDE Channel 13 Device Removed
• 2008-09-10T17:48 Looks like forcedeth, rebooted by price
• 2008-09-18T15:22 system became unresponsive; error message "do_IRQ: 2.189 No irq handler for vector" power cycled by gavin
• <Sep/27 03:10 pm>i Kernel panic - not syncing: softlockup: blocked tasks power cycled by bills
• 2008-10-08T15:53 soft lockup
• 2008-10-22T13:28:48 system became unresponsive; error message "do_IRQ: 0.179 No irq handler for vector" power cycled by gavin
• 2008-11-03T12:39:33 system became unresponsive; error message "do_IRQ: 0.179 No irq handler for vector" power cycled by gavin
• 2008-11-03T18:04 system unresponsive while processing stacks; panic on console (looks like spinlock problem); power cycled by pap
• 2008-11-12T17:00 ; power cycled

<Nov/12 02:06 pm>[768330.447278] do_IRQ: 2.187 No irq handler for vector <Nov/12 02:07 pm>[768402.246842] do_IRQ: 0.179 No irq handler for vector

• 2008-12-01T14:29 no response on console following large load (> 20 on Ganglia). Error message: <code>[1629646.647189] do_IRQ: 2.179 No irq handler for vector</code>. Power cycled by pap.
• 2008-12-18T10:48 system became unresponsive; error message "do_IRQ: 2.179 No irq handler for vector"; panic; power cycled by gavin
• 2008-12-18T11:35 system became unresponsive;no message displayed; power cycled by gavin
• 2008-12-19T18:57 system became unresponsive; error message "BUG: spinlock lockup on CPU#2, ppSub/16110, ffffe2000bd42628" ; power cycled by gavin
• 2008-12-20T17:30 crashed, no message on console, power cycled by rodney
• 2008-12-30T17:57:51 crash; no message on console; power cycled by gavin
• 2009-01-12T13:32:31 system unresponsive; error message "do_IRQ: 2.179 No irq handler for vector" power cycled by gavin
• 2010-05-10T19:45:39 system unresponsive;. console message [ 4519.328798] do_IRQ: 2.189 No irq handler for vector; power cycled by gavin
• 2009-01-13T12:24 system became unresponsive No error messages power cycled by bills
• 2009-01-13 system went down two more times. The second time a kernel panic occured due to a null pointer dereference ipp016-crash090113.
• 2009-01-20 kernel oops kernel panic, power cycled.
• 2009-01-23 kernel BUG dead lock, power cycled.
• 2009-01-23T14:17 kernel panic kernel panic, power cycled.
• 2009-01-28T07:07 kernel panic kernel panic, power cycled.
• 2009-01-28T11:21 do_IRQ do_IRQ on console, YP/RPC errors being printed on console, power cycled.
• 2009-01-28T16:28 system unresponsive; error message "do_IRQ: 1.189 No irq handler for vector" power cycled by gavin
• 2009-02-01T19:03 do_IRQ: 2.179 No irq handler for vector BUG: spinlock lockup on CPU#2, ppImage/25380, ffff88022b8dcca8 power cycled by bills 2009-02-02T09:00
• 2009-02-02T16:50 system unresponsive, no console messages. power cycled by eugene
• 2009-02-02T17:36 do_ypcall: clnt_call: RPC: Unable to receive; errno = No route to host, power cycled.
• 2009-02-02T18:00 do_IRQ: 1.189 No irq handler for vector (2x); power cycled
• 2009-03-05T13:50 do_IRQ: 2.179 No irq handler for vector; power cycled by EAM
• 2009-03-12T11:10 no console messages; power cycled by EAM
• 2009-04-22T11:37 system unable to mount /data/ipp004.0 bill hung console trying so power cycled.
• 2009-05-15T12:08 do_IRQ error; power cycled by PAP
• 2009-06-08T23 do_IRQ: 1.189 No irq handler for vector; power cycled by bills
• 2009-12-08T05:36:06 node down not responding no console or log message power cycle cindy
• 2009-12-29T13:13:51 IDE Channel Device 10 failed (replaced 2009-12-30T16:04:53)
• 2010-05-10T12:46 power cycled by bills
  • <May/10 12:46 pm>ipp016 login: [12451489.477939] do_IRQ: 1.179 No irq handler for vector
  • <May/10 12:48 pm>[12451563.482352] BUG: spinlock lockup on CPU#1, ppSub/16197, ffff880160fec948
  • <May/10 01:27 pm>[12453912.155470] do_IRQ: 0.189 No irq handler for vector
• 2010-05-10T17:24:19 system down, unresponsive from the console, [ 5250.746281] do_IRQ: 2.179 No irq handler for vector, power cycled by gavin
• 2010-05-10T19:45:39 system down, console message [ 4519.328798] do_IRQ: 2.189 No irq handler for vector; power cycled by gavin
• 2010-05-11T14:06:30 system down, do_IRQ: 0.179 No irq handler for vector ; power cycled by gavin
• 2010-05-28T07:44:16 IDE Channel #12 Device Failed (replaced 2010-05-28T09:34:08)
• 2010-05-28T10:00 Memory swap
• 2010-06-26T11:01:14 IDE Channel 18 Device Failed (replaced 2010-06-28T09:28:54)
• 2010-07-27T09:42:38 system unresponsive, ipp016-crash-20100727, power cycled by gavin
• 2010-08-01T15:29:45 IDE Channel 8 Device Failed (replaced 2010-08-02T11:11:46)
• 2010-08-08T04:43:43 IDE Channel 1 Device Failed (replaced 2010-08-09T09:37:40)
• 2010-09-03T10:00 number of processes blew up system became unresponsive nothing on console power cycled by bills
• 2010-09-30T17:00:30 IDE Channel 15 Device Failed (replaced 2010-09-30T18:00:30)
• 2010-10-01T06:32:06 IDE Channel 23 Device Failed (replaced 2010-10-01T16:44:01)
• 2010-10-3T9:52:32 DISK# 2 Device Failed replaced 2010-10-04
• 2010-10-06T23:41:12 system unresponsive, ipp016-crash-20101006, power cycled by gavin
• 2010-10-07T09:58:12 system unresponsive, console messages: [ 8757.412781] do_IRQ: 2.179 No irq handler for vector, power cycled by gavin
• 2010-10-19 6:32:6 Drive # 7 Device Failed replaced rebuilt
• 2010-11-05T06:34:04 IDE Channel 17 Device Failed (replaced 2010-11-05T09:41:21)
• 2010-11-21 Channel # 20 Device Failed replaced rebuilt
• 2010-11-25 4:4:32 Volume Degraded Channel # 9 Device Failed
• 2010-12-15T08:10:02 IDE Channel 5 Device Failed (replaced 2010-12-15T09:18:56)
• 2010-12-21T06:47:28 IDE Channel 14 Device Failed (replaced 2010-12-21T09:25:28)
• 2010-12-31T20:42:51 IDE Channel 16 Device Failed (replaced 2011-01-03T11:27:36)
• 2011-01-08T08:00:00 System down for over 7 hours Ipp016-crash-2011-01-08
• 2011-08-04 10:11:1 Controller # 1(PCI) IDE Channel #16 Reading Error replaced
• 2011-11-10 9:5:9 Controller#1(PCI) IDE Channel #2 Reading Error NOT REPLACED
• 2011-11-16 19:18:42 Controller#1(PCI) IDE Channel #2 Reading Error REPLACED
• 2012-02-16 17:37:26 Controller#1(PCI) IDE Channel #8 Reading Error
• 2012-03-04 13:44:17 Controller#1(PCI) IDE Channel #8 Reading Error REPLACED
• 2012-07-10 17:29:20 Controller#1(PCI) IDE Channel #21 Reading Error REPLACED
• 2012-07-26T17:24:47 IDE Channel 24 Reading Error (replaced 2012-07-27T10:12:30)
• 2012-08-28T15:43:56 IDE Channel 3 Reading Error (replaced 2012-08-28T17:06:44)
• 2012-09-03 12:40 rebooted to clear stalled mounts (see [wiki:PS1_IPP_Czarlog_20120903
• 2012-09-11 04:47:15 Controller#1(PCI) IDE Channel #15 Reading Error (replaced: 2012-9-11 9:22:29 Controller# 1(PCI) IDE Channel # 15 Device Inserted)
• 2012-10-01 20:30 power cycled after not recovering mounts (see czar log PS1_IPP_Czarlog_20121001
• 2012-10-06 07:32:20 Controller#1(PCI) IDE Channel #19 Reading Error (replaced: 2012-10-06 08:48:45)
• Note: The time reported by the Areca RAID controller is off by ~3.5 hours! 08:48:55 vs. 12:19:16
• 2012-10-16 07:55 Bills: ipp016 had a panic this morning just power cycled it. (from czar log PS1_IPP_Czarlog_20121015)

  <Oct/15 06:09 pm>ipp016 login: [1200542.035287] BUG: unable to handle kernel paging request at ffffffff800073d4
  <Oct/15 06:09 pm>[1200542.036617] IP: [<ffffffff80589612>] xprt_autoclose+0x19/0x4c
  <Oct/15 06:09 pm>[1200542.036617] PGD 203067 PUD 207063 PMD 0 
  <Oct/15 06:09 pm>[1200542.036617] Oops: 0000 [#1] SMP 
  <Oct/15 06:09 pm>[1200542.036617] last sysfs file: /sys/class/i2c-adapter/i2c-0/0-002f/temp6_alarm
  <Oct/15 06:09 pm>[1200542.036617] CPU 0 
  

• 2012-10-29 22:30 GPF and couldn't ssh into, power cycled Ipp016-crash-20121029
• 2012-12-02 08:55 powercycled

  BUG: unable to handle kernel paging request at ffffffff800073d4
  <Dec/02 05:15 am>[1457104.714295] IP: [<ffffffff80589612>] xprt_autoclose+0x19/0x4c
  <Dec/02 05:15 am>[1457104.714295] PGD 203067 PUD 207063 PMD 0 
  <Dec/02 05:15 am>[1457104.714295] Oops: 0000 [#1] SMP 
  
  <Dec/02 05:15 am>[1457104.714295] last sysfs file: /sys/class/i2c-adapter/i2c-0/0-002f/temp6_alarm
  

• 2013-01-24 ~04:00 panic'd with the BUG: unable to handle kernel paging request signature. Power cycled by bills
• 2013-10-06 ~02:00 unresponsive after heavy lockd problems, power cycled okay