| Version 12 (modified by , 17 years ago) ( diff ) |
|---|
*Disk reports
*ipp004
- 2008-01-10T06:45:08 IDE Channel 12 Device Failed
- 2008-03-20T00:07:08 IDE Channel 12 Device Failed
- 2008-04-07T11:23:04 IDE Channel 12 Device Removed
- 2008-04-07T11:38:09 IDE Channel 12 Device Failed
- 2008-04-07T15:09:22 IDE Channel 12 Device Failed
- 2008-05-08T06:32:55 IDE Channel 6 Device Failed
- 2008-05-29T05:44:36 IDE Channel 12 Device Removed
- 2008-05-29T11:34:28 IDE Channel 12 Device Removed
- 2008-05-29T14:22:27 IDE Channel 12 Device Removed
- 2008-05-29T14:43:31 IDE Channel 12 Device Failed
- 2008-05-29T14:54:44 IDE Channel 12 Device Removed
- 2008-05-29T15:10:57 IDE Channel 12 Device Failed
- 2008-05-29T19:59:58 IDE Channel 12 Device Failed
- (disk slot 12 has been determined to be broken)
- 2008-08-20T18:20 (approx) network became unresponsive. notified by nagios. Power cycled by gavin
- 2008-08-25T20:00 (approx) system became unresponsive. Power cycled by bills
- 2008-08-30T15:00 (approx) sshd stopped responding. debug notes
- 2008-08-31T22:00 (approx) sshd stopped responding. rebooted 2008-09-01T06:30
- 2008-09-02T18:04:33 network became unresponsive (eth0: too many iterations (6) in nv_nic_irq), notified by nagios. power cycled by gavin
- 2008-09-02T19:00 (approx) ipp004 power cycled due to forcedeth failure
*ipp005
- 2008-07-15T16:20:00 ethernet became unresponsive in manner consistent with forcedeth. problem power cycled by bills
- 2008-07-15T17:15-30 (approx) system became unresponsive. power cycled by gavin
- 2008-07-16T15:00 system became unresponsive power cycled by bills
- 2008-07-17T20:45:32 IDE Channel 9 Device Removed
- 2008-07-17T21:08:59 IDE Channel 9 Device Failed
- 2008-07-17T23:31:08 IDE Channel 9 Device Removed
- 2008-07-28T10:10:08 system became unresponsive. power cycled by gavin
- 2008-07-28T11:02:00 system became unresponsive. (alerted by nagios) power cycled by gavin
- 2008-08-18T17:15:00 system lost network connection. power cycled by EAM
- 2008-08-31T06:00:00 (approx) system became unresponsive, no console log. (alerted by nagios) power cycled by eam
- 2008-09-12T16:55:00 total deadlock, power cycled
- 2008-09-15T07:04 system unresponsive all night. console unresponsive. power cycled by bills
- 2008-09-16T09:19:05 system unresponsive; no msg on console. notified by nagios. power cycled by gavin
- 2008-09-16T16:40 system deadlocked during fsck, power cycled
- 2008-09-30T11:28 kernel BUG in dmesg
- 2008-10-14T06:00 system hung, no response on console, power cycled.
- 2008-10-14T15:46 system hung, no response on console, power cycled.
- 2008-10-15T09:30 system hung, no response on console, power cycled.
- 2008-10-19T10:46 system hung, no response on console, no output power cycled by bills
- 2008-12-11T12:17 system hung, no response on console, power cycled by gavin
*ipp006
- 2008-07-31 early AM system unresponsive power cycled by eugene
- 2008-08-01T21:20 approx system unresponsive. attempt to log into console got yp error power cycled by bills
- 2008-08-18T12:00 kernel x87 math error? : kernel stack dump
- 2008-08-19T11:33 network hung, typical forcedeth.c YP/RPC errors on console
- 2008-08-26T11:00 forcedeth
- 2008-08-03AM network hung, console uresposive assume forcedeth power cycled by bills
- 2008-08-03T11:01 network hung, console uresposive assume forcedeth power cycled by bills
- 2008-09-03T12:06:26 system became unresponsive; error message "do_IRQ: 2.189 No irq handler for vector" power cycled by gavin
- 2008-09-16T07:45? system unresponsive, power cycled by price
- 2008-10-15T18:18 system became unresponsive console frozen last message [1071427.660233] do_IRQ: 1.179 No irq handler for vector. Power cycled by bills
- 2008-10-16T08:18 nagios email messages about total processes and root partition
- 2008-10-26T09:18 Host Down Alert. No response from console. No message. power cycled by bills
- 2008-10-26T17:07 Host Down Alert. ipp006-crash-20081026 power cycled by eugene
- 2008-11-28T09:22 Host Down Alert. error message "do_IRQ: 2.179 No irq handler for vector" power cycled by gavin
- 2008-12-18T16:16 system became unresponsive; error message "do_IRQ: 2.179 No irq handler for vector"; panic; power cycled by gavin
*ipp007
- 2008-07-24T20:50 system became unresponsive (nagios notification) power cycled by gavin
- 2008-07-28 early AM system became unresponsive power cycled by eugene
- 2008-08-10T12:30 (approx) forcedeth? unresponsive, power cycled by eugene
*ipp008
- 2007-03-23T05:39:42 IDE Channel 1 Device Removed
- 2008-08-09T06:38:00 system became unresponsive; error message "do_IRQ: 1.189 No irq handler for vector" (power cycled by eam)
- 2008-08-10T02:29:00 system became unresponsive: error message "do_IRQ: 1.189 No irq handler for vector" (power cycled by eam)
- 2008-08-11T10:53:34 system became unresponsive: nagios error message "Socket timeout after 10 seconds" power cycled by gavin
- 2008-08-11T14:46:55 system became unresponsive: notified by kaiser power cycled by gavin
- 2008-08-11T15:28:28 system became unresponsive: nagios error message "Socket timeout after 10 seconds" power cycled by gavin
- 2008-09-03T07:40 power cycled by bills due to network hang
- 2008-09-05T05:04 system became unresponsive; error message "do_IRQ: 1.189 No irq handler for vector" (many times, also 0.189, 2.189) (power cycled by eam)
- 2008-10-19T12:30 system became unresponsive power cycled
- 2008-10-20T22:00 system became unresponsive power cycled (nothing on console)
- 2008-10-21T12:53:00 system became unresponsive power cycled by gavin (nothing on console)
- 2008-10-21T14:05:00 system became unresponsive power cycled by gavin (nothing on console)
- 2008-10-22T10:35:00 system became unresponsive power cycled by bills (nothing on console)
- 2008-10-22T10:50:00 failed pretty quickly. (warps were running) power cycled by bills
- 2008-10-30T12:22:00 system became unresponsive power cycled by gavin (nothing on console)
- 2008-11-03T11:20 system became unresponsive power cycled by bills (nothing on console) Paul was processing on it.
- 2008-12-13T17:13 system became unresponsive (time is nagios email) power cycled by bills 12-14T09:55 (nothing on console)
- 2008-12-18T08:00 system became unresponsive power cycled by gavin (nothing on console)
- 2008-12-19T14:39:00 crashed, no message on console, power cycled by eam & rodney
- 2008-12-19T18:39 system unresponsive, no message on console, power cycled by gavin
- 2008-12-20T17:33 crashed, no message on console, power cycled by rodney
- 2008-12-21T16:40 crashed, no message on console, power cycled by rodney
- 2009-03-01T14:00 system became unresponsive power cycled (nothing on console)
- 2009-03-03T11:00 system became unresponsive power cycled (nothing on console)
- 2009-03-05T13:40 system became unresponsive power cycled (nothing on console)
*ipp009
- 2008-02-07T17:43:55 IDE Channel 5 Device Failed
- 2008-07-12 time unknown forcedeth problem power cycled by bills
- 2008-07-03T00:00 (approx) forcedeth problem power cycled around 0600 by bills
- 2008-10-30T11:07:00 console hung, spinlock ipp009-crash-20081030
- 2008-11-15T15:22 no response nothing on console power cycled by bills
*ipp010
- 2008-02-08T10:34:33 IDE Channel 12 Device Removed
- 2008-02-08T10:44:12 IDE Channel 12 Device Removed
- 2008-02-08T10:48:01 IDE Channel 12 Device Removed
- 2008-07-21T11:45 system became unresponsive over network power cycled by bills
- 2008-07-31T07:35 system became unresponsive over network power cycled by bills
- 2008-09-10T16:24 unresponsive, power cycled by price
- 2008-10-16T10:05 kernel panic
- 2008-12-11T00:02 unresponsive no messages power cycled by bills around 08:55
- 2009-01-04T00:12:11 IDE Channel 1 Device Failed
*ipp011
- 2007-03-23T05:51:14 IDE Channel 24 Device Failed
- 2007-03-23T05:51:14 IDE Channel 22 Device Removed
- 2007-03-23T05:51:14 IDE Channel 23 Device Removed
- 2007-03-23T05:54:44 IDE Channel 1 Device Removed
- 2008-07-10T10:15:00 - forcedeth NIC failure - RPC errors on console
- 2008-07-15T07:00:00 approx forecdeth problem power cycled by bills
- 2008-08-11T17:23:32 forecdeth problem power cycled by gavin
*ipp012
- 2007-03-22T14:36:28 IDE Channel 15 Device Removed
*ipp013
- 2006-01-01T00:56:09 IDE Channel 22 Device Removed
- 2008-06-23T02:33:49 IDE Channel 11 Device Failed(SMART)
- 2008-07-09T10:41:59 IDE Channel 11 Device Removed
*ipp014
- 2008-07-10T00:42:03 IDE Channel 10 Device Failed(SMART)
- 2008-07-10T02:23:17 IDE Channel 10 Device Removed
- 2008-10-06T05:31 Machine became unresponsive. No message on console. Power cycled by bills 06:07
- 2008-10-16T05:51 kernel panic ipp014_panic_1016
- 2008-10-15T22:30 machine became unresponsive nothing on console power cyled by bills 8:52 10/16
- 2008-10-20T20:49 kernel panic unable to handle kernel paging request ipp014_panic_1020
- 2008-10-22T05:45 machine became unresponsive nothing on console power cycled by bills around 07:20
- 2008-10-25T11:55:39 machine hung (spinlock) ipp014-crash-20081025
- 2008-10-25T11:55:39 machine hung (spinlock) ipp014-crash-20081026
- 2008-10-31T00:00:00 machine hung, no stack trace (a few segfault warnings only)
- 2008-11-05T14:10:00 machine hung (spinlock)
- 2008-11-26T11:50 (approx) console unresponsive; general protection fault on console; power cycled by pap
- 2008-11-26T11:00 (approx) console unresponsive; ipp014-crash-20081127; power cycled by eam
- 2008-12-05 crashed overnight, multiple traces produced; power cycled by pap
- 2008-12-11T01:54 unresponsive no messages power cycled by bills around 08:55
- 2009-03-02T17:00 (approx) console unresponsive; ipp014-crash-20090302; power cycled by eam
- 2009-03-05T18:23 kernel panic ipp014_crash-20090305; power cycled by bills
- 2009-03-11T06:00 kernel panic ipp014_crash-20090311; power cycled by eam
- 2009-03-11T16:00 kernel panic ipp014_crash-20090311b; power cycled by eam
*ipp015
- 2008-07-03T23:37:05 IDE Channel 21 Device Removed
- 2008-07-04T01:27:23 IDE Channel 3 Device Failed
- 2008-07-04T01:40:07 IDE Channel 3 Device Removed
- 2008-07-04T01:43:16 IDE Channel 4 Device Failed
- 2008-07-09T16:31:25 IDE Channel 7 Device Removed
- 2008-03-22T23:47:35 IDE Channel 7 Device Removed
- 2008-03-22T23:57:31 IDE Channel 7 Device Inserted
- 2008-3-22T23:47:35 IDE Channel 7 Device Removed
- 2008-3-22T23:57:31 IDE Channel 7 Device Inserted
- 2008-3-23T12:27:16 ARC-1280-VOL#00 Complete Rebuild 012:29:44
- 2008-07-09 ipp015 disk set moved into a wave #2 chassis (ipp025)
*ipp016
- 2008-02-26T10:35:50 IDE Channel 13 Device Removed
- 2008-09-10T17:48 Looks like forcedeth, rebooted by price
- 2008-09-18T15:22 system became unresponsive; error message "do_IRQ: 2.189 No irq handler for vector" power cycled by gavin
- <Sep/27 03:10 pm>i Kernel panic - not syncing: softlockup: blocked tasks power cycled by bills
- 2008-10-08T15:53 soft lockup
- 2008-10-22T13:28:48 system became unresponsive; error message "do_IRQ: 0.179 No irq handler for vector" power cycled by gavin
- 2008-11-03T12:39:33 system became unresponsive; error message "do_IRQ: 0.179 No irq handler for vector" power cycled by gavin
- 2008-11-03T18:04 system unresponsive while processing stacks; panic on console (looks like spinlock problem); power cycled by pap
- 2008-11-12T17:00 ; power cycled
<Nov/12 02:06 pm>[768330.447278] do_IRQ: 2.187 No irq handler for vector <Nov/12 02:07 pm>[768402.246842] do_IRQ: 0.179 No irq handler for vector
- 2008-12-01T14:29 no response on console following large load (> 20 on Ganglia). Error message: <code>[1629646.647189] do_IRQ: 2.179 No irq handler for vector</code>. Power cycled by pap.
- 2008-12-18T10:48 system became unresponsive; error message "do_IRQ: 2.179 No irq handler for vector"; panic; power cycled by gavin
- 2008-12-18T11:35 system became unresponsive;no message displayed; power cycled by gavin
- 2008-12-19T18:57 system became unresponsive; error message "BUG: spinlock lockup on CPU#2, ppSub/16110, ffffe2000bd42628" ; power cycled by gavin
- 2008-12-20T17:30 crashed, no message on console, power cycled by rodney
- 2008-12-30T17:57:51 crash; no message on console; power cycled by gavin
- 2009-01-12T13:32:31] system became unresponsive; error message "do_IRQ: 2.179 No irq handler for vector" power cycled by gavin
- 2009-01-13T12:24 system became unresponsive No error messages power cycled by bills
- 2009-01-13 system went down two more times. The second time a kernel panic occured due to a null pointer dereference ipp016-crash090113.
- 2009-01-20 kernel oops kernel panic, power cycled.
- 2009-01-23 kernel BUG dead lock, power cycled.
- 2009-01-23T14:17 kernel panic kernel panic, power cycled.
- 2009-01-28T07:07 kernel panic kernel panic, power cycled.
- 2009-01-28T11:21 do_IRQ do_IRQ on console, YP/RPC errors being printed on console, power cycled.
- 2009-01-28T16:28 system unresponsive; error message "do_IRQ: 1.189 No irq handler for vector" power cycled by gavin
- 2009-02-01T19:03 do_IRQ: 2.179 No irq handler for vector BUG: spinlock lockup on CPU#2, ppImage/25380, ffff88022b8dcca8 power cycled by bills 2009-02-02T09:00
- 2009-02-02T16:50 system unresponsive, no console messages. power cycled by eugene
- 2009-02-02T17:36 do_ypcall: clnt_call: RPC: Unable to receive; errno = No route to host, power cycled.
- 2009-02-02T18:00 do_IRQ: 1.189 No irq handler for vector (2x); power cycled
- 2009-03-05T13:50 do_IRQ: 2.179 No irq handler for vector; power cycled by EAM
- 2009-03-12T11:10 no console messages; power cycled by EAM
*ipp017
*ipp018
- 2008-01-09T07:52:30 IDE Channel 23 Device Removed
- 2008-01-09T07:52:59 IDE Channel 3 Device Removed
- 2008-01-09T07:52:59 IDE Channel 4 Device Removed
- 2008-01-09T07:53:39 IDE Channel 3 Device Removed
- 2008-01-09T07:53:42 IDE Channel 4 Device Removed
- 2008-01-09T07:59:26 IDE Channel 23 Device Removed
- 2008-08-07T00:00 (approx) ipp018 network became unresponsive. Power cycled by bills
- 2008-08-13T16:54 (~) classical forcedeth network lockup. Power cycled
- 2008-08-15T06:00 forcedeth : kernel stack dump
- 2008-08-17T21:05 forcedeth : kernel stack dump
- 2008-08-19T21:41 notified by kaiser to reboot, required fsck /dev/sda3 by gavin
- 2008-09-11T1335 kernel oops
- 2008-09-15T11:00 kernel oops; ssh was dead but was able to login and attempt to reboot; reboot hung forever... had to power cycle
- 2008-12-19T14:45 crashed with kernel panic : stack trace ipp018-crash-20081219
*ipp019
- 2008-09-03T22:10:27 found ipp019@ippcon booted from livecd, notified by nagios. unable to boot from raidset
*ipp020
- 2007-06-21T07:02:03 IDE Channel 19 Device Failed
- 2007-06-21T07:02:09 IDE Channel 19 Device Failed
- 2007-06-21T07:02:16 IDE Channel 20 Device Removed
- 2008-07-15:12:00:00 (approx) system unresponsive * power cycled
- 2008-08-20T17:51:57 forcdeth network death
- 2008-09-10T12:50:00 (approx) system unresponsive * power cycled
- 2008-09-16T21:58 kernel panic power cycled by bills around 2008-09-17T07:00
- 2008-09-19T04:49 [172920.752681] Kernel panic - not syncing: softlockup: blocked tasks power cycled by bills at 06:53 failure occured while iqanalysis was starting up processing
*ipp021
- 2008-08-26T17:00 forcedeth hang power cycled by bills
- 2008-09-10T18:48 forcedeth problem power cycled by gavin @ ~19:31
- 2008-11-05T18:49 load went to 11, console hung. power cycled by eam
*ipp023
- 2009-02-20T13:03:49 IDE Channel 6 Reading Error - Reseated 2009-02-20T13:34:50
*ipp025
- 2008-10-21T11:36 power cycled by gavin BUG: soft lockup
- 2008-10-22T12:17 console hung message: [85568.599373] BUG: spinlock lockup on CPU#3, pswarp/14919, ffff88013624a100 power cycled by bills
- 2008-10-22T14:16:36 console messages: BUG: spinlock on CPU#2
- 2008-10-24T16:29:00 console hung: ipp025-crash-2008-10-24
- 2008-10-24T16:29:00 console hung: ipp025-crash-20081026
- 2008-10-26T15:45 kernel panic ipp025-crash-20081026b power cycled by bills
- 2008-10-30T10:59:00 console hung, spinlock: ipp025-crash-20081030
- 2008-11-03T10:07 down (do_IRQ: 0.179 No irq handler for vector) during pap MD processing (ppStack 4 threads), power cycled by pap
- Suspect memory problems
*ipp027
- 2008-09-21T21:00 (approx) kernel oops
- 2008-10-04T09:30 machine non-responsive. Console frozen with no messages. Power cycled by bills.
- 2008-11-15T15:57 no response nothing on console power cycled by bills
- 2008-11-15T22:20 no response nothing on console power cycled by bills
- 2008-11-20T00 (approx) no response on console, power cycled by pap
- 2008-11-26T11:06 no response on console, power cycled by pap; was under heavy load
- 2008-11-27T08:30 no response on console, power cycled by bills; was under heavy load
- 2009-01-23T14:15 no response on console; was under heavy load
- 2009-02-19T08:35:05 IDE Channel 20 Device Failed
- 2009-03-23T10:22:50 IDE Channel 20 Device Failed
*ipp028
- 2008-09-22T16:30 ipp028-oops-2008-09-22
- 2008-09-26T10:40 ipp028-oops-2008-09-26
- 2008-10-23T05:50:09 IDE Channel 21 Device Failed
*ipp030
- 2009-02-20T10:03:49 IDE Channel 17 Reading Error - Reseated 2009-02-20T10:34:16
*ipp033
- 2008-09-22T04:00 (approx) kernel oops
*ipp036
*2008-10-29T06:34:58 IDE Channel 3 Device Failed
- ASA serial<->node name map
<pre> Note all nums start with 190314xx (where "xx" is the two digits).
ipp004: 07 ipp005: 10 ipp006: 12 ipp007: 11 ipp008: 01 ipp009: 13 ipp010: 09 ipp011: 08 ipp012: 02 ipp013: 04 ipp014: 03 ipp016: 05 ipp018: 14 ipp025: 06 ipp037: 00 </pre>
Note:
See TracWiki
for help on using the wiki.
