• 2010-01-14T09:32:37 IDE Channel Device 2 Failed (replaced 2009-01-14T?) Areca Controller did not detect replacement disk, re-seated & replaced new disk. no response. (rebooted IPP026 2009-01-15T17:00) Brad was required to re-seat once again after reboot in order for Areca to detect new disk.
• 2010-01-19T09:42:37 IDE Channel 9 Media/Timeout error (replaced 2010-1-19T09:45:38)
• 2011-4-2 13:36:31 Controller#1(PCI) IDE Channel # 17 Reading Error disc replaced
• 2011-5-13 5:46:38 Controller#1(PCI) IDE Channel #10 Reading Error
• 2011-07-06T17:05:14 system unresponsive, kernel panic, power cycled by gavin
• 2011-07-12T15:00:00 nfs went insane, rebooted by EAM
• 2011-07-13T08:00:00 kernel panic last night, rebooted by EAM
• 2011-07-26T13:50:00 kernel panic, rebooted by EAM ipp026-2011-07-26-panic
• 2011-07-27T12:30 Kernel panic rebooted by Serge

  <Jul/27 10:15 pm>[80552.646321] HARDWARE ERROR
  <Jul/27 10:15 pm>[80552.646321] CPU 1: Machine Check Exception:                5 Bank 5: b200001084200e0f
  <Jul/27 10:15 pm>[80552.646321] RIP !INEXACT! 33:<00007fcf717506c4> 
  <Jul/27 10:15 pm>[80552.646321] TSC ea1511662cd0 
  <Jul/27 10:15 pm>[80552.646321] This is not a software problem!
  <Jul/27 10:15 pm>[80552.646321] Run through mcelog --ascii to decode and contact your hardware vendor
  <Jul/27 10:15 pm>[80552.646321] 
  <Jul/27 10:15 pm>[80552.646321] HARDWARE ERROR
  <Jul/27 10:15 pm>[80552.646321] CPU 7: Machine Check Exception:                4 Bank 5: b200002008000e0f
  <Jul/27 10:15 pm>[80552.646321] TSC ea1511664498 
  <Jul/27 10:15 pm>[80552.646321] This is not a software problem!
  <Jul/27 10:15 pm>[80552.646321] Run through mcelog --ascii to decode and contact your hardware vendor
  <Jul/27 10:15 pm>[80552.646321] Kernel panic - not syncing: Machine check
  <Jul/27 10:15 pm>[80552.646321] ------------[ cut here ]------------
  <Jul/27 10:15 pm>[80552.646321] WARNING: at kernel/smp.c:333 smp_call_function_mask+0x37/0x1d7()
  <Jul/27 10:15 pm>[80552.646321] Modules linked in: coretemp w83627hf w83793 hwmon_vid k8temp autofs4 i2c_i801 iTCO_wdt i2c_core e1000e tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
  <Jul/27 10:15 pm>[80552.646321] Pid: 7776, comm: ppSub Tainted: G   M  D W  2.6.28-rc7-00105-gfeaf384 #4
  <Jul/27 10:15 pm>[80552.646321] Call Trace:
  <Jul/27 10:15 pm>[80552.646321]  <#MC>  [<ffffffff802397db>] warn_on_slowpath+0x51/0x6d
  <Jul/27 10:15 pm>[80552.646321]  [<ffffffff803eec86>] notify_update+0x2b/0x30
  <Jul/27 10:15 pm>[80552.646321]  [<ffffffff8025bc31>] smp_call_function_mask+0x37/0x1d7
  <Jul/27 10:15 pm>[80552.646321]  [<ffffffff80264013>] crash_kexec+0x17/0xef
  
  <Jul/27 10:15 pm>[80552.646321]  [<ffffffff802640e2>] crash_kexec+0xe6/0xef
  <Jul/27 10:15 pm>[80552.646321]  [<ffffffff8021d53e>] native_smp_send_stop+0x1a/0x26
  <Jul/27 10:15 pm>[80552.646321]  [<ffffffff80239896>] panic+0x95/0x13f
  <Jul/27 10:15 pm>[80552.646321]  [<ffffffff80239d5d>] release_console_sem+0x3e/0x1a5
  <Jul/27 10:15 pm>[80552.646321]  [<ffffffff80239d5d>] release_console_sem+0x3e/0x1a5
  <Jul/27 10:15 pm>[80552.646321]  [<ffffffff80239d5d>] release_console_sem+0x3e/0x1a5
  <Jul/27 10:15 pm>[80552.646321]  [<ffffffff805b9e66>] __atomic_notifier_call_chain+0x74/0x83
  <Jul/27 10:15 pm>[80552.646321]  [<ffffffff805b9df2>] __atomic_notifier_call_chain+0x0/0x83
  <Jul/27 10:15 pm>[80552.646321]  [<ffffffff8023969e>] oops_enter+0x9/0x10
  <Jul/27 10:15 pm>[80552.646321]  [<ffffffff80217b55>] mce_log+0x0/0x7f
  <Jul/27 10:15 pm>[80552.646321]  [<ffffffff80217eef>] do_machine_check+0x2d5/0x378
  <Jul/27 10:15 pm>[80552.646321]  [<ffffffff8020d04f>] machine_check+0x7f/0x90
  <Jul/27 10:15 pm>[80552.646321]  <<EOE>> <4>---[ end trace 4eaa2a86a8e2da22 ]---
  

• 2011-08-10T03:25 kernel panic ipp026-crash-20110810, rebooted by eam
• 2011-09-13T15:14 kernel panic ipp026-crash-20110913, same error as the previous two it seems, rebooted by CZW
• 2011-09-15T18:10 kernel panic ipp026-crash-20110915, same error, rebooted by CZW, removed from processing list
• 2011-09-30T12:36 kernel panic, same error, rebooted by SC
• 2012-11-09 New ARECA RAID controller installed and all data remained accessible. After this successful install/test all RAID disks swapped with new 2TB disks. OS installed and all backed up data reloaded.
• 2012-12-09 10:15 many faults, trap invalid opcode and BUG: unable to handle kernel NULL pointer dereference at 0000000000000000. may want a reboot but MEH doesn't like to do on weekend unless absolutely necessary... not a full crash but.. ipp026-crash-20121209