• [4551966.920235] Kernel panic - not syncing: Attempted to kill init!
• 20090814T1227 System hung. Following messages on console Power cycled by bills

       [229478.284767] BUG: spinlock lockup on CPU#1, scsi_eh_5/2771, ffff88083a9b17c0
       [229363.893762] BUG: spinlock lockup on CPU#3, syslog-ng/7829, ffff88083a9b17c0
       [229464.858611] BUG: spinlock lockup on CPU#6, sshd/12753, ffff88083a9b17c0
  
  

• 20090819T2000 appprox Kernel panic - not syncing: Attempted to kill init! power cycled by bills
• 20090825T2040 or so system became unresponsive. Not noticed until 0439 by iqanalysis folks. No response on console at 0641. Power cycled by bills. At the time of the failure summit copy operations were firing up. Also, at the same time cronjob mysql dumps running to back up database on other nodes. Times were changed to start at 4AM.
• 2009-09-08T16:00:37 system became unresponsive ; nothing on console ; 3ware logs have been removed after reboot (unable to to tell if auto verify caused the unresponsiveness. power cycled by gavin.
• 2009-09-12T00:48 system became unresponsive, load extremely high ; nothing on console ; power cycled by gavin
• 2009-09-14T08:06:24 system became unresponsive, load extremely high ; nothing on console ; power cycled by gavin.
• 2009-09-15 12:32:04 HST 2009 3ware controller Firmware upgrade done, node required reboot by cindy
• 2009-09-28T08:12:54 system unresponsive, extremely high load, nothing on console, power cycled by gavin
• 2009-11-04T console msg: [3209490.099538] Kernel panic - not syncing: Attempted to kill init!, ippdb02_crash-20091104, power cycled by gavin (2009-11-12T10:20:36)
• 2009-11-12T04:26:35 system unresponsive, nothing displayed on console and messages, ganglia reported high load​ / memory​ / CPU-wait​, power cycled by gavin
• Sat Nov 14 21:15:49 HST 2009 system unresponsive, nothing displayed on console and messages, power cycled by cindy
• Sun Nov 22 21:28:03 HST 2009 nagios started complaining about problems communicating with ippdb02 Host down alert at 22:07 console shows hardware error See ippdb02_error20091122 System did not come back up after cycling power twice. Left it with power on. Bill Sweeney
• 2009-12-03T03:34:27 high load​, cpu​, memory​, 504 processes, nothing on console ; power cycled by gavin
• 2010-01-27T10:30:00 Motherboard swap - Aberdeen
• 2010-04-22T00:06:25 system became unresponsive, nothing on console, power cycled by gavin (rebooted 2010-04-22T05:59)
• 2010-06-17T08:01:38 system unresponsive, Kernel panic - not syncing: Attempted to kill init!, ippdb02_crash_20100617, power cycled by gavin (rebooted 2010-06-17T09:16)
• 11-1-11 drive replaced
• 2013-02-21 Swapped out all the 300GB (SAS 15K rpm) drives replacing with spare 1TB (SATA 3.0Gbps) drives we had from the upgrade of ipp027 - ipp030 from 1TB to 2TB. Rita/Haydn
• 2013-02-25 updated 3ware 9690SA-8I firmware to latest stable (4.10.00.027), from previous working one (4.10.00.007) due to device time-out and controller reset w/ 1TB SATA HDDs.
• 2013-02-26 Swapped out the 1TB drives that were installed 2-21-13 and replaced with the original 300GB (SAS 15K rmp) drives. The 1TB drives didn't seem to be compatible with the RAID controller. Luckily I was able to replace the disks in the correct location, so didn't have to rebuild the RAID array. Rita
• 2013-02-28 Swapped out the 300GB drives installed on 2/26 and replaced with new 2TB drives. IPP team will test to see if these are compatible with RAID controller.
• 2013-03-07 2TB drives were not compatible with RAID controller - unable to rsync. Meant to swap in the original 300GB drives but mistakenly swapped in the 1TB disks. Will return tomorrow to correct. -Rita
• 2013-03-08 1TB drives swapped with the original 300GB drives. All rebooted fine and IPP to proceed with rsync. -Rita
• 2013-03-27 Rita and Haydn swapped out the original 300GB drives with new 600GB. Configured new RAID array and installed OS. All rebooted fine. Informed IPP team so they could verify.