Checked ganglia this morning. Lots of failed jobs. ipp005 high load. Ganglia shows that started around 11pm last night. Here are the contents of the console.

Decided to cycle power. Prior to doing that I entered sync command. That command just hung. Cycled power.

<Oct/20 08:42 am>This is ipp005.ifa.hawaii.edu (Linux x86_64 2.6.28-rc7-00105-gfeaf384) 11:02:14

<Oct/29 06:08 am>ipp005 login: [768588.516988] 
<Oct/29 06:08 am>[768588.516991] =========================
<Oct/29 06:08 am>[768588.520874] [ BUG: held lock freed! ]
<Oct/29 06:08 am>[768588.520874] -------------------------
<Oct/29 06:08 am>[768588.520874] nfsd/7639 is freeing memory ffff8801a5d9c000-ffff8801a5d9c2b7, with a lock still held there!
<Oct/29 06:08 am>[768588.520874]  (&(&ip->i_lock)->mr_lock){....}, at: [<ffffffffa00c1bbb>] xfs_iget_core+0x3de/0x5ae [xfs]

<Oct/29 06:08 am>[768588.520874] 2 locks held by nfsd/7639:
<Oct/29 06:08 am>[768588.520874]  #0:  (hash_sem){....}, at: [<ffffffff8034f2fd>] nfsd+0x1b7/0x296
<Oct/29 06:08 am>[768588.520874]  #1:  (&(&ip->i_lock)->mr_lock){....}, at: [<ffffffffa00c1bbb>] xfs_iget_core+0x3de/0x5ae [xfs]
<Oct/29 06:08 am>[768588.520874] 
<Oct/29 06:08 am>[768588.520874] stack backtrace:
<Oct/29 06:08 am>[768588.520874] Pid: 7639, comm: nfsd Not tainted 2.6.28-rc7-00105-gfeaf384 #4
<Oct/29 06:08 am>[768588.520874] Call Trace:
                 <Oct/29 06:08 am>[768588.520874]  [<ffffffff80257acf>] debug_check_no_locks_freed+0xda/0x100
<Oct/29 06:08 am>[768588.520874]  [<ffffffffa00c1d70>] xfs_iget_core+0x593/0x5ae [xfs]
<Oct/29 06:08 am>[768588.520874]  [<ffffffff802a5d85>] kmem_cache_free+0x6a/0xab
<Oct/29 06:08 am>[768588.520874]  [<ffffffffa00c1d70>] xfs_iget_core+0x593/0x5ae [xfs]
<Oct/29 06:08 am>[768588.520874]  [<ffffffff805b7388>] _spin_unlock+0x17/0x20
<Oct/29 06:08 am>[768588.520874]  [<ffffffffa00c1e6d>] xfs_iget+0xe2/0x189 [xfs]
<Oct/29 06:08 am>[768588.520874]  [<ffffffffa00e0b78>] xfs_nfs_get_inode+0x39/0x88 [xfs]
<Oct/29 06:08 am>[768588.520874]  [<ffffffffa00e0c62>] xfs_fs_fh_to_dentry+0x69/0x76 [xfs]
<Oct/29 06:08 am>[768588.520874]  [<ffffffff8034e8ff>] exportfs_decode_fh+0x30/0x1cd
<Oct/29 06:08 am>[768588.520874]  [<ffffffff80351f21>] nfsd_acceptable+0x0/0xd3
<Oct/29 06:08 am>[768588.520874]  [<ffffffff803562ca>] exp_find_key+0x96/0xa9
<Oct/29 06:08 am>[768588.520874]  [<ffffffff8050ac9a>] sock_recvmsg+0xd5/0xed
<Oct/29 06:08 am>[768588.520874]  [<ffffffff8022eda5>] place_entity+0x52/0x75
<Oct/29 06:08 am>[768588.520874]  [<ffffffff80356346>] exp_find+0x69/0x74
<Oct/29 06:08 am>[768588.520874]  [<ffffffff803517a4>] fh_verify+0x269/0x54d

<Oct/29 06:08 am>[768588.681209]  [<ffffffff80353053>] nfsd_open+0x1f/0x17b
<Oct/29 06:08 am>[768588.681209]  [<ffffffff80353530>] nfsd_read+0x7f/0xc4
                 <Oct/29 06:08 am>[768588.681209]  [<ffffffff8035a34b>] nfsd3_proc_read+0xfe/0x141
<Oct/29 06:08 am>[768588.681209]  [<ffffffff8034ecd8>] nfsd_dispatch+0xde/0x1c2
<Oct/29 06:08 am>[768588.681209]  [<ffffffff8058febb>] svc_process+0x408/0x6e9
<Oct/29 06:08 am>[768588.681209]  [<ffffffff805b70a2>] __down_read+0x34/0x9e
<Oct/29 06:08 am>[768588.681209]  [<ffffffff8034f305>] nfsd+0x1bf/0x296
<Oct/29 06:08 am>[768588.681209]  [<ffffffff8034f146>] nfsd+0x0/0x296
<Oct/29 06:08 am>[768588.681209]  [<ffffffff8024becc>] kthread+0x47/0x73
<Oct/29 06:08 am>[768588.681209]  [<ffffffff80234852>] schedule_tail+0x27/0x5f
<Oct/29 06:08 am>[768588.681209]  [<ffffffff8020cd19>] child_rip+0xa/0x11
<Oct/29 06:08 am>[768588.681209]  [<ffffffff8024be85>] kthread+0x0/0x73
<Oct/29 06:08 am>[768588.681209]  [<ffffffff8020cd0f>] child_rip+0x0/0x11
<Oct/29 08:42 am>[777846.860683] eth0: too many iterations (61) in nv_nic_irq.
<Oct/29 08:49 am>[778280.004590] eth0: too many iterations (61) in nv_nic_irq.
<Oct/29 10:18 am>[783590.637909] eth0: too many iterations (61) in nv_nic_irq.
<Oct/29 10:50 am>[785500.050813] eth0: too many iterations (61) in nv_nic_irq.
<Oct/29 11:53 am>[789307.308652] eth0: too many iterations (61) in nv_nic_irq.
<Oct/29 12:04 pm>[789975.432215] eth0: too many iterations (61) in nv_nic_irq.
<Oct/29 12:18 pm>[790831.449968] eth0: too many iterations (61) in nv_nic_irq.
<Oct/29 12:18 pm>[790835.238038] eth0: too many iterations (61) in nv_nic_irq.
<Oct/29 02:20 pm>[798133.388035] eth0: too many iterations (61) in nv_nic_irq.

<Oct/29 03:42 pm>[803028.043815] eth0: too many iterations (61) in nv_nic_irq.
<Oct/29 04:21 pm>[805413.358445] eth0: too many iterations (61) in nv_nic_irq.
<Oct/29 04:25 pm>[805631.004822] eth0: too many iterations (61) in nv_nic_irq.
<Oct/29 04:45 pm>[806813.000336] eth0: too many iterations (61) in nv_nic_irq.
<Oct/29 05:12 pm>[808422.702304] eth0: too many iterations (61) in nv_nic_irq.
<Oct/29 08:31 pm>[820402.099219] BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
<Oct/29 08:31 pm>[820402.099219] IP: [<ffffffff8022dfe5>] __wake_up_common+0x28/0x74
<Oct/29 08:31 pm>[820402.099219] PGD 1c7d97067 PUD 1c7d50067 PMD 0 
<Oct/29 08:31 pm>[820402.099219] Oops: 0000 [#1] SMP 
<Oct/29 08:31 pm>[820402.099219] last sysfs file: /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq
<Oct/29 08:31 pm>[820402.099219] CPU 3 
<Oct/29 08:31 pm>[820402.099219] Modules linked in: w83627hf hwmon_vid autofs4 smsc37b787_wdt i2c_nforce2 k8temp i2c_core forcedeth tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
<Oct/29 08:31 pm>[820402.099219] Pid: 3168, comm: xfslogd/3 Not tainted 2.6.28-rc7-00105-gfeaf384 #4
<Oct/29 08:31 pm>[820402.099219] RIP: 0010:[<ffffffff8022dfe5>]  [<ffffffff8022dfe5>] __wake_up_common+0x28/0x74
<Oct/29 08:31 pm>[820402.099219] RSP: 0018:ffff88012e06bce0  EFLAGS: 00010097
<Oct/29 08:31 pm>[820402.099219] RAX: 0000000000000286 RBX: ffff880161aa7858 RCX: 0000000000000000
<Oct/29 08:31 pm>[820402.099219] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff880161aa7858
<Oct/29 08:31 pm>[820402.099219] RBP: ffff88012e06bd10 R08: ffffffffffffffe8 R09: 0000000000000001
<Oct/29 08:31 pm>[820402.099219] R10: 0000000000000000 R11: ffffffff8039d94e R12: 0000000000000000
<Oct/29 08:31 pm>[820402.099219] R13: 0000000000000000 R14: ffff880161aa7858 R15: 0000000000000000
<Oct/29 08:31 pm>[820402.099219] FS:  0000000040eeb950(0000) GS:ffff88022fa0d780(0000) knlGS:0000000000000000
<Oct/29 08:31 pm>[820402.099219] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b

<Oct/29 08:31 pm>[820402.099219] CR2: 0000000000000000 CR3: 00000001c7d5e000 CR4: 00000000000006e0
<Oct/29 08:31 pm>[820402.099219] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<Oct/29 08:31 pm>[820402.099219] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<Oct/29 08:31 pm>[820402.099219] Process xfslogd/3 (pid: 3168, threadinfo ffff88012e06a000, task ffff88012e1b0f30)
<Oct/29 08:31 pm>[820402.099219] Stack:
<Oct/29 08:31 pm>[820402.099219]  0000000300000000 ffff880161aa7858 0000000000000000 0000000000000000
<Oct/29 08:31 pm>[820402.099219]  0000000000000286 0000000000000003 ffff88012e06bd50 ffffffff8022f03e
<Oct/29 08:31 pm>[820402.099219]  ffffffffffffff6e ffff880016af5001 0000000000000000 ffff880161aa7800
<Oct/29 08:31 pm>[820402.099219] Call Trace:
<Oct/29 08:31 pm>[820402.099219]  [<ffffffff8022f03e>] __wake_up+0x38/0x4e
<Oct/29 08:31 pm>[820402.099219]  [<ffffffffa00b042f>] xfs_buf_item_unpin+0x25/0xa4 [xfs]
<Oct/29 08:31 pm>[820402.099219]  [<ffffffffa00d50d3>] ? xfs_trans_chunk_committed+0xba/0xf0 [xfs]
<Oct/29 08:31 pm>[820402.099219]  [<ffffffffa00d514d>] ? xfs_trans_committed+0x44/0xd2 [xfs]
<Oct/29 08:31 pm>[820402.099219]  [<ffffffffa00ca407>] ? xlog_state_do_callback+0x169/0x281 [xfs]
<Oct/29 08:31 pm>[820402.099219]  [<ffffffffa00e079c>] ? xfs_buf_iodone_work+0x0/0x74 [xfs]
<Oct/29 08:31 pm>[820402.099219]  [<ffffffff80248973>] ? run_workqueue+0xed/0x1ed
<Oct/29 08:31 pm>[820402.099219]  [<ffffffff8024891d>] ? run_workqueue+0x97/0x1ed
<Oct/29 08:31 pm>[820402.099219]  [<ffffffff80249511>] ? worker_thread+0xd8/0xe3
<Oct/29 08:31 pm>[820402.099219]  [<ffffffff8024bfda>] ? autoremove_wake_function+0x0/0x2e
<Oct/29 08:31 pm>[820402.099219]  [<ffffffff80249439>] ? worker_thread+0x0/0xe3
<Oct/29 08:31 pm>[820402.099219]  [<ffffffff8024becc>] ? kthread+0x47/0x73
<Oct/29 08:31 pm>[820402.099219]  [<ffffffff80234852>] ? schedule_tail+0x27/0x5f

<Oct/29 08:31 pm>[820402.099219]  [<ffffffff8020cd19>] ? child_rip+0xa/0x11
<Oct/29 08:31 pm>[820402.099219]  [<ffffffff80212496>] ? c1e_idle+0x0/0xe9
<Oct/29 08:31 pm>[820402.099219]  [<ffffffff8024be85>] ? kthread+0x0/0x73
<Oct/29 08:31 pm>[820402.099219]  [<ffffffff8020cd0f>] ? child_rip+0x0/0x11
<Oct/29 08:31 pm>[820402.099219] Code: e5 c9 c3 55 48 89 e5 41 57 4d 89 c7 41 56 49 89 fe 41 55 41 54 41 89 d4 53 48 83 ec 08 89 75 d4 89 4d d0 4c 8b 47 30 49 83 e8 18 <49> 8b 58 18 eb 29 45 8b 28 4c 89 f9 8b 55 d0 8b 75 d4 4c 89 c7 
<Oct/29 08:31 pm>[820402.099219] RIP  [<ffffffff8022dfe5>] __wake_up_common+0x28/0x74
<Oct/29 08:31 pm>[820402.099219]  RSP <ffff88012e06bce0>
<Oct/29 08:31 pm>[820402.099219] CR2: 0000000000000000
<Oct/29 08:31 pm>[820402.099219] ---[ end trace 06a2b6b5cc51d31c ]---

File erased !