{{{
<May/15 12:07 pm>ipp018 login: [686154.017001] do_IRQ: 2.179 No irq handler for vector
<May/15 12:07 pm>[686154.017006] do_IRQ: 0.179 No irq handler for vector
<May/15 12:07 pm>[686154.017012] do_IRQ: 1.179 No irq handler for vector
<May/15 12:11 pm>[686427.038519] general protection fault: 0000 [#1] SMP 
<May/15 12:11 pm>[686427.039998] last sysfs file: /sys/devices/system/cpu/sched_mc_power_savings
<May/15 12:11 pm>[686427.039998] CPU 2 
<May/15 12:11 pm>[686427.039998] Modules linked in: w83627hf hwmon_vid autofs4 smsc37b787_wdt k8temp forcedeth i2c_nforce2 i2c_core tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7<May/15 12:11 pm>n
                 <May/15 12:11 pm>[686427.039998] Pid: 9435, comm: pclient Tainted: G   M       2.6.28-rc7-00105-gfeaf384 #4
<May/15 12:11 pm>[686427.039998] RIP: 0010:[<ffffffff80235a72>]  [<ffffffff80235a72>] tg_shares_up+0x43/0x212
<May/15 12:11 pm>[686427.039998] RSP: 0018:ffff8801d7d47b08  EFLAGS: 00010097
<May/15 12:11 pm>[686427.039998] RAX: 2e6170614b2f706d RBX: ffff880135a445c0 RCX: 0000000000000020
<May/15 12:11 pm>[686427.039998] RDX: ffff88022fa9cda0 RSI: 0000000000000006 RDI: 0000000000000006
<May/15 12:11 pm>[686427.039998] RBP: ffff8801d7d47b98 R08: 0000000000000020 R09: ffff880135a445d8
<May/15 12:11 pm>[686427.039998] R10: 0000000000000000 R11: ffffffff8037ca54 R12: ffff88022e004de0
<May/15 12:11 pm>[686427.039998] R13: ffff880135a445c0 R14: ffffffff80235a2f R15: ffffffff8022d8c2
<May/15 12:11 pm>[686427.039998] FS:  00007f1c3efa46f0(0000) GS:ffff88022fa20400(0000) knlGS:0000000000000000

<May/15 12:11 pm>[686427.039998] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<May/15 12:11 pm>[686427.039998] CR2: 0000000000e8b018 CR3: 00000001623dc000 CR4: 00000000000006e0
<May/15 12:11 pm>[686427.039998] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<May/15 12:11 pm>[686427.039998] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<May/15 12:11 pm>[686427.039998] Process pclient (pid: 9435, threadinfo ffff8801d7d46000, task ffff880142a0cbf0)
<May/15 12:11 pm>[686427.039998] Stack:
<May/15 12:11 pm>[686427.039998]  0000000000000002 ffff88022fa9cda0 ffff880189573cc0 0000000000000046
<May/15 12:11 pm>[686427.039998]  ffff880135a4a480 0000000000000087 ffff88022e004de0 ffff880135a445d8
<May/15 12:11 pm>[686427.039998]  2f3a343030707265 0000000000000046 ffff8801d7d47bd8 ffff88022fa9cf00
<May/15 12:11 pm>[686427.039998] Call Trace:
<May/15 12:11 pm>[686427.039998]  [<ffffffff80235a2f>] ? tg_shares_up+0x0/0x212
<May/15 12:11 pm>[686427.039998]  [<ffffffff8022d8c2>] ? tg_nop+0x0/0x8
<May/15 12:11 pm>[686427.039998]  [<ffffffff8022f541>] walk_tg_tree+0x90/0xc7
<May/15 12:11 pm>[686427.039998]  [<ffffffff8022f4b1>] ? walk_tg_tree+0x0/0xc7
<May/15 12:11 pm>[686427.039998]  [<ffffffff80232657>] try_to_wake_up+0xb0/0x23b
<May/15 12:11 pm>[686427.039998]  [<ffffffff8022dffe>] __wake_up_common+0x41/0x74
<May/15 12:11 pm>[686427.039998]  [<ffffffff8022efea>] __wake_up_sync+0x3a/0x56
<May/15 12:11 pm>[686427.039998]  [<ffffffff8050eba6>] sock_def_readable+0x3d/0x6a
<May/15 12:11 pm>[686427.039998]  [<ffffffff8055e063>] ? unix_stream_sendmsg+0x22c/0x2ed
<May/15 12:11 pm>[686427.039998]  [<ffffffff8050a4bb>] ? sock_aio_write+0xd1/0xe0
<May/15 12:11 pm>[686427.039998]  [<ffffffff802abdd4>] ? do_sync_write+0xc9/0x10c
<May/15 12:11 pm>[686427.039998]  [<ffffffff8024bfda>] ? autoremove_wake_function+0x0/0x2e

<May/15 12:11 pm>[686427.039998]  [<ffffffff805b76fc>] ? _spin_unlock_irq+0x1f/0x22
<May/15 12:11 pm>[686427.039998]  [<ffffffff802430b9>] ? do_sigaction+0x140/0x154
<May/15 12:11 pm>[686427.039998]  [<ffffffff802ac544>] ? vfs_write+0xc0/0x136
<May/15 12:11 pm>[686427.039998]  [<ffffffff802ac9bb>] ? sys_write+0x45/0x6e
<May/15 12:11 pm>[686427.039998]  [<ffffffff8020bd2b>] ? system_call_fastpath+0x16/0x1b
<May/15 12:11 pm>[686427.039998] Code: 83 cf ff 48 c7 45 b0 00 00 00 00 48 c7 45 c8 00 00 00 00 eb 28 48 8b 55 a0 48 63 c7 48 8b 52 08 48 89 95 78 ff ff ff 48 8b 04 c2 <48> 8b 10 48 01 55 c8 48 8b 90 a0 00 00 00 48 01 55 b0 48 8d 43 
<May/15 12:11 pm>[686427.039998] RIP  [<ffffffff80235a72>] tg_shares_up+0x43/0x212
<May/15 12:11 pm>[686427.039998]  RSP <ffff8801d7d47b08>
<May/15 12:11 pm>[686427.039998] ---[ end trace 0ac761e6139243e1 ]---
<May/15 12:12 pm>[686446.385844] BUG: unable to handle kernel NULL pointer dereference at 00000000000000c4
<May/15 12:12 pm>[686446.385942] IP: [<ffffffff80514bce>] dev_kfree_skb_irq+0x1/0x3e
<May/15 12:12 pm>[686446.385942] PGD 180191067 PUD 1622b4067 PMD 0 
<May/15 12:12 pm>[686446.385942] Oops: 0002 [#2] SMP 
<May/15 12:12 pm>[686446.385942] last sysfs file: /sys/devices/system/cpu/sched_mc_power_savings
<May/15 12:12 pm>[686446.385942] CPU 3 
<May/15 12:12 pm>[686446.385942] Modules linked in: w83627hf hwmon_vid autofs4 smsc37b787_wdt k8temp forcedeth i2c_nforce2 i2c_core tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
<May/15 12:12 pm>[686446.385942] Pid: 14068, comm: ppStack Tainted: G   M  D    2.6.28-rc7-00105-gfeaf384 #4
<May/15 12:12 pm>[686446.385942] RIP: 0010:[<ffffffff80514bce>]  [<ffffffff80514bce>] dev_kfree_skb_irq+0x1/0x3e
<May/15 12:12 pm>[686446.385942] RSP: 0018:ffff88012fadfeb8  EFLAGS: 00010006
<May/15 12:12 pm>[686446.385942] RAX: 0000000000010000 RBX: ffff88012e0fa900 RCX: 0000000000000001
<May/15 12:12 pm>[686446.385942] RDX: 0000000020000000 RSI: 0000000000000058 RDI: 0000000000000000
<May/15 12:12 pm>[686446.385942] RBP: 00000000200000f6 R08: 0000000000000001 R09: 0000000000000001

<May/15 12:12 pm>[686446.385942] R10: 0000000000000000 R11: ffffffff8039d94e R12: ffff88012e0fa000
<May/15 12:12 pm>[686446.385942] R13: 0000000000000040 R14: ffff88012e01f7a0 R15: 0000000000000000
<May/15 12:12 pm>[686446.385942] FS:  0000000040ae4950(0063) GS:ffff88022fa20880(0000) knlGS:0000000000000000
<May/15 12:12 pm>[686446.385942] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<May/15 12:12 pm>[686446.385942] CR2: 00000000000000c4 CR3: 0000000160e52000 CR4: 00000000000006e0
<May/15 12:12 pm>[686446.385942] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<May/15 12:12 pm>[686446.385942] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<May/15 12:12 pm>[686446.385942] Process ppStack (pid: 14068, threadinfo ffff880<May/15 12:12 pm>fff88021742ad90)
                 <May/15 12:12 pm>[686446.385942] Stack:
<May/15 12:12 pm>[686446.385942]  ffff88012e0fa900 ffffffffa01706a8 ffff88012e0fa900 ffffc200020ac000
<May/15 12:12 pm>[686446.385942]  0000000000000010 ffff88012e0fa000 0000000000000000 ffffffffa0171066
<May/15 12:12 pm>[686446.385942]  ffff88022e468440 0000000000000000 0000000000000000 0000000000000011
<May/15 12:12 pm>[686446.385942] Call Trace:
<May/15 12:12 pm>[686446.385942]  <IRQ> <0> [<ffffffffa01706a8>] ? nv_tx_done_optimized+0xd0/0x20d [forcedeth]
<May/15 12:12 pm>[686446.385942]  [<ffffffffa0171066>] ? nv_nic_irq_optimized+0x80/0x232 [forcedeth]
<May/15 12:12 pm>[686446.385942]  [<ffffffff80277305>] ? handle_IRQ_event+0x1e/0x4c
<May/15 12:12 pm>[686446.385942]  [<ffffffff80278cbb>] ? handle_fasteoi_irq+0x90/0xcf
<May/15 12:12 pm>[686446.385942]  [<ffffffff8020e15b>] ? do_IRQ+0xf7/0x16c
<May/15 12:12 pm>[686446.385942]  [<ffffffff8020c2f6>] ? ret_from_intr+0x0/0xa
<May/15 12:12 pm>[686446.385942]  <EOI> <0> [<ffffffff8039d94e>] ? delay_tsc+0x0/0x42
<May/15 12:12 pm>[686446.385942]  [<ffffffff8021d9d0>] ? native_flush_tlb_others+0x91/0xb6
<May/15 12:12 pm>[686446.385942]  [<ffffffff8021d9c9>] ? native_flush_tlb_others+0x8a/0xb6

<May/15 12:12 pm>[686446.385942]  [<ffffffff8021dc02>] ? flush_tlb_mm+0x5e/0x64
<May/15 12:12 pm>[686446.385942]  [<ffffffff8028b91f>] ? zap_page_range+0xc6/0xf0
<May/15 12:12 pm>[686446.385942]  [<ffffffff80289ff1>] ? sys_madvise+0x3c9/0x4d4
<May/15 12:12 pm>[686446.385942]  [<ffffffff8020bd2b>] ? system_call_fastpath+0x16/0x1b
<May/15 12:12 pm>[686446.385942] Code: 08 48 89 f0 48 89 fa 81 e1 61 00 1d 00 81 e2 61 00 1d 00 48 83 c8 1e 48 09 d0 48 89 c2 48 21 c8 80 ce 18 48 21 d7 48 09 f8 c3 53 <f0> ff 8f c4 00 00 00 0f 94 c0 84 c0 74 2d 9c 5b fa 65 48 8b 14 
<May/15 12:12 pm>[686446.385942] RIP  [<ffffffff80514bce>] dev_kfree_skb_irq+0x1/0x3e
<May/15 12:12 pm>[686446.385942]  RSP <ffff88012fadfeb8>
<May/15 12:12 pm>[686446.385942] CR2: 00000000000000c4
<May/15 12:12 pm>[686446.385942] Kernel panic - not syncing: Fatal exception in interrupt
<May/15 12:12 pm>[686446.385942] ------------[ cut here ]------------
<May/15 12:12 pm>[686446.385942] WARNING: at kernel/smp.c:333 smp_call_function_mask+0x37/0x1d7()
<May/15 12:12 pm>[686446.385942] Modules linked in: w83627hf hwmon_vid autofs4 smsc37b787_wdt k8temp forcedeth i2c_nforce2 i2c_core tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
<May/15 12:12 pm>[686446.385942] Pid: 14068, comm: ppStack Tainted: G   M  D    2.6.28-rc7-00105-gfeaf384 #4
<May/15 12:12 pm>[686446.385942] Call Trace:
<May/15 12:12 pm>[686446.385942]  <IRQ>  [<ffffffff802397db>] warn_on_slowpath+0x51/0x6d
<May/15 12:12 pm>[686446.385942]  [<ffffffff805b9e66>] __atomic_notifier_call_chain+0x74/0x83
<May/15 12:12 pm>[686446.385942]  [<ffffffff805b9df2>] __atomic_notifier_call_chain+0x0/0x83
<May/15 12:12 pm>[686446.385942]  [<ffffffff803eec86>] notify_update+0x2b/0x30
<May/15 12:12 pm>[686446.385942]  [<ffffffff8025bc31>] smp_call_function_mask+0x37/0x1d7
<May/15 12:12 pm>[686446.385942]  [<ffffffff80264013>] crash_kexec+0x17/0xef
<May/15 12:12 pm>[686446.385942]  [<ffffffff802640e2>] crash_kexec+0xe6/0xef
<May/15 12:12 pm>[686446.385942]  [<ffffffff80239d5d>] release_console_sem+0x3e/0x1a5

<May/15 12:12 pm>[686446.385942]  [<ffffffff80239d5d>] release_console_sem+0x3e/0x1a5
<May/15 12:12 pm>[686446.385942]  [<ffffffff8021d53e>] native_smp_send_stop+0x1a/0x26
<May/15 12:12 pm>[686446.385942]  [<ffffffff80239896>] panic+0x95/0x13f
<May/15 12:12 pm>[686446.385942]  [<ffffffff80239d5d>] release_console_sem+0x3e/0x1a5
<May/15 12:12 pm>[686446.385942]  [<ffffffff80239d5d>] release_console_sem+0x3e/0x1a5
<May/15 12:12 pm>[686446.385942]  [<ffffffff80239d5d>] release_console_sem+0x3e/0x1a5
<May/15 12:12 pm>[686446.385942]  [<ffffffff8023a475>] printk+0x4e/0x56
<May/15 12:12 pm>[686446.385942]  [<ffffffff8020e925>] show_registers+0x1eb/0x20e
<May/15 12:12 pm>[686446.385942]  [<ffffffff803f0656>] do_unblank_screen+0xd/0x10d
<May/15 12:12 pm>[686446.385942]  [<ffffffff805b83ed>] oops_end+0x7b/0x88
<May/15 12:12 pm>[686446.385942]  [<ffffffff805b9ce3>] do_page_fault+0x726/0x7df
<May/15 12:12 pm>[686446.385942]  [<ffffffff805b795d>] error_exit+0x0/0x9a
<May/15 12:12 pm>[686446.385942]  [<ffffffff8039d94e>] delay_tsc+0x0/0x42
<May/15 12:12 pm>[686446.385942]  [<ffffffff80514bce>] dev_kfree_skb_irq+0x1/0x3e
<May/15 12:12 pm>[686446.385942]  [<ffffffffa01706a8>] nv_tx_done_optimized+0xd0/0x20d [forcedeth]
<May/15 12:12 pm>[686446.385942]  [<ffffffffa0171066>] nv_nic_irq_optimized+0x80/0x232 [forcedeth]
<May/15 12:12 pm>[686446.385942]  [<ffffffff80277305>] handle_IRQ_event+0x1e/0x4c
<May/15 12:12 pm>[686446.385942]  [<ffffffff80278cbb>] handle_fasteoi_irq+0x90/0xcf
<May/15 12:12 pm>[686446.385942]  [<ffffffff8020e15b>] do_IRQ+0xf7/0x16c
<May/15 12:12 pm>[686446.385942]  [<ffffffff8020c2f6>] ret_from_intr+0x0/0xa
<May/15 12:12 pm>[686446.385942]  <EOI>  [<ffffffff8039d94e>] delay_tsc+0x0/0x42
<May/15 12:12 pm>[686446.385942]  [<ffffffff8021d9d0>] native_flush_tlb_others+0x91/0xb6

<May/15 12:12 pm>[686446.385942]  [<ffffffff8021d9c9>] native_flush_tlb_others+0x8a/0xb6
<May/15 12:12 pm>[686446.385942]  [<ffffffff8021dc02>] flush_tlb_mm+0x5e/0x64
<May/15 12:12 pm>[686446.385942]  [<ffffffff8028b91f>] zap_page_range+0xc6/0xf0
<May/15 12:12 pm>[686446.385942]  [<ffffffff80289ff1>] sys_madvise+0x3c9/0x4d4
<May/15 12:12 pm>[686446.385942]  [<ffffffff8020bd2b>] system_call_fastpath+0x16/0x1b
<May/15 12:12 pm>[686446.385942] ---[ end trace 0ac761e6139243e1 ]---
<May/15 12:12 pm>[686446.385942] BUG: spinlock lockup on CPU#2, tcsh/9276, ffff880227897530
<May/15 12:12 pm>[686446.385942] Pid: 9276, comm: tcsh Tainted: G   M  D W  2.6.28-rc7-00105-gfeaf384 #4
<May/15 12:12 pm>[686446.385942] Call Trace:
<May/15 12:12 pm>[686446.385942]  [<ffffffff803a103f>] _raw_spin_lock+0xfb/0x122
<May/15 12:12 pm>[686446.385942]  [<ffffffff805b7758>] _spin_lock_irqsave+0x37/0x3f
<May/15 12:12 pm>[686446.385942]  [<ffffffff8022efce>] __wake_up_sync+0x1e/0x56
                 <May/15 12:12 pm>[686446.385942]  [<ffffffff8022efce>] __wake_up_sync+0x1e/0x56
<May/15 12:12 pm>[686446.385942]  [<ffffffff8050eba6>] sock_def_readable+0x3d/0x6a
<May/15 12:12 pm>[686446.385942]  [<ffffffff8055e063>] unix_stream_sendmsg+0x22c/0x2ed
<May/15 12:12 pm>[686446.385942]  [<ffffffff8050a4bb>] sock_aio_write+0xd1/0xe0
<May/15 12:12 pm>[686446.385942]  [<ffffffff802abdd4>] do_sync_write+0xc9/0x10c
<May/15 12:12 pm>[686446.385942]  [<ffffffff8024bfda>] autoremove_wake_function+0x0/0x2e
<May/15 12:12 pm>[686446.385942]  [<ffffffff802ac544>] vfs_write+0xc0/0x136
<May/15 12:12 pm>[686446.385942]  [<ffffffff802ac9bb>] sys_write+0x45/0x6e
                 <May/15 12:12 pm>[686446.385942]  [<ffffffff8020bd2b>] system_call_fastpath+0x16/0x1b

}}}