IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links
wiki:ipp054-2012-01-02-crash

From console:

File content:
1973]  [<ffffffff8151cc0e>] schedule+0x25a/0x86a
<Feb/01 11:56 am>[4661030.797691]  [<ffffffff8151d264>] io_schedule+0x46/0x5e
<Feb/01 11:56 am>[4661030.803489]  [<ffffffff810ac90d>] sync_page+0x44/0x48
<Feb/01 11:56 am>[4661030.809095]  [<ffffffff8151d619>] __wait_on_bit_lock+0x45/0x8c
<Feb/01 11:56 am>[4661030.815502]  [<ffffffff810ac8c9>] ? sync_page+0x0/0x48
<Feb/01 11:56 am>[4661030.821203]  [<ffffffff810ac8b5>] __lock_page+0x63/0x6a
<Feb/01 11:56 am>[4661030.826994]  [<ffffffff81054ba4>] ? wake_bit_function+0x0/0x2a
<Feb/01 11:56 am>[4661030.833408]  [<ffffffff810c4f57>] handle_mm_fault+0x4c1/0x73e
<Feb/01 11:56 am>[4661030.839753]  [<ffffffff815227e9>] do_page_fault+0x351/0x375
<Feb/01 11:56 am>[4661030.845905]  [<ffffffff810348d8>] ? finish_task_switch+0x69/0xdd
<Feb/01 11:56 am>[4661030.852511]  [<ffffffff8103486f>] ? finish_task_switch+0x0/0xdd
<Feb/01 11:56 am>[4661030.859030]  [<ffffffff8151d152>] ? schedule+0x79e/0x86a
<Feb/01 11:56 am>[4661030.864912]  [<ffffffff8151ef6f>] ? lockdep_sys_exit_thunk+0x35/0x67
<Feb/01 11:56 am>[4661030.871899]  [<ffffffff81057962>] ? __run_hrtimer+0xff/0x12d
<Feb/01 11:56 am>[4661030.878156]  [<ffffffff8151fb5f>] page_fault+0x1f/0x30
<Feb/01 11:56 am>[4661030.883865] NMI backtrace for cpu 4
<Feb/01 11:56 am>[4661030.887865] CPU 4 
<Feb/01 11:56 am>[4661030.889990] Modules linked in: w83793 hwmon_vid autofs4 i2c_i801 i2c_core iTCO_wdt e1000e ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
<Feb/01 11:56 am>[4661030.925976] 
<Feb/01 11:56 am>[4661030.927899] Pid: 5693, comm: ppStack Tainted: G      D    2.6.34.4 #3 S7010/empty
<Feb/01 11:56 am>[4661030.936169] RIP: 0010:[<ffffffff812c08a2>]  [<ffffffff812c08a2>] delay_tsc+0x18/0x59

<Feb/01 11:56 am>[4661030.944787] RSP: 0000:ffff8803edf0daf0  EFLAGS: 00000086
<Feb/01 11:56 am>[4661030.950670] RAX: 00000000e48ab078 RBX: 00000000244f7331 RCX: 00000000e48ab034
<Feb/01 11:56 am>[4661030.958631] RDX: 00000000002c28af RSI: 0000000000000004 RDI: 0000000000000001
<Feb/01 11:56 am>[4661030.966579] RBP: ffff8803edf0daf8 R08: 0000000000000004 R09: 0000000000000004
<Feb/01 11:56 am>[4661030.974518] R10: 0000000000000000 R11: ffff8803edf0d928 R12: ffff880623308460
<Feb/01 11:56 am>[4661030.982432] R13: 0000000000000000 R14: ffff88052a510000 R15: ffff88052a510488
<Feb/01 11:56 am>[4661030.990390] FS:  0000000041f46950(0063) GS:ffff880002680000(0000) knlGS:0000000000000000
<Feb/01 11:56 am>[4661030.999319] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<Feb/01 11:56 am>[4661031.005664] CR2: 00007ff195fad000 CR3: 000000043b231000 CR4: 00000000000006e0
<Feb/01 11:56 am>[4661031.013615] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<Feb/01 11:56 am>[4661031.021555] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<Feb/01 11:56 am>[4661031.029501] Process ppStack (pid: 5693, threadinfo ffff8803edf0c000, task ffff88052a510000)
<Feb/01 11:56 am>[4661031.038711] Stack:
                 <Feb/01 11:56 am>[4661031.041185]  ffffffff81019da9 ffff8803edf0db08 ffffffff812c08fe ffff8803edf0db48
<Feb/01 11:56 am>[4661031.049157] <0> ffffffff812c4e65 000000009ef2dfac ffff880623308460 ffff880623308000
<Feb/01 11:56 am>[4661031.057867] <0> ffff88062341c300 ffff880623308000 0000000000000000 ffff8803edf0db68
<Feb/01 11:56 am>[4661031.071469] Call Trace:
<Feb/01 11:56 am>[4661031.074382]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
<Feb/01 11:56 am>[4661031.082206]  [<ffffffff812c08fe>] __delay+0xa/0xc
<Feb/01 11:56 am>[4661031.087459]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
<Feb/01 11:56 am>[4661031.093791]  [<ffffffff8151f115>] _raw_spin_lock_irq+0x35/0x39
<Feb/01 11:56 am>[4661031.100195]  [<ffffffff812aacd7>] ? __make_request+0x5e/0x445

<Feb/01 11:56 am>[4661031.106522]  [<ffffffff812aacd7>] __make_request+0x5e/0x445
<Feb/01 11:56 am>[4661031.112684]  [<ffffffff812a946b>] generic_make_request+0x32a/0x39c
<Feb/01 11:56 am>[4661031.119464]  [<ffffffff811100ac>] ? bio_alloc_bioset+0x48/0xc0
<Feb/01 11:56 am>[4661031.125893]  [<ffffffff812a9591>] submit_bio+0xb4/0xbd
<Feb/01 11:56 am>[4661031.131610]  [<ffffffff810d127d>] ? get_swap_bio+0x31/0x77
<Feb/01 11:56 am>[4661031.137670]  [<ffffffff810d130a>] swap_readpage+0x47/0x4d
<Feb/01 11:56 am>[4661031.143642]  [<ffffffff810d161b>] read_swap_cache_async+0xb1/0xf2
<Feb/01 11:56 am>[4661031.150334]  [<ffffffff810d216d>] ? valid_swaphandles+0x6b/0xfd
<Feb/01 11:56 am>[4661031.156851]  [<ffffffff810d16b4>] swapin_readahead+0x58/0x97
<Feb/01 11:56 am>[4661031.163108]  [<ffffffff810c4ef1>] handle_mm_fault+0x45b/0x73e
<Feb/01 11:56 am>[4661031.169444]  [<ffffffff815227e9>] do_page_fault+0x351/0x375
<Feb/01 11:56 am>[4661031.175580]  [<ffffffff810348d8>] ? finish_task_switch+0x69/0xdd
<Feb/01 11:56 am>[4661031.182186]  [<ffffffff8103486f>] ? finish_task_switch+0x0/0xdd
<Feb/01 11:56 am>[4661031.188711]  [<ffffffff8151d152>] ? schedule+0x79e/0x86a
<Feb/01 11:56 am>[4661031.194604]  [<ffffffff8151ef6f>] ? lockdep_sys_exit_thunk+0x35/0x67
                 <Feb/01 11:56 am>[4661031.201582]  [<ffffffff81057962>] ? __run_hrtimer+0xff/0x12d
<Feb/01 11:56 am>[4661031.207811]  [<ffffffff8151fb5f>] page_fault+0x1f/0x30
                 <Feb/01 11:56 am>[4661031.213513] Code: 2e 0f 1f 84 00 00 00 00 00 48 ff c8 75 fb 48 ff c8 c9 c3 55 48 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae e8 0f 31 <89> c1 41 89 f1 66 66 90 0f ae e8 0f 31 89 c6 48 89 f0 48 29 c8 
<Feb/01 11:56 am>[4661031.236617] Call Trace:
<Feb/01 11:56 am>[4661031.239532]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
<Feb/01 11:56 am>[4661031.247371]  [<ffffffff812c08fe>] __delay+0xa/0xc
<Feb/01 11:56 am>[4661031.252625]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147

<Feb/01 11:56 am>[4661031.258973]  [<ffffffff8151f115>] _raw_spin_lock_irq+0x35/0x39
<Feb/01 11:56 am>[4661031.265413]  [<ffffffff812aacd7>] ? __make_request+0x5e/0x445
<Feb/01 11:56 am>[4661031.271756]  [<ffffffff812aacd7>] __make_request+0x5e/0x445
<Feb/01 11:56 am>[4661031.277919]  [<ffffffff812a946b>] generic_make_request+0x32a/0x39c
<Feb/01 11:56 am>[4661031.284699]  [<ffffffff811100ac>] ? bio_alloc_bioset+0x48/0xc0
<Feb/01 11:56 am>[4661031.291128]  [<ffffffff812a9591>] submit_bio+0xb4/0xbd
<Feb/01 11:56 am>[4661031.296854]  [<ffffffff810d127d>] ? get_swap_bio+0x31/0x77
<Feb/01 11:56 am>[4661031.302924]  [<ffffffff810d130a>] swap_readpage+0x47/0x4d
<Feb/01 11:56 am>[4661031.308920]  [<ffffffff810d161b>] read_swap_cache_async+0xb1/0xf2
<Feb/01 11:56 am>[4661031.315608]  [<ffffffff810d216d>] ? valid_swaphandles+0x6b/0xfd
<Feb/01 11:56 am>[4661031.322123]  [<ffffffff810d16b4>] swapin_readahead+0x58/0x97
<Feb/01 11:56 am>[4661031.328361]  [<ffffffff810c4ef1>] handle_mm_fault+0x45b/0x73e
<Feb/01 11:56 am>[4661031.334707]  [<ffffffff815227e9>] do_page_fault+0x351/0x375
<Feb/01 11:56 am>[4661031.340860]  [<ffffffff810348d8>] ? finish_task_switch+0x69/0xdd
<Feb/01 11:56 am>[4661031.347457]  [<ffffffff8103486f>] ? finish_task_switch+0x0/0xdd
<Feb/01 11:56 am>[4661031.353948]  [<ffffffff8151d152>] ? schedule+0x79e/0x86a
<Feb/01 11:56 am>[4661031.359822]  [<ffffffff8151ef6f>] ? lockdep_sys_exit_thunk+0x35/0x67
<Feb/01 11:56 am>[4661031.366803]  [<ffffffff81057962>] ? __run_hrtimer+0xff/0x12d
<Feb/01 11:56 am>[4661031.373031]  [<ffffffff8151fb5f>] page_fault+0x1f/0x30
<Feb/01 11:56 am>[4661031.378715] Pid: 5693, comm: ppStack Tainted: G      D    2.6.34.4 #3
<Feb/01 11:56 am>[4661031.385738] Call Trace:
<Feb/01 11:56 am>[4661031.388675]  <NMI>  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a

<Feb/01 11:56 am>[4661031.395125]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
<Feb/01 11:56 am>[4661031.401562]  [<ffffffff81520360>] do_nmi+0xb2/0x259
<Feb/01 11:56 am>[4661031.406967]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
<Feb/01 11:56 am>[4661031.412063]  [<ffffffff812c08a2>] ? delay_tsc+0x18/0x59
<Feb/01 11:56 am>[4661031.417845]  <<EOE>>  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
<Feb/01 11:56 am>[4661031.426583]  [<ffffffff812c08fe>] __delay+0xa/0xc
<Feb/01 11:56 am>[4661031.431834]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
<Feb/01 11:56 am>[4661031.438156]  [<ffffffff8151f115>] _raw_spin_lock_irq+0x35/0x39
<Feb/01 11:56 am>[4661031.444587]  [<ffffffff812aacd7>] ? __make_request+0x5e/0x445
<Feb/01 11:56 am>[4661031.450931]  [<ffffffff812aacd7>] __make_request+0x5e/0x445
<Feb/01 11:56 am>[4661031.457066]  [<ffffffff812a946b>] generic_make_request+0x32a/0x39c
<Feb/01 11:56 am>[4661031.463854]  [<ffffffff811100ac>] ? bio_alloc_bioset+0x48/0xc0
<Feb/01 11:56 am>[4661031.470241]  [<ffffffff812a9591>] submit_bio+0xb4/0xbd
<Feb/01 11:56 am>[4661031.475941]  [<ffffffff810d127d>] ? get_swap_bio+0x31/0x77
<Feb/01 11:56 am>[4661031.482010]  [<ffffffff810d130a>] swap_readpage+0x47/0x4d
<Feb/01 11:56 am>[4661031.487998]  [<ffffffff810d161b>] read_swap_cache_async+0xb1/0xf2
<Feb/01 11:56 am>[4661031.494682]  [<ffffffff810d216d>] ? valid_swaphandles+0x6b/0xfd
<Feb/01 11:56 am>[4661031.501190]  [<ffffffff810d16b4>] swapin_readahead+0x58/0x97
<Feb/01 11:56 am>[4661031.507430]  [<ffffffff810c4ef1>] handle_mm_fault+0x45b/0x73e
<Feb/01 11:56 am>[4661031.513758]  [<ffffffff815227e9>] do_page_fault+0x351/0x375
<Feb/01 11:56 am>[4661031.519919]  [<ffffffff810348d8>] ? finish_task_switch+0x69/0xdd
<Feb/01 11:56 am>[4661031.526499]  [<ffffffff8103486f>] ? finish_task_switch+0x0/0xdd

<Feb/01 11:56 am>[4661031.533033]  [<ffffffff8151d152>] ? schedule+0x79e/0x86a
<Feb/01 11:56 am>[4661031.538925]  [<ffffffff8151ef6f>] ? lockdep_sys_exit_thunk+0x35/0x67
<Feb/01 11:56 am>[4661031.545905]  [<ffffffff81057962>] ? __run_hrtimer+0xff/0x12d
<Feb/01 11:56 am>[4661031.552160]  [<ffffffff8151fb5f>] page_fault+0x1f/0x30
<Feb/01 11:56 am>[4661031.557860] NMI backtrace for cpu 8
<Feb/01 11:56 am>[4661031.561842] CPU 8 
<Feb/01 11:56 am>[4661031.563960] Modules linked in: w83793 hwmon_vid autofs4 i2c_i801 i2c_core iTCO_wdt e1000e ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
<Feb/01 11:56 am>[4661031.600007] 
<Feb/01 11:56 am>[4661031.601938] Pid: 8506, comm: nfsd Tainted: G      D    2.6.34.4 #3 S7010/empty
<Feb/01 11:56 am>[4661031.609983] RIP: 0010:[<ffffffff812c088a>]  [<ffffffff812c088a>] delay_tsc+0x0/0x59
<Feb/01 11:56 am>[4661031.618506] RSP: 0018:ffff88061b7633e8  EFLAGS: 00000002
<Feb/01 11:56 am>[4661031.624381] RAX: 0000000000000000 RBX: 0000000024539cc2 RCX: 00000000e49afc28
<Feb/01 11:56 am>[4661031.632333] RDX: 0000000000005600 RSI: 00000000e49afc44 RDI: 0000000000000001
<Feb/01 11:56 am>[4661031.640291] RBP: ffff88061b7633f0 R08: 0000000000000008 R09: 0000000000000008
<Feb/01 11:56 am>[4661031.648256] R10: 0000000000000000 R11: ffff88061b763210 R12: ffff880002673c00
<Feb/01 11:56 am>[4661031.656211] R13: 0000000000000000 R14: ffff88061b768000 R15: ffff88061b768488
<Feb/01 11:56 am>[4661031.664159] FS:  0000000000000000(0000) GS:ffff880655240000(0000) knlGS:0000000000000000
<Feb/01 11:56 am>[4661031.673105] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
<Feb/01 11:56 am>[4661031.679434] CR2: 00007ff195e94068 CR3: 0000000001a8a000 CR4: 00000000000006e0
<Feb/01 11:56 am>[4661031.687367] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<Feb/01 11:56 am>[4661031.695299] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<Feb/01 11:56 am>[4661031.703241] Process nfsd (pid: 8506, threadinfo ffff88061b762000, task ffff88061b768000)

<Feb/01 11:56 am>[4661031.712125] Stack:
<Feb/01 11:56 am>[4661031.714599]  ffffffff812c08fe ffff88061b763430 ffffffff812c4e65 000000009ef2dfac
<Feb/01 11:56 am>[4661031.722625] <0> ffff880002673c00 ffff880623bb2060 0000000000013c00 ffff88061b7634a0
<Feb/01 11:56 am>[4661031.731370] <0> 0000000000000008 ffff88061b763450 ffffffff8151f193 ffffffff8102fb6b
<Feb/01 11:56 am>[4661031.740517] Call Trace:
<Feb/01 11:56 am>[4661031.743436]  [<ffffffff812c08fe>] ? __delay+0xa/0xc
<Feb/01 11:56 am>[4661031.748855]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
<Feb/01 11:56 am>[4661031.755203]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
<Feb/01 11:56 am>[4661031.761285]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
<Feb/01 11:56 am>[4661031.767335]  [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
<Feb/01 11:56 am>[4661031.773236]  [<ffffffff81036578>] try_to_wake_up+0x2e/0x342
<Feb/01 11:56 am>[4661031.779391]  [<ffffffff810368b8>] wake_up_process+0x10/0x12
                 <Feb/01 11:56 am>[4661031.785562]  [<ffffffff8110649a>] bdi_queue_work+0xa2/0xb9
<Feb/01 11:56 am>[4661031.791637]  [<ffffffff811068ae>] bdi_alloc_queue_work+0xa4/0xc4
<Feb/01 11:56 am>[4661031.798227]  [<ffffffff81106a1f>] wakeup_flusher_threads+0x93/0xc6
<Feb/01 11:56 am>[4661031.805029]  [<ffffffff811069bf>] ? wakeup_flusher_threads+0x33/0xc6
<Feb/01 11:56 am>[4661031.811991]  [<ffffffff810b9854>] do_try_to_free_pages+0x299/0x37a
<Feb/01 11:56 am>[4661031.818760]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
<Feb/01 11:56 am>[4661031.825076]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
<Feb/01 11:56 am>[4661031.831863]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
<Feb/01 11:56 am>[4661031.838823]  [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
<Feb/01 11:56 am>[4661031.845330]  [<ffffffff810acb60>] __page_cache_alloc+0x59/0x5e

<Feb/01 11:56 am>[4661031.851752]  [<ffffffff810b4860>] __do_page_cache_readahead+0xe5/0x1fc
<Feb/01 11:56 am>[4661031.858897]  [<ffffffff810b47f6>] ? __do_page_cache_readahead+0x7b/0x1fc
<Feb/01 11:56 am>[4661031.866230]  [<ffffffff810b4993>] ra_submit+0x1c/0x20
<Feb/01 11:56 am>[4661031.871841]  [<ffffffff810b4cb7>] ondemand_readahead+0x243/0x256
<Feb/01 11:56 am>[4661031.878466]  [<ffffffff810ac48a>] ? find_get_pages_contig+0x0/0x118
<Feb/01 11:56 am>[4661031.885364]  [<ffffffff810b4daa>] page_cache_sync_readahead+0x38/0x3a
<Feb/01 11:56 am>[4661031.892430]  [<ffffffff81109bf2>] __generic_file_splice_read+0xfa/0x41c
<Feb/01 11:56 am>[4661031.899649]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Feb/01 11:56 am>[4661031.906078]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Feb/01 11:56 am>[4661031.912483]  [<ffffffff812615b1>] ? xfs_ilock+0x49/0xb2
<Feb/01 11:56 am>[4661031.918287]  [<ffffffff811086db>] ? spd_release_page+0x0/0x14
<Feb/01 11:56 am>[4661031.924624]  [<ffffffff81109f57>] generic_file_splice_read+0x43/0x74
<Feb/01 11:56 am>[4661031.931561]  [<ffffffff8128191a>] xfs_file_splice_read+0xde/0x125
<Feb/01 11:56 am>[4661031.938241]  [<ffffffff8110875d>] vfs_splice_to+0x6e/0x81
<Feb/01 11:56 am>[4661031.944219]  [<ffffffff81108837>] splice_direct_to_actor+0xc7/0x19c
<Feb/01 11:56 am>[4661031.951077]  [<ffffffff811f562c>] ? nfsd_direct_splice_actor+0x0/0x12
<Feb/01 11:56 am>[4661031.958107]  [<ffffffff811f54ef>] nfsd_vfs_read+0x263/0x3a0
<Feb/01 11:56 am>[4661031.964282]  [<ffffffff81281760>] ? xfs_file_open+0x0/0x3f
<Feb/01 11:56 am>[4661031.970358]  [<ffffffff811f5c4e>] nfsd_read+0xa7/0xc3
<Feb/01 11:56 am>[4661031.975975]  [<ffffffff811fcd79>] nfsd3_proc_read+0xfe/0x142
<Feb/01 11:56 am>[4661031.982217]  [<ffffffff811f0854>] nfsd_dispatch+0xe9/0x1ca
<Feb/01 11:56 am>[4661031.988274]  [<ffffffff814f04c5>] svc_process+0x425/0x72a

<Feb/01 11:56 am>[4661031.994270]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
<Feb/01 11:56 am>[4661031.999580]  [<ffffffff811f0dc2>] nfsd+0xf3/0x139
<Feb/01 11:56 am>[4661032.004841]  [<ffffffff810546e2>] kthread+0x7d/0x85
<Feb/01 11:56 am>[4661032.010277]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
<Feb/01 11:56 am>[4661032.016799]  [<ffffffff81054665>] ? kthread+0x0/0x85
<Feb/01 11:56 am>[4661032.022315]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
<Feb/01 11:56 am>[4661032.029000] Code: 90 55 48 89 f8 48 89 e5 48 85 c0 74 19 eb 02 66 90 eb 0e 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 48 ff c8 75 fb 48 ff c8 c9 c3 <55> 48 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae 
<Feb/01 11:56 am>[4661032.052181] Call Trace:
<Feb/01 11:56 am>[4661032.055094]  [<ffffffff812c08fe>] ? __delay+0xa/0xc
<Feb/01 11:56 am>[4661032.060520]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
<Feb/01 11:56 am>[4661032.066840]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
<Feb/01 11:56 am>[4661032.072924]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
<Feb/01 11:56 am>[4661032.078966]  [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
<Feb/01 11:56 am>[4661032.084822]  [<ffffffff81036578>] try_to_wake_up+0x2e/0x342
<Feb/01 11:56 am>[4661032.090977]  [<ffffffff810368b8>] wake_up_process+0x10/0x12
<Feb/01 11:56 am>[4661032.101637]  [<ffffffff8110649a>] bdi_queue_work+0xa2/0xb9
<Feb/01 11:56 am>[4661032.107687]  [<ffffffff811068ae>] bdi_alloc_queue_work+0xa4/0xc4
<Feb/01 11:56 am>[4661032.114301]  [<ffffffff81106a1f>] wakeup_flusher_threads+0x93/0xc6
                 <Feb/01 11:56 am>[4661032.121087]  [<ffffffff811069bf>] ? wakeup_flusher_threads+0x33/0xc6
<Feb/01 11:56 am>[4661032.128058]  [<ffffffff810b9854>] do_try_to_free_pages+0x299/0x37a
<Feb/01 11:56 am>[4661032.134852]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
<Feb/01 11:56 am>[4661032.141178]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f

<Feb/01 11:56 am>[4661032.147948]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
<Feb/01 11:56 am>[4661032.154926]  [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
<Feb/01 11:56 am>[4661032.161450]  [<ffffffff810acb60>] __page_cache_alloc+0x59/0x5e
<Feb/01 11:56 am>[4661032.167863]  [<ffffffff810b4860>] __do_page_cache_readahead+0xe5/0x1fc
<Feb/01 11:56 am>[4661032.175007]  [<ffffffff810b47f6>] ? __do_page_cache_readahead+0x7b/0x1fc
<Feb/01 11:56 am>[4661032.182340]  [<ffffffff810b4993>] ra_submit+0x1c/0x20
<Feb/01 11:56 am>[4661032.187966]  [<ffffffff810b4cb7>] ondemand_readahead+0x243/0x256
<Feb/01 11:56 am>[4661032.194558]  [<ffffffff810ac48a>] ? find_get_pages_contig+0x0/0x118
<Feb/01 11:56 am>[4661032.201449]  [<ffffffff810b4daa>] page_cache_sync_readahead+0x38/0x3a
<Feb/01 11:56 am>[4661032.208513]  [<ffffffff81109bf2>] __generic_file_splice_read+0xfa/0x41c
<Feb/01 11:56 am>[4661032.215749]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Feb/01 11:56 am>[4661032.222170]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Feb/01 11:56 am>[4661032.228609]  [<ffffffff812615b1>] ? xfs_ilock+0x49/0xb2
<Feb/01 11:56 am>[4661032.234406]  [<ffffffff811086db>] ? spd_release_page+0x0/0x14
<Feb/01 11:56 am>[4661032.240751]  [<ffffffff81109f57>] generic_file_splice_read+0x43/0x74
<Feb/01 11:56 am>[4661032.247713]  [<ffffffff8128191a>] xfs_file_splice_read+0xde/0x125
<Feb/01 11:56 am>[4661032.254395]  [<ffffffff8110875d>] vfs_splice_to+0x6e/0x81
<Feb/01 11:56 am>[4661032.260381]  [<ffffffff81108837>] splice_direct_to_actor+0xc7/0x19c
<Feb/01 11:56 am>[4661032.267248]  [<ffffffff811f562c>] ? nfsd_direct_splice_actor+0x0/0x12
<Feb/01 11:56 am>[4661032.274312]  [<ffffffff811f54ef>] nfsd_vfs_read+0x263/0x3a0
<Feb/01 11:56 am>[4661032.280445]  [<ffffffff81281760>] ? xfs_file_open+0x0/0x3f
<Feb/01 11:56 am>[4661032.286495]  [<ffffffff811f5c4e>] nfsd_read+0xa7/0xc3

<Feb/01 11:56 am>[4661032.292093]  [<ffffffff811fcd79>] nfsd3_proc_read+0xfe/0x142
<Feb/01 11:56 am>[4661032.298326]  [<ffffffff811f0854>] nfsd_dispatch+0xe9/0x1ca
<Feb/01 11:56 am>[4661032.304375]  [<ffffffff814f04c5>] svc_process+0x425/0x72a
<Feb/01 11:56 am>[4661032.310364]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
<Feb/01 11:56 am>[4661032.315725]  [<ffffffff811f0dc2>] nfsd+0xf3/0x139
<Feb/01 11:56 am>[4661032.320977]  [<ffffffff810546e2>] kthread+0x7d/0x85
<Feb/01 11:56 am>[4661032.326430]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
<Feb/01 11:56 am>[4661032.332952]  [<ffffffff81054665>] ? kthread+0x0/0x85
<Feb/01 11:56 am>[4661032.338460]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
<Feb/01 11:56 am>[4661032.345162] Pid: 8506, comm: nfsd Tainted: G      D    2.6.34.4 #3
<Feb/01 11:56 am>[4661032.351966] Call Trace:
<Feb/01 11:56 am>[4661032.354884]  <NMI>  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
<Feb/01 11:56 am>[4661032.361320]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
<Feb/01 11:56 am>[4661032.367748]  [<ffffffff81520360>] do_nmi+0xb2/0x259
<Feb/01 11:56 am>[4661032.373206]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
<Feb/01 11:56 am>[4661032.378232]  [<ffffffff812c088a>] ? delay_tsc+0x0/0x59
<Feb/01 11:56 am>[4661032.383910]  <<EOE>>  [<ffffffff812c08fe>] ? __delay+0xa/0xc
<Feb/01 11:56 am>[4661032.390171]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
<Feb/01 11:56 am>[4661032.396515]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
<Feb/01 11:56 am>[4661032.402572]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
<Feb/01 11:56 am>[4661032.408656]  [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
<Feb/01 11:56 am>[4661032.414557]  [<ffffffff81036578>] try_to_wake_up+0x2e/0x342

<Feb/01 11:56 am>[4661032.420687]  [<ffffffff810368b8>] wake_up_process+0x10/0x12
<Feb/01 11:56 am>[4661032.426865]  [<ffffffff8110649a>] bdi_queue_work+0xa2/0xb9
<Feb/01 11:56 am>[4661032.432932]  [<ffffffff811068ae>] bdi_alloc_queue_work+0xa4/0xc4
<Feb/01 11:56 am>[4661032.439546]  [<ffffffff81106a1f>] wakeup_flusher_threads+0x93/0xc6
<Feb/01 11:56 am>[4661032.446351]  [<ffffffff811069bf>] ? wakeup_flusher_threads+0x33/0xc6
<Feb/01 11:56 am>[4661032.453302]  [<ffffffff810b9854>] do_try_to_free_pages+0x299/0x37a
<Feb/01 11:56 am>[4661032.460088]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
<Feb/01 11:56 am>[4661032.466413]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
<Feb/01 11:56 am>[4661032.473185]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
<Feb/01 11:56 am>[4661032.480146]  [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
<Feb/01 11:56 am>[4661032.486635]  [<ffffffff810acb60>] __page_cache_alloc+0x59/0x5e
<Feb/01 11:56 am>[4661032.493048]  [<ffffffff810b4860>] __do_page_cache_readahead+0xe5/0x1fc
<Feb/01 11:56 am>[4661032.500175]  [<ffffffff810b47f6>] ? __do_page_cache_readahead+0x7b/0x1fc
<Feb/01 11:56 am>[4661032.507509]  [<ffffffff810b4993>] ra_submit+0x1c/0x20
<Feb/01 11:56 am>[4661032.513117]  [<ffffffff810b4cb7>] ondemand_readahead+0x243/0x256
<Feb/01 11:56 am>[4661032.519665]  [<ffffffff810ac48a>] ? find_get_pages_contig+0x0/0x118
<Feb/01 11:56 am>[4661032.526547]  [<ffffffff810b4daa>] page_cache_sync_readahead+0x38/0x3a
<Feb/01 11:56 am>[4661032.533568]  [<ffffffff81109bf2>] __generic_file_splice_read+0xfa/0x41c
<Feb/01 11:56 am>[4661032.540815]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Feb/01 11:56 am>[4661032.547217]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Feb/01 11:56 am>[4661032.553648]  [<ffffffff812615b1>] ? xfs_ilock+0x49/0xb2
<Feb/01 11:56 am>[4661032.559443]  [<ffffffff811086db>] ? spd_release_page+0x0/0x14

<Feb/01 11:56 am>[4661032.565790]  [<ffffffff81109f57>] generic_file_splice_read+0x43/0x74
<Feb/01 11:56 am>[4661032.572768]  [<ffffffff8128191a>] xfs_file_splice_read+0xde/0x125
<Feb/01 11:56 am>[4661032.579450]  [<ffffffff8110875d>] vfs_splice_to+0x6e/0x81
                 <Feb/01 11:56 am>[4661032.585427]  [<ffffffff81108837>] splice_direct_to_actor+0xc7/0x19c
<Feb/01 11:56 am>[4661032.592321]  [<ffffffff811f562c>] ? nfsd_direct_splice_actor+0x0/0x12
<Feb/01 11:56 am>[4661032.599377]  [<ffffffff811f54ef>] nfsd_vfs_read+0x263/0x3a0
<Feb/01 11:56 am>[4661032.605535]  [<ffffffff81281760>] ? xfs_file_open+0x0/0x3f
<Feb/01 11:56 am>[4661032.611585]  [<ffffffff811f5c4e>] nfsd_read+0xa7/0xc3
<Feb/01 11:56 am>[4661032.617207]  [<ffffffff811fcd79>] nfsd3_proc_read+0xfe/0x142
<Feb/01 11:56 am>[4661032.623442]  [<ffffffff811f0854>] nfsd_dispatch+0xe9/0x1ca
<Feb/01 11:56 am>[4661032.629509]  [<ffffffff814f04c5>] svc_process+0x425/0x72a
<Feb/01 11:56 am>[4661032.635479]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
<Feb/01 11:56 am>[4661032.640841]  [<ffffffff811f0dc2>] nfsd+0xf3/0x139
<Feb/01 11:56 am>[4661032.646085]  [<ffffffff810546e2>] kthread+0x7d/0x85
<Feb/01 11:56 am>[4661032.651521]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
<Feb/01 11:56 am>[4661032.658023]  [<ffffffff81054665>] ? kthread+0x0/0x85
<Feb/01 11:56 am>[4661032.663550]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
<Feb/01 11:56 am>[4661032.670211] NMI backtrace for cpu 5
<Feb/01 11:56 am>[4661032.674206] CPU 5 
<Feb/01 11:56 am>[4661032.676323] Modules linked in: w83793 hwmon_vid autofs4 i2c_i801 i2c_core iTCO_wdt e1000e ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
<Feb/01 11:56 am>[4661032.712431] 
<Feb/01 11:56 am>[4661032.714359] Pid: 8141, comm: psphotStack Tainted: G      D    2.6.34.4 #3 S7010/empty

<Feb/01 11:56 am>[4661032.723023] RIP: 0010:[<ffffffff812c08af>]  [<ffffffff812c08af>] delay_tsc+0x25/0x59
<Feb/01 11:56 am>[4661032.731642] RSP: 0018:ffff88033a5d1500  EFLAGS: 00000086
<Feb/01 11:56 am>[4661032.737514] RAX: 00000000e48ec350 RBX: 0000000024400779 RCX: 00000000e48ec330
<Feb/01 11:56 am>[4661032.745457] RDX: 00000000002c28af RSI: 0000000000000005 RDI: 0000000000000001
<Feb/01 11:56 am>[4661032.753406] RBP: ffff88033a5d1508 R08: 0000000000000005 R09: 0000000000000005
<Feb/01 11:56 am>[4661032.761318] R10: 0000000000000000 R11: ffff88033a5d1338 R12: ffff880623308460
<Feb/01 11:56 am>[4661032.769268] R13: 0000000000000000 R14: ffff8800634f3090 R15: ffff8800634f3518
<Feb/01 11:56 am>[4661032.777212] FS:  0000000042611950(0063) GS:ffff8800026a0000(0000) knlGS:0000000000000000
<Feb/01 11:56 am>[4661032.786136] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
<Feb/01 11:56 am>[4661032.792482] CR2: 00007fc17fdb1000 CR3: 00000005a4b03000 CR4: 00000000000006e0
<Feb/01 11:56 am>[4661032.800449] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<Feb/01 11:56 am>[4661032.808382] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<Feb/01 11:56 am>[4661032.816337] Process psphotStack (pid: 8141, threadinfo ffff88033a5d0000, task ffff8800634f3090)
<Feb/01 11:56 am>[4661032.825901] Stack:
<Feb/01 11:56 am>[4661032.828375]  ffffffff81019da9 ffff88033a5d1518 ffffffff812c08fe ffff88033a5d1558
<Feb/01 11:56 am>[4661032.836382] <0> ffffffff812c4e65 000000009ef2dfac ffff880623308460 ffff880623308000
                 <Feb/01 11:56 am>[4661032.845136] <0> ffff8800ad50f740 ffff880623308000 0000000000000001 ffff88033a5d1578
<Feb/01 11:56 am>[4661032.854247] Call Trace:
                 <Feb/01 11:56 am>[4661032.857171]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
<Feb/01 11:56 am>[4661032.865003]  [<ffffffff812c08fe>] __delay+0xa/0xc
<Feb/01 11:56 am>[4661032.870266]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
<Feb/01 11:56 am>[4661032.876561]  [<ffffffff8151f115>] _raw_spin_lock_irq+0x35/0x39

<Feb/01 11:56 am>[4661032.883009]  [<ffffffff812aacd7>] ? __make_request+0x5e/0x445
<Feb/01 11:56 am>[4661032.889363]  [<ffffffff812aacd7>] __make_request+0x5e/0x445
<Feb/01 11:56 am>[4661032.895506]  [<ffffffff812a946b>] generic_make_request+0x32a/0x39c
<Feb/01 11:56 am>[4661032.902303]  [<ffffffff810b33f2>] ? test_set_page_writeback+0x53/0x10e
<Feb/01 11:56 am>[4661032.909438]  [<ffffffff812a9591>] submit_bio+0xb4/0xbd
<Feb/01 11:56 am>[4661032.915118]  [<ffffffff810ac9a9>] ? unlock_page+0x22/0x26
<Feb/01 11:56 am>[4661032.921100]  [<ffffffff810d139c>] swap_writepage+0x8c/0x97
<Feb/01 11:56 am>[4661032.927150]  [<ffffffff810b7612>] shrink_page_list+0x3c2/0x68f
<Feb/01 11:56 am>[4661032.933591]  [<ffffffff810e5fb2>] ? mem_cgroup_del_lru_list+0x7b/0x8e
<Feb/01 11:56 am>[4661032.940649]  [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
<Feb/01 11:56 am>[4661032.946640]  [<ffffffff8102fc7b>] ? cpuacct_charge+0x76/0x96
<Feb/01 11:56 am>[4661032.952882]  [<ffffffff81058774>] ? up_read+0x1e/0x36
<Feb/01 11:56 am>[4661032.958488]  [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
<Feb/01 11:56 am>[4661032.965252]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
<Feb/01 11:56 am>[4661032.971611]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
<Feb/01 11:56 am>[4661032.978416]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
<Feb/01 11:56 am>[4661032.985377]  [<ffffffff810da90f>] alloc_page_vma+0x148/0x14d
<Feb/01 11:56 am>[4661032.991606]  [<ffffffff810d15c5>] read_swap_cache_async+0x5b/0xf2
<Feb/01 11:56 am>[4661032.998290]  [<ffffffff810d216d>] ? valid_swaphandles+0x6b/0xfd
<Feb/01 11:56 am>[4661033.004781]  [<ffffffff810d16b4>] swapin_readahead+0x58/0x97
<Feb/01 11:56 am>[4661033.011036]  [<ffffffff810c4ef1>] handle_mm_fault+0x45b/0x73e
<Feb/01 11:56 am>[4661033.017382]  [<ffffffff815227e9>] do_page_fault+0x351/0x375

<Feb/01 11:56 am>[4661033.023526]  [<ffffffff810348d8>] ? finish_task_switch+0x69/0xdd
<Feb/01 11:56 am>[4661033.030140]  [<ffffffff8103486f>] ? finish_task_switch+0x0/0xdd
<Feb/01 11:56 am>[4661033.036675]  [<ffffffff8151d152>] ? schedule+0x79e/0x86a
<Feb/01 11:56 am>[4661033.042549]  [<ffffffff8151ef6f>] ? lockdep_sys_exit_thunk+0x35/0x67
<Feb/01 11:56 am>[4661033.049513]  [<ffffffff810dce8c>] ? get_partial_node+0x1b/0x96
<Feb/01 11:56 am>[4661033.055908]  [<ffffffff8151fb5f>] page_fault+0x1f/0x30
<Feb/01 11:56 am>[4661033.061607] Code: fb 48 ff c8 c9 c3 55 48 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae e8 0f 31 89 c1 41 89 f1 66 66 90 0f ae e8 0f 31 <89> c6 48 89 f0 48 29 c8 48 39 f8 73 25 f3 90 65 44 8b 04 25 f0 
<Feb/01 11:56 am>[4661033.084720] Call Trace:
<Feb/01 11:56 am>[4661033.087643]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
<Feb/01 11:56 am>[4661033.095510]  [<ffffffff812c08fe>] __delay+0xa/0xc
<Feb/01 11:56 am>[4661033.100771]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
<Feb/01 11:56 am>[4661033.107103]  [<ffffffff8151f115>] _raw_spin_lock_irq+0x35/0x39
<Feb/01 11:56 am>[4661033.113535]  [<ffffffff812aacd7>] ? __make_request+0x5e/0x445
<Feb/01 11:56 am>[4661033.119869]  [<ffffffff812aacd7>] __make_request+0x5e/0x445
<Feb/01 11:56 am>[4661033.126005]  [<ffffffff812a946b>] generic_make_request+0x32a/0x39c
<Feb/01 11:56 am>[4661033.132768]  [<ffffffff810b33f2>] ? test_set_page_writeback+0x53/0x10e
<Feb/01 11:56 am>[4661033.139903]  [<ffffffff812a9591>] submit_bio+0xb4/0xbd
                 <Feb/01 11:56 am>[4661033.145598]  [<ffffffff810ac9a9>] ? unlock_page+0x22/0x26
<Feb/01 11:56 am>[4661033.156079]  [<ffffffff810d139c>] swap_writepage+0x8c/0x97
<Feb/01 11:56 am>[4661033.162161]  [<ffffffff810b7612>] shrink_page_list+0x3c2/0x68f
<Feb/01 11:56 am>[4661033.168569]  [<ffffffff810e5fb2>] ? mem_cgroup_del_lru_list+0x7b/0x8e
<Feb/01 11:56 am>[4661033.175617]  [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86

<Feb/01 11:56 am>[4661033.181593]  [<ffffffff8102fc7b>] ? cpuacct_charge+0x76/0x96
<Feb/01 11:56 am>[4661033.187852]  [<ffffffff81058774>] ? up_read+0x1e/0x36
<Feb/01 11:56 am>[4661033.193447]  [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
<Feb/01 11:56 am>[4661033.200221]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
<Feb/01 11:56 am>[4661033.206547]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
<Feb/01 11:56 am>[4661033.213309]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
<Feb/01 11:56 am>[4661033.220262]  [<ffffffff810da90f>] alloc_page_vma+0x148/0x14d
<Feb/01 11:56 am>[4661033.226499]  [<ffffffff810d15c5>] read_swap_cache_async+0x5b/0xf2
<Feb/01 11:56 am>[4661033.233190]  [<ffffffff810d216d>] ? valid_swaphandles+0x6b/0xfd
<Feb/01 11:56 am>[4661033.239707]  [<ffffffff810d16b4>] swapin_readahead+0x58/0x97
<Feb/01 11:56 am>[4661033.245964]  [<ffffffff810c4ef1>] handle_mm_fault+0x45b/0x73e
<Feb/01 11:56 am>[4661033.252308]  [<ffffffff815227e9>] do_page_fault+0x351/0x375
<Feb/01 11:56 am>[4661033.258462]  [<ffffffff810348d8>] ? finish_task_switch+0x69/0xdd
<Feb/01 11:56 am>[4661033.265067]  [<ffffffff8103486f>] ? finish_task_switch+0x0/0xdd
<Feb/01 11:56 am>[4661033.271584]  [<ffffffff8151d152>] ? schedule+0x79e/0x86a
<Feb/01 11:56 am>[4661033.277475]  [<ffffffff8151ef6f>] ? lockdep_sys_exit_thunk+0x35/0x67
<Feb/01 11:56 am>[4661033.284431]  [<ffffffff810dce8c>] ? get_partial_node+0x1b/0x96
<Feb/01 11:56 am>[4661033.290868]  [<ffffffff8151fb5f>] page_fault+0x1f/0x30
<Feb/01 11:56 am>[4661033.296594] Pid: 8141, comm: psphotStack Tainted: G      D    2.6.34.4 #3
<Feb/01 11:56 am>[4661033.304009] Call Trace:
<Feb/01 11:56 am>[4661033.306924]  <NMI>  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
<Feb/01 11:56 am>[4661033.313404]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5

<Feb/01 11:56 am>[4661033.319842]  [<ffffffff81520360>] do_nmi+0xb2/0x259
<Feb/01 11:56 am>[4661033.325272]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
<Feb/01 11:56 am>[4661033.330350]  [<ffffffff812c08af>] ? delay_tsc+0x25/0x59
<Feb/01 11:56 am>[4661033.336116]  <<EOE>>  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
<Feb/01 11:56 am>[4661033.344817]  [<ffffffff812c08fe>] __delay+0xa/0xc
<Feb/01 11:56 am>[4661033.350077]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
<Feb/01 11:56 am>[4661033.356390]  [<ffffffff8151f115>] _raw_spin_lock_irq+0x35/0x39
<Feb/01 11:56 am>[4661033.362804]  [<ffffffff812aacd7>] ? __make_request+0x5e/0x445
<Feb/01 11:56 am>[4661033.369087]  [<ffffffff812aacd7>] __make_request+0x5e/0x445
<Feb/01 11:56 am>[4661033.375258]  [<ffffffff812a946b>] generic_make_request+0x32a/0x39c
<Feb/01 11:56 am>[4661033.382046]  [<ffffffff810b33f2>] ? test_set_page_writeback+0x53/0x10e
<Feb/01 11:56 am>[4661033.389198]  [<ffffffff812a9591>] submit_bio+0xb4/0xbd
<Feb/01 11:56 am>[4661033.394895]  [<ffffffff810ac9a9>] ? unlock_page+0x22/0x26
<Feb/01 11:56 am>[4661033.400850]  [<ffffffff810d139c>] swap_writepage+0x8c/0x97
<Feb/01 11:56 am>[4661033.406927]  [<ffffffff810b7612>] shrink_page_list+0x3c2/0x68f
<Feb/01 11:56 am>[4661033.413369]  [<ffffffff810e5fb2>] ? mem_cgroup_del_lru_list+0x7b/0x8e
<Feb/01 11:56 am>[4661033.420442]  [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
<Feb/01 11:56 am>[4661033.426428]  [<ffffffff8102fc7b>] ? cpuacct_charge+0x76/0x96
<Feb/01 11:56 am>[4661033.432652]  [<ffffffff81058774>] ? up_read+0x1e/0x36
<Feb/01 11:56 am>[4661033.438232]  [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
<Feb/01 11:56 am>[4661033.445021]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
<Feb/01 11:56 am>[4661033.451346]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f

<Feb/01 11:56 am>[4661033.458133]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
<Feb/01 11:56 am>[4661033.465079]  [<ffffffff810da90f>] alloc_page_vma+0x148/0x14d
<Feb/01 11:56 am>[4661033.471331]  [<ffffffff810d15c5>] read_swap_cache_async+0x5b/0xf2
<Feb/01 11:56 am>[4661033.478005]  [<ffffffff810d216d>] ? valid_swaphandles+0x6b/0xfd
<Feb/01 11:56 am>[4661033.484524]  [<ffffffff810d16b4>] swapin_readahead+0x58/0x97
<Feb/01 11:56 am>[4661033.490763]  [<ffffffff810c4ef1>] handle_mm_fault+0x45b/0x73e
                 <Feb/01 11:56 am>[4661033.497116]  [<ffffffff815227e9>] do_page_fault+0x351/0x375
<Feb/01 11:56 am>[4661033.503286]  [<ffffffff810348d8>] ? finish_task_switch+0x69/0xdd
<Feb/01 11:56 am>[4661033.509909]  [<ffffffff8103486f>] ? finish_task_switch+0x0/0xdd
<Feb/01 11:56 am>[4661033.516427]  [<ffffffff8151d152>] ? schedule+0x79e/0x86a
<Feb/01 11:56 am>[4661033.522310]  [<ffffffff8151ef6f>] ? lockdep_sys_exit_thunk+0x35/0x67
<Feb/01 11:56 am>[4661033.529290]  [<ffffffff810dce8c>] ? get_partial_node+0x1b/0x96
<Feb/01 11:56 am>[4661033.535720]  [<ffffffff8151fb5f>] page_fault+0x1f/0x30
<Feb/01 11:56 am>[4661033.541436] NMI backtrace for cpu 7
<Feb/01 11:56 am>[4661033.545427] CPU 7 
<Feb/01 11:56 am>[4661033.547545] Modules linked in: w83793 hwmon_vid autofs4 i2c_i801 i2c_core iTCO_wdt e1000e ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
<Feb/01 11:56 am>[4661033.583688] 
<Feb/01 11:56 am>[4661033.585617] Pid: 0, comm: swapper Tainted: G      D    2.6.34.4 #3 S7010/empty
<Feb/01 11:56 am>[4661033.593636] RIP: 0010:[<ffffffff812c08b7>]  [<ffffffff812c08b7>] delay_tsc+0x2d/0x59
<Feb/01 11:56 am>[4661033.602273] RSP: 0018:ffff880655223b58  EFLAGS: 00000012
<Feb/01 11:56 am>[4661033.608165] RAX: 000000000000001c RBX: 0000000024530af5 RCX: 00000000e496e978
<Feb/01 11:56 am>[4661033.616101] RDX: 00000000002c28af RSI: 00000000e496e994 RDI: 0000000000000001

<Feb/01 11:56 am>[4661033.624066] RBP: ffff880655223b60 R08: 0000000000000007 R09: 0000000000000007
<Feb/01 11:56 am>[4661033.632019] R10: 0000000000000000 R11: ffff880655223990 R12: ffff8806552b3c00
<Feb/01 11:56 am>[4661033.639970] R13: 0000000000000000 R14: ffff880c24c8c0c0 R15: ffff880c24c8c548
<Feb/01 11:56 am>[4661033.647874] FS:  0000000000000000(0000) GS:ffff880655220000(0000) knlGS:0000000000000000
<Feb/01 11:56 am>[4661033.656821] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
<Feb/01 11:56 am>[4661033.663140] CR2: 00007ff195e94016 CR3: 0000000001a8a000 CR4: 00000000000006e0
<Feb/01 11:56 am>[4661033.671100] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<Feb/01 11:56 am>[4661033.679040] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<Feb/01 11:56 am>[4661033.686978] Process swapper (pid: 0, threadinfo ffff880624d38000, task ffff880c24c8c0c0)
<Feb/01 11:56 am>[4661033.695848] Stack:
<Feb/01 11:56 am>[4661033.698316]  ffffffff81019da9 ffff880655223b70 ffffffff812c08fe ffff880655223bb0
<Feb/01 11:56 am>[4661033.706321] <0> ffffffff812c4e65 000000009ef2dfac ffff8806552b3c00 ffff8806552b3c00
<Feb/01 11:56 am>[4661033.715074] <0> ffff88065522e3f0 00000000000004aa 0000000000001000 ffff880655223bd0
<Feb/01 11:56 am>[4661033.724205] Call Trace:
<Feb/01 11:56 am>[4661033.727118]  <IRQ> 
<Feb/01 11:56 am>[4661033.729709]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
<Feb/01 11:56 am>[4661033.737577]  [<ffffffff812c08fe>] __delay+0xa/0xc
<Feb/01 11:56 am>[4661033.742821]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
<Feb/01 11:56 am>[4661033.749169]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
<Feb/01 11:56 am>[4661033.755861]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76
<Feb/01 11:56 am>[4661033.762118]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
<Feb/01 11:56 am>[4661033.768201]  [<ffffffff81037672>] load_balance+0xbb6/0xf63

<Feb/01 11:56 am>[4661033.774258]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Feb/01 11:56 am>[4661033.780743]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Feb/01 11:56 am>[4661033.787165]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
                 <Feb/01 11:56 am>[4661033.793598]  [<ffffffff81047c8b>] ? run_timer_softirq+0x4d/0x30c
<Feb/01 11:56 am>[4661033.800192]  [<ffffffff81037af9>] rebalance_domains+0xda/0x150
<Feb/01 11:56 am>[4661033.806624]  [<ffffffff81037bb2>] run_rebalance_domains+0x43/0xcf
<Feb/01 11:56 am>[4661033.813316]  [<ffffffff8104103c>] __do_softirq+0xd8/0x194
<Feb/01 11:56 am>[4661033.819301]  [<ffffffff8100388c>] call_softirq+0x1c/0x28
<Feb/01 11:56 am>[4661033.825186]  [<ffffffff81004e7f>] do_softirq+0x33/0x68
<Feb/01 11:56 am>[4661033.830880]  [<ffffffff81040ec2>] irq_exit+0x36/0x75
<Feb/01 11:56 am>[4661033.836408]  [<ffffffff810197aa>] smp_apic_timer_interrupt+0x87/0x95
<Feb/01 11:56 am>[4661033.843364]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
<Feb/01 11:56 am>[4661033.849992]  <EOI> 
<Feb/01 11:56 am>[4661033.852599]  [<ffffffff8151cc4a>] ? schedule+0x296/0x86a
<Feb/01 11:56 am>[4661033.858466]  [<ffffffff81303870>] ? acpi_idle_enter_c1+0xdd/0xf8
<Feb/01 11:56 am>[4661033.865040]  [<ffffffff81303851>] ? acpi_idle_enter_c1+0xbe/0xf8
<Feb/01 11:56 am>[4661033.871608]  [<ffffffff81430151>] cpuidle_idle_call+0x95/0xf0
<Feb/01 11:56 am>[4661033.877919]  [<ffffffff81001d4b>] cpu_idle+0x55/0x8d
<Feb/01 11:56 am>[4661033.883444]  [<ffffffff815191ca>] start_secondary+0x1ae/0x1b2
<Feb/01 11:56 am>[4661033.889790] Code: 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae e8 0f 31 89 c1 41 89 f1 66 66 90 0f ae e8 0f 31 89 c6 48 89 f0 48 29 c8 <48> 39 f8 73 25 f3 90 65 44 8b 04 25 f0 d1 00 00 45 39 c1 74 db 
<Feb/01 11:56 am>[4661033.912946] Call Trace:
<Feb/01 11:56 am>[4661033.915849]  <IRQ>  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d

<Feb/01 11:56 am>[4661033.924346]  [<ffffffff812c08fe>] __delay+0xa/0xc
<Feb/01 11:56 am>[4661033.929603]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
<Feb/01 11:56 am>[4661033.935943]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
<Feb/01 11:56 am>[4661033.942626]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76
<Feb/01 11:56 am>[4661033.948857]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
<Feb/01 11:56 am>[4661033.954922]  [<ffffffff81037672>] load_balance+0xbb6/0xf63
<Feb/01 11:56 am>[4661033.961006]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Feb/01 11:56 am>[4661033.967412]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Feb/01 11:56 am>[4661033.973853]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Feb/01 11:56 am>[4661033.980257]  [<ffffffff81047c8b>] ? run_timer_softirq+0x4d/0x30c
<Feb/01 11:56 am>[4661033.986880]  [<ffffffff81037af9>] rebalance_domains+0xda/0x150
<Feb/01 11:56 am>[4661033.993319]  [<ffffffff81037bb2>] run_rebalance_domains+0x43/0xcf
<Feb/01 11:56 am>[4661034.000027]  [<ffffffff8104103c>] __do_softirq+0xd8/0x194
<Feb/01 11:56 am>[4661034.005997]  [<ffffffff8100388c>] call_softirq+0x1c/0x28
<Feb/01 11:56 am>[4661034.011838]  [<ffffffff81004e7f>] do_softirq+0x33/0x68
<Feb/01 11:56 am>[4661034.017558]  [<ffffffff81040ec2>] irq_exit+0x36/0x75
<Feb/01 11:56 am>[4661034.023086]  [<ffffffff810197aa>] smp_apic_timer_interrupt+0x87/0x95
<Feb/01 11:56 am>[4661034.030033]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
<Feb/01 11:56 am>[4661034.036636]  <EOI>  [<ffffffff8151cc4a>] ? schedule+0x296/0x86a
<Feb/01 11:56 am>[4661034.043199]  [<ffffffff81303870>] ? acpi_idle_enter_c1+0xdd/0xf8
<Feb/01 11:56 am>[4661034.049793]  [<ffffffff81303851>] ? acpi_idle_enter_c1+0xbe/0xf8
<Feb/01 11:56 am>[4661034.056397]  [<ffffffff81430151>] cpuidle_idle_call+0x95/0xf0

<Feb/01 11:56 am>[4661034.062707]  [<ffffffff81001d4b>] cpu_idle+0x55/0x8d
<Feb/01 11:56 am>[4661034.068189]  [<ffffffff815191ca>] start_secondary+0x1ae/0x1b2
<Feb/01 11:56 am>[4661034.074501] Pid: 0, comm: swapper Tainted: G      D    2.6.34.4 #3
<Feb/01 11:56 am>[4661034.081271] Call Trace:
<Feb/01 11:56 am>[4661034.084188]  <NMI>  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
<Feb/01 11:56 am>[4661034.090633]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
<Feb/01 11:56 am>[4661034.097071]  [<ffffffff81520360>] do_nmi+0xb2/0x259
<Feb/01 11:56 am>[4661034.102518]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
<Feb/01 11:56 am>[4661034.107597]  [<ffffffff812c08b7>] ? delay_tsc+0x2d/0x59
<Feb/01 11:56 am>[4661034.113379]  <<EOE>>  <IRQ>  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
<Feb/01 11:56 am>[4661034.122776]  [<ffffffff812c08fe>] __delay+0xa/0xc
<Feb/01 11:56 am>[4661034.128034]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
<Feb/01 11:56 am>[4661034.134382]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
<Feb/01 11:56 am>[4661034.141074]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76
<Feb/01 11:56 am>[4661034.147322]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
<Feb/01 11:56 am>[4661034.153379]  [<ffffffff81037672>] load_balance+0xbb6/0xf63
<Feb/01 11:56 am>[4661034.159464]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Feb/01 11:56 am>[4661034.165913]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Feb/01 11:56 am>[4661034.172335]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Feb/01 11:56 am>[4661034.183265]  [<ffffffff81047c8b>] ? run_timer_softirq+0x4d/0x30c
<Feb/01 11:56 am>[4661034.189843]  [<ffffffff81037af9>] rebalance_domains+0xda/0x150
<Feb/01 11:56 am>[4661034.196264]  [<ffffffff81037bb2>] run_rebalance_domains+0x43/0xcf

<Feb/01 11:56 am>[4661034.202946]  [<ffffffff8104103c>] __do_softirq+0xd8/0x194
<Feb/01 11:56 am>[4661034.208915]  [<ffffffff8100388c>] call_softirq+0x1c/0x28
<Feb/01 11:56 am>[4661034.214809]  [<ffffffff81004e7f>] do_softirq+0x33/0x68
<Feb/01 11:56 am>[4661034.220522]  [<ffffffff81040ec2>] irq_exit+0x36/0x75
<Feb/01 11:56 am>[4661034.226058]  [<ffffffff810197aa>] smp_apic_timer_interrupt+0x87/0x95
<Feb/01 11:56 am>[4661034.233023]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
<Feb/01 11:56 am>[4661034.239626]  <EOI>  [<ffffffff8151cc4a>] ? schedule+0x296/0x86a
<Feb/01 11:56 am>[4661034.246188]  [<ffffffff81303870>] ? acpi_idle_enter_c1+0xdd/0xf8
<Feb/01 11:56 am>[4661034.252792]  [<ffffffff81303851>] ? acpi_idle_enter_c1+0xbe/0xf8
<Feb/01 11:56 am>[4661034.259403]  [<ffffffff81430151>] cpuidle_idle_call+0x95/0xf0
<Feb/01 11:56 am>[4661034.265756]  [<ffffffff81001d4b>] cpu_idle+0x55/0x8d
<Feb/01 11:56 am>[4661034.271299]  [<ffffffff815191ca>] start_secondary+0x1ae/0x1b2
<Feb/01 11:56 am>[4661034.277637] NMI backtrace for cpu 1
                 <Feb/01 11:56 am>[4661034.281643] CPU 1 
<Feb/01 11:56 am>[4661034.283761] Modules linked in: w83793 hwmon_vid autofs4 i2c_i801 i2c_core iTCO_wdt e1000e ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
<Feb/01 11:56 am>[4661034.319807] 
<Feb/01 11:56 am>[4661034.321738] Pid: 29943, comm: pclient Tainted: G      D    2.6.34.4 #3 S7010/empty
<Feb/01 11:56 am>[4661034.330119] RIP: 0010:[<ffffffff812c08a4>]  [<ffffffff812c08a4>] delay_tsc+0x1a/0x59
<Feb/01 11:56 am>[4661034.338721] RSP: 0018:ffff880295cfd530  EFLAGS: 00000086
<Feb/01 11:56 am>[4661034.344588] RAX: 00000000abd8e664 RBX: 000000001fd84c96 RCX: 00000000abd8e664
<Feb/01 11:56 am>[4661034.352548] RDX: 00000000002c28ae RSI: 0000000000000001 RDI: 0000000000000001
<Feb/01 11:56 am>[4661034.360497] RBP: ffff880295cfd538 R08: 0000000000000001 R09: 0000000000000001

<Feb/01 11:56 am>[4661034.368470] R10: 0000000000000000 R11: ffff880295cfd368 R12: ffff8806552b3c00
<Feb/01 11:56 am>[4661034.376409] R13: 0000000000000000 R14: ffff8806239f50f0 R15: ffff8806239f5578
<Feb/01 11:56 am>[4661034.384348] FS:  00007fcb6b8b76f0(0000) GS:ffff880002620000(0000) knlGS:0000000000000000
<Feb/01 11:56 am>[4661034.393262] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<Feb/01 11:56 am>[4661034.399591] CR2: 000000000040fe90 CR3: 000000022edd0000 CR4: 00000000000006e0
<Feb/01 11:56 am>[4661034.407509] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<Feb/01 11:56 am>[4661034.415445] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<Feb/01 11:56 am>[4661034.423395] Process pclient (pid: 29943, threadinfo ffff880295cfc000, task ffff8806239f50f0)
<Feb/01 11:56 am>[4661034.432656] Stack:
<Feb/01 11:56 am>[4661034.435129]  ffffffff81019da9 ffff880295cfd548 ffffffff812c08fe ffff880295cfd588
<Feb/01 11:56 am>[4661034.443118] <0> ffffffff812c4e65 000000009ef2dfac ffff8806552b3c00 ffff8806552b3c00
<Feb/01 11:56 am>[4661034.451863] <0> ffff88000262e650 00000000000007fe 0000000000001000 ffff880295cfd5a8
<Feb/01 11:56 am>[4661034.460984] Call Trace:
<Feb/01 11:56 am>[4661034.463915]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
<Feb/01 11:56 am>[4661034.471765]  [<ffffffff812c08fe>] __delay+0xa/0xc
<Feb/01 11:56 am>[4661034.477036]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
<Feb/01 11:56 am>[4661034.483375]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
<Feb/01 11:56 am>[4661034.490067]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76
<Feb/01 11:56 am>[4661034.496314]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
<Feb/01 11:56 am>[4661034.502381]  [<ffffffff81037672>] load_balance+0xbb6/0xf63
<Feb/01 11:56 am>[4661034.508465]  [<ffffffff810cc693>] ? page_referenced_one+0xc9/0xe5
<Feb/01 11:56 am>[4661034.515139]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc

<Feb/01 11:56 am>[4661034.521554]  [<ffffffff8151cc0e>] schedule+0x25a/0x86a
<Feb/01 11:56 am>[4661034.527228]  [<ffffffff8151e2a0>] schedule_hrtimeout_range+0xce/0x103
<Feb/01 11:56 am>[4661034.534262]  [<ffffffff81057842>] ? hrtimer_wakeup+0x0/0x21
<Feb/01 11:56 am>[4661034.540386]  [<ffffffff810583de>] ? hrtimer_start_range_ns+0xf/0x11
<Feb/01 11:56 am>[4661034.547261]  [<ffffffff810f87a3>] poll_schedule_timeout+0x46/0x69
                 <Feb/01 11:56 am>[4661034.553934]  [<ffffffff810f0b80>] ? pipe_poll+0x2e/0x8b
<Feb/01 11:56 am>[4661034.559729]  [<ffffffff810f945d>] do_select+0x5ad/0x5f1
<Feb/01 11:56 am>[4661034.565509]  [<ffffffff810f8eb0>] ? do_select+0x0/0x5f1
<Feb/01 11:56 am>[4661034.571299]  [<ffffffff810f9994>] ? __pollwait+0x0/0xcf
<Feb/01 11:56 am>[4661034.577106]  [<ffffffff810f9a63>] ? pollwake+0x0/0x59
<Feb/01 11:56 am>[4661034.582714]  [<ffffffff8102fc93>] ? cpuacct_charge+0x8e/0x96
<Feb/01 11:56 am>[4661034.588939]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
<Feb/01 11:56 am>[4661034.595013]  [<ffffffff8103687a>] ? try_to_wake_up+0x330/0x342
<Feb/01 11:56 am>[4661034.601445]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Feb/01 11:56 am>[4661034.607876]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Feb/01 11:56 am>[4661034.614315]  [<ffffffff810f1c3b>] ? pipe_read+0x81/0x3d6
<Feb/01 11:56 am>[4661034.620216]  [<ffffffff810f94de>] ? core_sys_select+0x3d/0x295
<Feb/01 11:56 am>[4661034.626631]  [<ffffffff810f9688>] core_sys_select+0x1e7/0x295
<Feb/01 11:56 am>[4661034.632949]  [<ffffffff810f94de>] ? core_sys_select+0x3d/0x295
<Feb/01 11:56 am>[4661034.639399]  [<ffffffff810ea38f>] ? do_sync_read+0xc7/0x10d
<Feb/01 11:56 am>[4661034.645568]  [<ffffffff8105cc46>] ? ktime_get_ts+0xb1/0xbe
<Feb/01 11:56 am>[4661034.651652]  [<ffffffff810f996c>] sys_select+0x94/0xbc

<Feb/01 11:56 am>[4661034.657379]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
<Feb/01 11:56 am>[4661034.663995] Code: 1f 84 00 00 00 00 00 48 ff c8 75 fb 48 ff c8 c9 c3 55 48 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae e8 0f 31 89 c1 <41> 89 f1 66 66 90 0f ae e8 0f 31 89 c6 48 89 f0 48 29 c8 48 39 
<Feb/01 11:56 am>[4661034.687175] Call Trace:
                 <Feb/01 11:56 am>[4661034.690088]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
<Feb/01 11:56 am>[4661034.697964]  [<ffffffff812c08fe>] __delay+0xa/0xc
<Feb/01 11:56 am>[4661034.703218]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
<Feb/01 11:56 am>[4661034.709566]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
<Feb/01 11:56 am>[4661034.716248]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76
<Feb/01 11:56 am>[4661034.722505]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
<Feb/01 11:56 am>[4661034.728572]  [<ffffffff81037672>] load_balance+0xbb6/0xf63
<Feb/01 11:56 am>[4661034.734656]  [<ffffffff810cc693>] ? page_referenced_one+0xc9/0xe5
                 <Feb/01 11:56 am>[4661034.741357]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Feb/01 11:56 am>[4661034.747786]  [<ffffffff8151cc0e>] schedule+0x25a/0x86a
<Feb/01 11:56 am>[4661034.753496]  [<ffffffff8151e2a0>] schedule_hrtimeout_range+0xce/0x103
<Feb/01 11:56 am>[4661034.760564]  [<ffffffff81057842>] ? hrtimer_wakeup+0x0/0x21
<Feb/01 11:56 am>[4661034.766740]  [<ffffffff810583de>] ? hrtimer_start_range_ns+0xf/0x11
<Feb/01 11:56 am>[4661034.773573]  [<ffffffff810f87a3>] poll_schedule_timeout+0x46/0x69
<Feb/01 11:56 am>[4661034.780255]  [<ffffffff810f0b80>] ? pipe_poll+0x2e/0x8b
<Feb/01 11:56 am>[4661034.786058]  [<ffffffff810f945d>] do_select+0x5ad/0x5f1
<Feb/01 11:56 am>[4661034.791838]  [<ffffffff810f8eb0>] ? do_select+0x0/0x5f1
<Feb/01 11:56 am>[4661034.797629]  [<ffffffff810f9994>] ? __pollwait+0x0/0xcf
<Feb/01 11:56 am>[4661034.803426]  [<ffffffff810f9a63>] ? pollwake+0x0/0x59

<Feb/01 11:56 am>[4661034.809008]  [<ffffffff8102fc93>] ? cpuacct_charge+0x8e/0x96
<Feb/01 11:56 am>[4661034.815215]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
<Feb/01 11:56 am>[4661034.821300]  [<ffffffff8103687a>] ? try_to_wake_up+0x330/0x342
<Feb/01 11:56 am>[4661034.827748]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Feb/01 11:56 am>[4661034.834171]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Feb/01 11:56 am>[4661034.840577]  [<ffffffff810f1c3b>] ? pipe_read+0x81/0x3d6
<Feb/01 11:56 am>[4661034.846460]  [<ffffffff810f94de>] ? core_sys_select+0x3d/0x295
<Feb/01 11:56 am>[4661034.852874]  [<ffffffff810f9688>] core_sys_select+0x1e7/0x295
<Feb/01 11:56 am>[4661034.859192]  [<ffffffff810f94de>] ? core_sys_select+0x3d/0x295
<Feb/01 11:56 am>[4661034.865598]  [<ffffffff810ea38f>] ? do_sync_read+0xc7/0x10d
<Feb/01 11:56 am>[4661034.871767]  [<ffffffff8105cc46>] ? ktime_get_ts+0xb1/0xbe
                 <Feb/01 11:56 am>[4661034.877852]  [<ffffffff810f996c>] sys_select+0x94/0xbc
<Feb/01 11:56 am>[4661034.883563]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
<Feb/01 11:56 am>[4661034.890177] Pid: 29943, comm: pclient Tainted: G      D    2.6.34.4 #3
<Feb/01 11:56 am>[4661034.897319] Call Trace:
<Feb/01 11:56 am>[4661034.900244]  <NMI>  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
<Feb/01 11:56 am>[4661034.906706]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
<Feb/01 11:56 am>[4661034.913145]  [<ffffffff81520360>] do_nmi+0xb2/0x259
<Feb/01 11:56 am>[4661034.918591]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
<Feb/01 11:56 am>[4661034.923670]  [<ffffffff812c08a4>] ? delay_tsc+0x1a/0x59
<Feb/01 11:56 am>[4661034.929469]  <<EOE>>  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
<Feb/01 11:56 am>[4661034.938165]  [<ffffffff812c08fe>] __delay+0xa/0xc

<Feb/01 11:56 am>[4661034.943414]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
<Feb/01 11:56 am>[4661034.949736]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
<Feb/01 11:56 am>[4661034.956429]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76
<Feb/01 11:56 am>[4661034.962677]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
<Feb/01 11:56 am>[4661034.968751]  [<ffffffff81037672>] load_balance+0xbb6/0xf63
<Feb/01 11:56 am>[4661034.974837]  [<ffffffff810cc693>] ? page_referenced_one+0xc9/0xe5
<Feb/01 11:56 am>[4661034.981510]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Feb/01 11:56 am>[4661034.987924]  [<ffffffff8151cc0e>] schedule+0x25a/0x86a
<Feb/01 11:56 am>[4661034.993625]  [<ffffffff8151e2a0>] schedule_hrtimeout_range+0xce/0x103
<Feb/01 11:56 am>[4661035.000665]  [<ffffffff81057842>] ? hrtimer_wakeup+0x0/0x21
<Feb/01 11:56 am>[4661035.006808]  [<ffffffff810583de>] ? hrtimer_start_range_ns+0xf/0x11
<Feb/01 11:56 am>[4661035.013675]  [<ffffffff810f87a3>] poll_schedule_timeout+0x46/0x69
<Feb/01 11:56 am>[4661035.020356]  [<ffffffff810f0b80>] ? pipe_poll+0x2e/0x8b
<Feb/01 11:56 am>[4661035.026151]  [<ffffffff810f945d>] do_select+0x5ad/0x5f1
<Feb/01 11:56 am>[4661035.031949]  [<ffffffff810f8eb0>] ? do_select+0x0/0x5f1
<Feb/01 11:56 am>[4661035.037757]  [<ffffffff810f9994>] ? __pollwait+0x0/0xcf
<Feb/01 11:56 am>[4661035.043553]  [<ffffffff810f9a63>] ? pollwake+0x0/0x59
<Feb/01 11:56 am>[4661035.049161]  [<ffffffff8102fc93>] ? cpuacct_charge+0x8e/0x96
<Feb/01 11:56 am>[4661035.055386]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
<Feb/01 11:56 am>[4661035.061427]  [<ffffffff8103687a>] ? try_to_wake_up+0x330/0x342
<Feb/01 11:56 am>[4661035.067842]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Feb/01 11:56 am>[4661035.074237]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc

<Feb/01 11:56 am>[4661035.080668]  [<ffffffff810f1c3b>] ? pipe_read+0x81/0x3d6
<Feb/01 11:56 am>[4661035.086560]  [<ffffffff810f94de>] ? core_sys_select+0x3d/0x295
<Feb/01 11:56 am>[4661035.092994]  [<ffffffff810f9688>] core_sys_select+0x1e7/0x295
<Feb/01 11:56 am>[4661035.099327]  [<ffffffff810f94de>] ? core_sys_select+0x3d/0x295
<Feb/01 11:56 am>[4661035.105750]  [<ffffffff810ea38f>] ? do_sync_read+0xc7/0x10d
<Feb/01 11:56 am>[4661035.111913]  [<ffffffff8105cc46>] ? ktime_get_ts+0xb1/0xbe
<Feb/01 11:56 am>[4661035.117961]  [<ffffffff810f996c>] sys_select+0x94/0xbc
<Feb/01 11:56 am>[4661035.123690]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
<Feb/01 11:56 am>[4661035.130312] NMI backtrace for cpu 10
<Feb/01 11:56 am>[4661035.134378] CPU 10 
<Feb/01 11:56 am>[4661035.136574] Modules linked in: w83793 hwmon_vid autofs4 i2c_i801 i2c_core iTCO_wdt e1000e ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
<Feb/01 11:56 am>[4661035.172716] 
<Feb/01 11:56 am>[4661035.174647] Pid: 5694, comm: ppStack Tainted: G      D    2.6.34.4 #3 S7010/empty
<Feb/01 11:56 am>[4661035.182935] RIP: 0010:[<ffffffff812c08e1>]  [<ffffffff812c08e1>] delay_tsc+0x57/0x59
<Feb/01 11:56 am>[4661035.191536] RSP: 0018:ffff88012129b840  EFLAGS: 00000006
<Feb/01 11:56 am>[4661035.201907] RAX: 000000000000001c RBX: 0000000024529cb0 RCX: 00000000e4a3215c
<Feb/01 11:56 am>[4661035.209852] RDX: 00000000002c28af RSI: 00000000e4a32178 RDI: 0000000000000001
<Feb/01 11:56 am>[4661035.217774] RBP: ffff88012129b848 R08: 000000000000000a R09: 000000000000000a
<Feb/01 11:56 am>[4661035.225695] R10: 0000000000000000 R11: ffff88012129b678 R12: ffff8806552b3c00
<Feb/01 11:56 am>[4661035.233599] R13: 0000000000000000 R14: ffff88052a5140c0 R15: ffff88052a514548
<Feb/01 11:56 am>[4661035.241547] FS:  0000000042747950(0063) GS:ffff880655280000(0000) knlGS:0000000000000000
<Feb/01 11:56 am>[4661035.250477] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033

<Feb/01 11:56 am>[4661035.256806] CR2: 00007ff195f55000 CR3: 000000043b231000 CR4: 00000000000006e0
<Feb/01 11:56 am>[4661035.264738] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<Feb/01 11:56 am>[4661035.272660] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<Feb/01 11:56 am>[4661035.280582] Process ppStack (pid: 5694, threadinfo ffff88012129a000, task ffff88052a5140c0)
<Feb/01 11:56 am>[4661035.289765] Stack:
<Feb/01 11:56 am>[4661035.292215]  ffffffff81019da9 ffff88012129b858 ffffffff812c08fe ffff88012129b898
<Feb/01 11:56 am>[4661035.300213] <0> ffffffff812c4e65 000000009ef2dfac ffff8806552b3c00 ffff8806552b3c00
<Feb/01 11:56 am>[4661035.308940] <0> ffff88065528e3f0 0000000000000400 0000000000001000 ffff88012129b8b8
<Feb/01 11:56 am>[4661035.318043] Call Trace:
<Feb/01 11:56 am>[4661035.320941]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
<Feb/01 11:56 am>[4661035.328763]  [<ffffffff812c08fe>] __delay+0xa/0xc
<Feb/01 11:56 am>[4661035.334025]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
<Feb/01 11:56 am>[4661035.340356]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
<Feb/01 11:56 am>[4661035.347065]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76
<Feb/01 11:56 am>[4661035.353297]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
<Feb/01 11:56 am>[4661035.359361]  [<ffffffff81037672>] load_balance+0xbb6/0xf63
<Feb/01 11:56 am>[4661035.365437]  [<ffffffff812b59f1>] ? cfq_set_request+0x313/0x36b
<Feb/01 11:56 am>[4661035.371948]  [<ffffffff8151cc0e>] schedule+0x25a/0x86a
<Feb/01 11:56 am>[4661035.377639]  [<ffffffff81066393>] futex_wait_queue_me+0xbb/0xd6
<Feb/01 11:56 am>[4661035.384151]  [<ffffffff810664f1>] futex_wait+0x143/0x2b1
<Feb/01 11:56 am>[4661035.390034]  [<ffffffff8151f67a>] ? _raw_spin_unlock+0x26/0x2b
<Feb/01 11:56 am>[4661035.396459]  [<ffffffff8106675f>] ? futex_wake+0x60/0xf2

<Feb/01 11:56 am>[4661035.402342]  [<ffffffff810667e0>] ? futex_wake+0xe1/0xf2
<Feb/01 11:56 am>[4661035.408244]  [<ffffffff81068312>] do_futex+0x9e/0x9e3
<Feb/01 11:56 am>[4661035.413849]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Feb/01 11:56 am>[4661035.420267]  [<ffffffff8102fc7b>] ? cpuacct_charge+0x76/0x96
<Feb/01 11:56 am>[4661035.426516]  [<ffffffff810348d8>] ? finish_task_switch+0x69/0xdd
<Feb/01 11:56 am>[4661035.433104]  [<ffffffff8151f61f>] ? _raw_spin_unlock_irq+0x26/0x2c
<Feb/01 11:56 am>[4661035.439881]  [<ffffffff810348d8>] ? finish_task_switch+0x69/0xdd
<Feb/01 11:56 am>[4661035.446476]  [<ffffffff8103486f>] ? finish_task_switch+0x0/0xdd
<Feb/01 11:56 am>[4661035.452976]  [<ffffffff8151d152>] ? schedule+0x79e/0x86a
<Feb/01 11:56 am>[4661035.458868]  [<ffffffff81068d65>] sys_futex+0x10e/0x12c
<Feb/01 11:56 am>[4661035.464665]  [<ffffffff8151ef6f>] ? lockdep_sys_exit_thunk+0x35/0x67
<Feb/01 11:56 am>[4661035.471636]  [<ffffffff8151ef6f>] ? lockdep_sys_exit_thunk+0x35/0x67
<Feb/01 11:56 am>[4661035.478614]  [<ffffffff8106675f>] ? futex_wake+0x60/0xf2
<Feb/01 11:56 am>[4661035.484486]  [<ffffffff8106675f>] ? futex_wake+0x60/0xf2
<Feb/01 11:56 am>[4661035.490354]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
<Feb/01 11:56 am>[4661035.496976] Code: c8 48 39 f8 73 25 f3 90 65 44 8b 04 25 f0 d1 00 00 45 39 c1 74 db 48 29 f1 48 01 cf 66 66 90 0f ae e8 0f 31 89 c1 45 89 c1 eb c6 <c9> c3 55 48 c7 05 71 4c 80 00 8a 08 2c 81 48 89 e5 c9 c3 55 48 
<Feb/01 11:56 am>[4661035.520089] Call Trace:
<Feb/01 11:56 am>[4661035.523010]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
<Feb/01 11:56 am>[4661035.530877]  [<ffffffff812c08fe>] __delay+0xa/0xc
<Feb/01 11:56 am>[4661035.536114]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
<Feb/01 11:56 am>[4661035.542436]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
<Feb/01 11:56 am>[4661035.549129]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76

<Feb/01 11:56 am>[4661035.555366]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
<Feb/01 11:56 am>[4661035.561424]  [<ffffffff81037672>] load_balance+0xbb6/0xf63
<Feb/01 11:56 am>[4661035.567508]  [<ffffffff812b59f1>] ? cfq_set_request+0x313/0x36b
<Feb/01 11:56 am>[4661035.574011]  [<ffffffff8151cc0e>] schedule+0x25a/0x86a
<Feb/01 11:56 am>[4661035.579711]  [<ffffffff81066393>] futex_wait_queue_me+0xbb/0xd6
<Feb/01 11:56 am>[4661035.586212]  [<ffffffff810664f1>] futex_wait+0x143/0x2b1
<Feb/01 11:56 am>[4661035.592088]  [<ffffffff8151f67a>] ? _raw_spin_unlock+0x26/0x2b
                 <Feb/01 11:56 am>[4661035.598521]  [<ffffffff8106675f>] ? futex_wake+0x60/0xf2
<Feb/01 11:56 am>[4661035.604412]  [<ffffffff810667e0>] ? futex_wake+0xe1/0xf2
<Feb/01 11:56 am>[4661035.610280]  [<ffffffff81068312>] do_futex+0x9e/0x9e3
<Feb/01 11:56 am>[4661035.615886]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Feb/01 11:56 am>[4661035.622318]  [<ffffffff8102fc7b>] ? cpuacct_charge+0x76/0x96
<Feb/01 11:56 am>[4661035.628542]  [<ffffffff810348d8>] ? finish_task_switch+0x69/0xdd
<Feb/01 11:56 am>[4661035.635147]  [<ffffffff8151f61f>] ? _raw_spin_unlock_irq+0x26/0x2c
                 <Feb/01 11:56 am>[4661035.641926]  [<ffffffff810348d8>] ? finish_task_switch+0x69/0xdd
<Feb/01 11:56 am>[4661035.648520]  [<ffffffff8103486f>] ? finish_task_switch+0x0/0xdd
<Feb/01 11:56 am>[4661035.655037]  [<ffffffff8151d152>] ? schedule+0x79e/0x86a
<Feb/01 11:56 am>[4661035.660929]  [<ffffffff81068d65>] sys_futex+0x10e/0x12c
<Feb/01 11:56 am>[4661035.666710]  [<ffffffff8151ef6f>] ? lockdep_sys_exit_thunk+0x35/0x67
<Feb/01 11:56 am>[4661035.673689]  [<ffffffff8151ef6f>] ? lockdep_sys_exit_thunk+0x35/0x67
<Feb/01 11:56 am>[4661035.680632]  [<ffffffff8106675f>] ? futex_wake+0x60/0xf2
<Feb/01 11:56 am>[4661035.686531]  [<ffffffff8106675f>] ? futex_wake+0x60/0xf2

<Feb/01 11:56 am>[4661035.692417]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
<Feb/01 11:56 am>[4661035.699024] Pid: 5694, comm: ppStack Tainted: G      D    2.6.34.4 #3
<Feb/01 11:56 am>[4661035.706095] Call Trace:
<Feb/01 11:56 am>[4661035.709020]  <NMI>  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
<Feb/01 11:56 am>[4661035.715499]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
<Feb/01 11:56 am>[4661035.721929]  [<ffffffff81520360>] do_nmi+0xb2/0x259
<Feb/01 11:56 am>[4661035.727368]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
<Feb/01 11:56 am>[4661035.732447]  [<ffffffff812c08e1>] ? delay_tsc+0x57/0x59
<Feb/01 11:56 am>[4661035.738246]  <<EOE>>  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
<Feb/01 11:56 am>[4661035.746975]  [<ffffffff812c08fe>] __delay+0xa/0xc
<Feb/01 11:56 am>[4661035.752243]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
<Feb/01 11:56 am>[4661035.758572]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
<Feb/01 11:56 am>[4661035.765258]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76
<Feb/01 11:56 am>[4661035.771479]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
<Feb/01 11:56 am>[4661035.777544]  [<ffffffff81037672>] load_balance+0xbb6/0xf63
<Feb/01 11:56 am>[4661035.783621]  [<ffffffff812b59f1>] ? cfq_set_request+0x313/0x36b
<Feb/01 11:56 am>[4661035.790131]  [<ffffffff8151cc0e>] schedule+0x25a/0x86a
<Feb/01 11:56 am>[4661035.795856]  [<ffffffff81066393>] futex_wait_queue_me+0xbb/0xd6
<Feb/01 11:56 am>[4661035.802341]  [<ffffffff810664f1>] futex_wait+0x143/0x2b1
<Feb/01 11:56 am>[4661035.808215]  [<ffffffff8151f67a>] ? _raw_spin_unlock+0x26/0x2b
<Feb/01 11:56 am>[4661035.814620]  [<ffffffff8106675f>] ? futex_wake+0x60/0xf2
<Feb/01 11:56 am>[4661035.820489]  [<ffffffff810667e0>] ? futex_wake+0xe1/0xf2

<Feb/01 11:56 am>[4661035.826390]  [<ffffffff81068312>] do_futex+0x9e/0x9e3
<Feb/01 11:56 am>[4661035.832022]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Feb/01 11:56 am>[4661035.838430]  [<ffffffff8102fc7b>] ? cpuacct_charge+0x76/0x96
<Feb/01 11:56 am>[4661035.844670]  [<ffffffff810348d8>] ? finish_task_switch+0x69/0xdd
<Feb/01 11:56 am>[4661035.851285]  [<ffffffff8151f61f>] ? _raw_spin_unlock_irq+0x26/0x2c
<Feb/01 11:56 am>[4661035.858054]  [<ffffffff810348d8>] ? finish_task_switch+0x69/0xdd
<Feb/01 11:56 am>[4661035.864674]  [<ffffffff8103486f>] ? finish_task_switch+0x0/0xdd
<Feb/01 11:56 am>[4661035.871165]  [<ffffffff8151d152>] ? schedule+0x79e/0x86a
<Feb/01 11:56 am>[4661035.877024]  [<ffffffff81068d65>] sys_futex+0x10e/0x12c
<Feb/01 11:56 am>[4661035.882829]  [<ffffffff8151ef6f>] ? lockdep_sys_exit_thunk+0x35/0x67
<Feb/01 11:56 am>[4661035.889793]  [<ffffffff8151ef6f>] ? lockdep_sys_exit_thunk+0x35/0x67
<Feb/01 11:56 am>[4661035.896770]  [<ffffffff8106675f>] ? futex_wake+0x60/0xf2
<Feb/01 11:56 am>[4661035.902660]  [<ffffffff8106675f>] ? futex_wake+0x60/0xf2
<Feb/01 11:56 am>[4661035.908536]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
<Feb/01 11:56 am>[4661035.915141] NMI backtrace for cpu 11
<Feb/01 11:56 am>[4661035.919217] CPU 11 
<Feb/01 11:56 am>[4661035.921412] Modules linked in: w83793 hwmon_vid autofs4 i2c_i801 i2c_core iTCO_wdt e1000e ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c10<Feb/01 11:56 am>
                 <Feb/01 11:56 am>[4661035.957511] 
<Feb/01 11:56 am>[4661035.959449] Pid: 8467, comm: nfsd Tainted: G      D    2.6.34.4 #3 S7010/empty
<Feb/01 11:56 am>[4661035.967493] RIP: 0010:[<ffffffff812c08ad>]  [<ffffffff812c08ad>] delay_tsc+0x23/0x59
<Feb/01 11:56 am>[4661035.976104] RSP: 0018:ffff88061def13e8  EFLAGS: 00000096
<Feb/01 11:56 am>[4661035.982006] RAX: 00000000e4a73418 RBX: 000000002453a7ea RCX: 00000000e4a73418

<Feb/01 11:56 am>[4661035.989950] RDX: 00000000002c28af RSI: 000000000000000b RDI: 0000000000000001
<Feb/01 11:56 am>[4661035.997897] RBP: ffff88061def13f0 R08: 000000000000000b R09: 000000000000000b
<Feb/01 11:56 am>[4661036.005819] R10: 0000000000000000 R11: ffff88061def1220 R12: ffff880002673c00
<Feb/01 11:56 am>[4661036.013768] R13: 0000000000000000 R14: ffff88061e3d3090 R15: ffff88061e3d3518
<Feb/01 11:56 am>[4661036.021732] FS:  0000000000000000(0000) GS:ffff8806552a0000(0000) knlGS:0000000000000000
<Feb/01 11:56 am>[4661036.030662] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
<Feb/01 11:56 am>[4661036.036998] CR2: 00007f53a10add5c CR3: 0000000001a8a000 CR4: 00000000000006e0
<Feb/01 11:56 am>[4661036.044950] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<Feb/01 11:56 am>[4661036.052889] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<Feb/01 11:56 am>[4661036.060845] Process nfsd (pid: 8467, threadinfo ffff88061def0000, task ffff88061e3d3090)
<Feb/01 11:56 am>[4661036.069749] Stack:
<Feb/01 11:56 am>[4661036.072215]  ffffffff81019da9 ffff88061def1400 ffffffff812c08fe ffff88061def1440
<Feb/01 11:56 am>[4661036.080240] <0> ffffffff812c4e65 000000009ef2dfac ffff880002673c00 ffff880623bb2060
<Feb/01 11:56 am>[4661036.088951] <0> 0000000000013c00 ffff88061def14b0 000000000000000b ffff88061def1460
<Feb/01 11:56 am>[4661036.098055] Call Trace:
<Feb/01 11:56 am>[4661036.100976]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
<Feb/01 11:56 am>[4661036.108843]  [<ffffffff812c08fe>] __delay+0xa/0xc
<Feb/01 11:56 am>[4661036.114106]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
<Feb/01 11:56 am>[4661036.120428]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
<Feb/01 11:56 am>[4661036.126511]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
<Feb/01 11:56 am>[4661036.132568]  [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
<Feb/01 11:56 am>[4661036.138419]  [<ffffffff81036578>] try_to_wake_up+0x2e/0x342

<Feb/01 11:56 am>[4661036.144547]  [<ffffffff810368b8>] wake_up_process+0x10/0x12
<Feb/01 11:56 am>[4661036.150719]  [<ffffffff8110649a>] bdi_queue_work+0xa2/0xb9
<Feb/01 11:56 am>[4661036.156784]  [<ffffffff811068ae>] bdi_alloc_queue_work+0xa4/0xc4
<Feb/01 11:56 am>[4661036.167888]  [<ffffffff81106a1f>] wakeup_flusher_threads+0x93/0xc6
<Feb/01 11:56 am>[4661036.174649]  [<ffffffff811069bf>] ? wakeup_flusher_threads+0x33/0xc6
<Feb/01 11:56 am>[4661036.181600]  [<ffffffff810b9854>] do_try_to_free_pages+0x299/0x37a
<Feb/01 11:56 am>[4661036.188369]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
<Feb/01 11:56 am>[4661036.194705]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
<Feb/01 11:56 am>[4661036.201491]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
<Feb/01 11:56 am>[4661036.208426]  [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
<Feb/01 11:56 am>[4661036.214932]  [<ffffffff810acb60>] __page_cache_alloc+0x59/0x5e
<Feb/01 11:56 am>[4661036.221338]  [<ffffffff810b4860>] __do_page_cache_readahead+0xe5/0x1fc
<Feb/01 11:56 am>[4661036.228481]  [<ffffffff810b47f6>] ? __do_page_cache_readahead+0x7b/0x1fc
<Feb/01 11:56 am>[4661036.235779]  [<ffffffff812bc23c>] ? radix_tree_prev_hole+0x30/0x53
<Feb/01 11:56 am>[4661036.242547]  [<ffffffff810b4c40>] ondemand_readahead+0x1cc/0x256
<Feb/01 11:56 am>[4661036.249167]  [<ffffffff810b4ba0>] ? ondemand_readahead+0x12c/0x256
<Feb/01 11:56 am>[4661036.255929]  [<ffffffff810b3c7a>] ? determine_dirtyable_memory+0x15/0x27
<Feb/01 11:56 am>[4661036.263254]  [<ffffffff810b4daa>] page_cache_sync_readahead+0x38/0x3a
<Feb/01 11:56 am>[4661036.270290]  [<ffffffff81109bf2>] __generic_file_splice_read+0xfa/0x41c
<Feb/01 11:56 am>[4661036.277519]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Feb/01 11:56 am>[4661036.283956]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Feb/01 11:56 am>[4661036.290301]  [<ffffffff812615b1>] ? xfs_ilock+0x49/0xb2

<Feb/01 11:56 am>[4661036.296080]  [<ffffffff811086db>] ? spd_release_page+0x0/0x14
<Feb/01 11:56 am>[4661036.302416]  [<ffffffff81109f57>] generic_file_splice_read+0x43/0x74
<Feb/01 11:56 am>[4661036.309396]  [<ffffffff8128191a>] xfs_file_splice_read+0xde/0x125
<Feb/01 11:56 am>[4661036.316094]  [<ffffffff8110875d>] vfs_splice_to+0x6e/0x81
                 <Feb/01 11:56 am>[4661036.322072]  [<ffffffff81108837>] splice_direct_to_actor+0xc7/0x19c
<Feb/01 11:56 am>[4661036.328964]  [<ffffffff811f562c>] ? nfsd_direct_splice_actor+0x0/0x12
<Feb/01 11:56 am>[4661036.336012]  [<ffffffff811f54ef>] nfsd_vfs_read+0x263/0x3a0
<Feb/01 11:56 am>[4661036.342171]  [<ffffffff81281760>] ? xfs_file_open+0x0/0x3f
<Feb/01 11:56 am>[4661036.348256]  [<ffffffff811f5c4e>] nfsd_read+0xa7/0xc3
<Feb/01 11:56 am>[4661036.353887]  [<ffffffff811fcd79>] nfsd3_proc_read+0xfe/0x142
<Feb/01 11:56 am>[4661036.360112]  [<ffffffff811f0854>] nfsd_dispatch+0xe9/0x1ca
<Feb/01 11:56 am>[4661036.366179]  [<ffffffff814f04c5>] svc_process+0x425/0x72a
<Feb/01 11:56 am>[4661036.372140]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
<Feb/01 11:56 am>[4661036.377478]  [<ffffffff811f0dc2>] nfsd+0xf3/0x139
<Feb/01 11:56 am>[4661036.382721]  [<ffffffff810546e2>] kthread+0x7d/0x85
<Feb/01 11:56 am>[4661036.388164]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
<Feb/01 11:56 am>[4661036.394694]  [<ffffffff81054665>] ? kthread+0x0/0x85
<Feb/01 11:56 am>[4661036.400211]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
<Feb/01 11:56 am>[4661036.406898] Code: c8 75 fb 48 ff c8 c9 c3 55 48 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae e8 0f 31 89 c1 41 89 f1 66 66 90 0f ae e8 <0f> 31 89 c6 48 89 f0 48 29 c8 48 39 f8 73 25 f3 90 65 44 8b 04 
<Feb/01 11:56 am>[4661036.429990] Call Trace:
<Feb/01 11:56 am>[4661036.432904]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
<Feb/01 11:56 am>[4661036.440771]  [<ffffffff812c08fe>] __delay+0xa/0xc

<Feb/01 11:56 am>[4661036.446041]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
<Feb/01 11:56 am>[4661036.452365]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
<Feb/01 11:56 am>[4661036.458438]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
<Feb/01 11:56 am>[4661036.464480]  [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
<Feb/01 11:56 am>[4661036.470356]  [<ffffffff81036578>] try_to_wake_up+0x2e/0x342
<Feb/01 11:56 am>[4661036.476528]  [<ffffffff810368b8>] wake_up_process+0x10/0x12
<Feb/01 11:56 am>[4661036.482680]  [<ffffffff8110649a>] bdi_queue_work+0xa2/0xb9
<Feb/01 11:56 am>[4661036.488730]  [<ffffffff811068ae>] bdi_alloc_queue_work+0xa4/0xc4
<Feb/01 11:56 am>[4661036.495301]  [<ffffffff81106a1f>] wakeup_flusher_threads+0x93/0xc6
<Feb/01 11:56 am>[4661036.502097]  [<ffffffff811069bf>] ? wakeup_flusher_threads+0x33/0xc6
<Feb/01 11:56 am>[4661036.509074]  [<ffffffff810b9854>] do_try_to_free_pages+0x299/0x37a
<Feb/01 11:56 am>[4661036.515859]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
<Feb/01 11:56 am>[4661036.522202]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
<Feb/01 11:56 am>[4661036.528990]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
<Feb/01 11:56 am>[4661036.535968]  [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
<Feb/01 11:56 am>[4661036.542483]  [<ffffffff810acb60>] __page_cache_alloc+0x59/0x5e
<Feb/01 11:56 am>[4661036.548898]  [<ffffffff810b4860>] __do_page_cache_readahead+0xe5/0x1fc
<Feb/01 11:56 am>[4661036.556041]  [<ffffffff810b47f6>] ? __do_page_cache_readahead+0x7b/0x1fc
<Feb/01 11:56 am>[4661036.563358]  [<ffffffff812bc23c>] ? radix_tree_prev_hole+0x30/0x53
<Feb/01 11:56 am>[4661036.570142]  [<ffffffff810b4c40>] ondemand_readahead+0x1cc/0x256
<Feb/01 11:56 am>[4661036.576736]  [<ffffffff810b4ba0>] ? ondemand_readahead+0x12c/0x256
<Feb/01 11:56 am>[4661036.583523]  [<ffffffff810b3c7a>] ? determine_dirtyable_memory+0x15/0x27

<Feb/01 11:56 am>[4661036.590839]  [<ffffffff810b4daa>] page_cache_sync_readahead+0x38/0x3a
<Feb/01 11:56 am>[4661036.597885]  [<ffffffff81109bf2>] __generic_file_splice_read+0xfa/0x41c
<Feb/01 11:56 am>[4661036.605131]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Feb/01 11:56 am>[4661036.611551]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Feb/01 11:56 am>[4661036.617974]  [<ffffffff812615b1>] ? xfs_ilock+0x49/0xb2
<Feb/01 11:56 am>[4661036.623769]  [<ffffffff811086db>] ? spd_release_page+0x0/0x14
<Feb/01 11:56 am>[4661036.630097]  [<ffffffff81109f57>] generic_file_splice_read+0x43/0x74
<Feb/01 11:56 am>[4661036.637069]  [<ffffffff8128191a>] xfs_file_splice_read+0xde/0x125
<Feb/01 11:56 am>[4661036.643757]  [<ffffffff8110875d>] vfs_splice_to+0x6e/0x81
                 <Feb/01 11:56 am>[4661036.649728]  [<ffffffff81108837>] splice_direct_to_actor+0xc7/0x19c
<Feb/01 11:56 am>[4661036.656585]  [<ffffffff811f562c>] ? nfsd_direct_splice_actor+0x0/0x12
<Feb/01 11:56 am>[4661036.663643]  [<ffffffff811f54ef>] nfsd_vfs_read+0x263/0x3a0
<Feb/01 11:56 am>[4661036.669783]  [<ffffffff81281760>] ? xfs_file_open+0x0/0x3f
<Feb/01 11:56 am>[4661036.675849]  [<ffffffff811f5c4e>] nfsd_read+0xa7/0xc3
<Feb/01 11:56 am>[4661036.681473]  [<ffffffff811fcd79>] nfsd3_proc_read+0xfe/0x142
<Feb/01 11:56 am>[4661036.687714]  [<ffffffff811f0854>] nfsd_dispatch+0xe9/0x1ca
<Feb/01 11:56 am>[4661036.693781]  [<ffffffff814f04c5>] svc_process+0x425/0x72a
<Feb/01 11:56 am>[4661036.699762]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
<Feb/01 11:56 am>[4661036.705107]  [<ffffffff811f0dc2>] nfsd+0xf3/0x139
<Feb/01 11:56 am>[4661036.710360]  [<ffffffff810546e2>] kthread+0x7d/0x85
<Feb/01 11:56 am>[4661036.715776]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
<Feb/01 11:56 am>[4661036.722254]  [<ffffffff81054665>] ? kthread+0x0/0x85

<Feb/01 11:56 am>[4661036.727754]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
<Feb/01 11:56 am>[4661036.734440] Pid: 8467, comm: nfsd Tainted: G      D    2.6.34.4 #3
<Feb/01 11:56 am>[4661036.741226] Call Trace:
<Feb/01 11:56 am>[4661036.744159]  <NMI>  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
<Feb/01 11:56 am>[4661036.750622]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
<Feb/01 11:56 am>[4661036.757026]  [<ffffffff81520360>] do_nmi+0xb2/0x259
<Feb/01 11:56 am>[4661036.762457]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
<Feb/01 11:56 am>[4661036.767526]  [<ffffffff812c08ad>] ? delay_tsc+0x23/0x59
<Feb/01 11:56 am>[4661036.773325]  <<EOE>>  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
<Feb/01 11:56 am>[4661036.782038]  [<ffffffff812c08fe>] __delay+0xa/0xc
<Feb/01 11:56 am>[4661036.787304]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
<Feb/01 11:56 am>[4661036.793600]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
<Feb/01 11:56 am>[4661036.799657]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
<Feb/01 11:56 am>[4661036.805733]  [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
<Feb/01 11:56 am>[4661036.811618]  [<ffffffff81036578>] try_to_wake_up+0x2e/0x342
<Feb/01 11:56 am>[4661036.817779]  [<ffffffff810368b8>] wake_up_process+0x10/0x12
<Feb/01 11:56 am>[4661036.823933]  [<ffffffff8110649a>] bdi_queue_work+0xa2/0xb9
<Feb/01 11:56 am>[4661036.829992]  [<ffffffff811068ae>] bdi_alloc_queue_work+0xa4/0xc4
<Feb/01 11:56 am>[4661036.836589]  [<ffffffff81106a1f>] wakeup_flusher_threads+0x93/0xc6
<Feb/01 11:56 am>[4661036.843376]  [<ffffffff811069bf>] ? wakeup_flusher_threads+0x33/0xc6
<Feb/01 11:56 am>[4661036.850327]  [<ffffffff810b9854>] do_try_to_free_pages+0x299/0x37a
<Feb/01 11:56 am>[4661036.857104]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78

<Feb/01 11:56 am>[4661036.863429]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
<Feb/01 11:56 am>[4661036.870201]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
<Feb/01 11:56 am>[4661036.877161]  [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
<Feb/01 11:56 am>[4661036.883669]  [<ffffffff810acb60>] __page_cache_alloc+0x59/0x5e
<Feb/01 11:56 am>[4661036.890064]  [<ffffffff810b4860>] __do_page_cache_readahead+0xe5/0x1fc
<Feb/01 11:56 am>[4661036.897191]  [<ffffffff810b47f6>] ? __do_page_cache_readahead+0x7b/0x1fc
<Feb/01 11:56 am>[4661036.904480]  [<ffffffff812bc23c>] ? radix_tree_prev_hole+0x30/0x53
<Feb/01 11:56 am>[4661036.911255]  [<ffffffff810b4c40>] ondemand_readahead+0x1cc/0x256
<Feb/01 11:56 am>[4661036.917868]  [<ffffffff810b4ba0>] ? ondemand_readahead+0x12c/0x256
<Feb/01 11:56 am>[4661036.924639]  [<ffffffff810b3c7a>] ? determine_dirtyable_memory+0x15/0x27
<Feb/01 11:56 am>[4661036.931945]  [<ffffffff810b4daa>] page_cache_sync_readahead+0x38/0x3a
<Feb/01 11:56 am>[4661036.939017]  [<ffffffff81109bf2>] __generic_file_splice_read+0xfa/0x41c
<Feb/01 11:56 am>[4661036.946221]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Feb/01 11:56 am>[4661036.952631]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Feb/01 11:56 am>[4661036.959070]  [<ffffffff812615b1>] ? xfs_ilock+0x49/0xb2
<Feb/01 11:56 am>[4661036.964884]  [<ffffffff811086db>] ? spd_release_page+0x0/0x14
<Feb/01 11:56 am>[4661036.971231]  [<ffffffff81109f57>] generic_file_splice_read+0x43/0x74
<Feb/01 11:56 am>[4661036.978201]  [<ffffffff8128191a>] xfs_file_splice_read+0xde/0x125
<Feb/01 11:56 am>[4661036.984881]  [<ffffffff8110875d>] vfs_splice_to+0x6e/0x81
<Feb/01 11:56 am>[4661036.990841]  [<ffffffff81108837>] splice_direct_to_actor+0xc7/0x19c
<Feb/01 11:56 am>[4661036.997708]  [<ffffffff811f562c>] ? nfsd_direct_splice_actor+0x0/0x12
<Feb/01 11:56 am>[4661037.004772]  [<ffffffff811f54ef>] nfsd_vfs_read+0x263/0x3a0

<Feb/01 11:56 am>[4661037.010906]  [<ffffffff81281760>] ? xfs_file_open+0x0/0x3f
<Feb/01 11:56 am>[4661037.016956]  [<ffffffff811f5c4e>] nfsd_read+0xa7/0xc3
<Feb/01 11:56 am>[4661037.022588]  [<ffffffff811fcd79>] nfsd3_proc_read+0xfe/0x142
<Feb/01 11:56 am>[4661037.028805]  [<ffffffff811f0854>] nfsd_dispatch+0xe9/0x1ca
<Feb/01 11:56 am>[4661037.034879]  [<ffffffff814f04c5>] svc_process+0x425/0x72a
<Feb/01 11:56 am>[4661037.040851]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
<Feb/01 11:56 am>[4661037.046212]  [<ffffffff811f0dc2>] nfsd+0xf3/0x139
<Feb/01 11:56 am>[4661037.051466]  [<ffffffff810546e2>] kthread+0x7d/0x85
<Feb/01 11:56 am>[4661037.056882]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
<Feb/01 11:56 am>[4661037.063361]  [<ffffffff81054665>] ? kthread+0x0/0x85
<Feb/01 11:56 am>[4661037.068860]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
<Feb/01 11:56 am>[4661037.075546] NMI backtrace for cpu 6
<Feb/01 11:56 am>[4661037.079550] CPU 6 
<Feb/01 11:56 am>[4661037.081651] Modules linked in: w83793 hwmon_vid autofs4 i2c_i801 i2c_core iTCO_wdt e1000e ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
<Feb/01 11:56 am>[4661037.117750] 
<Feb/01 11:56 am>[4661037.119672] Pid: 0, comm: swapper Tainted: G      D    2.6.34.4 #3 S7010/empty
<Feb/01 11:56 am>[4661037.127716] RIP: 0010:[<ffffffff812c08af>]  [<ffffffff812c08af>] delay_tsc+0x25/0x59
<Feb/01 11:56 am>[4661037.136360] RSP: 0018:ffff880655203b58  EFLAGS: 00000092
<Feb/01 11:56 am>[4661037.142245] RAX: 00000000e492d634 RBX: 0000000024520b3e RCX: 00000000e492d618
<Feb/01 11:56 am>[4661037.150181] RDX: 00000000002c28af RSI: 0000000000000006 RDI: 0000000000000001
<Feb/01 11:56 am>[4661037.158119] RBP: ffff880655203b60 R08: 0000000000000006 R09: 0000000000000006
<Feb/01 11:56 am>[4661037.166041] R10: 0000000000000000 R11: ffff880655203990 R12: ffff8806552b3c00

<Feb/01 11:56 am>[4661037.173989] R13: 0000000000000000 R14: ffff880c24c88000 R15: ffff880c24c88488
<Feb/01 11:56 am>[4661037.181928] FS:  0000000000000000(0000) GS:ffff880655200000(0000) knlGS:0000000000000000
<Feb/01 11:56 am>[4661037.190867] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
<Feb/01 11:56 am>[4661037.197203] CR2: 000000001d9921e0 CR3: 0000000001a8a000 CR4: 00000000000006e0
<Feb/01 11:56 am>[4661037.205165] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<Feb/01 11:56 am>[4661037.213121] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<Feb/01 11:56 am>[4661037.221085] Process swapper (pid: 0, threadinfo ffff880624d28000, task ffff880c24c88000)
<Feb/01 11:56 am>[4661037.229996] Stack:
<Feb/01 11:56 am>[4661037.232473]  ffffffff81019da9 ffff880655203b70 ffffffff812c08fe ffff880655203bb0
<Feb/01 11:56 am>[4661037.244996] <0> ffffffff812c4e65 000000009ef2dfac ffff8806552b3c00 ffff8806552b3c00
<Feb/01 11:56 am>[4661037.253714] <0> ffff88065520e3f0 00000000000004aa 0000000000001000 ffff880655203bd0
<Feb/01 11:56 am>[4661037.262836] Call Trace:
<Feb/01 11:56 am>[4661037.265756]  <IRQ> 
<Feb/01 11:56 am>[4661037.268357]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
<Feb/01 11:56 am>[4661037.276189]  [<ffffffff812c08fe>] __delay+0xa/0xc
<Feb/01 11:56 am>[4661037.281436]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
<Feb/01 11:56 am>[4661037.287766]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
<Feb/01 11:56 am>[4661037.294449]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76
<Feb/01 11:56 am>[4661037.300687]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
<Feb/01 11:56 am>[4661037.306761]  [<ffffffff81037672>] load_balance+0xbb6/0xf63
<Feb/01 11:56 am>[4661037.312812]  [<ffffffff81361d5e>] ? scsi_request_fn+0x39d/0x459
<Feb/01 11:56 am>[4661037.319321]  [<ffffffff8151f115>] ? _raw_spin_lock_irq+0x35/0x39

<Feb/01 11:56 am>[4661037.325944]  [<ffffffff81361d5e>] ? scsi_request_fn+0x39d/0x459
<Feb/01 11:56 am>[4661037.332426]  [<ffffffff81047c8b>] ? run_timer_softirq+0x4d/0x30c
<Feb/01 11:56 am>[4661037.339020]  [<ffffffff81037af9>] rebalance_domains+0xda/0x150
<Feb/01 11:56 am>[4661037.345436]  [<ffffffff81037bb2>] run_rebalance_domains+0x43/0xcf
<Feb/01 11:56 am>[4661037.352144]  [<ffffffff8104103c>] __do_softirq+0xd8/0x194
<Feb/01 11:56 am>[4661037.358122]  [<ffffffff8100388c>] call_softirq+0x1c/0x28
<Feb/01 11:56 am>[4661037.364023]  [<ffffffff81004e7f>] do_softirq+0x33/0x68
<Feb/01 11:56 am>[4661037.369708]  [<ffffffff81040ec2>] irq_exit+0x36/0x75
<Feb/01 11:56 am>[4661037.375238]  [<ffffffff810197aa>] smp_apic_timer_interrupt+0x87/0x95
<Feb/01 11:56 am>[4661037.382176]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
<Feb/01 11:56 am>[4661037.388795]  <EOI> 
<Feb/01 11:56 am>[4661037.391391]  [<ffffffff8151cc4a>] ? schedule+0x296/0x86a
<Feb/01 11:56 am>[4661037.397289]  [<ffffffff81303870>] ? acpi_idle_enter_c1+0xdd/0xf8
<Feb/01 11:56 am>[4661037.403911]  [<ffffffff81303851>] ? acpi_idle_enter_c1+0xbe/0xf8
<Feb/01 11:56 am>[4661037.410533]  [<ffffffff81430151>] cpuidle_idle_call+0x95/0xf0
<Feb/01 11:56 am>[4661037.416841]  [<ffffffff81001d4b>] cpu_idle+0x55/0x8d
<Feb/01 11:56 am>[4661037.422351]  [<ffffffff815191ca>] start_secondary+0x1ae/0x1b2
<Feb/01 11:56 am>[4661037.428680] Code: fb 48 ff c8 c9 c3 55 48 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae e8 0f 31 89 c1 41 89 f1 66 66 90 0f ae e8 0f 31 <89> c6 48 89 f0 48 29 c8 48 39 f8 73 25 f3 90 65 44 8b 04 25 f0 
<Feb/01 11:56 am>[4661037.451861] Call Trace:
<Feb/01 11:56 am>[4661037.454773]  <IRQ>  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
<Feb/01 11:56 am>[4661037.463296]  [<ffffffff812c08fe>] __delay+0xa/0xc
<Feb/01 11:56 am>[4661037.468537]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147

<Feb/01 11:56 am>[4661037.474824]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
<Feb/01 11:56 am>[4661037.481535]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76
<Feb/01 11:56 am>[4661037.487764]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
<Feb/01 11:56 am>[4661037.493830]  [<ffffffff81037672>] load_balance+0xbb6/0xf63
<Feb/01 11:56 am>[4661037.499889]  [<ffffffff81361d5e>] ? scsi_request_fn+0x39d/0x459
<Feb/01 11:56 am>[4661037.506408]  [<ffffffff8151f115>] ? _raw_spin_lock_irq+0x35/0x39
<Feb/01 11:56 am>[4661037.513004]  [<ffffffff81361d5e>] ? scsi_request_fn+0x39d/0x459
<Feb/01 11:56 am>[4661037.519531]  [<ffffffff81047c8b>] ? run_timer_softirq+0x4d/0x30c
                 <Feb/01 11:56 am>[4661037.526126]  [<ffffffff81037af9>] rebalance_domains+0xda/0x150
<Feb/01 11:56 am>[4661037.532539]  [<ffffffff81037bb2>] run_rebalance_domains+0x43/0xcf
<Feb/01 11:56 am>[4661037.539221]  [<ffffffff8104103c>] __do_softirq+0xd8/0x194
<Feb/01 11:56 am>[4661037.545207]  [<ffffffff8100388c>] call_softirq+0x1c/0x28
<Feb/01 11:56 am>[4661037.551092]  [<ffffffff81004e7f>] do_softirq+0x33/0x68
<Feb/01 11:56 am>[4661037.556794]  [<ffffffff81040ec2>] irq_exit+0x36/0x75
<Feb/01 11:56 am>[4661037.562298]  [<ffffffff810197aa>] smp_apic_timer_interrupt+0x87/0x95
<Feb/01 11:56 am>[4661037.569244]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
<Feb/01 11:56 am>[4661037.575848]  <EOI>  [<ffffffff8151cc4a>] ? schedule+0x296/0x86a
<Feb/01 11:56 am>[4661037.582419]  [<ffffffff81303870>] ? acpi_idle_enter_c1+0xdd/0xf8
<Feb/01 11:56 am>[4661037.589013]  [<ffffffff81303851>] ? acpi_idle_enter_c1+0xbe/0xf8
<Feb/01 11:56 am>[4661037.595634]  [<ffffffff81430151>] cpuidle_idle_call+0x95/0xf0
<Feb/01 11:56 am>[4661037.601951]  [<ffffffff81001d4b>] cpu_idle+0x55/0x8d
<Feb/01 11:56 am>[4661037.607445]  [<ffffffff815191ca>] start_secondary+0x1ae/0x1b2

<Feb/01 11:56 am>[4661037.613791] Pid: 0, comm: swapper Tainted: G      D    2.6.34.4 #3
<Feb/01 11:56 am>[4661037.620578] Call Trace:
<Feb/01 11:56 am>[4661037.623503]  <NMI>  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
                 <Feb/01 11:56 am>[4661037.629965]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
<Feb/01 11:56 am>[4661037.636394]  [<ffffffff81520360>] do_nmi+0xb2/0x259
<Feb/01 11:56 am>[4661037.641808]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
<Feb/01 11:56 am>[4661037.646895]  [<ffffffff812c08af>] ? delay_tsc+0x25/0x59
<Feb/01 11:56 am>[4661037.652704]  <<EOE>>  <IRQ>  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
<Feb/01 11:56 am>[4661037.662091]  [<ffffffff812c08fe>] __delay+0xa/0xc
<Feb/01 11:56 am>[4661037.667357]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
<Feb/01 11:56 am>[4661037.673637]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
<Feb/01 11:56 am>[4661037.680338]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76
<Feb/01 11:56 am>[4661037.686592]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
<Feb/01 11:56 am>[4661037.692668]  [<ffffffff81037672>] load_balance+0xbb6/0xf63
<Feb/01 11:56 am>[4661037.698726]  [<ffffffff81361d5e>] ? scsi_request_fn+0x39d/0x459
<Feb/01 11:56 am>[4661037.705245]  [<ffffffff8151f115>] ? _raw_spin_lock_irq+0x35/0x39
<Feb/01 11:56 am>[4661037.711860]  [<ffffffff81361d5e>] ? scsi_request_fn+0x39d/0x459
<Feb/01 11:56 am>[4661037.718350]  [<ffffffff81047c8b>] ? run_timer_softirq+0x4d/0x30c
<Feb/01 11:56 am>[4661037.724945]  [<ffffffff81037af9>] rebalance_domains+0xda/0x150
<Feb/01 11:56 am>[4661037.731359]  [<ffffffff81037bb2>] run_rebalance_domains+0x43/0xcf
<Feb/01 11:56 am>[4661037.738058]  [<ffffffff8104103c>] __do_softirq+0xd8/0x194
<Feb/01 11:56 am>[4661037.744027]  [<ffffffff8100388c>] call_softirq+0x1c/0x28

<Feb/01 11:56 am>[4661037.749912]  [<ffffffff81004e7f>] do_softirq+0x33/0x68
<Feb/01 11:56 am>[4661037.755615]  [<ffffffff81040ec2>] irq_exit+0x36/0x75
<Feb/01 11:56 am>[4661037.761143]  [<ffffffff810197aa>] smp_apic_timer_interrupt+0x87/0x95
<Feb/01 11:56 am>[4661037.768091]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
<Feb/01 11:56 am>[4661037.774650]  <EOI>  [<ffffffff8151cc4a>] ? schedule+0x296/0x86a
<Feb/01 11:56 am>[4661037.781205]  [<ffffffff81303870>] ? acpi_idle_enter_c1+0xdd/0xf8
<Feb/01 11:56 am>[4661037.787781]  [<ffffffff81303851>] ? acpi_idle_enter_c1+0xbe/0xf8
                 <Feb/01 11:56 am>[4661037.794385]  [<ffffffff81430151>] cpuidle_idle_call+0x95/0xf0
<Feb/01 11:56 am>[4661037.800728]  [<ffffffff81001d4b>] cpu_idle+0x55/0x8d
<Feb/01 11:56 am>[4661037.806256]  [<ffffffff815191ca>] start_secondary+0x1ae/0x1b2
Last modified 15 years ago Last modified on Feb 1, 2012, 12:13:24 PM
Note: See TracWiki for help on using the wiki.