On console
n+0x0/0xf
<Apr/09 11:03 pm>[2979142.271348] [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
<Apr/09 11:03 pm>[2979142.271350] [<ffffffff811f0d78>] nfsd+0xa9/0x139
<Apr/09 11:03 pm>[2979142.271352] [<ffffffff810546e2>] kthread+0x7d/0x85
<Apr/09 11:03 pm>[2979142.271354] [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
<Apr/09 11:03 pm>[2979142.271357] [<ffffffff81054665>] ? kthread+0x0/0x85
<Apr/09 11:03 pm>[2979142.271358] [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
<Apr/09 11:03 pm>[2979142.271360] Pid: 8541, comm: nfsd Tainted: G D 2.6.34.4 #3
<Apr/09 11:03 pm>[2979142.271361] Call Trace:
<Apr/09 11:03 pm>[2979142.271362] <NMI> [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
<Apr/09 11:03 pm>[2979142.271367] [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
<Apr/09 11:03 pm>[2979142.271369] [<ffffffff81520360>] do_nmi+0xb2/0x259
<Apr/09 11:03 pm>[2979142.271371] [<ffffffff8151fdfa>] nmi+0x1a/0x20
<Apr/09 11:03 pm>[2979142.271373] [<ffffffff812c08a0>] ? delay_tsc+0x16/0x59
<Apr/09 11:03 pm>[2979142.271374] <<EOE>> [<ffffffff812c093d>] __const_udelay+0x3d/0x3f
<Apr/09 11:03 pm>[2979142.271378] [<ffffffff810197e9>] native_safe_apic_wait_icr_idle+0x31/0x48
<Apr/09 11:03 pm>[2979142.271380] [<ffffffff81019c76>] default_send_IPI_mask_sequence_phys+0x49/0xab
<Apr/09 11:03 pm>[2979142.271383] [<ffffffff8101c5d8>] physflat_send_IPI_all+0x14/0x16
<Apr/09 11:03 pm>[2979142.271385] [<ffffffff81019d94>] arch_trigger_all_cpu_backtrace+0x3c/0x5d
<Apr/09 11:03 pm>[2979142.271387] [<ffffffff812c4ea4>] do_raw_spin_lock+0x11d/0x147
<Apr/09 11:03 pm>[2979142.271390] [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
<Apr/09 11:03 pm>[2979142.271392] [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
<Apr/09 11:03 pm>[2979142.271394] [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
<Apr/09 11:03 pm>[2979142.271396] [<ffffffff81036578>] try_to_wake_up+0x2e/0x342
<Apr/09 11:03 pm>[2979142.271398] [<ffffffff81036899>] default_wake_function+0xd/0xf
<Apr/09 11:03 pm>[2979142.271400] [<ffffffff8102c08d>] __wake_up_common+0x46/0x76
<Apr/09 11:03 pm>[2979142.271402] [<ffffffff8102faa6>] __wake_up+0x38/0x4f
<Apr/09 11:03 pm>[2979142.271404] [<ffffffff814fbb75>] svc_xprt_enqueue+0x1e2/0x23d
<Apr/09 11:03 pm>[2979142.271406] [<ffffffff814fbc26>] svc_xprt_received+0x24/0x26
<Apr/09 11:03 pm>[2979142.271409] [<ffffffff814f1a12>] svc_udp_recvfrom+0x15f/0x2f9
<Apr/09 11:03 pm>[2979142.271411] [<ffffffff814fc980>] svc_recv+0x758/0x7f5
<Apr/09 11:03 pm>[2979142.271413] [<ffffffff814fc1d9>] ? svc_send+0x85/0xa1
<Apr/09 11:03 pm>[2979142.271415] [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
<Apr/09 11:03 pm>[2979142.271417] [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
<Apr/09 11:03 pm>[2979142.271419] [<ffffffff811f0d78>] nfsd+0xa9/0x139
<Apr/09 11:03 pm>[2979142.271421] [<ffffffff810546e2>] kthread+0x7d/0x85
<Apr/09 11:03 pm>[2979142.271423] [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
<Apr/09 11:03 pm>[2979142.271425] [<ffffffff81054665>] ? kthread+0x0/0x85
<Apr/09 11:03 pm>[2979142.271427] [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
<Apr/09 11:03 pm>[2979142.271530] NMI backtrace for cpu 6
<Apr/09 11:03 pm>[2979142.271531] CPU 6
<Apr/09 11:03 pm>[2979142.271532] Modules linked in: w83793 hwmon_vid autofs4 iTCO_wdt e1000e i2c_i801 i2c_core ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
<Apr/09 11:03 pm>[2979142.271546]
<Apr/09 11:03 pm>[2979142.271547] Pid: 8590, comm: nfsd Tainted: G D 2.6.34.4 #3 S7010/empty
<Apr/09 11:03 pm>[2979142.271549] RIP: 0010:[<ffffffff812c08a2>] [<ffffffff812c08a2>] delay_tsc+0x18/0x59
<Apr/09 11:03 pm>[2979142.271553] RSP: 0018:ffff880655203de0 EFLAGS: 00000082
<Apr/09 11:03 pm>[2979142.271555] RAX: 000000009106eca8 RBX: 000000009aeab2fb RCX: 000000009106ec64
<Apr/09 11:03 pm>[2979142.271556] RDX: 00000000001c399e RSI: 0000000000000006 RDI: 0000000000000001
<Apr/09 11:03 pm>[2979142.271558] RBP: ffff880655203de8 R08: 0000000000000001 R09: 0000000000000006
<Apr/09 11:03 pm>[2979142.271559] R10: 0000000100000000 R11: 0000000000000000 R12: ffff880655213c00
<Apr/09 11:03 pm>[2979142.271561] R13: 0000000000000001 R14: ffff8806227ed0f0 R15: ffff8806227ed578
<Apr/09 11:03 pm>[2979142.271563] FS: 0000000000000000(0000) GS:ffff880655200000(0000) knlGS:0000000000000000
<Apr/09 11:03 pm>[2979142.271565] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
<Apr/09 11:03 pm>[2979142.271566] CR2: 0000000389ca1c18 CR3: 0000000001a8a000 CR4: 00000000000006e0
<Apr/09 11:03 pm>[2979142.271568] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<Apr/09 11:03 pm>[2979142.271569] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<Apr/09 11:03 pm>[2979142.271571] Process nfsd (pid: 8590, threadinfo ffff880622754000, task ffff8806227ed0f0)
<Apr/09 11:03 pm>[2979142.271572] Stack:
<Apr/09 11:03 pm>[2979142.271573] ffff880655213c00 ffff880655203df8 ffffffff812c08fe ffff880655203e38
<Apr/09 11:03 pm>[2979142.271575] <0> ffffffff812c4e65 000000009ef30658 ffff880655213c00 ffff880655213c00
<Apr/09 11:03 pm>[2979142.271577] <0> 0000000000000006 ffff88065520eae0 ffff88065520ea70 ffff880655203e58
<Apr/09 11:03 pm>[2979142.271580] Call Trace:
<Apr/09 11:03 pm>[2979142.271581] <IRQ>
<Apr/09 11:03 pm>[2979142.271583] [<ffffffff812c08fe>] __delay+0xa/0xc
<Apr/09 11:03 pm>[2979142.271585] [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
<Apr/09 11:03 pm>[2979142.271587] [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
<Apr/09 11:03 pm>[2979142.271590] [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
<Apr/09 11:03 pm>[2979142.271592] [<ffffffff81038588>] scheduler_tick+0x3d/0x236
<Apr/09 11:03 pm>[2979142.271594] [<ffffffff810483d1>] update_process_times+0x49/0x59
<Apr/09 11:03 pm>[2979142.271597] [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
<Apr/09 11:03 pm>[2979142.271599] [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
<Apr/09 11:03 pm>[2979142.271601] [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
<Apr/09 11:03 pm>[2979142.271603] [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:03 pm>[2979142.271606] [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
<Apr/09 11:03 pm>[2979142.271611] [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
<Apr/09 11:03 pm>[2979142.271612] <EOI>
<Apr/09 11:03 pm>[2979142.271613] Code: 2e 0f 1f 84 00 00 00 00 00 48 ff c8 75 fb 48 ff c8 c9 c3 55 48 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae e8 0f 31 <89> c1 41 89 f1 66 66 90 0f ae e8 0f 31 89 c6 48 89 f0 48 29 c8
<Apr/09 11:03 pm>[2979142.271629] Call Trace:
<Apr/09 11:03 pm>[2979142.271629] <IRQ> [<ffffffff812c08fe>] __delay+0xa/0xc
<Apr/09 11:03 pm>[2979142.271633] [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
<Apr/09 11:03 pm>[2979142.271635] [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
<Apr/09 11:03 pm>[2979142.271638] [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
<Apr/09 11:03 pm>[2979142.271640] [<ffffffff81038588>] scheduler_tick+0x3d/0x236
<Apr/09 11:03 pm>[2979142.271642] [<ffffffff810483d1>] update_process_times+0x49/0x59
<Apr/09 11:03 pm>[2979142.271644] [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
<Apr/09 11:03 pm>[2979142.271646] [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
<Apr/09 11:03 pm>[2979142.271648] [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
<Apr/09 11:03 pm>[2979142.271650] [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:03 pm>[2979142.271652] [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
<Apr/09 11:03 pm>[2979142.271655] [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
<Apr/09 11:03 pm>[2979142.271656] <EOI> Pid: 8590, comm: nfsd Tainted: G D 2.6.34.4 #3
<Apr/09 11:03 pm>[2979142.271658] Call Trace:
<Apr/09 11:03 pm>[2979142.271659] <NMI> [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
<Apr/09 11:03 pm>[2979142.271663] [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
<Apr/09 11:03 pm>[2979142.271665] [<ffffffff81520360>] do_nmi+0xb2/0x259
<Apr/09 11:03 pm>[2979142.271667] [<ffffffff8151fdfa>] nmi+0x1a/0x20
<Apr/09 11:03 pm>[2979142.271669] [<ffffffff812c08a2>] ? delay_tsc+0x18/0x59
<Apr/09 11:03 pm>[2979142.271670] <<EOE>> <IRQ> [<ffffffff812c08fe>] __delay+0xa/0xc
<Apr/09 11:03 pm>[2979142.271674] [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
<Apr/09 11:03 pm>[2979142.271676] [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
<Apr/09 11:03 pm>[2979142.271678] [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
<Apr/09 11:03 pm>[2979142.271680] [<ffffffff81038588>] scheduler_tick+0x3d/0x236
<Apr/09 11:03 pm>[2979142.271683] [<ffffffff810483d1>] update_process_times+0x49/0x59
<Apr/09 11:03 pm>[2979142.271685] [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
<Apr/09 11:03 pm>[2979142.271687] [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
<Apr/09 11:03 pm>[2979142.271689] [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
<Apr/09 11:03 pm>[2979142.271691] [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:03 pm>[2979142.271693] [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
<Apr/09 11:03 pm>[2979142.271695] [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
<Apr/09 11:03 pm>[2979142.271696] <EOI>
<Apr/09 11:03 pm>[2979142.271697] NMI backtrace for cpu 7
<Apr/09 11:03 pm>[2979142.271698] CPU 7
<Apr/09 11:03 pm>[2979142.271699] Modules linked in: w83793 hwmon_vid autofs4 iTCO_wdt e1000e i2c_i801 i2c_core ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
<Apr/09 11:03 pm>[2979142.271713]
<Apr/09 11:03 pm>[2979142.271715] Pid: 8574, comm: nfsd Tainted: G D 2.6.34.4 #3 S7010/empty
<Apr/09 11:03 pm>[2979142.271717] RIP: 0010:[<ffffffff8133423b>] [<ffffffff8133423b>] io_serial_in+0x10/0x15
<Apr/09 11:03 pm>[2979142.271722] RSP: 0018:ffff880655223938 EFLAGS: 00000002
<Apr/09 11:03 pm>[2979142.271723] RAX: 0000000000000a00 RBX: 0000000000000000 RCX: 0000000000000000
<Apr/09 11:03 pm>[2979142.271725] RDX: 00000000000003fd RSI: 0000000000000005 RDI: ffffffff82400ac0
<Apr/09 11:03 pm>[2979142.271727] RBP: ffff880655223938 R08: 0000000000000007 R09: 0000000000000007
<Apr/09 11:03 pm>[2979142.271728] R10: 0000000000000000 R11: ffffffff81334e4e R12: ffffffff82400ac0
<Apr/09 11:03 pm>[2979142.271730] R13: 000000000000270f R14: 0000000000000020 R15: 0000000000000044
<Apr/09 11:03 pm>[2979142.271732] FS: 0000000000000000(0000) GS:ffff880655220000(0000) knlGS:0000000000000000
<Apr/09 11:03 pm>[2979142.271733] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
<Apr/09 11:03 pm>[2979142.271735] CR2: 00007f7053196000 CR3: 0000000001a8a000 CR4: 00000000000006e0
<Apr/09 11:03 pm>[2979142.271736] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<Apr/09 11:03 pm>[2979142.271738] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<Apr/09 11:03 pm>[2979142.271740] Process nfsd (pid: 8574, threadinfo ffff880623fec000, task ffff8806227d3090)
<Apr/09 11:03 pm>[2979142.271741] Stack:
<Apr/09 11:03 pm>[2979142.271742] ffff880655223968 ffffffff81334767 ffffffff82400ac0 0000000000000062
<Apr/09 11:03 pm>[2979142.271744] <0> ffffffff813347d5 ffffffff82400ac0 ffff880655223988 ffffffff813347ec
<Apr/09 11:03 pm>[2979142.271746] <0> ffffffff81c39048 0000000000000022 ffff8806552239c8 ffffffff813311e8
<Apr/09 11:03 pm>[2979142.271749] Call Trace:
<Apr/09 11:03 pm>[2979142.271750] <IRQ>
<Apr/09 11:03 pm>[2979142.271752] [<ffffffff81334767>] wait_for_xmitr+0x24/0x92
<Apr/09 11:03 pm>[2979142.271755] [<ffffffff813347d5>] ? serial8250_console_putchar+0x0/0x27
<Apr/09 11:03 pm>[2979142.271757] [<ffffffff813347ec>] serial8250_console_putchar+0x17/0x27
<Apr/09 11:03 pm>[2979142.271760] [<ffffffff813311e8>] uart_console_write+0x41/0x52
<Apr/09 11:03 pm>[2979142.271762] [<ffffffff81334eb0>] serial8250_console_write+0xb6/0x108
<Apr/09 11:03 pm>[2979142.271765] [<ffffffff8103b339>] __call_console_drivers+0x67/0x79
<Apr/09 11:03 pm>[2979142.271768] [<ffffffff8103b3a6>] _call_console_drivers+0x5b/0x5f
<Apr/09 11:03 pm>[2979142.271770] [<ffffffff8103b87b>] release_console_sem+0x121/0x1c6
<Apr/09 11:03 pm>[2979142.271773] [<ffffffff8103beb2>] vprintk+0x36b/0x3a0
<Apr/09 11:03 pm>[2979142.271775] [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:03 pm>[2979142.271777] [<ffffffff8151c589>] printk+0x67/0x69
<Apr/09 11:03 pm>[2979142.271779] [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:03 pm>[2979142.271781] [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:03 pm>[2979142.271784] [<ffffffff8151c589>] ? printk+0x67/0x69
<Apr/09 11:03 pm>[2979142.271786] [<ffffffff81019eb4>] ? touch_nmi_watchdog+0x5b/0x60
<Apr/09 11:03 pm>[2979142.271789] [<ffffffff810060bb>] print_trace_address+0x4e/0x57
<Apr/09 11:03 pm>[2979142.271791] [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:03 pm>[2979142.271793] [<ffffffff810061e9>] print_context_stack+0x62/0xbd
<Apr/09 11:03 pm>[2979142.271795] [<ffffffff810054e2>] dump_trace+0x23d/0x327
<Apr/09 11:03 pm>[2979142.271798] [<ffffffff8100604f>] show_trace_log_lvl+0x4c/0x58
<Apr/09 11:03 pm>[2979142.271800] [<ffffffff8100606b>] show_trace+0x10/0x12
<Apr/09 11:03 pm>[2979142.271802] [<ffffffff8151c401>] dump_stack+0x72/0x7b
<Apr/09 11:03 pm>[2979142.271804] [<ffffffff812c4e9f>] do_raw_spin_lock+0x118/0x147
<Apr/09 11:03 pm>[2979142.271806] [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
<Apr/09 11:03 pm>[2979142.271808] [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
<Apr/09 11:03 pm>[2979142.271811] [<ffffffff81038588>] scheduler_tick+0x3d/0x236
<Apr/09 11:03 pm>[2979142.271813] [<ffffffff810483d1>] update_process_times+0x49/0x59
<Apr/09 11:03 pm>[2979142.271815] [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
<Apr/09 11:03 pm>[2979142.271817] [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
<Apr/09 11:03 pm>[2979142.271819] [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
<Apr/09 11:03 pm>[2979142.271821] [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:03 pm>[2979142.271823] [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
<Apr/09 11:03 pm>[2979142.271826] [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
<Apr/09 11:03 pm>[2979142.271827] <EOI>
<Apr/09 11:03 pm>[2979142.271829] [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
<Apr/09 11:03 pm>[2979142.271831] [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
<Apr/09 11:03 pm>[2979142.271833] [<ffffffff810d48fd>] ? get_swap_page+0x13c/0x5bb
<Apr/09 11:03 pm>[2979142.271836] [<ffffffff810d17e9>] add_to_swap+0xf/0x48
<Apr/09 11:03 pm>[2979142.271838] [<ffffffff810b7464>] shrink_page_list+0x214/0x68f
<Apr/09 11:03 pm>[2979142.271841] [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Apr/09 11:03 pm>[2979142.271844] [<ffffffff810e5fb2>] ? mem_cgroup_del_lru_list+0x7b/0x8e
<Apr/09 11:03 pm>[2979142.271846] [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
<Apr/09 11:03 pm>[2979142.271851] [<ffffffff8144b300>] ? release_sock+0xdb/0xe4
<Apr/09 11:03 pm>[2979142.271853] [<ffffffff8144b254>] ? release_sock+0x2f/0xe4
<Apr/09 11:03 pm>[2979142.271855] [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
<Apr/09 11:03 pm>[2979142.271857] [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
<Apr/09 11:03 pm>[2979142.271860] [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
<Apr/09 11:03 pm>[2979142.271862] [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
<Apr/09 11:03 pm>[2979142.271865] [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
<Apr/09 11:03 pm>[2979142.271869] [<ffffffff814fc30d>] svc_recv+0xe5/0x7f5
<Apr/09 11:03 pm>[2979142.271871] [<ffffffff814fc1d9>] ? svc_send+0x85/0xa1
<Apr/09 11:03 pm>[2979142.271873] [<ffffffff811f9bb1>] ? exp_readunlock+0x10/0x12
<Apr/09 11:03 pm>[2979142.271876] [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
<Apr/09 11:03 pm>[2979142.271878] [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
<Apr/09 11:03 pm>[2979142.271879] [<ffffffff811f0d78>] nfsd+0xa9/0x139
<Apr/09 11:03 pm>[2979142.271882] [<ffffffff810546e2>] kthread+0x7d/0x85
<Apr/09 11:03 pm>[2979142.271884] [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
<Apr/09 11:03 pm>[2979142.271886] [<ffffffff81054665>] ? kthread+0x0/0x85
<Apr/09 11:03 pm>[2979142.271888] [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
<Apr/09 11:03 pm>[2979142.271890] Code: 49 03 40 38 88 10 83 fe 01 77 0c be 01 00 00 00 4c 89 c7 41 ff 50 40 58 5b c9 c3 0f b6 4f 69 55 89 f2 48 89 e5 d3 e2 03 57 30 ec <0f> b6 c0 c9 c3 0f b6 4f 69 55 88 d0 48 89 e5 d3 e6 03 77 30 89
<Apr/09 11:03 pm>[2979142.271905] Call Trace:
<Apr/09 11:03 pm>[2979142.271906] <IRQ> [<ffffffff81334767>] wait_for_xmitr+0x24/0x92
<Apr/09 11:03 pm>[2979142.271910] [<ffffffff813347d5>] ? serial8250_console_putchar+0x0/0x27
<Apr/09 11:03 pm>[2979142.271913] [<ffffffff813347ec>] serial8250_console_putchar+0x17/0x27
<Apr/09 11:03 pm>[2979142.271915] [<ffffffff813311e8>] uart_console_write+0x41/0x52
<Apr/09 11:03 pm>[2979142.271917] [<ffffffff81334eb0>] serial8250_console_write+0xb6/0x108
<Apr/09 11:03 pm>[2979142.271920] [<ffffffff8103b339>] __call_console_drivers+0x67/0x79
<Apr/09 11:03 pm>[2979142.271922] [<ffffffff8103b3a6>] _call_console_drivers+0x5b/0x5f
<Apr/09 11:03 pm>[2979142.271925] [<ffffffff8103b87b>] release_console_sem+0x121/0x1c6
<Apr/09 11:03 pm>[2979142.271927] [<ffffffff8103beb2>] vprintk+0x36b/0x3a0
<Apr/09 11:03 pm>[2979142.271930] [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:03 pm>[2979142.271932] [<ffffffff8151c589>] printk+0x67/0x69
<Apr/09 11:03 pm>[2979142.271934] [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:03 pm>[2979142.271936] [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:03 pm>[2979142.271938] [<ffffffff8151c589>] ? printk+0x67/0x69
<Apr/09 11:03 pm>[2979142.271940] [<ffffffff81019eb4>] ? touch_nmi_watchdog+0x5b/0x60
<Apr/09 11:03 pm>[2979142.271942] [<ffffffff810060bb>] print_trace_address+0x4e/0x57
<Apr/09 11:03 pm>[2979142.271944] [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:03 pm>[2979142.271946] [<ffffffff810061e9>] print_context_stack+0x62/0xbd
<Apr/09 11:03 pm>[2979142.271949] [<ffffffff810054e2>] dump_trace+0x23d/0x327
<Apr/09 11:03 pm>[2979142.271951] [<ffffffff8100604f>] show_trace_log_lvl+0x4c/0x58
<Apr/09 11:03 pm>[2979142.271953] [<ffffffff8100606b>] show_trace+0x10/0x12
<Apr/09 11:03 pm>[2979142.271955] [<ffffffff8151c401>] dump_stack+0x72/0x7b
<Apr/09 11:03 pm>[2979142.271957] [<ffffffff812c4e9f>] do_raw_spin_lock+0x118/0x147
<Apr/09 11:03 pm>[2979142.271959] [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
<Apr/09 11:03 pm>[2979142.271962] [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
<Apr/09 11:03 pm>[2979142.271964] [<ffffffff81038588>] scheduler_tick+0x3d/0x236
<Apr/09 11:03 pm>[2979142.271966] [<ffffffff810483d1>] update_process_times+0x49/0x59
<Apr/09 11:03 pm>[2979142.271968] [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
<Apr/09 11:03 pm>[2979142.271970] [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
<Apr/09 11:03 pm>[2979142.271972] [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
<Apr/09 11:03 pm>[2979142.271974] [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:03 pm>[2979142.271976] [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
<Apr/09 11:03 pm>[2979142.271979] [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
<Apr/09 11:03 pm>[2979142.271980] <EOI> [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
<Apr/09 11:03 pm>[2979142.271984] [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
<Apr/09 11:03 pm>[2979142.271986] [<ffffffff810d48fd>] ? get_swap_page+0x13c/0x5bb
<Apr/09 11:03 pm>[2979142.271988] [<ffffffff810d17e9>] add_to_swap+0xf/0x48
<Apr/09 11:03 pm>[2979142.271990] [<ffffffff810b7464>] shrink_page_list+0x214/0x68f
<Apr/09 11:03 pm>[2979142.271992] [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Apr/09 11:03 pm>[2979142.271995] [<ffffffff810e5fb2>] ? mem_cgroup_del_lru_list+0x7b/0x8e
<Apr/09 11:03 pm>[2979142.271997] [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
<Apr/09 11:03 pm>[2979142.272000] [<ffffffff8144b300>] ? release_sock+0xdb/0xe4
<Apr/09 11:03 pm>[2979142.272002] [<ffffffff8144b254>] ? release_sock+0x2f/0xe4
<Apr/09 11:03 pm>[2979142.272004] [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
<Apr/09 11:03 pm>[2979142.272006] [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
<Apr/09 11:03 pm>[2979142.272009] [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
<Apr/09 11:03 pm>[2979142.272011] [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
<Apr/09 11:03 pm>[2979142.272014] [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
<Apr/09 11:03 pm>[2979142.272016] [<ffffffff814fc30d>] svc_recv+0xe5/0x7f5
<Apr/09 11:03 pm>[2979142.272018] [<ffffffff814fc1d9>] ? svc_send+0x85/0xa1
<Apr/09 11:03 pm>[2979142.272020] [<ffffffff811f9bb1>] ? exp_readunlock+0x10/0x12
<Apr/09 11:03 pm>[2979142.272023] [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
<Apr/09 11:03 pm>[2979142.272025] [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
<Apr/09 11:03 pm>[2979142.272027] [<ffffffff811f0d78>] nfsd+0xa9/0x139
<Apr/09 11:03 pm>[2979142.272029] [<ffffffff810546e2>] kthread+0x7d/0x85
<Apr/09 11:03 pm>[2979142.272031] [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
<Apr/09 11:03 pm>[2979142.272033] [<ffffffff81054665>] ? kthread+0x0/0x85
<Apr/09 11:03 pm>[2979142.272035] [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
<Apr/09 11:03 pm>[2979142.272037] Pid: 8574, comm: nfsd Tainted: G D 2.6.34.4 #3
<Apr/09 11:03 pm>[2979142.272038] Call Trace:
<Apr/09 11:03 pm>[2979142.272039] <NMI> [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
<Apr/09 11:03 pm>[2979142.272043] [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
<Apr/09 11:03 pm>[2979142.272045] [<ffffffff81520360>] do_nmi+0xb2/0x259
<Apr/09 11:03 pm>[2979142.272047] [<ffffffff8151fdfa>] nmi+0x1a/0x20
<Apr/09 11:03 pm>[2979142.272049] [<ffffffff81334e4e>] ? serial8250_console_write+0x54/0x108
<Apr/09 11:03 pm>[2979142.272052] [<ffffffff8133423b>] ? io_serial_in+0x10/0x15
<Apr/09 11:03 pm>[2979142.272053] <<EOE>> <IRQ> [<ffffffff81334767>] wait_for_xmitr+0x24/0x92
<Apr/09 11:03 pm>[2979142.272057] [<ffffffff813347d5>] ? serial8250_console_putchar+0x0/0x27
<Apr/09 11:03 pm>[2979142.272060] [<ffffffff813347ec>] serial8250_console_putchar+0x17/0x27
<Apr/09 11:03 pm>[2979142.272062] [<ffffffff813311e8>] uart_console_write+0x41/0x52
<Apr/09 11:03 pm>[2979142.272064] [<ffffffff81334eb0>] serial8250_console_write+0xb6/0x108
<Apr/09 11:03 pm>[2979142.272067] [<ffffffff8103b339>] __call_console_drivers+0x67/0x79
<Apr/09 11:03 pm>[2979142.272069] [<ffffffff8103b3a6>] _call_console_drivers+0x5b/0x5f
<Apr/09 11:03 pm>[2979142.272072] [<ffffffff8103b87b>] release_console_sem+0x121/0x1c6
<Apr/09 11:03 pm>[2979142.272074] [<ffffffff8103beb2>] vprintk+0x36b/0x3a0
<Apr/09 11:03 pm>[2979142.272076] [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:03 pm>[2979142.272078] [<ffffffff8151c589>] printk+0x67/0x69
<Apr/09 11:03 pm>[2979142.272080] [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:03 pm>[2979142.272082] [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:03 pm>[2979142.272085] [<ffffffff8151c589>] ? printk+0x67/0x69
<Apr/09 11:03 pm>[2979142.272087] [<ffffffff81019eb4>] ? touch_nmi_watchdog+0x5b/0x60
<Apr/09 11:03 pm>[2979142.272089] [<ffffffff810060bb>] print_trace_address+0x4e/0x57
<Apr/09 11:03 pm>[2979142.272091] [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:03 pm>[2979142.272093] [<ffffffff810061e9>] print_context_stack+0x62/0xbd
<Apr/09 11:03 pm>[2979142.272095] [<ffffffff810054e2>] dump_trace+0x23d/0x327
<Apr/09 11:03 pm>[2979142.272098] [<ffffffff8100604f>] show_trace_log_lvl+0x4c/0x58
<Apr/09 11:03 pm>[2979142.272100] [<ffffffff8100606b>] show_trace+0x10/0x12
<Apr/09 11:03 pm>[2979142.272102] [<ffffffff8151c401>] dump_stack+0x72/0x7b
<Apr/09 11:03 pm>[2979142.272104] [<ffffffff812c4e9f>] do_raw_spin_lock+0x118/0x147
<Apr/09 11:03 pm>[2979142.272106] [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
<Apr/09 11:03 pm>[2979142.272108] [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
<Apr/09 11:03 pm>[2979142.272111] [<ffffffff81038588>] scheduler_tick+0x3d/0x236
<Apr/09 11:03 pm>[2979142.272113] [<ffffffff810483d1>] update_process_times+0x49/0x59
<Apr/09 11:03 pm>[2979142.272115] [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
<Apr/09 11:03 pm>[2979142.272117] [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
<Apr/09 11:03 pm>[2979142.272119] [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
<Apr/09 11:03 pm>[2979142.272121] [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:03 pm>[2979142.272123] [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
<Apr/09 11:03 pm>[2979142.272126] [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
<Apr/09 11:03 pm>[2979142.272127] <EOI> [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
<Apr/09 11:03 pm>[2979142.272130] [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
<Apr/09 11:03 pm>[2979142.272132] [<ffffffff810d48fd>] ? get_swap_page+0x13c/0x5bb
<Apr/09 11:03 pm>[2979142.272134] [<ffffffff810d17e9>] add_to_swap+0xf/0x48
<Apr/09 11:03 pm>[2979142.272136] [<ffffffff810b7464>] shrink_page_list+0x214/0x68f
<Apr/09 11:03 pm>[2979142.272139] [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Apr/09 11:03 pm>[2979142.272141] [<ffffffff810e5fb2>] ? mem_cgroup_del_lru_list+0x7b/0x8e
<Apr/09 11:03 pm>[2979142.272143] [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
<Apr/09 11:03 pm>[2979142.272146] [<ffffffff8144b300>] ? release_sock+0xdb/0xe4
<Apr/09 11:03 pm>[2979142.272148] [<ffffffff8144b254>] ? release_sock+0x2f/0xe4
<Apr/09 11:03 pm>[2979142.272151] [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
<Apr/09 11:03 pm>[2979142.272153] [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
<Apr/09 11:03 pm>[2979142.272155] [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
<Apr/09 11:03 pm>[2979142.272158] [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
<Apr/09 11:03 pm>[2979142.272160] [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
<Apr/09 11:03 pm>[2979142.272162] [<ffffffff814fc30d>] svc_recv+0xe5/0x7f5
<Apr/09 11:03 pm>[2979142.272164] [<ffffffff814fc1d9>] ? svc_send+0x85/0xa1
<Apr/09 11:03 pm>[2979142.272167] [<ffffffff811f9bb1>] ? exp_readunlock+0x10/0x12
<Apr/09 11:03 pm>[2979142.272169] [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
<Apr/09 11:03 pm>[2979142.272171] [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
<Apr/09 11:03 pm>[2979142.272172] [<ffffffff811f0d78>] nfsd+0xa9/0x139
<Apr/09 11:03 pm>[2979142.272175] [<ffffffff810546e2>] kthread+0x7d/0x85
<Apr/09 11:03 pm>[2979142.272177] [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
<Apr/09 11:03 pm>[2979142.272179] [<ffffffff81054665>] ? kthread+0x0/0x85
<Apr/09 11:03 pm>[2979142.272181] [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
<Apr/09 11:03 pm>[2979144.047629] BUG: spinlock lockup on CPU#6, nfsd/8590, ffff880655213c00
<Apr/09 11:03 pm>[2979144.047630] Pid: 8590, comm: nfsd Tainted: G D 2.6.34.4 #3
<Apr/09 11:03 pm>[2979144.047631] Call Trace:
<Apr/09 11:03 pm>[2979144.047632] <IRQ> [<ffffffff812c4e9f>] do_raw_spin_lock+0x118/0x147
<Apr/09 11:03 pm>[2979144.047636] [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
<Apr/09 11:03 pm>[2979144.047638] [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
<Apr/09 11:03 pm>[2979144.047640] [<ffffffff81038588>] scheduler_tick+0x3d/0x236
<Apr/09 11:03 pm>[2979144.047642] [<ffffffff810483d1>] update_process_times+0x49/0x59
<Apr/09 11:03 pm>[2979144.047644] [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
<Apr/09 11:03 pm>[2979144.047646] [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
<Apr/09 11:03 pm>[2979144.047648] [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
<Apr/09 11:03 pm>[2979144.047650] [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:03 pm>[2979144.047653] [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
<Apr/09 11:03 pm>[2979144.047655] [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
<Apr/09 11:03 pm>[2979144.047656] <EOI>
<Apr/09 11:03 pm>[2979144.047657] sending NMI to all CPUs:
<Apr/09 11:03 pm>[2979144.047659] NMI backtrace for cpu 5
<Apr/09 11:03 pm>[2979144.047660] CPU 5
<Apr/09 11:03 pm>[2979144.047661] Modules linked in: w83793 hwmon_vid autofs4 iTCO_wdt e1000e i2c_i801 i2c_core ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
<Apr/09 11:03 pm>[2979144.047674]
<Apr/09 11:03 pm>[2979144.047675] Pid: 8541, comm: nfsd Tainted: G D 2.6.34.4 #3 S7010/empty
<Apr/09 11:03 pm>[2979144.047677] RIP: 0010:[<ffffffff812c08a2>] [<ffffffff812c08a2>] delay_tsc+0x18/0x59
<Apr/09 11:03 pm>[2979144.047680] RSP: 0018:ffff880621edfb48 EFLAGS: 00000096
<Apr/09 11:03 pm>[2979144.047681] RAX: 00000000ab55d014 RBX: 000000000407cd3a RCX: 00000000ab55cfd0
<Apr/09 11:03 pm>[2979144.047683] RDX: 00000000001c399f RSI: 0000000000000005 RDI: 0000000000000001
<Apr/09 11:03 pm>[2979144.047684] RBP: ffff880621edfb50 R08: 0000000000000005 R09: 0000000000000005
<Apr/09 11:03 pm>[2979144.047686] R10: 0000000000000000 R11: ffff880621edf980 R12: ffff880655213c00
<Apr/09 11:03 pm>[2979144.047687] R13: 0000000000000000 R14: ffff8806231bb090 R15: ffff8806231bb518
<Apr/09 11:03 pm>[2979144.047689] FS: 0000000000000000(0000) GS:ffff8800026a0000(0000) knlGS:0000000000000000
<Apr/09 11:03 pm>[2979144.047691] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
<Apr/09 11:04 pm>[2979144.047692] CR2: 00007ff62c558028 CR3: 0000000001a8a000 CR4: 00000000000006e0
<Apr/09 11:04 pm>[2979144.047694] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<Apr/09 11:04 pm>[2979144.047695] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<Apr/09 11:04 pm>[2979144.047697] Process nfsd (pid: 8541, threadinfo ffff880621ede000, task ffff8806231bb090)
<Apr/09 11:04 pm>[2979144.047698] Stack:
<Apr/09 11:04 pm>[2979144.047699] ffffffff81019da9 ffff880621edfb60 ffffffff812c08fe ffff880621edfba0
<Apr/09 11:04 pm>[2979144.047701] <0> ffffffff812c4e65 000000009ef30658 ffff880655213c00 ffff8806230b3090
<Apr/09 11:04 pm>[2979144.047703] <0> 0000000000013c00 ffff880621edfc10 0000000000000005 ffff880621edfbc0
<Apr/09 11:04 pm>[2979144.047705] Call Trace:
<Apr/09 11:04 pm>[2979144.047708] [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
<Apr/09 11:04 pm>[2979144.047710] [<ffffffff812c08fe>] __delay+0xa/0xc
<Apr/09 11:04 pm>[2979144.047712] [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
<Apr/09 11:04 pm>[2979144.047714] [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
<Apr/09 11:04 pm>[2979144.047716] [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
<Apr/09 11:04 pm>[2979144.047718] [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
<Apr/09 11:04 pm>[2979144.047720] [<ffffffff81036578>] try_to_wake_up+0x2e/0x342
<Apr/09 11:04 pm>[2979144.047722] [<ffffffff81036899>] default_wake_function+0xd/0xf
<Apr/09 11:04 pm>[2979144.047724] [<ffffffff8102c08d>] __wake_up_common+0x46/0x76
<Apr/09 11:04 pm>[2979144.047726] [<ffffffff8102faa6>] __wake_up+0x38/0x4f
<Apr/09 11:04 pm>[2979144.047728] [<ffffffff814fbb75>] svc_xprt_enqueue+0x1e2/0x23d
<Apr/09 11:04 pm>[2979144.047731] [<ffffffff814fbc26>] svc_xprt_received+0x24/0x26
<Apr/09 11:04 pm>[2979144.047733] [<ffffffff814f1a12>] svc_udp_recvfrom+0x15f/0x2f9
<Apr/09 11:04 pm>[2979144.047735] [<ffffffff814fc980>] svc_recv+0x758/0x7f5
<Apr/09 11:04 pm>[2979144.047737] [<ffffffff814fc1d9>] ? svc_send+0x85/0xa1
<Apr/09 11:04 pm>[2979144.047739] [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
<Apr/09 11:04 pm>[2979144.047741] [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
<Apr/09 11:04 pm>[2979144.047743] [<ffffffff811f0d78>] nfsd+0xa9/0x139
<Apr/09 11:04 pm>[2979144.047745] [<ffffffff810546e2>] kthread+0x7d/0x85
<Apr/09 11:04 pm>[2979144.047747] [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
<Apr/09 11:04 pm>[2979144.047749] [<ffffffff81054665>] ? kthread+0x0/0x85
<Apr/09 11:04 pm>[2979144.047751] [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
<Apr/09 11:04 pm>[2979144.047752] Code: 2e 0f 1f 84 00 00 00 00 00 48 ff c8 75 fb 48 ff c8 c9 c3 55 48 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae e8 0f 31 <89> c1 41 89 f1 66 66 90 0f ae e8 0f 31 89 c6 48 89 f0 48 29 c8
<Apr/09 11:04 pm>[2979144.047767] Call Trace:
<Apr/09 11:04 pm>[2979144.047770] [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
<Apr/09 11:04 pm>[2979144.047772] [<ffffffff812c08fe>] __delay+0xa/0xc
<Apr/09 11:04 pm>[2979144.047774] [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
<Apr/09 11:04 pm>[2979144.047776] [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
<Apr/09 11:04 pm>[2979144.047778] [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
<Apr/09 11:04 pm>[2979144.047780] [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
<Apr/09 11:04 pm>[2979144.047782] [<ffffffff81036578>] try_to_wake_up+0x2e/0x342
<Apr/09 11:04 pm>[2979144.047784] [<ffffffff81036899>] default_wake_function+0xd/0xf
<Apr/09 11:04 pm>[2979144.047786] [<ffffffff8102c08d>] __wake_up_common+0x46/0x76
<Apr/09 11:04 pm>[2979144.047788] [<ffffffff8102faa6>] __wake_up+0x38/0x4f
<Apr/09 11:04 pm>[2979144.047790] [<ffffffff814fbb75>] svc_xprt_enqueue+0x1e2/0x23d
<Apr/09 11:04 pm>[2979144.047792] [<ffffffff814fbc26>] svc_xprt_received+0x24/0x26
<Apr/09 11:04 pm>[2979144.047794] [<ffffffff814f1a12>] svc_udp_recvfrom+0x15f/0x2f9
<Apr/09 11:04 pm>[2979144.047796] [<ffffffff814fc980>] svc_recv+0x758/0x7f5
<Apr/09 11:04 pm>[2979144.047798] [<ffffffff814fc1d9>] ? svc_send+0x85/0xa1
<Apr/09 11:04 pm>[2979144.047801] [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
<Apr/09 11:04 pm>[2979144.047803] [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
<Apr/09 11:04 pm>[2979144.047804] [<ffffffff811f0d78>] nfsd+0xa9/0x139
<Apr/09 11:04 pm>[2979144.047806] [<ffffffff810546e2>] kthread+0x7d/0x85
<Apr/09 11:04 pm>[2979144.047808] [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
<Apr/09 11:04 pm>[2979144.047811] [<ffffffff81054665>] ? kthread+0x0/0x85
<Apr/09 11:04 pm>[2979144.047813] [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
<Apr/09 11:04 pm>[2979144.047814] Pid: 8541, comm: nfsd Tainted: G D 2.6.34.4 #3
<Apr/09 11:04 pm>[2979144.047815] Call Trace:
<Apr/09 11:04 pm>[2979144.047816] <NMI> [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
<Apr/09 11:04 pm>[2979144.047820] [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
<Apr/09 11:04 pm>[2979144.047822] [<ffffffff81520360>] do_nmi+0xb2/0x259
<Apr/09 11:04 pm>[2979144.047823] [<ffffffff8151fdfa>] nmi+0x1a/0x20
<Apr/09 11:04 pm>[2979144.047826] [<ffffffff812c08a2>] ? delay_tsc+0x18/0x59
<Apr/09 11:04 pm>[2979144.047827] <<EOE>> [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
<Apr/09 11:04 pm>[2979144.047830] [<ffffffff812c08fe>] __delay+0xa/0xc
<Apr/09 11:04 pm>[2979144.047832] [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
<Apr/09 11:04 pm>[2979144.047835] [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
<Apr/09 11:04 pm>[2979144.047836] [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
<Apr/09 11:04 pm>[2979144.047838] [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
<Apr/09 11:04 pm>[2979144.047840] [<ffffffff81036578>] try_to_wake_up+0x2e/0x342
<Apr/09 11:04 pm>[2979144.047843] [<ffffffff81036899>] default_wake_function+0xd/0xf
<Apr/09 11:04 pm>[2979144.047845] [<ffffffff8102c08d>] __wake_up_common+0x46/0x76
<Apr/09 11:04 pm>[2979144.047847] [<ffffffff8102faa6>] __wake_up+0x38/0x4f
<Apr/09 11:04 pm>[2979144.047849] [<ffffffff814fbb75>] svc_xprt_enqueue+0x1e2/0x23d
<Apr/09 11:04 pm>[2979144.047851] [<ffffffff814fbc26>] svc_xprt_received+0x24/0x26
<Apr/09 11:04 pm>[2979144.047853] [<ffffffff814f1a12>] svc_udp_recvfrom+0x15f/0x2f9
<Apr/09 11:04 pm>[2979144.047855] [<ffffffff814fc980>] svc_recv+0x758/0x7f5
<Apr/09 11:04 pm>[2979144.047857] [<ffffffff814fc1d9>] ? svc_send+0x85/0xa1
<Apr/09 11:04 pm>[2979144.047860] [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
<Apr/09 11:04 pm>[2979144.047861] [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
<Apr/09 11:04 pm>[2979144.047863] [<ffffffff811f0d78>] nfsd+0xa9/0x139
<Apr/09 11:04 pm>[2979144.047865] [<ffffffff810546e2>] kthread+0x7d/0x85
<Apr/09 11:04 pm>[2979144.047867] [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
<Apr/09 11:04 pm>[2979144.047870] [<ffffffff81054665>] ? kthread+0x0/0x85
<Apr/09 11:04 pm>[2979144.047871] [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
<Apr/09 11:04 pm>[2979144.047873] NMI backtrace for cpu 6
<Apr/09 11:04 pm>[2979144.047874] CPU 6
<Apr/09 11:04 pm>[2979144.047875] Modules linked in: w83793 hwmon_vid autofs4 iTCO_wdt e1000e i2c_i801 i2c_core ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
<Apr/09 11:04 pm>[2979144.047889]
<Apr/09 11:04 pm>[2979144.047890] Pid: 8590, comm: nfsd Tainted: G D 2.6.34.4 #3 S7010/empty
<Apr/09 11:04 pm>[2979144.047891] RIP: 0010:[<ffffffff812c08af>] [<ffffffff812c08af>] delay_tsc+0x25/0x59
<Apr/09 11:04 pm>[2979144.047894] RSP: 0018:ffff880655203d50 EFLAGS: 00000046
<Apr/09 11:04 pm>[2979144.047896] RAX: 00000000ab59e364 RBX: 0000000000000000 RCX: 00000000ab59e31c
<Apr/09 11:04 pm>[2979144.047897] RDX: 00000000001c399f RSI: 00000000ab59e338 RDI: 00000000000411b3
<Apr/09 11:04 pm>[2979144.047899] RBP: ffff880655203d58 R08: 0000000000000006 R09: 0000000000000006
<Apr/09 11:04 pm>[2979144.047900] R10: 000000000000000b R11: ffff880655203c18 R12: 0000000000001000
<Apr/09 11:04 pm>[2979144.047902] R13: 0000000000000002 R14: 0000000000000092 R15: ffff8806227e0022
<Apr/09 11:04 pm>[2979144.047904] FS: 0000000000000000(0000) GS:ffff880655200000(0000) knlGS:0000000000000000
<Apr/09 11:04 pm>[2979144.047905] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
<Apr/09 11:04 pm>[2979144.047907] CR2: 0000000389ca1c18 CR3: 0000000001a8a000 CR4: 00000000000006e0
<Apr/09 11:04 pm>[2979144.047908] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<Apr/09 11:04 pm>[2979144.047910] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<Apr/09 11:04 pm>[2979144.047911] Process nfsd (pid: 8590, threadinfo ffff880622754000, task ffff8806227ed0f0)
<Apr/09 11:04 pm>[2979144.047913] Stack:
<Apr/09 11:04 pm>[2979144.047913] 0000000000000000 ffff880655203d68 ffffffff812c093d ffff880655203d88
<Apr/09 11:04 pm>[2979144.047915] <0> ffffffff810197e9 0000000000000007 000000000000d200 ffff880655203dc8
<Apr/09 11:04 pm>[2979144.047918] <0> ffffffff81019c76 ffffffff81b24210 0000000000000000 ffff880655213c00
<Apr/09 11:04 pm>[2979144.047920] Call Trace:
<Apr/09 11:04 pm>[2979144.047921] <IRQ>
<Apr/09 11:04 pm>[2979144.047923] [<ffffffff812c093d>] __const_udelay+0x3d/0x3f
<Apr/09 11:04 pm>[2979144.047925] [<ffffffff810197e9>] native_safe_apic_wait_icr_idle+0x31/0x48
<Apr/09 11:04 pm>[2979144.047928] [<ffffffff81019c76>] default_send_IPI_mask_sequence_phys+0x49/0xab
<Apr/09 11:04 pm>[2979144.047930] [<ffffffff8101c5d8>] physflat_send_IPI_all+0x14/0x16
<Apr/09 11:04 pm>[2979144.047933] [<ffffffff81019d94>] arch_trigger_all_cpu_backtrace+0x3c/0x5d
<Apr/09 11:04 pm>[2979144.047935] [<ffffffff812c4ea4>] do_raw_spin_lock+0x11d/0x147
<Apr/09 11:04 pm>[2979144.047937] [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
<Apr/09 11:04 pm>[2979144.047939] [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
<Apr/09 11:04 pm>[2979144.047941] [<ffffffff81038588>] scheduler_tick+0x3d/0x236
<Apr/09 11:04 pm>[2979144.047944] [<ffffffff810483d1>] update_process_times+0x49/0x59
<Apr/09 11:04 pm>[2979144.047946] [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
<Apr/09 11:04 pm>[2979144.047948] [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
<Apr/09 11:04 pm>[2979144.047950] [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
<Apr/09 11:04 pm>[2979144.047952] [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:04 pm>[2979144.047954] [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
<Apr/09 11:04 pm>[2979144.047956] [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
<Apr/09 11:04 pm>[2979144.047958] <EOI>
<Apr/09 11:04 pm>[2979144.047958] Code: fb 48 ff c8 c9 c3 55 48 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae e8 0f 31 89 c1 41 89 f1 66 66 90 0f ae e8 0f 31 <89> c6 48 89 f0 48 29 c8 48 39 f8 73 25 f3 90 65 44 8b 04 25 f0
<Apr/09 11:04 pm>[2979144.047974] Call Trace:
<Apr/09 11:04 pm>[2979144.047975] <IRQ> [<ffffffff812c093d>] __const_udelay+0x3d/0x3f
<Apr/09 11:04 pm>[2979144.047978] [<ffffffff810197e9>] native_safe_apic_wait_icr_idle+0x31/0x48
<Apr/09 11:04 pm>[2979144.047981] [<ffffffff81019c76>] default_send_IPI_mask_sequence_phys+0x49/0xab
<Apr/09 11:04 pm>[2979144.047984] [<ffffffff8101c5d8>] physflat_send_IPI_all+0x14/0x16
<Apr/09 11:04 pm>[2979144.047986] [<ffffffff81019d94>] arch_trigger_all_cpu_backtrace+0x3c/0x5d
<Apr/09 11:04 pm>[2979144.047988] [<ffffffff812c4ea4>] do_raw_spin_lock+0x11d/0x147
<Apr/09 11:04 pm>[2979144.047990] [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
<Apr/09 11:04 pm>[2979144.047993] [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
<Apr/09 11:04 pm>[2979144.047995] [<ffffffff81038588>] scheduler_tick+0x3d/0x236
<Apr/09 11:04 pm>[2979144.047997] [<ffffffff810483d1>] update_process_times+0x49/0x59
<Apr/09 11:04 pm>[2979144.047999] [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
<Apr/09 11:04 pm>[2979144.048001] [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
<Apr/09 11:04 pm>[2979144.048003] [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
<Apr/09 11:04 pm>[2979144.048005] [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:04 pm>[2979144.048007] [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
<Apr/09 11:04 pm>[2979144.048009] [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
<Apr/09 11:04 pm>[2979144.048011] <EOI> Pid: 8590, comm: nfsd Tainted: G D 2.6.34.4 #3
<Apr/09 11:04 pm>[2979144.048013] Call Trace:
<Apr/09 11:04 pm>[2979144.048013] <NMI> [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
<Apr/09 11:04 pm>[2979144.048017] [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
<Apr/09 11:04 pm>[2979144.048019] [<ffffffff81520360>] do_nmi+0xb2/0x259
<Apr/09 11:04 pm>[2979144.048021] [<ffffffff8151fdfa>] nmi+0x1a/0x20
<Apr/09 11:04 pm>[2979144.048023] [<ffffffff812c08af>] ? delay_tsc+0x25/0x59
<Apr/09 11:04 pm>[2979144.048024] <<EOE>> <IRQ> [<ffffffff812c093d>] __const_udelay+0x3d/0x3f
<Apr/09 11:04 pm>[2979144.048028] [<ffffffff810197e9>] native_safe_apic_wait_icr_idle+0x31/0x48
<Apr/09 11:04 pm>[2979144.048030] [<ffffffff81019c76>] default_send_IPI_mask_sequence_phys+0x49/0xab
<Apr/09 11:04 pm>[2979144.048033] [<ffffffff8101c5d8>] physflat_send_IPI_all+0x14/0x16
<Apr/09 11:04 pm>[2979144.048035] [<ffffffff81019d94>] arch_trigger_all_cpu_backtrace+0x3c/0x5d
<Apr/09 11:04 pm>[2979144.048037] [<ffffffff812c4ea4>] do_raw_spin_lock+0x11d/0x147
<Apr/09 11:04 pm>[2979144.048040] [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
<Apr/09 11:04 pm>[2979144.048042] [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
<Apr/09 11:04 pm>[2979144.048044] [<ffffffff81038588>] scheduler_tick+0x3d/0x236
<Apr/09 11:04 pm>[2979144.048046] [<ffffffff810483d1>] update_process_times+0x49/0x59
<Apr/09 11:04 pm>[2979144.048048] [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
<Apr/09 11:04 pm>[2979144.048050] [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
<Apr/09 11:04 pm>[2979144.048052] [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
<Apr/09 11:04 pm>[2979144.048054] [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:04 pm>[2979144.048056] [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
<Apr/09 11:04 pm>[2979144.048059] [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
<Apr/09 11:04 pm>[2979144.048060] <EOI>
<Apr/09 11:04 pm>[2979144.048062] NMI backtrace for cpu 7
<Apr/09 11:04 pm>[2979144.048063] CPU 7
<Apr/09 11:04 pm>[2979144.048063] Modules linked in: w83793 hwmon_vid autofs4 iTCO_wdt e1000e i2c_i801 i2c_core ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
<Apr/09 11:04 pm>[2979144.048077]
<Apr/09 11:04 pm>[2979144.048078] Pid: 8574, comm: nfsd Tainted: G D 2.6.34.4 #3 S7010/empty
<Apr/09 11:04 pm>[2979144.048079] RIP: 0010:[<ffffffff812c08ad>] [<ffffffff812c08ad>] delay_tsc+0x23/0x59
<Apr/09 11:04 pm>[2979144.048083] RSP: 0018:ffff880655223920 EFLAGS: 00000046
<Apr/09 11:04 pm>[2979144.048084] RAX: 00000000000002fc RBX: 0000000000000000 RCX: 00000000ab663114
<Apr/09 11:04 pm>[2979144.048086] RDX: 00000000001c399f RSI: 00000000ab663410 RDI: 0000000000000a6b
<Apr/09 11:04 pm>[2979144.048087] RBP: ffff880655223928 R08: 0000000000000007 R09: 0000000000000007
<Apr/09 11:04 pm>[2979144.048089] R10: ffff8800000b8820 R11: ffffffff81334e4e R12: ffffffff82400ac0
<Apr/09 11:04 pm>[2979144.048090] R13: 00000000000026f3 R14: 0000000000000020 R15: 0000000000000044
<Apr/09 11:04 pm>[2979144.048092] FS: 0000000000000000(0000) GS:ffff880655220000(0000) knlGS:0000000000000000
<Apr/09 11:04 pm>[2979144.048094] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
<Apr/09 11:04 pm>[2979144.048095] CR2: 00007f7053196000 CR3: 0000000001a8a000 CR4: 00000000000006e0
<Apr/09 11:04 pm>[2979144.048097] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<Apr/09 11:04 pm>[2979144.048099] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<Apr/09 11:04 pm>[2979144.048100] Process nfsd (pid: 8574, threadinfo ffff880623fec000, task ffff8806227d3090)
<Apr/09 11:04 pm>[2979144.048102] Stack:
<Apr/09 11:04 pm>[2979144.048102] 0000000000000000 ffff880655223938 ffffffff812c093d ffff880655223968
<Apr/09 11:04 pm>[2979144.048104] <0> ffffffff81334786 ffffffff82400ac0 0000000000000031 ffffffff813347d5
<Apr/09 11:04 pm>[2979144.048107] <0> ffffffff82400ac0 ffff880655223988 ffffffff813347ec ffffffff81c3dc36
<Apr/09 11:04 pm>[2979144.048109] Call Trace:
<Apr/09 11:04 pm>[2979144.048110] <IRQ>
<Apr/09 11:04 pm>[2979144.048112] [<ffffffff812c093d>] __const_udelay+0x3d/0x3f
<Apr/09 11:04 pm>[2979144.048114] [<ffffffff81334786>] wait_for_xmitr+0x43/0x92
<Apr/09 11:04 pm>[2979144.048116] [<ffffffff813347d5>] ? serial8250_console_putchar+0x0/0x27
<Apr/09 11:04 pm>[2979144.048119] [<ffffffff813347ec>] serial8250_console_putchar+0x17/0x27
<Apr/09 11:04 pm>[2979144.048121] [<ffffffff813311e8>] uart_console_write+0x41/0x52
<Apr/09 11:04 pm>[2979144.048123] [<ffffffff81334eb0>] serial8250_console_write+0xb6/0x108
<Apr/09 11:04 pm>[2979144.048126] [<ffffffff8103b339>] __call_console_drivers+0x67/0x79
<Apr/09 11:04 pm>[2979144.048128] [<ffffffff8103b3a6>] _call_console_drivers+0x5b/0x5f
<Apr/09 11:04 pm>[2979144.048131] [<ffffffff8103b87b>] release_console_sem+0x121/0x1c6
<Apr/09 11:04 pm>[2979144.048133] [<ffffffff8103beb2>] vprintk+0x36b/0x3a0
<Apr/09 11:04 pm>[2979144.048135] [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:04 pm>[2979144.048138] [<ffffffff8151c589>] printk+0x67/0x69
<Apr/09 11:04 pm>[2979144.048139] [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:04 pm>[2979144.048141] [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:04 pm>[2979144.048144] [<ffffffff8151c589>] ? printk+0x67/0x69
<Apr/09 11:04 pm>[2979144.048146] [<ffffffff81019eb4>] ? touch_nmi_watchdog+0x5b/0x60
<Apr/09 11:04 pm>[2979144.048148] [<ffffffff810060bb>] print_trace_address+0x4e/0x57
<Apr/09 11:04 pm>[2979144.048150] [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:04 pm>[2979144.048152] [<ffffffff810061e9>] print_context_stack+0x62/0xbd
<Apr/09 11:04 pm>[2979144.048154] [<ffffffff810054e2>] dump_trace+0x23d/0x327
<Apr/09 11:04 pm>[2979144.048156] [<ffffffff8100604f>] show_trace_log_lvl+0x4c/0x58
<Apr/09 11:04 pm>[2979144.048158] [<ffffffff8100606b>] show_trace+0x10/0x12
<Apr/09 11:04 pm>[2979144.048160] [<ffffffff8151c401>] dump_stack+0x72/0x7b
<Apr/09 11:04 pm>[2979144.048162] [<ffffffff812c4e9f>] do_raw_spin_lock+0x118/0x147
<Apr/09 11:04 pm>[2979144.048165] [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
<Apr/09 11:04 pm>[2979144.048167] [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
<Apr/09 11:04 pm>[2979144.048169] [<ffffffff81038588>] scheduler_tick+0x3d/0x236
<Apr/09 11:04 pm>[2979144.048171] [<ffffffff810483d1>] update_process_times+0x49/0x59
<Apr/09 11:04 pm>[2979144.048173] [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
<Apr/09 11:04 pm>[2979144.048175] [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
<Apr/09 11:04 pm>[2979144.048177] [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
<Apr/09 11:04 pm>[2979144.048179] [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:04 pm>[2979144.048181] [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
<Apr/09 11:04 pm>[2979144.048184] [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
<Apr/09 11:04 pm>[2979144.048185] <EOI>
<Apr/09 11:04 pm>[2979144.048187] [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
<Apr/09 11:04 pm>[2979144.048188] [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
<Apr/09 11:04 pm>[2979144.048190] [<ffffffff810d48fd>] ? get_swap_page+0x13c/0x5bb
<Apr/09 11:04 pm>[2979144.048193] [<ffffffff810d17e9>] add_to_swap+0xf/0x48
<Apr/09 11:04 pm>[2979144.048195] [<ffffffff810b7464>] shrink_page_list+0x214/0x68f
<Apr/09 11:04 pm>[2979144.048197] [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Apr/09 11:04 pm>[2979144.048199] [<ffffffff810e5fb2>] ? mem_cgroup_del_lru_list+0x7b/0x8e
<Apr/09 11:04 pm>[2979144.048202] [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
<Apr/09 11:04 pm>[2979144.048204] [<ffffffff8144b300>] ? release_sock+0xdb/0xe4
<Apr/09 11:04 pm>[2979144.048207] [<ffffffff8144b254>] ? release_sock+0x2f/0xe4
<Apr/09 11:04 pm>[2979144.048209] [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
<Apr/09 11:04 pm>[2979144.048211] [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
<Apr/09 11:04 pm>[2979144.048213] [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
<Apr/09 11:04 pm>[2979144.048216] [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
<Apr/09 11:04 pm>[2979144.048219] [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
<Apr/09 11:04 pm>[2979144.048221] [<ffffffff814fc30d>] svc_recv+0xe5/0x7f5
<Apr/09 11:04 pm>[2979144.048223] [<ffffffff814fc1d9>] ? svc_send+0x85/0xa1
<Apr/09 11:04 pm>[2979144.048225] [<ffffffff811f9bb1>] ? exp_readunlock+0x10/0x12
<Apr/09 11:04 pm>[2979144.048227] [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
<Apr/09 11:04 pm>[2979144.048229] [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
<Apr/09 11:04 pm>[2979144.048231] [<ffffffff811f0d78>] nfsd+0xa9/0x139
<Apr/09 11:04 pm>[2979144.048233] [<ffffffff810546e2>] kthread+0x7d/0x85
<Apr/09 11:04 pm>[2979144.048235] [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
<Apr/09 11:04 pm>[2979144.048237] [<ffffffff81054665>] ? kthread+0x0/0x85
<Apr/09 11:04 pm>[2979144.048239] [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
<Apr/09 11:04 pm>[2979144.048240] Code: c8 75 fb 48 ff c8 c9 c3 55 48 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae e8 0f 31 89 c1 41 89 f1 66 66 90 0f ae e8 <0f> 31 89 c6 48 89 f0 48 29 c8 48 39 f8 73 25 f3 90 65 44 8b 04
<Apr/09 11:04 pm>[2979144.048255] Call Trace:
<Apr/09 11:04 pm>[2979144.048256] <IRQ> [<ffffffff812c093d>] __const_udelay+0x3d/0x3f
<Apr/09 11:04 pm>[2979144.048260] [<ffffffff81334786>] wait_for_xmitr+0x43/0x92
<Apr/09 11:04 pm>[2979144.048262] [<ffffffff813347d5>] ? serial8250_console_putchar+0x0/0x27
<Apr/09 11:04 pm>[2979144.048264] [<ffffffff813347ec>] serial8250_console_putchar+0x17/0x27
<Apr/09 11:04 pm>[2979144.048267] [<ffffffff813311e8>] uart_console_write+0x41/0x52
<Apr/09 11:04 pm>[2979144.048269] [<ffffffff81334eb0>] serial8250_console_write+0xb6/0x108
<Apr/09 11:04 pm>[2979144.048272] [<ffffffff8103b339>] __call_console_drivers+0x67/0x79
<Apr/09 11:04 pm>[2979144.048274] [<ffffffff8103b3a6>] _call_console_drivers+0x5b/0x5f
<Apr/09 11:04 pm>[2979144.048276] [<ffffffff8103b87b>] release_console_sem+0x121/0x1c6
<Apr/09 11:04 pm>[2979144.048279] [<ffffffff8103beb2>] vprintk+0x36b/0x3a0
<Apr/09 11:04 pm>[2979144.048281] [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:04 pm>[2979144.048283] [<ffffffff8151c589>] printk+0x67/0x69
<Apr/09 11:04 pm>[2979144.048285] [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:04 pm>[2979144.048287] [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:04 pm>[2979144.048289] [<ffffffff8151c589>] ? printk+0x67/0x69
<Apr/09 11:04 pm>[2979144.048291] [<ffffffff81019eb4>] ? touch_nmi_watchdog+0x5b/0x60
<Apr/09 11:04 pm>[2979144.048293] [<ffffffff810060bb>] print_trace_address+0x4e/0x57
<Apr/09 11:04 pm>[2979144.048295] [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:04 pm>[2979144.048297] [<ffffffff810061e9>] print_context_stack+0x62/0xbd
<Apr/09 11:04 pm>[2979144.048299] [<ffffffff810054e2>] dump_trace+0x23d/0x327
<Apr/09 11:04 pm>[2979144.048302] [<ffffffff8100604f>] show_trace_log_lvl+0x4c/0x58
<Apr/09 11:04 pm>[2979144.048304] [<ffffffff8100606b>] show_trace+0x10/0x12
<Apr/09 11:04 pm>[2979144.048306] [<ffffffff8151c401>] dump_stack+0x72/0x7b
<Apr/09 11:04 pm>[2979144.048308] [<ffffffff812c4e9f>] do_raw_spin_lock+0x118/0x147
<Apr/09 11:04 pm>[2979144.048310] [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
<Apr/09 11:04 pm>[2979144.048312] [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
<Apr/09 11:04 pm>[2979144.048314] [<ffffffff81038588>] scheduler_tick+0x3d/0x236
<Apr/09 11:04 pm>[2979144.048316] [<ffffffff810483d1>] update_process_times+0x49/0x59
<Apr/09 11:04 pm>[2979144.048318] [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
<Apr/09 11:04 pm>[2979144.048320] [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
<Apr/09 11:04 pm>[2979144.048322] [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
<Apr/09 11:04 pm>[2979144.048324] [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:04 pm>[2979144.048326] [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
<Apr/09 11:04 pm>[2979144.048329] [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
<Apr/09 11:04 pm>[2979144.048330] <EOI> [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
<Apr/09 11:04 pm>[2979144.048333] [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
<Apr/09 11:04 pm>[2979144.048335] [<ffffffff810d48fd>] ? get_swap_page+0x13c/0x5bb
<Apr/09 11:04 pm>[2979144.048337] [<ffffffff810d17e9>] add_to_swap+0xf/0x48
<Apr/09 11:04 pm>[2979144.048339] [<ffffffff810b7464>] shrink_page_list+0x214/0x68f
<Apr/09 11:04 pm>[2979144.048342] [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Apr/09 11:04 pm>[2979144.048344] [<ffffffff810e5fb2>] ? mem_cgroup_del_lru_list+0x7b/0x8e
<Apr/09 11:04 pm>[2979144.048347] [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
<Apr/09 11:04 pm>[2979144.048349] [<ffffffff8144b300>] ? release_sock+0xdb/0xe4
<Apr/09 11:04 pm>[2979144.048352] [<ffffffff8144b254>] ? release_sock+0x2f/0xe4
<Apr/09 11:04 pm>[2979144.048354] [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
<Apr/09 11:04 pm>[2979144.048356] [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
<Apr/09 11:04 pm>[2979144.048358] [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
<Apr/09 11:04 pm>[2979144.048361] [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
<Apr/09 11:04 pm>[2979144.048363] [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
<Apr/09 11:04 pm>[2979144.048366] [<ffffffff814fc30d>] svc_recv+0xe5/0x7f5
<Apr/09 11:04 pm>[2979144.048367] [<ffffffff814fc1d9>] ? svc_send+0x85/0xa1
<Apr/09 11:04 pm>[2979144.048370] [<ffffffff811f9bb1>] ? exp_readunlock+0x10/0x12
<Apr/09 11:04 pm>[2979144.048372] [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
<Apr/09 11:04 pm>[2979144.048374] [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
<Apr/09 11:04 pm>[2979144.048375] [<ffffffff811f0d78>] nfsd+0xa9/0x139
<Apr/09 11:04 pm>[2979144.048377] [<ffffffff810546e2>] kthread+0x7d/0x85
<Apr/09 11:04 pm>[2979144.048379] [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
<Apr/09 11:04 pm>[2979144.048382] [<ffffffff81054665>] ? kthread+0x0/0x85
<Apr/09 11:04 pm>[2979144.048383] [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
<Apr/09 11:04 pm>[2979144.048385] Pid: 8574, comm: nfsd Tainted: G D 2.6.34.4 #3
<Apr/09 11:04 pm>[2979144.048386] Call Trace:
<Apr/09 11:04 pm>[2979144.048387] <NMI> [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
<Apr/09 11:04 pm>[2979144.048391] [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
<Apr/09 11:04 pm>[2979144.048392] [<ffffffff81520360>] do_nmi+0xb2/0x259
<Apr/09 11:04 pm>[2979144.048394] [<ffffffff8151fdfa>] nmi+0x1a/0x20
<Apr/09 11:04 pm>[2979144.048397] [<ffffffff81334e4e>] ? serial8250_console_write+0x54/0x108
<Apr/09 11:04 pm>[2979144.048399] [<ffffffff812c08ad>] ? delay_tsc+0x23/0x59
<Apr/09 11:04 pm>[2979144.048400] <<EOE>> <IRQ> [<ffffffff812c093d>] __const_udelay+0x3d/0x3f
<Apr/09 11:04 pm>[2979144.048404] [<ffffffff81334786>] wait_for_xmitr+0x43/0x92
<Apr/09 11:04 pm>[2979144.048406] [<ffffffff813347d5>] ? serial8250_console_putchar+0x0/0x27
<Apr/09 11:04 pm>[2979144.048409] [<ffffffff813347ec>] serial8250_console_putchar+0x17/0x27
<Apr/09 11:04 pm>[2979144.048411] [<ffffffff813311e8>] uart_console_write+0x41/0x52
<Apr/09 11:04 pm>[2979144.048413] [<ffffffff81334eb0>] serial8250_console_write+0xb6/0x108
<Apr/09 11:04 pm>[2979144.048416] [<ffffffff8103b339>] __call_console_drivers+0x67/0x79
<Apr/09 11:04 pm>[2979144.048418] [<ffffffff8103b3a6>] _call_console_drivers+0x5b/0x5f
<Apr/09 11:04 pm>[2979144.048420] [<ffffffff8103b87b>] release_console_sem+0x121/0x1c6
<Apr/09 11:04 pm>[2979144.048423] [<ffffffff8103beb2>] vprintk+0x36b/0x3a0
<Apr/09 11:04 pm>[2979144.048425] [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:04 pm>[2979144.048427] [<ffffffff8151c589>] printk+0x67/0x69
<Apr/09 11:04 pm>[2979144.048429] [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:04 pm>[2979144.048431] [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:04 pm>[2979144.048433] [<ffffffff8151c589>] ? printk+0x67/0x69
<Apr/09 11:04 pm>[2979144.048435] [<ffffffff81019eb4>] ? touch_nmi_watchdog+0x5b/0x60
<Apr/09 11:04 pm>[2979144.048438] [<ffffffff810060bb>] print_trace_address+0x4e/0x57
<Apr/09 11:04 pm>[2979144.048440] [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:04 pm>[2979144.048442] [<ffffffff810061e9>] print_context_stack+0x62/0xbd
<Apr/09 11:04 pm>[2979144.048444] [<ffffffff810054e2>] dump_trace+0x23d/0x327
<Apr/09 11:04 pm>[2979144.048446] [<ffffffff8100604f>] show_trace_log_lvl+0x4c/0x58
<Apr/09 11:04 pm>[2979144.048448] [<ffffffff8100606b>] show_trace+0x10/0x12
<Apr/09 11:04 pm>[2979144.048450] [<ffffffff8151c401>] dump_stack+0x72/0x7b
<Apr/09 11:04 pm>[2979144.048452] [<ffffffff812c4e9f>] do_raw_spin_lock+0x118/0x147
<Apr/09 11:04 pm>[2979144.048454] [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
<Apr/09 11:04 pm>[2979144.048456] [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
<Apr/09 11:04 pm>[2979144.048458] [<ffffffff81038588>] scheduler_tick+0x3d/0x236
<Apr/09 11:04 pm>[2979144.048461] [<ffffffff810483d1>] update_process_times+0x49/0x59
<Apr/09 11:04 pm>[2979144.048463] [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
<Apr/09 11:04 pm>[2979144.048465] [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
<Apr/09 11:04 pm>[2979144.048466] [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
<Apr/09 11:04 pm>[2979144.048469] [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:04 pm>[2979144.048471] [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
<Apr/09 11:04 pm>[2979144.048473] [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
<Apr/09 11:04 pm>[2979144.048474] <EOI> [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
<Apr/09 11:04 pm>[2979144.048478] [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
<Apr/09 11:04 pm>[2979144.048480] [<ffffffff810d48fd>] ? get_swap_page+0x13c/0x5bb
<Apr/09 11:04 pm>[2979144.048482] [<ffffffff810d17e9>] add_to_swap+0xf/0x48
<Apr/09 11:04 pm>[2979144.048484] [<ffffffff810b7464>] shrink_page_list+0x214/0x68f
<Apr/09 11:04 pm>[2979144.048486] [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Apr/09 11:04 pm>[2979144.048489] [<ffffffff810e5fb2>] ? mem_cgroup_del_lru_list+0x7b/0x8e
<Apr/09 11:04 pm>[2979144.048491] [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
<Apr/09 11:04 pm>[2979144.048494] [<ffffffff8144b300>] ? release_sock+0xdb/0xe4
<Apr/09 11:04 pm>[2979144.048496] [<ffffffff8144b254>] ? release_sock+0x2f/0xe4
<Apr/09 11:04 pm>[2979144.048498] [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
<Apr/09 11:04 pm>[2979144.048500] [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
<Apr/09 11:04 pm>[2979144.048503] [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
<Apr/09 11:04 pm>[2979144.048505] [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
<Apr/09 11:04 pm>[2979144.048508] [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
<Apr/09 11:04 pm>[2979144.048510] [<ffffffff814fc30d>] svc_recv+0xe5/0x7f5
<Apr/09 11:04 pm>[2979144.048512] [<ffffffff814fc1d9>] ? svc_send+0x85/0xa1
<Apr/09 11:04 pm>[2979144.048514] [<ffffffff811f9bb1>] ? exp_readunlock+0x10/0x12
<Apr/09 11:04 pm>[2979144.048516] [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
<Apr/09 11:04 pm>[2979144.048518] [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
<Apr/09 11:04 pm>[2979144.048519] [<ffffffff811f0d78>] nfsd+0xa9/0x139
<Apr/09 11:04 pm>[2979144.048521] [<ffffffff810546e2>] kthread+0x7d/0x85
<Apr/09 11:04 pm>[2979144.048524] [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
<Apr/09 11:04 pm>[2979144.048526] [<ffffffff81054665>] ? kthread+0x0/0x85
<Apr/09 11:04 pm>[2979144.048528] [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
<Apr/09 11:04 pm>[2979147.680342] [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
<Apr/09 11:04 pm>[2979147.687277] [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
<Apr/09 11:04 pm>[2979147.693896] <EOI> [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
<Apr/09 11:04 pm>[2979147.700832] [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
<Apr/09 11:04 pm>[2979147.707094] [<ffffffff810d48fd>] ? get_swap_page+0x13c/0x5bb
<Apr/09 11:04 pm>[2979147.713439] [<ffffffff810d17e9>] add_to_swap+0xf/0x48
<Apr/09 11:04 pm>[2979147.719149] [<ffffffff810b7464>] shrink_page_list+0x214/0x68f
<Apr/09 11:04 pm>[2979147.725582] [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Apr/09 11:04 pm>[2979147.732030] [<ffffffff810e5fb2>] ? mem_cgroup_del_lru_list+0x7b/0x8e
<Apr/09 11:04 pm>[2979147.739095] [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
<Apr/09 11:04 pm>[2979147.745084] [<ffffffff8144b300>] ? release_sock+0xdb/0xe4
<Apr/09 11:04 pm>[2979147.751158] [<ffffffff8144b254>] ? release_sock+0x2f/0xe4
<Apr/09 11:04 pm>[2979147.757242] [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
<Apr/09 11:04 pm>[2979147.764031] [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
<Apr/09 11:04 pm>[2979147.770357] [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
<Apr/09 11:04 pm>[2979147.777135] [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
<Apr/09 11:04 pm>[2979147.784114] [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
<Apr/09 11:04 pm>[2979147.790637] [<ffffffff814fc30d>] svc_recv+0xe5/0x7f5
<Apr/09 11:04 pm>[2979147.796242] [<ffffffff814fc1d9>] ? svc_send+0x85/0xa1
<Apr/09 11:04 pm>[2979147.801919] [<ffffffff811f9bb1>] ? exp_readunlock+0x10/0x12
<Apr/09 11:04 pm>[2979147.808143] [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
<Apr/09 11:04 pm>[2979147.814827] [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
<Apr/09 11:04 pm>[2979147.820144] [<ffffffff811f0d78>] nfsd+0xa9/0x139
<Apr/09 11:04 pm>[2979147.825362] [<ffffffff810546e2>] kthread+0x7d/0x85
<Apr/09 11:04 pm>[2979147.830788] [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
<Apr/09 11:04 pm>[2979147.837317] [<ffffffff81054665>] ? kthread+0x0/0x85
<Apr/09 11:04 pm>[2979147.842844] [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
<Apr/09 11:04 pm>[2979147.849545] sending NMI to all CPUs:
<Apr/09 11:04 pm>[2979147.853637] NMI backtrace for cpu 5
<Apr/09 11:04 pm>[2979147.857632] CPU 5
<Apr/09 11:04 pm>[2979147.859760] Modules linked in: w83793 hwmon_vid autofs4 iTCO_wdt e1000e i2c_i801 i2c_core ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
<Apr/09 11:04 pm>[2979147.895719]
<Apr/09 11:04 pm>[2979147.897633] Pid: 8541, comm: nfsd Tainted: G D 2.6.34.4 #3 S7010/empty
<Apr/09 11:04 pm>[2979147.905657] RIP: 0010:[<ffffffff812c08af>] [<ffffffff812c08af>] delay_tsc+0x25/0x59
<Apr/09 11:04 pm>[2979147.914295] RSP: 0018:ffff880621edfb48 EFLAGS: 00000096
<Apr/09 11:04 pm>[2979147.920170] RAX: 0000000008477830 RBX: 000000000cac0b19 RCX: 0000000008477814
<Apr/09 11:04 pm>[2979147.928106] RDX: 00000000001c39a2 RSI: 0000000000000005 RDI: 0000000000000001
<Apr/09 11:04 pm>[2979147.936018] RBP: ffff880621edfb50 R08: 0000000000000005 R09: 0000000000000005
<Apr/09 11:04 pm>[2979147.943985] R10: 0000000000000000 R11: ffff880621edf980 R12: ffff880655213c00
<Apr/09 11:04 pm>[2979147.951912] R13: 0000000000000000 R14: ffff8806231bb090 R15: ffff8806231bb518
<Apr/09 11:04 pm>[2979147.959853] FS: 0000000000000000(0000) GS:ffff8800026a0000(0000) knlGS:0000000000000000
<Apr/09 11:04 pm>[2979147.968784] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
<Apr/09 11:04 pm>[2979147.975104] CR2: 00007ff62c558028 CR3: 0000000001a8a000 CR4: 00000000000006e0
<Apr/09 11:04 pm>[2979147.983027] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<Apr/09 11:04 pm>[2979147.990983] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<Apr/09 11:04 pm>[2979147.998941] Process nfsd (pid: 8541, threadinfo ffff880621ede000, task ffff8806231bb090)
<Apr/09 11:04 pm>[2979148.007896] Stack:
<Apr/09 11:04 pm>[2979148.010354] ffffffff81019da9 ffff880621edfb60 ffffffff812c08fe ffff880621edfba0
<Apr/09 11:04 pm>[2979148.018370] <0> ffffffff812c4e65 000000009ef30658 ffff880655213c00 ffff8806230b3090
<Apr/09 11:04 pm>[2979148.027115] <0> 0000000000013c00 ffff880621edfc10 0000000000000005 ffff880621edfbc0
<Apr/09 11:04 pm>[2979148.036220] Call Trace:
<Apr/09 11:04 pm>[2979148.039141] [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
<Apr/09 11:04 pm>[2979148.046998] [<ffffffff812c08fe>] __delay+0xa/0xc
<Apr/09 11:04 pm>[2979148.052262] [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
<Apr/09 11:04 pm>[2979148.058577] [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
<Apr/09 11:04 pm>[2979148.064641] [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
<Apr/09 11:04 pm>[2979148.070700] [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
<Apr/09 11:04 pm>[2979148.076594] [<ffffffff81036578>] try_to_wake_up+0x2e/0x342
<Apr/09 11:04 pm>[2979148.082729] [<ffffffff81036899>] default_wake_function+0xd/0xf
<Apr/09 11:04 pm>[2979148.089240] [<ffffffff8102c08d>] __wake_up_common+0x46/0x76
<Apr/09 11:04 pm>[2979148.099950] [<ffffffff8102faa6>] __wake_up+0x38/0x4f
<Apr/09 11:04 pm>[2979148.105563] [<ffffffff814fbb75>] svc_xprt_enqueue+0x1e2/0x23d
<Apr/09 11:04 pm>[2979148.111972] [<ffffffff814fbc26>] svc_xprt_received+0x24/0x26
<Apr/09 11:04 pm>[2979148.118290] [<ffffffff814f1a12>] svc_udp_recvfrom+0x15f/0x2f9
<Apr/09 11:04 pm>[2979148.124703] [<ffffffff814fc980>] svc_recv+0x758/0x7f5
<Apr/09 11:04 pm>[2979148.130404] [<ffffffff814fc1d9>] ? svc_send+0x85/0xa1
<Apr/09 11:04 pm>[2979148.136109] [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
<Apr/09 11:04 pm>[2979148.142802] [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
<Apr/09 11:04 pm>[2979148.148162] [<ffffffff811f0d78>] nfsd+0xa9/0x139
<Apr/09 11:04 pm>[2979148.153363] [<ffffffff810546e2>] kthread+0x7d/0x85
<Apr/09 11:04 pm>[2979148.158798] [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
<Apr/09 11:04 pm>[2979148.165318] [<ffffffff81054665>] ? kthread+0x0/0x85
<Apr/09 11:04 pm>[2979148.170844] [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
<Apr/09 11:04 pm>[2979148.177537] Code: fb 48 ff c8 c9 c3 55 48 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae e8 0f 31 89 c1 41 89 f1 66 66 90 0f ae e8 0f 31 <89> c6 48 89 f0 48 29 c8 48 39 f8 73 25 f3 90 65 44 8b 04 25 f0
<Apr/09 11:04 pm>[2979148.200631] Call Trace:
<Apr/09 11:04 pm>[2979148.203536] [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
<Apr/09 11:04 pm>[2979148.211349] [<ffffffff812c08fe>] __delay+0xa/0xc
<Apr/09 11:04 pm>[2979148.216597] [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
<Apr/09 11:04 pm>[2979148.222919] [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
<Apr/09 11:04 pm>[2979148.229001] [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
<Apr/09 11:04 pm>[2979148.235042] [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
<Apr/09 11:04 pm>[2979148.240935] [<ffffffff81036578>] try_to_wake_up+0x2e/0x342
<Apr/09 11:04 pm>[2979148.247081] [<ffffffff81036899>] default_wake_function+0xd/0xf
<Apr/09 11:04 pm>[2979148.253582] [<ffffffff8102c08d>] __wake_up_common+0x46/0x76
<Apr/09 11:04 pm>[2979148.259830] [<ffffffff8102faa6>] __wake_up+0x38/0x4f
<Apr/09 11:04 pm>[2979148.265444] [<ffffffff814fbb75>] svc_xprt_enqueue+0x1e2/0x23d
<Apr/09 11:04 pm>[2979148.271886] [<ffffffff814fbc26>] svc_xprt_received+0x24/0x26
<Apr/09 11:04 pm>[2979148.278230] [<ffffffff814f1a12>] svc_udp_recvfrom+0x15f/0x2f9
<Apr/09 11:04 pm>[2979148.284636] [<ffffffff814fc980>] svc_recv+0x758/0x7f5
<Apr/09 11:04 pm>[2979148.290337] [<ffffffff814fc1d9>] ? svc_send+0x85/0xa1
<Apr/09 11:04 pm>[2979148.296031] [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
<Apr/09 11:04 pm>[2979148.302732] [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
<Apr/09 11:04 pm>[2979148.308085] [<ffffffff811f0d78>] nfsd+0xa9/0x139
<Apr/09 11:04 pm>[2979148.313337] [<ffffffff810546e2>] kthread+0x7d/0x85
<Apr/09 11:04 pm>[2979148.318764] [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
<Apr/09 11:04 pm>[2979148.325258] [<ffffffff81054665>] ? kthread+0x0/0x85
<Apr/09 11:04 pm>[2979148.330793] [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
<Apr/09 11:04 pm>[2979148.337486] Pid: 8541, comm: nfsd Tainted: G D 2.6.34.4 #3
<Apr/09 11:04 pm>[2979148.344257] Call Trace:
<Apr/09 11:04 pm>[2979148.347165] <NMI> [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
<Apr/09 11:04 pm>[2979148.353661] [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
<Apr/09 11:04 pm>[2979148.360082] [<ffffffff81520360>] do_nmi+0xb2/0x259
<Apr/09 11:04 pm>[2979148.365531] [<ffffffff8151fdfa>] nmi+0x1a/0x20
<Apr/09 11:04 pm>[2979148.370600] [<ffffffff812c08af>] ? delay_tsc+0x25/0x59
<Apr/09 11:04 pm>[2979148.376381] <<EOE>> [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
<Apr/09 11:04 pm>[2979148.385101] [<ffffffff812c08fe>] __delay+0xa/0xc
<Apr/09 11:04 pm>[2979148.390353] [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
<Apr/09 11:04 pm>[2979148.396692] [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
<Apr/09 11:04 pm>[2979148.402748] [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
<Apr/09 11:04 pm>[2979148.408798] [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
<Apr/09 11:04 pm>[2979148.414655] [<ffffffff81036578>] try_to_wake_up+0x2e/0x342
<Apr/09 11:04 pm>[2979148.420802] [<ffffffff81036899>] default_wake_function+0xd/0xf
<Apr/09 11:04 pm>[2979148.427329] [<ffffffff8102c08d>] __wake_up_common+0x46/0x76
<Apr/09 11:04 pm>[2979148.433559] [<ffffffff8102faa6>] __wake_up+0x38/0x4f
<Apr/09 11:04 pm>[2979148.439183] [<ffffffff814fbb75>] svc_xprt_enqueue+0x1e2/0x23d
<Apr/09 11:04 pm>[2979148.445591] [<ffffffff814fbc26>] svc_xprt_received+0x24/0x26
<Apr/09 11:04 pm>[2979148.451900] [<ffffffff814f1a12>] svc_udp_recvfrom+0x15f/0x2f9
<Apr/09 11:04 pm>[2979148.458313] [<ffffffff814fc980>] svc_recv+0x758/0x7f5
<Apr/09 11:04 pm>[2979148.464016] [<ffffffff814fc1d9>] ? svc_send+0x85/0xa1
<Apr/09 11:04 pm>[2979148.469708] [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
<Apr/09 11:04 pm>[2979148.476377] [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
<Apr/09 11:04 pm>[2979148.481718] [<ffffffff811f0d78>] nfsd+0xa9/0x139
<Apr/09 11:04 pm>[2979148.486954] [<ffffffff810546e2>] kthread+0x7d/0x85
<Apr/09 11:04 pm>[2979148.492364] [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
<Apr/09 11:04 pm>[2979148.498867] [<ffffffff81054665>] ? kthread+0x0/0x85
<Apr/09 11:04 pm>[2979148.504411] [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
<Apr/09 11:04 pm>[2979148.511087] NMI backtrace for cpu 7
<Apr/09 11:04 pm>[2979148.515091] CPU 7
<Apr/09 11:04 pm>[2979148.517218] Modules linked in: w83793 hwmon_vid autofs4 iTCO_wdt e1000e i2c_i801 i2c_core ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
<Apr/09 11:04 pm>[2979148.553301]
<Apr/09 11:04 pm>[2979148.555221] Pid: 8574, comm: nfsd Tainted: G D 2.6.34.4 #3 S7010/empty
<Apr/09 11:04 pm>[2979148.563221] RIP: 0010:[<ffffffff812c08af>] [<ffffffff812c08af>] delay_tsc+0x25/0x59
<Apr/09 11:04 pm>[2979148.571833] RSP: 0018:ffff880655223dc0 EFLAGS: 00000046
<Apr/09 11:04 pm>[2979148.577716] RAX: 00000000084f9dbc RBX: 0000000000000001 RCX: 00000000084f9d44
<Apr/09 11:04 pm>[2979148.585644] RDX: 00000000001c39a2 RSI: 00000000084f9d90 RDI: 000000000028b0fc
<Apr/09 11:04 pm>[2979148.593549] RBP: ffff880655223dc8 R08: 0000000000000007 R09: 0000000000000007
<Apr/09 11:04 pm>[2979148.601479] R10: ffff8800000bd000 R11: 0000000000000086 R12: ffff880655233c00
<Apr/09 11:04 pm>[2979148.609373] R13: 0000000000000000 R14: ffff8806227d3090 R15: ffff8806227d3518
<Apr/09 11:04 pm>[2979148.617296] FS: 0000000000000000(0000) GS:ffff880655220000(0000) knlGS:0000000000000000
<Apr/09 11:04 pm>[2979148.626245] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
<Apr/09 11:04 pm>[2979148.632580] CR2: 00007f7053196000 CR3: 0000000001a8a000 CR4: 00000000000006e0
<Apr/09 11:04 pm>[2979148.640497] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<Apr/09 11:04 pm>[2979148.648436] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<Apr/09 11:04 pm>[2979148.656357] Process nfsd (pid: 8574, threadinfo ffff880623fec000, task ffff8806227d3090)
<Apr/09 11:04 pm>[2979148.665279] Stack:
<Apr/09 11:04 pm>[2979148.667745] ffff8806227d3518 ffff880655223dd8 ffffffff812c093d ffff880655223df8
<Apr/09 11:04 pm>[2979148.675761] <0> ffffffff81019da9 000000009ef30658 000000009ef30658 ffff880655223e38
<Apr/09 11:04 pm>[2979148.684516] <0> ffffffff812c4ea4 000000009ef30658 ffff880655233c00 ffff880655233c00
<Apr/09 11:04 pm>[2979148.693662] Call Trace:
<Apr/09 11:04 pm>[2979148.696582] <IRQ>
<Apr/09 11:04 pm>[2979148.699192] [<ffffffff812c093d>] __const_udelay+0x3d/0x3f
<Apr/09 11:04 pm>[2979148.705261] [<ffffffff81019da9>] arch_trigger_all_cpu_backtrace+0x51/0x5d
<Apr/09 11:04 pm>[2979148.712763] [<ffffffff812c4ea4>] do_raw_spin_lock+0x11d/0x147
<Apr/09 11:04 pm>[2979148.719200] [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
<Apr/09 11:04 pm>[2979148.725239] [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
<Apr/09 11:04 pm>[2979148.731593] [<ffffffff81038588>] scheduler_tick+0x3d/0x236
<Apr/09 11:04 pm>[2979148.737755] [<ffffffff810483d1>] update_process_times+0x49/0x59
<Apr/09 11:04 pm>[2979148.744359] [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
<Apr/09 11:04 pm>[2979148.750617] [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
<Apr/09 11:04 pm>[2979148.756960] [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
<Apr/09 11:04 pm>[2979148.763027] [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:04 pm>[2979148.769468] [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
<Apr/09 11:04 pm>[2979148.776437] [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
<Apr/09 11:04 pm>[2979148.783005] <EOI>
<Apr/09 11:04 pm>[2979148.785627] [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
<Apr/09 11:04 pm>[2979148.791880] [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
<Apr/09 11:04 pm>[2979148.798119] [<ffffffff810d48fd>] ? get_swap_page+0x13c/0x5bb
<Apr/09 11:04 pm>[2979148.804438] [<ffffffff810d17e9>] add_to_swap+0xf/0x48
<Apr/09 11:04 pm>[2979148.810138] [<ffffffff810b7464>] shrink_page_list+0x214/0x68f
<Apr/09 11:04 pm>[2979148.816520] [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Apr/09 11:04 pm>[2979148.822951] [<ffffffff810e5fb2>] ? mem_cgroup_del_lru_list+0x7b/0x8e
<Apr/09 11:04 pm>[2979148.830017] [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
<Apr/09 11:04 pm>[2979148.836003] [<ffffffff8144b300>] ? release_sock+0xdb/0xe4
<Apr/09 11:04 pm>[2979148.842061] [<ffffffff8144b254>] ? release_sock+0x2f/0xe4
<Apr/09 11:04 pm>[2979148.848119] [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
<Apr/09 11:04 pm>[2979148.854882] [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
<Apr/09 11:04 pm>[2979148.861191] [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
<Apr/09 11:04 pm>[2979148.867987] [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
<Apr/09 11:04 pm>[2979148.874930] [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
<Apr/09 11:04 pm>[2979148.881444] [<ffffffff814fc30d>] svc_recv+0xe5/0x7f5
<Apr/09 11:04 pm>[2979148.887059] [<ffffffff814fc1d9>] ? svc_send+0x85/0xa1
<Apr/09 11:04 pm>[2979148.892736] [<ffffffff811f9bb1>] ? exp_readunlock+0x10/0x12
<Apr/09 11:04 pm>[2979148.898978] [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
<Apr/09 11:04 pm>[2979148.905654] [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
<Apr/09 11:04 pm>[2979148.910961] [<ffffffff811f0d78>] nfsd+0xa9/0x139
<Apr/09 11:04 pm>[2979148.916214] [<ffffffff810546e2>] kthread+0x7d/0x85
<Apr/09 11:04 pm>[2979148.921648] [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
<Apr/09 11:04 pm>[2979148.928169] [<ffffffff81054665>] ? kthread+0x0/0x85
<Apr/09 11:04 pm>[2979148.933686] [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
<Apr/09 11:04 pm>[2979148.940337] Code: fb 48 ff c8 c9 c3 55 48 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae e8 0f 31 89 c1 41 89 f1 66 66 90 0f ae e8 0f 31 <89> c6 48 89 f0 48 29 c8 48 39 f8 73 25 f3 90 65 44 8b 04 25 f0
<Apr/09 11:04 pm>[2979148.963508] Call Trace:
<Apr/09 11:04 pm>[2979148.966436] <IRQ> [<ffffffff812c093d>] __const_udelay+0x3d/0x3f
<Apr/09 11:04 pm>[2979148.973108] [<ffffffff81019da9>] arch_trigger_all_cpu_backtrace+0x51/0x5d
<Apr/09 11:04 pm>[2979148.980615] [<ffffffff812c4ea4>] do_raw_spin_lock+0x11d/0x147
<Apr/09 11:04 pm>[2979148.987017] [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
<Apr/09 11:04 pm>[2979148.993100] [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
<Apr/09 11:04 pm>[2979148.999411] [<ffffffff81038588>] scheduler_tick+0x3d/0x236
<Apr/09 11:04 pm>[2979149.005512] [<ffffffff810483d1>] update_process_times+0x49/0x59
<Apr/09 11:04 pm>[2979149.012108] [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
<Apr/09 11:04 pm>[2979149.018347] [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
<Apr/09 11:04 pm>[2979149.024674] [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
<Apr/09 11:04 pm>[2979149.030741] [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:04 pm>[2979149.037157] [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
<Apr/09 11:04 pm>[2979149.044126] [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
<Apr/09 11:04 pm>[2979149.050719] <EOI> [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
<Apr/09 11:04 pm>[2979149.057613] [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
<Apr/09 11:04 pm>[2979149.063859] [<ffffffff810d48fd>] ? get_swap_page+0x13c/0x5bb
<Apr/09 11:04 pm>[2979149.070203] [<ffffffff810d17e9>] add_to_swap+0xf/0x48
<Apr/09 11:04 pm>[2979149.075938] [<ffffffff810b7464>] shrink_page_list+0x214/0x68f
<Apr/09 11:04 pm>[2979149.082328] [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Apr/09 11:04 pm>[2979149.088733] [<ffffffff810e5fb2>] ? mem_cgroup_del_lru_list+0x7b/0x8e
<Apr/09 11:04 pm>[2979149.095788] [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
<Apr/09 11:04 pm>[2979149.101776] [<ffffffff8144b300>] ? release_sock+0xdb/0xe4
<Apr/09 11:04 pm>[2979149.107851] [<ffffffff8144b254>] ? release_sock+0x2f/0xe4
<Apr/09 11:04 pm>[2979149.113939] [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
<Apr/09 11:04 pm>[2979149.125252] [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
<Apr/09 11:04 pm>[2979149.131556] [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
<Apr/09 11:04 pm>[2979149.138352] [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
<Apr/09 11:04 pm>[2979149.145314] [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
<Apr/09 11:04 pm>[2979149.151819] [<ffffffff814fc30d>] svc_recv+0xe5/0x7f5
<Apr/09 11:04 pm>[2979149.157451] [<ffffffff814fc1d9>] ? svc_send+0x85/0xa1
<Apr/09 11:04 pm>[2979149.163144] [<ffffffff811f9bb1>] ? exp_readunlock+0x10/0x12
<Apr/09 11:04 pm>[2979149.169395] [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
<Apr/09 11:04 pm>[2979149.176097] [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
<Apr/09 11:04 pm>[2979149.181438] [<ffffffff811f0d78>] nfsd+0xa9/0x139
<Apr/09 11:04 pm>[2979149.186691] [<ffffffff810546e2>] kthread+0x7d/0x85
<Apr/09 11:04 pm>[2979149.192127] [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
<Apr/09 11:04 pm>[2979149.198622] [<ffffffff81054665>] ? kthread+0x0/0x85
<Apr/09 11:04 pm>[2979149.204139] [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
<Apr/09 11:04 pm>[2979149.210826] Pid: 8574, comm: nfsd Tainted: G D 2.6.34.4 #3
<Apr/09 11:04 pm>[2979149.217611] Call Trace:
<Apr/09 11:04 pm>[2979149.220525] <NMI> [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
<Apr/09 11:04 pm>[2979149.226989] [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
<Apr/09 11:04 pm>[2979149.233394] [<ffffffff81520360>] do_nmi+0xb2/0x259
<Apr/09 11:04 pm>[2979149.238823] [<ffffffff8151fdfa>] nmi+0x1a/0x20
<Apr/09 11:04 pm>[2979149.243893] [<ffffffff812c08af>] ? delay_tsc+0x25/0x59
<Apr/09 11:04 pm>[2979149.249666] <<EOE>> <IRQ> [<ffffffff812c093d>] __const_udelay+0x3d/0x3f
<Apr/09 11:04 pm>[2979149.257247] [<ffffffff81019da9>] arch_trigger_all_cpu_backtrace+0x51/0x5d
<Apr/09 11:04 pm>[2979149.264728] [<ffffffff812c4ea4>] do_raw_spin_lock+0x11d/0x147
<Apr/09 11:04 pm>[2979149.271138] [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
<Apr/09 11:04 pm>[2979149.277168] [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
<Apr/09 11:04 pm>[2979149.283505] [<ffffffff81038588>] scheduler_tick+0x3d/0x236
<Apr/09 11:04 pm>[2979149.289675] [<ffffffff810483d1>] update_process_times+0x49/0x59
<Apr/09 11:04 pm>[2979149.296288] [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
<Apr/09 11:04 pm>[2979149.302528] [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
<Apr/09 11:04 pm>[2979149.308856] [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
<Apr/09 11:04 pm>[2979149.314923] [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:04 pm>[2979149.321346] [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
<Apr/09 11:04 pm>[2979149.328315] [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
<Apr/09 11:04 pm>[2979149.334893] <EOI> [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
<Apr/09 11:04 pm>[2979149.341803] [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
<Apr/09 11:04 pm>[2979149.348013] [<ffffffff810d48fd>] ? get_swap_page+0x13c/0x5bb
<Apr/09 11:04 pm>[2979149.354331] [<ffffffff810d17e9>] add_to_swap+0xf/0x48
<Apr/09 11:04 pm>[2979149.360040] [<ffffffff810b7464>] shrink_page_list+0x214/0x68f
<Apr/09 11:04 pm>[2979149.366482] [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Apr/09 11:04 pm>[2979149.372906] [<ffffffff810e5fb2>] ? mem_cgroup_del_lru_list+0x7b/0x8e
<Apr/09 11:04 pm>[2979149.379947] [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
<Apr/09 11:04 pm>[2979149.385905] [<ffffffff8144b300>] ? release_sock+0xdb/0xe4
<Apr/09 11:04 pm>[2979149.391962] [<ffffffff8144b254>] ? release_sock+0x2f/0xe4
<Apr/09 11:04 pm>[2979149.398030] [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
<Apr/09 11:04 pm>[2979149.404819] [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
<Apr/09 11:04 pm>[2979149.411171] [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
<Apr/09 11:04 pm>[2979149.417975] [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
<Apr/09 11:04 pm>[2979149.424953] [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
<Apr/09 11:04 pm>[2979149.431460] [<ffffffff814fc30d>] svc_recv+0xe5/0x7f5
<Apr/09 11:04 pm>[2979149.437047] [<ffffffff814fc1d9>] ? svc_send+0x85/0xa1
<Apr/09 11:04 pm>[2979149.442733] [<ffffffff811f9bb1>] ? exp_readunlock+0x10/0x12
<Apr/09 11:04 pm>[2979149.448967] [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
<Apr/09 11:04 pm>[2979149.455649] [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
<Apr/09 11:04 pm>[2979149.460958] [<ffffffff811f0d78>] nfsd+0xa9/0x139
<Apr/09 11:04 pm>[2979149.466201] [<ffffffff810546e2>] kthread+0x7d/0x85
<Apr/09 11:04 pm>[2979149.471636] [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
<Apr/09 11:04 pm>[2979149.478123] [<ffffffff81054665>] ? kthread+0x0/0x85
<Apr/09 11:04 pm>[2979149.483657] [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
<Apr/09 11:04 pm>[2979149.490283] NMI backtrace for cpu 6
<Apr/09 11:04 pm>[2979149.494243] CPU 6
<Apr/09 11:04 pm>[2979149.496362] Modules linked in: w83793 hwmon_vid autofs4 iTCO_wdt e1000e i2c_i801 i2c_core ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
<Apr/09 11:04 pm>[2979149.532444]
<Apr/09 11:04 pm>[2979149.534381] Pid: 8590, comm: nfsd Tainted: G D 2.6.34.4 #3 S7010/empty
<Apr/09 11:04 pm>[2979149.542400] RIP: 0010:[<ffffffff812c08af>] [<ffffffff812c08af>] delay_tsc+0x25/0x59
<Apr/09 11:04 pm>[2979149.551003] RSP: 0018:ffff880655203de0 EFLAGS: 00000082
<Apr/09 11:04 pm>[2979149.556859] RAX: 00000000084b8b04 RBX: 0000000008a39bf5 RCX: 00000000084b8ae8
<Apr/09 11:04 pm>[2979149.564795] RDX: 00000000001c39a2 RSI: 0000000000000006 RDI: 0000000000000001
<Apr/09 11:04 pm>[2979149.572752] RBP: ffff880655203de8 R08: 0000000000000006 R09: 0000000000000006
<Apr/09 11:04 pm>[2979149.580707] R10: 000000000000000b R11: ffff880655203c18 R12: ffff880655213c00
<Apr/09 11:04 pm>[2979149.588637] R13: 0000000000000000 R14: ffff8806227ed0f0 R15: ffff8806227ed578
<Apr/09 11:04 pm>[2979149.596585] FS: 0000000000000000(0000) GS:ffff880655200000(0000) knlGS:0000000000000000
<Apr/09 11:04 pm>[2979149.605481] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
<Apr/09 11:04 pm>[2979149.611792] CR2: 0000000389ca1c18 CR3: 0000000001a8a000 CR4: 00000000000006e0
<Apr/09 11:04 pm>[2979149.619718] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<Apr/09 11:04 pm>[2979149.627674] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<Apr/09 11:04 pm>[2979149.635588] Process nfsd (pid: 8590, threadinfo ffff880622754000, task ffff8806227ed0f0)
<Apr/09 11:04 pm>[2979149.644526] Stack:
<Apr/09 11:04 pm>[2979149.646993] ffffffff81019da9 ffff880655203df8 ffffffff812c08fe ffff880655203e38
<Apr/09 11:04 pm>[2979149.654991] <0> ffffffff812c4e65 000000009ef30658 ffff880655213c00 ffff880655213c00
<Apr/09 11:04 pm>[2979149.663735] <0> 0000000000000006 ffff88065520eae0 ffff88065520ea70 ffff880655203e58
<Apr/09 11:04 pm>[2979149.672874] Call Trace:
<Apr/09 11:04 pm>[2979149.675795] <IRQ>
<Apr/09 11:04 pm>[2979149.678395] [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
<Apr/09 11:04 pm>[2979149.686254] [<ffffffff812c08fe>] __delay+0xa/0xc
<Apr/09 11:04 pm>[2979149.691489] [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
<Apr/09 11:04 pm>[2979149.697804] [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
<Apr/09 11:04 pm>[2979149.703878] [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
<Apr/09 11:04 pm>[2979149.710188] [<ffffffff81038588>] scheduler_tick+0x3d/0x236
<Apr/09 11:04 pm>[2979149.716325] [<ffffffff810483d1>] update_process_times+0x49/0x59
<Apr/09 11:04 pm>[2979149.722938] [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
<Apr/09 11:04 pm>[2979149.729187] [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
<Apr/09 11:04 pm>[2979149.735514] [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
<Apr/09 11:04 pm>[2979149.741580] [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:04 pm>[2979149.748012] [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
<Apr/09 11:04 pm>[2979149.754973] [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
<Apr/09 11:04 pm>[2979149.761558] <EOI>
<Apr/09 11:04 pm>[2979149.763773] Code: fb 48 ff c8 c9 c3 55 48 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae e8 0f 31 89 c1 41 89 f1 66 66 90 0f ae e8 0f 31 <89> c6 48 89 f0 48 29 c8 48 39 f8 73 25 f3 90 65 44 8b 04 25 f0
<Apr/09 11:04 pm>[2979149.787287] Call Trace:
<Apr/09 11:04 pm>[2979149.790216] <IRQ> [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
<Apr/09 11:04 pm>[2979149.798755] [<ffffffff812c08fe>] __delay+0xa/0xc
<Apr/09 11:04 pm>[2979149.803987] [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
<Apr/09 11:04 pm>[2979149.810309] [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
<Apr/09 11:04 pm>[2979149.816367] [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
<Apr/09 11:04 pm>[2979149.822705] [<ffffffff81038588>] scheduler_tick+0x3d/0x236
<Apr/09 11:04 pm>[2979149.828848] [<ffffffff810483d1>] update_process_times+0x49/0x59
<Apr/09 11:04 pm>[2979149.835436] [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
<Apr/09 11:04 pm>[2979149.841665] [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
<Apr/09 11:04 pm>[2979149.847959] [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
<Apr/09 11:04 pm>[2979149.853990] [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:04 pm>[2979149.860431] [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
<Apr/09 11:04 pm>[2979149.867392] [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
<Apr/09 11:04 pm>[2979149.873986] <EOI> Pid: 8590, comm: nfsd Tainted: G D 2.6.34.4 #3
<Apr/09 11:04 pm>[2979149.881469] Call Trace:
<Apr/09 11:04 pm>[2979149.884400] <NMI> [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
<Apr/09 11:04 pm>[2979149.890881] [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
<Apr/09 11:04 pm>[2979149.897310] [<ffffffff81520360>] do_nmi+0xb2/0x259
<Apr/09 11:04 pm>[2979149.902740] [<ffffffff8151fdfa>] nmi+0x1a/0x20
<Apr/09 11:04 pm>[2979149.907809] [<ffffffff812c08af>] ? delay_tsc+0x25/0x59
<Apr/09 11:04 pm>[2979149.913610] <<EOE>> <IRQ> [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
<Apr/09 11:04 pm>[2979149.922945] [<ffffffff812c08fe>] __delay+0xa/0xc
<Apr/09 11:04 pm>[2979149.928194] [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
<Apr/09 11:04 pm>[2979149.934534] [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
<Apr/09 11:04 pm>[2979149.940556] [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
<Apr/09 11:04 pm>[2979149.946884] [<ffffffff81038588>] scheduler_tick+0x3d/0x236
<Apr/09 11:04 pm>[2979149.953028] [<ffffffff810483d1>] update_process_times+0x49/0x59
<Apr/09 11:04 pm>[2979149.959618] [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
<Apr/09 11:04 pm>[2979149.965830] [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
<Apr/09 11:04 pm>[2979149.972157] [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
<Apr/09 11:04 pm>[2979149.978214] [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
<Apr/09 11:04 pm>[2979149.984638] [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
<Apr/09 11:04 pm>[2979149.991582] [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
Last modified
14 years ago
Last modified on Apr 9, 2012, 11:22:33 PM
Note:
See TracWiki
for help on using the wiki.
