IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Initial Version and Version 1 of ipp056-2012-04-09-crash


Ignore:
Timestamp:
Apr 9, 2012, 11:22:33 PM (14 years ago)
Author:
Mark Huber
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • ipp056-2012-04-09-crash

    v1 v1  
     1On console
     2{{{
     3n+0x0/0xf
     4<Apr/09 11:03 pm>[2979142.271348]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
     5<Apr/09 11:03 pm>[2979142.271350]  [<ffffffff811f0d78>] nfsd+0xa9/0x139
     6<Apr/09 11:03 pm>[2979142.271352]  [<ffffffff810546e2>] kthread+0x7d/0x85
     7<Apr/09 11:03 pm>[2979142.271354]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     8<Apr/09 11:03 pm>[2979142.271357]  [<ffffffff81054665>] ? kthread+0x0/0x85
     9<Apr/09 11:03 pm>[2979142.271358]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     10<Apr/09 11:03 pm>[2979142.271360] Pid: 8541, comm: nfsd Tainted: G      D    2.6.34.4 #3
     11<Apr/09 11:03 pm>[2979142.271361] Call Trace:
     12<Apr/09 11:03 pm>[2979142.271362]  <NMI>  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
     13<Apr/09 11:03 pm>[2979142.271367]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
     14<Apr/09 11:03 pm>[2979142.271369]  [<ffffffff81520360>] do_nmi+0xb2/0x259
     15<Apr/09 11:03 pm>[2979142.271371]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
     16<Apr/09 11:03 pm>[2979142.271373]  [<ffffffff812c08a0>] ? delay_tsc+0x16/0x59
     17<Apr/09 11:03 pm>[2979142.271374]  <<EOE>>  [<ffffffff812c093d>] __const_udelay+0x3d/0x3f
     18<Apr/09 11:03 pm>[2979142.271378]  [<ffffffff810197e9>] native_safe_apic_wait_icr_idle+0x31/0x48
     19<Apr/09 11:03 pm>[2979142.271380]  [<ffffffff81019c76>] default_send_IPI_mask_sequence_phys+0x49/0xab
     20<Apr/09 11:03 pm>[2979142.271383]  [<ffffffff8101c5d8>] physflat_send_IPI_all+0x14/0x16
     21<Apr/09 11:03 pm>[2979142.271385]  [<ffffffff81019d94>] arch_trigger_all_cpu_backtrace+0x3c/0x5d
     22<Apr/09 11:03 pm>[2979142.271387]  [<ffffffff812c4ea4>] do_raw_spin_lock+0x11d/0x147
     23<Apr/09 11:03 pm>[2979142.271390]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     24
     25<Apr/09 11:03 pm>[2979142.271392]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
     26<Apr/09 11:03 pm>[2979142.271394]  [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
     27<Apr/09 11:03 pm>[2979142.271396]  [<ffffffff81036578>] try_to_wake_up+0x2e/0x342
     28<Apr/09 11:03 pm>[2979142.271398]  [<ffffffff81036899>] default_wake_function+0xd/0xf
     29<Apr/09 11:03 pm>[2979142.271400]  [<ffffffff8102c08d>] __wake_up_common+0x46/0x76
     30<Apr/09 11:03 pm>[2979142.271402]  [<ffffffff8102faa6>] __wake_up+0x38/0x4f
     31<Apr/09 11:03 pm>[2979142.271404]  [<ffffffff814fbb75>] svc_xprt_enqueue+0x1e2/0x23d
     32<Apr/09 11:03 pm>[2979142.271406]  [<ffffffff814fbc26>] svc_xprt_received+0x24/0x26
     33<Apr/09 11:03 pm>[2979142.271409]  [<ffffffff814f1a12>] svc_udp_recvfrom+0x15f/0x2f9
     34<Apr/09 11:03 pm>[2979142.271411]  [<ffffffff814fc980>] svc_recv+0x758/0x7f5
     35<Apr/09 11:03 pm>[2979142.271413]  [<ffffffff814fc1d9>] ? svc_send+0x85/0xa1
     36<Apr/09 11:03 pm>[2979142.271415]  [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
     37<Apr/09 11:03 pm>[2979142.271417]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
     38<Apr/09 11:03 pm>[2979142.271419]  [<ffffffff811f0d78>] nfsd+0xa9/0x139
     39<Apr/09 11:03 pm>[2979142.271421]  [<ffffffff810546e2>] kthread+0x7d/0x85
     40<Apr/09 11:03 pm>[2979142.271423]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     41<Apr/09 11:03 pm>[2979142.271425]  [<ffffffff81054665>] ? kthread+0x0/0x85
     42<Apr/09 11:03 pm>[2979142.271427]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     43<Apr/09 11:03 pm>[2979142.271530] NMI backtrace for cpu 6
     44<Apr/09 11:03 pm>[2979142.271531] CPU 6
     45<Apr/09 11:03 pm>[2979142.271532] Modules linked in: w83793 hwmon_vid autofs4 iTCO_wdt e1000e i2c_i801 i2c_core ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
     46<Apr/09 11:03 pm>[2979142.271546]
     47
     48<Apr/09 11:03 pm>[2979142.271547] Pid: 8590, comm: nfsd Tainted: G      D    2.6.34.4 #3 S7010/empty
     49<Apr/09 11:03 pm>[2979142.271549] RIP: 0010:[<ffffffff812c08a2>]  [<ffffffff812c08a2>] delay_tsc+0x18/0x59
     50<Apr/09 11:03 pm>[2979142.271553] RSP: 0018:ffff880655203de0  EFLAGS: 00000082
     51<Apr/09 11:03 pm>[2979142.271555] RAX: 000000009106eca8 RBX: 000000009aeab2fb RCX: 000000009106ec64
     52<Apr/09 11:03 pm>[2979142.271556] RDX: 00000000001c399e RSI: 0000000000000006 RDI: 0000000000000001
     53<Apr/09 11:03 pm>[2979142.271558] RBP: ffff880655203de8 R08: 0000000000000001 R09: 0000000000000006
     54<Apr/09 11:03 pm>[2979142.271559] R10: 0000000100000000 R11: 0000000000000000 R12: ffff880655213c00
     55<Apr/09 11:03 pm>[2979142.271561] R13: 0000000000000001 R14: ffff8806227ed0f0 R15: ffff8806227ed578
     56<Apr/09 11:03 pm>[2979142.271563] FS:  0000000000000000(0000) GS:ffff880655200000(0000) knlGS:0000000000000000
     57<Apr/09 11:03 pm>[2979142.271565] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
     58<Apr/09 11:03 pm>[2979142.271566] CR2: 0000000389ca1c18 CR3: 0000000001a8a000 CR4: 00000000000006e0
     59<Apr/09 11:03 pm>[2979142.271568] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     60<Apr/09 11:03 pm>[2979142.271569] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     61<Apr/09 11:03 pm>[2979142.271571] Process nfsd (pid: 8590, threadinfo ffff880622754000, task ffff8806227ed0f0)
     62<Apr/09 11:03 pm>[2979142.271572] Stack:
     63<Apr/09 11:03 pm>[2979142.271573]  ffff880655213c00 ffff880655203df8 ffffffff812c08fe ffff880655203e38
     64<Apr/09 11:03 pm>[2979142.271575] <0> ffffffff812c4e65 000000009ef30658 ffff880655213c00 ffff880655213c00
     65<Apr/09 11:03 pm>[2979142.271577] <0> 0000000000000006 ffff88065520eae0 ffff88065520ea70 ffff880655203e58
     66<Apr/09 11:03 pm>[2979142.271580] Call Trace:
     67<Apr/09 11:03 pm>[2979142.271581]  <IRQ>
     68<Apr/09 11:03 pm>[2979142.271583]  [<ffffffff812c08fe>] __delay+0xa/0xc
     69<Apr/09 11:03 pm>[2979142.271585]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     70
     71<Apr/09 11:03 pm>[2979142.271587]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     72<Apr/09 11:03 pm>[2979142.271590]  [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
     73<Apr/09 11:03 pm>[2979142.271592]  [<ffffffff81038588>] scheduler_tick+0x3d/0x236
     74<Apr/09 11:03 pm>[2979142.271594]  [<ffffffff810483d1>] update_process_times+0x49/0x59
     75<Apr/09 11:03 pm>[2979142.271597]  [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
     76<Apr/09 11:03 pm>[2979142.271599]  [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
     77<Apr/09 11:03 pm>[2979142.271601]  [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
     78<Apr/09 11:03 pm>[2979142.271603]  [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
     79<Apr/09 11:03 pm>[2979142.271606]  [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
     80<Apr/09 11:03 pm>[2979142.271611]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
     81<Apr/09 11:03 pm>[2979142.271612]  <EOI>
     82<Apr/09 11:03 pm>[2979142.271613] Code: 2e 0f 1f 84 00 00 00 00 00 48 ff c8 75 fb 48 ff c8 c9 c3 55 48 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae e8 0f 31 <89> c1 41 89 f1 66 66 90 0f ae e8 0f 31 89 c6 48 89 f0 48 29 c8
     83<Apr/09 11:03 pm>[2979142.271629] Call Trace:
     84<Apr/09 11:03 pm>[2979142.271629]  <IRQ>  [<ffffffff812c08fe>] __delay+0xa/0xc
     85<Apr/09 11:03 pm>[2979142.271633]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     86<Apr/09 11:03 pm>[2979142.271635]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     87<Apr/09 11:03 pm>[2979142.271638]  [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
     88<Apr/09 11:03 pm>[2979142.271640]  [<ffffffff81038588>] scheduler_tick+0x3d/0x236
     89<Apr/09 11:03 pm>[2979142.271642]  [<ffffffff810483d1>] update_process_times+0x49/0x59
     90<Apr/09 11:03 pm>[2979142.271644]  [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
     91                 <Apr/09 11:03 pm>[2979142.271646]  [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
     92<Apr/09 11:03 pm>[2979142.271648]  [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
     93
     94<Apr/09 11:03 pm>[2979142.271650]  [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
     95<Apr/09 11:03 pm>[2979142.271652]  [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
     96<Apr/09 11:03 pm>[2979142.271655]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
     97<Apr/09 11:03 pm>[2979142.271656]  <EOI> Pid: 8590, comm: nfsd Tainted: G      D    2.6.34.4 #3
     98<Apr/09 11:03 pm>[2979142.271658] Call Trace:
     99<Apr/09 11:03 pm>[2979142.271659]  <NMI>  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
     100<Apr/09 11:03 pm>[2979142.271663]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
     101<Apr/09 11:03 pm>[2979142.271665]  [<ffffffff81520360>] do_nmi+0xb2/0x259
     102<Apr/09 11:03 pm>[2979142.271667]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
     103<Apr/09 11:03 pm>[2979142.271669]  [<ffffffff812c08a2>] ? delay_tsc+0x18/0x59
     104<Apr/09 11:03 pm>[2979142.271670]  <<EOE>>  <IRQ>  [<ffffffff812c08fe>] __delay+0xa/0xc
     105<Apr/09 11:03 pm>[2979142.271674]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     106<Apr/09 11:03 pm>[2979142.271676]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     107<Apr/09 11:03 pm>[2979142.271678]  [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
     108<Apr/09 11:03 pm>[2979142.271680]  [<ffffffff81038588>] scheduler_tick+0x3d/0x236
     109<Apr/09 11:03 pm>[2979142.271683]  [<ffffffff810483d1>] update_process_times+0x49/0x59
     110<Apr/09 11:03 pm>[2979142.271685]  [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
     111<Apr/09 11:03 pm>[2979142.271687]  [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
     112<Apr/09 11:03 pm>[2979142.271689]  [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
     113<Apr/09 11:03 pm>[2979142.271691]  [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
     114<Apr/09 11:03 pm>[2979142.271693]  [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
     115<Apr/09 11:03 pm>[2979142.271695]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
     116
     117<Apr/09 11:03 pm>[2979142.271696]  <EOI>
     118<Apr/09 11:03 pm>[2979142.271697] NMI backtrace for cpu 7
     119<Apr/09 11:03 pm>[2979142.271698] CPU 7
     120<Apr/09 11:03 pm>[2979142.271699] Modules linked in: w83793 hwmon_vid autofs4 iTCO_wdt e1000e i2c_i801 i2c_core ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
     121<Apr/09 11:03 pm>[2979142.271713]
     122<Apr/09 11:03 pm>[2979142.271715] Pid: 8574, comm: nfsd Tainted: G      D    2.6.34.4 #3 S7010/empty
     123<Apr/09 11:03 pm>[2979142.271717] RIP: 0010:[<ffffffff8133423b>]  [<ffffffff8133423b>] io_serial_in+0x10/0x15
     124<Apr/09 11:03 pm>[2979142.271722] RSP: 0018:ffff880655223938  EFLAGS: 00000002
     125<Apr/09 11:03 pm>[2979142.271723] RAX: 0000000000000a00 RBX: 0000000000000000 RCX: 0000000000000000
     126<Apr/09 11:03 pm>[2979142.271725] RDX: 00000000000003fd RSI: 0000000000000005 RDI: ffffffff82400ac0
     127<Apr/09 11:03 pm>[2979142.271727] RBP: ffff880655223938 R08: 0000000000000007 R09: 0000000000000007
     128<Apr/09 11:03 pm>[2979142.271728] R10: 0000000000000000 R11: ffffffff81334e4e R12: ffffffff82400ac0
     129<Apr/09 11:03 pm>[2979142.271730] R13: 000000000000270f R14: 0000000000000020 R15: 0000000000000044
     130<Apr/09 11:03 pm>[2979142.271732] FS:  0000000000000000(0000) GS:ffff880655220000(0000) knlGS:0000000000000000
     131<Apr/09 11:03 pm>[2979142.271733] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
     132<Apr/09 11:03 pm>[2979142.271735] CR2: 00007f7053196000 CR3: 0000000001a8a000 CR4: 00000000000006e0
     133<Apr/09 11:03 pm>[2979142.271736] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     134<Apr/09 11:03 pm>[2979142.271738] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     135<Apr/09 11:03 pm>[2979142.271740] Process nfsd (pid: 8574, threadinfo ffff880623fec000, task ffff8806227d3090)
     136<Apr/09 11:03 pm>[2979142.271741] Stack:
     137<Apr/09 11:03 pm>[2979142.271742]  ffff880655223968 ffffffff81334767 ffffffff82400ac0 0000000000000062
     138<Apr/09 11:03 pm>[2979142.271744] <0> ffffffff813347d5 ffffffff82400ac0 ffff880655223988 ffffffff813347ec
     139
     140<Apr/09 11:03 pm>[2979142.271746] <0> ffffffff81c39048 0000000000000022 ffff8806552239c8 ffffffff813311e8
     141<Apr/09 11:03 pm>[2979142.271749] Call Trace:
     142<Apr/09 11:03 pm>[2979142.271750]  <IRQ>
     143<Apr/09 11:03 pm>[2979142.271752]  [<ffffffff81334767>] wait_for_xmitr+0x24/0x92
     144<Apr/09 11:03 pm>[2979142.271755]  [<ffffffff813347d5>] ? serial8250_console_putchar+0x0/0x27
     145<Apr/09 11:03 pm>[2979142.271757]  [<ffffffff813347ec>] serial8250_console_putchar+0x17/0x27
     146<Apr/09 11:03 pm>[2979142.271760]  [<ffffffff813311e8>] uart_console_write+0x41/0x52
     147<Apr/09 11:03 pm>[2979142.271762]  [<ffffffff81334eb0>] serial8250_console_write+0xb6/0x108
     148<Apr/09 11:03 pm>[2979142.271765]  [<ffffffff8103b339>] __call_console_drivers+0x67/0x79
     149<Apr/09 11:03 pm>[2979142.271768]  [<ffffffff8103b3a6>] _call_console_drivers+0x5b/0x5f
     150<Apr/09 11:03 pm>[2979142.271770]  [<ffffffff8103b87b>] release_console_sem+0x121/0x1c6
     151<Apr/09 11:03 pm>[2979142.271773]  [<ffffffff8103beb2>] vprintk+0x36b/0x3a0
     152<Apr/09 11:03 pm>[2979142.271775]  [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
     153<Apr/09 11:03 pm>[2979142.271777]  [<ffffffff8151c589>] printk+0x67/0x69
     154<Apr/09 11:03 pm>[2979142.271779]  [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
     155<Apr/09 11:03 pm>[2979142.271781]  [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
     156<Apr/09 11:03 pm>[2979142.271784]  [<ffffffff8151c589>] ? printk+0x67/0x69
     157<Apr/09 11:03 pm>[2979142.271786]  [<ffffffff81019eb4>] ? touch_nmi_watchdog+0x5b/0x60
     158<Apr/09 11:03 pm>[2979142.271789]  [<ffffffff810060bb>] print_trace_address+0x4e/0x57
     159<Apr/09 11:03 pm>[2979142.271791]  [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
     160<Apr/09 11:03 pm>[2979142.271793]  [<ffffffff810061e9>] print_context_stack+0x62/0xbd
     161<Apr/09 11:03 pm>[2979142.271795]  [<ffffffff810054e2>] dump_trace+0x23d/0x327
     162
     163<Apr/09 11:03 pm>[2979142.271798]  [<ffffffff8100604f>] show_trace_log_lvl+0x4c/0x58
     164<Apr/09 11:03 pm>[2979142.271800]  [<ffffffff8100606b>] show_trace+0x10/0x12
     165<Apr/09 11:03 pm>[2979142.271802]  [<ffffffff8151c401>] dump_stack+0x72/0x7b
     166<Apr/09 11:03 pm>[2979142.271804]  [<ffffffff812c4e9f>] do_raw_spin_lock+0x118/0x147
     167<Apr/09 11:03 pm>[2979142.271806]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     168<Apr/09 11:03 pm>[2979142.271808]  [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
     169<Apr/09 11:03 pm>[2979142.271811]  [<ffffffff81038588>] scheduler_tick+0x3d/0x236
     170<Apr/09 11:03 pm>[2979142.271813]  [<ffffffff810483d1>] update_process_times+0x49/0x59
     171<Apr/09 11:03 pm>[2979142.271815]  [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
     172<Apr/09 11:03 pm>[2979142.271817]  [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
     173<Apr/09 11:03 pm>[2979142.271819]  [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
     174<Apr/09 11:03 pm>[2979142.271821]  [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
     175<Apr/09 11:03 pm>[2979142.271823]  [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
     176<Apr/09 11:03 pm>[2979142.271826]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
     177<Apr/09 11:03 pm>[2979142.271827]  <EOI>
     178<Apr/09 11:03 pm>[2979142.271829]  [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
     179<Apr/09 11:03 pm>[2979142.271831]  [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
     180<Apr/09 11:03 pm>[2979142.271833]  [<ffffffff810d48fd>] ? get_swap_page+0x13c/0x5bb
     181<Apr/09 11:03 pm>[2979142.271836]  [<ffffffff810d17e9>] add_to_swap+0xf/0x48
     182<Apr/09 11:03 pm>[2979142.271838]  [<ffffffff810b7464>] shrink_page_list+0x214/0x68f
     183<Apr/09 11:03 pm>[2979142.271841]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     184<Apr/09 11:03 pm>[2979142.271844]  [<ffffffff810e5fb2>] ? mem_cgroup_del_lru_list+0x7b/0x8e
     185
     186<Apr/09 11:03 pm>[2979142.271846]  [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
     187<Apr/09 11:03 pm>[2979142.271851]  [<ffffffff8144b300>] ? release_sock+0xdb/0xe4
     188<Apr/09 11:03 pm>[2979142.271853]  [<ffffffff8144b254>] ? release_sock+0x2f/0xe4
     189<Apr/09 11:03 pm>[2979142.271855]  [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
     190<Apr/09 11:03 pm>[2979142.271857]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
     191<Apr/09 11:03 pm>[2979142.271860]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
     192<Apr/09 11:03 pm>[2979142.271862]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
     193<Apr/09 11:03 pm>[2979142.271865]  [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
     194<Apr/09 11:03 pm>[2979142.271869]  [<ffffffff814fc30d>] svc_recv+0xe5/0x7f5
     195<Apr/09 11:03 pm>[2979142.271871]  [<ffffffff814fc1d9>] ? svc_send+0x85/0xa1
     196<Apr/09 11:03 pm>[2979142.271873]  [<ffffffff811f9bb1>] ? exp_readunlock+0x10/0x12
     197<Apr/09 11:03 pm>[2979142.271876]  [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
     198<Apr/09 11:03 pm>[2979142.271878]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
     199<Apr/09 11:03 pm>[2979142.271879]  [<ffffffff811f0d78>] nfsd+0xa9/0x139
     200<Apr/09 11:03 pm>[2979142.271882]  [<ffffffff810546e2>] kthread+0x7d/0x85
     201<Apr/09 11:03 pm>[2979142.271884]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     202<Apr/09 11:03 pm>[2979142.271886]  [<ffffffff81054665>] ? kthread+0x0/0x85
     203<Apr/09 11:03 pm>[2979142.271888]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     204<Apr/09 11:03 pm>[2979142.271890] Code: 49 03 40 38 88 10 83 fe 01 77 0c be 01 00 00 00 4c 89 c7 41 ff 50 40 58 5b c9 c3 0f b6 4f 69 55 89 f2 48 89 e5 d3 e2 03 57 30 ec <0f> b6 c0 c9 c3 0f b6 4f 69 55 88 d0 48 89 e5 d3 e6 03 77 30 89
     205<Apr/09 11:03 pm>[2979142.271905] Call Trace:
     206<Apr/09 11:03 pm>[2979142.271906]  <IRQ>  [<ffffffff81334767>] wait_for_xmitr+0x24/0x92
     207<Apr/09 11:03 pm>[2979142.271910]  [<ffffffff813347d5>] ? serial8250_console_putchar+0x0/0x27
     208
     209<Apr/09 11:03 pm>[2979142.271913]  [<ffffffff813347ec>] serial8250_console_putchar+0x17/0x27
     210<Apr/09 11:03 pm>[2979142.271915]  [<ffffffff813311e8>] uart_console_write+0x41/0x52
     211<Apr/09 11:03 pm>[2979142.271917]  [<ffffffff81334eb0>] serial8250_console_write+0xb6/0x108
     212<Apr/09 11:03 pm>[2979142.271920]  [<ffffffff8103b339>] __call_console_drivers+0x67/0x79
     213<Apr/09 11:03 pm>[2979142.271922]  [<ffffffff8103b3a6>] _call_console_drivers+0x5b/0x5f
     214<Apr/09 11:03 pm>[2979142.271925]  [<ffffffff8103b87b>] release_console_sem+0x121/0x1c6
     215<Apr/09 11:03 pm>[2979142.271927]  [<ffffffff8103beb2>] vprintk+0x36b/0x3a0
     216<Apr/09 11:03 pm>[2979142.271930]  [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
     217<Apr/09 11:03 pm>[2979142.271932]  [<ffffffff8151c589>] printk+0x67/0x69
     218<Apr/09 11:03 pm>[2979142.271934]  [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
     219<Apr/09 11:03 pm>[2979142.271936]  [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
     220<Apr/09 11:03 pm>[2979142.271938]  [<ffffffff8151c589>] ? printk+0x67/0x69
     221<Apr/09 11:03 pm>[2979142.271940]  [<ffffffff81019eb4>] ? touch_nmi_watchdog+0x5b/0x60
     222<Apr/09 11:03 pm>[2979142.271942]  [<ffffffff810060bb>] print_trace_address+0x4e/0x57
     223<Apr/09 11:03 pm>[2979142.271944]  [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
     224<Apr/09 11:03 pm>[2979142.271946]  [<ffffffff810061e9>] print_context_stack+0x62/0xbd
     225<Apr/09 11:03 pm>[2979142.271949]  [<ffffffff810054e2>] dump_trace+0x23d/0x327
     226<Apr/09 11:03 pm>[2979142.271951]  [<ffffffff8100604f>] show_trace_log_lvl+0x4c/0x58
     227<Apr/09 11:03 pm>[2979142.271953]  [<ffffffff8100606b>] show_trace+0x10/0x12
     228<Apr/09 11:03 pm>[2979142.271955]  [<ffffffff8151c401>] dump_stack+0x72/0x7b
     229<Apr/09 11:03 pm>[2979142.271957]  [<ffffffff812c4e9f>] do_raw_spin_lock+0x118/0x147
     230<Apr/09 11:03 pm>[2979142.271959]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     231
     232<Apr/09 11:03 pm>[2979142.271962]  [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
     233<Apr/09 11:03 pm>[2979142.271964]  [<ffffffff81038588>] scheduler_tick+0x3d/0x236
     234<Apr/09 11:03 pm>[2979142.271966]  [<ffffffff810483d1>] update_process_times+0x49/0x59
     235<Apr/09 11:03 pm>[2979142.271968]  [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
     236<Apr/09 11:03 pm>[2979142.271970]  [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
     237<Apr/09 11:03 pm>[2979142.271972]  [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
     238<Apr/09 11:03 pm>[2979142.271974]  [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
     239<Apr/09 11:03 pm>[2979142.271976]  [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
     240                 <Apr/09 11:03 pm>[2979142.271979]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
     241<Apr/09 11:03 pm>[2979142.271980]  <EOI>  [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
     242<Apr/09 11:03 pm>[2979142.271984]  [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
     243<Apr/09 11:03 pm>[2979142.271986]  [<ffffffff810d48fd>] ? get_swap_page+0x13c/0x5bb
     244<Apr/09 11:03 pm>[2979142.271988]  [<ffffffff810d17e9>] add_to_swap+0xf/0x48
     245<Apr/09 11:03 pm>[2979142.271990]  [<ffffffff810b7464>] shrink_page_list+0x214/0x68f
     246<Apr/09 11:03 pm>[2979142.271992]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     247<Apr/09 11:03 pm>[2979142.271995]  [<ffffffff810e5fb2>] ? mem_cgroup_del_lru_list+0x7b/0x8e
     248<Apr/09 11:03 pm>[2979142.271997]  [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
     249<Apr/09 11:03 pm>[2979142.272000]  [<ffffffff8144b300>] ? release_sock+0xdb/0xe4
     250<Apr/09 11:03 pm>[2979142.272002]  [<ffffffff8144b254>] ? release_sock+0x2f/0xe4
     251<Apr/09 11:03 pm>[2979142.272004]  [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
     252<Apr/09 11:03 pm>[2979142.272006]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
     253<Apr/09 11:03 pm>[2979142.272009]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
     254
     255<Apr/09 11:03 pm>[2979142.272011]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
     256<Apr/09 11:03 pm>[2979142.272014]  [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
     257<Apr/09 11:03 pm>[2979142.272016]  [<ffffffff814fc30d>] svc_recv+0xe5/0x7f5
     258                 <Apr/09 11:03 pm>[2979142.272018]  [<ffffffff814fc1d9>] ? svc_send+0x85/0xa1
     259<Apr/09 11:03 pm>[2979142.272020]  [<ffffffff811f9bb1>] ? exp_readunlock+0x10/0x12
     260                 <Apr/09 11:03 pm>[2979142.272023]  [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
     261<Apr/09 11:03 pm>[2979142.272025]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
     262<Apr/09 11:03 pm>[2979142.272027]  [<ffffffff811f0d78>] nfsd+0xa9/0x139
     263<Apr/09 11:03 pm>[2979142.272029]  [<ffffffff810546e2>] kthread+0x7d/0x85
     264<Apr/09 11:03 pm>[2979142.272031]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     265<Apr/09 11:03 pm>[2979142.272033]  [<ffffffff81054665>] ? kthread+0x0/0x85
     266<Apr/09 11:03 pm>[2979142.272035]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     267<Apr/09 11:03 pm>[2979142.272037] Pid: 8574, comm: nfsd Tainted: G      D    2.6.34.4 #3
     268<Apr/09 11:03 pm>[2979142.272038] Call Trace:
     269<Apr/09 11:03 pm>[2979142.272039]  <NMI>  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
     270<Apr/09 11:03 pm>[2979142.272043]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
     271                 <Apr/09 11:03 pm>[2979142.272045]  [<ffffffff81520360>] do_nmi+0xb2/0x259
     272<Apr/09 11:03 pm>[2979142.272047]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
     273<Apr/09 11:03 pm>[2979142.272049]  [<ffffffff81334e4e>] ? serial8250_console_write+0x54/0x108
     274<Apr/09 11:03 pm>[2979142.272052]  [<ffffffff8133423b>] ? io_serial_in+0x10/0x15
     275<Apr/09 11:03 pm>[2979142.272053]  <<EOE>>  <IRQ>  [<ffffffff81334767>] wait_for_xmitr+0x24/0x92
     276                 <Apr/09 11:03 pm>[2979142.272057]  [<ffffffff813347d5>] ? serial8250_console_putchar+0x0/0x27
     277
     278<Apr/09 11:03 pm>[2979142.272060]  [<ffffffff813347ec>] serial8250_console_putchar+0x17/0x27
     279<Apr/09 11:03 pm>[2979142.272062]  [<ffffffff813311e8>] uart_console_write+0x41/0x52
     280<Apr/09 11:03 pm>[2979142.272064]  [<ffffffff81334eb0>] serial8250_console_write+0xb6/0x108
     281<Apr/09 11:03 pm>[2979142.272067]  [<ffffffff8103b339>] __call_console_drivers+0x67/0x79
     282<Apr/09 11:03 pm>[2979142.272069]  [<ffffffff8103b3a6>] _call_console_drivers+0x5b/0x5f
     283<Apr/09 11:03 pm>[2979142.272072]  [<ffffffff8103b87b>] release_console_sem+0x121/0x1c6
     284<Apr/09 11:03 pm>[2979142.272074]  [<ffffffff8103beb2>] vprintk+0x36b/0x3a0
     285<Apr/09 11:03 pm>[2979142.272076]  [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
     286                 <Apr/09 11:03 pm>[2979142.272078]  [<ffffffff8151c589>] printk+0x67/0x69
     287<Apr/09 11:03 pm>[2979142.272080]  [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
     288<Apr/09 11:03 pm>[2979142.272082]  [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
     289<Apr/09 11:03 pm>[2979142.272085]  [<ffffffff8151c589>] ? printk+0x67/0x69
     290<Apr/09 11:03 pm>[2979142.272087]  [<ffffffff81019eb4>] ? touch_nmi_watchdog+0x5b/0x60
     291<Apr/09 11:03 pm>[2979142.272089]  [<ffffffff810060bb>] print_trace_address+0x4e/0x57
     292<Apr/09 11:03 pm>[2979142.272091]  [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
     293<Apr/09 11:03 pm>[2979142.272093]  [<ffffffff810061e9>] print_context_stack+0x62/0xbd
     294<Apr/09 11:03 pm>[2979142.272095]  [<ffffffff810054e2>] dump_trace+0x23d/0x327
     295<Apr/09 11:03 pm>[2979142.272098]  [<ffffffff8100604f>] show_trace_log_lvl+0x4c/0x58
     296<Apr/09 11:03 pm>[2979142.272100]  [<ffffffff8100606b>] show_trace+0x10/0x12
     297<Apr/09 11:03 pm>[2979142.272102]  [<ffffffff8151c401>] dump_stack+0x72/0x7b
     298<Apr/09 11:03 pm>[2979142.272104]  [<ffffffff812c4e9f>] do_raw_spin_lock+0x118/0x147
     299<Apr/09 11:03 pm>[2979142.272106]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     300
     301<Apr/09 11:03 pm>[2979142.272108]  [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
     302<Apr/09 11:03 pm>[2979142.272111]  [<ffffffff81038588>] scheduler_tick+0x3d/0x236
     303<Apr/09 11:03 pm>[2979142.272113]  [<ffffffff810483d1>] update_process_times+0x49/0x59
     304<Apr/09 11:03 pm>[2979142.272115]  [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
     305<Apr/09 11:03 pm>[2979142.272117]  [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
     306<Apr/09 11:03 pm>[2979142.272119]  [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
     307<Apr/09 11:03 pm>[2979142.272121]  [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
     308<Apr/09 11:03 pm>[2979142.272123]  [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
     309<Apr/09 11:03 pm>[2979142.272126]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
     310<Apr/09 11:03 pm>[2979142.272127]  <EOI>  [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
     311<Apr/09 11:03 pm>[2979142.272130]  [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
     312<Apr/09 11:03 pm>[2979142.272132]  [<ffffffff810d48fd>] ? get_swap_page+0x13c/0x5bb
     313<Apr/09 11:03 pm>[2979142.272134]  [<ffffffff810d17e9>] add_to_swap+0xf/0x48
     314<Apr/09 11:03 pm>[2979142.272136]  [<ffffffff810b7464>] shrink_page_list+0x214/0x68f
     315<Apr/09 11:03 pm>[2979142.272139]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     316<Apr/09 11:03 pm>[2979142.272141]  [<ffffffff810e5fb2>] ? mem_cgroup_del_lru_list+0x7b/0x8e
     317<Apr/09 11:03 pm>[2979142.272143]  [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
     318<Apr/09 11:03 pm>[2979142.272146]  [<ffffffff8144b300>] ? release_sock+0xdb/0xe4
     319<Apr/09 11:03 pm>[2979142.272148]  [<ffffffff8144b254>] ? release_sock+0x2f/0xe4
     320<Apr/09 11:03 pm>[2979142.272151]  [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
     321<Apr/09 11:03 pm>[2979142.272153]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
     322<Apr/09 11:03 pm>[2979142.272155]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
     323
     324<Apr/09 11:03 pm>[2979142.272158]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
     325<Apr/09 11:03 pm>[2979142.272160]  [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
     326<Apr/09 11:03 pm>[2979142.272162]  [<ffffffff814fc30d>] svc_recv+0xe5/0x7f5
     327<Apr/09 11:03 pm>[2979142.272164]  [<ffffffff814fc1d9>] ? svc_send+0x85/0xa1
     328<Apr/09 11:03 pm>[2979142.272167]  [<ffffffff811f9bb1>] ? exp_readunlock+0x10/0x12
     329<Apr/09 11:03 pm>[2979142.272169]  [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
     330<Apr/09 11:03 pm>[2979142.272171]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
     331<Apr/09 11:03 pm>[2979142.272172]  [<ffffffff811f0d78>] nfsd+0xa9/0x139
     332<Apr/09 11:03 pm>[2979142.272175]  [<ffffffff810546e2>] kthread+0x7d/0x85
     333<Apr/09 11:03 pm>[2979142.272177]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     334<Apr/09 11:03 pm>[2979142.272179]  [<ffffffff81054665>] ? kthread+0x0/0x85
     335<Apr/09 11:03 pm>[2979142.272181]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     336<Apr/09 11:03 pm>[2979144.047629] BUG: spinlock lockup on CPU#6, nfsd/8590, ffff880655213c00
     337<Apr/09 11:03 pm>[2979144.047630] Pid: 8590, comm: nfsd Tainted: G      D    2.6.34.4 #3
     338<Apr/09 11:03 pm>[2979144.047631] Call Trace:
     339<Apr/09 11:03 pm>[2979144.047632]  <IRQ>  [<ffffffff812c4e9f>] do_raw_spin_lock+0x118/0x147
     340<Apr/09 11:03 pm>[2979144.047636]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     341<Apr/09 11:03 pm>[2979144.047638]  [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
     342<Apr/09 11:03 pm>[2979144.047640]  [<ffffffff81038588>] scheduler_tick+0x3d/0x236
     343<Apr/09 11:03 pm>[2979144.047642]  [<ffffffff810483d1>] update_process_times+0x49/0x59
     344                 <Apr/09 11:03 pm>[2979144.047644]  [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
     345<Apr/09 11:03 pm>[2979144.047646]  [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
     346
     347<Apr/09 11:03 pm>[2979144.047648]  [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
     348<Apr/09 11:03 pm>[2979144.047650]  [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
     349<Apr/09 11:03 pm>[2979144.047653]  [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
     350<Apr/09 11:03 pm>[2979144.047655]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
     351<Apr/09 11:03 pm>[2979144.047656]  <EOI>
     352<Apr/09 11:03 pm>[2979144.047657] sending NMI to all CPUs:
     353<Apr/09 11:03 pm>[2979144.047659] NMI backtrace for cpu 5
     354<Apr/09 11:03 pm>[2979144.047660] CPU 5
     355<Apr/09 11:03 pm>[2979144.047661] Modules linked in: w83793 hwmon_vid autofs4 iTCO_wdt e1000e i2c_i801 i2c_core ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
     356<Apr/09 11:03 pm>[2979144.047674]
     357<Apr/09 11:03 pm>[2979144.047675] Pid: 8541, comm: nfsd Tainted: G      D    2.6.34.4 #3 S7010/empty
     358<Apr/09 11:03 pm>[2979144.047677] RIP: 0010:[<ffffffff812c08a2>]  [<ffffffff812c08a2>] delay_tsc+0x18/0x59
     359<Apr/09 11:03 pm>[2979144.047680] RSP: 0018:ffff880621edfb48  EFLAGS: 00000096
     360<Apr/09 11:03 pm>[2979144.047681] RAX: 00000000ab55d014 RBX: 000000000407cd3a RCX: 00000000ab55cfd0
     361<Apr/09 11:03 pm>[2979144.047683] RDX: 00000000001c399f RSI: 0000000000000005 RDI: 0000000000000001
     362<Apr/09 11:03 pm>[2979144.047684] RBP: ffff880621edfb50 R08: 0000000000000005 R09: 0000000000000005
     363<Apr/09 11:03 pm>[2979144.047686] R10: 0000000000000000 R11: ffff880621edf980 R12: ffff880655213c00
     364<Apr/09 11:03 pm>[2979144.047687] R13: 0000000000000000 R14: ffff8806231bb090 R15: ffff8806231bb518
     365<Apr/09 11:03 pm>[2979144.047689] FS:  0000000000000000(0000) GS:ffff8800026a0000(0000) knlGS:0000000000000000
     366<Apr/09 11:03 pm>[2979144.047691] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
     367<Apr/09 11:04 pm>[2979144.047692] CR2: 00007ff62c558028 CR3: 0000000001a8a000 CR4: 00000000000006e0
     368<Apr/09 11:04 pm>[2979144.047694] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     369
     370<Apr/09 11:04 pm>[2979144.047695] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     371<Apr/09 11:04 pm>[2979144.047697] Process nfsd (pid: 8541, threadinfo ffff880621ede000, task ffff8806231bb090)
     372<Apr/09 11:04 pm>[2979144.047698] Stack:
     373<Apr/09 11:04 pm>[2979144.047699]  ffffffff81019da9 ffff880621edfb60 ffffffff812c08fe ffff880621edfba0
     374<Apr/09 11:04 pm>[2979144.047701] <0> ffffffff812c4e65 000000009ef30658 ffff880655213c00 ffff8806230b3090
     375<Apr/09 11:04 pm>[2979144.047703] <0> 0000000000013c00 ffff880621edfc10 0000000000000005 ffff880621edfbc0
     376<Apr/09 11:04 pm>[2979144.047705] Call Trace:
     377<Apr/09 11:04 pm>[2979144.047708]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     378<Apr/09 11:04 pm>[2979144.047710]  [<ffffffff812c08fe>] __delay+0xa/0xc
     379<Apr/09 11:04 pm>[2979144.047712]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     380<Apr/09 11:04 pm>[2979144.047714]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     381<Apr/09 11:04 pm>[2979144.047716]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
     382<Apr/09 11:04 pm>[2979144.047718]  [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
     383<Apr/09 11:04 pm>[2979144.047720]  [<ffffffff81036578>] try_to_wake_up+0x2e/0x342
     384<Apr/09 11:04 pm>[2979144.047722]  [<ffffffff81036899>] default_wake_function+0xd/0xf
     385<Apr/09 11:04 pm>[2979144.047724]  [<ffffffff8102c08d>] __wake_up_common+0x46/0x76
     386<Apr/09 11:04 pm>[2979144.047726]  [<ffffffff8102faa6>] __wake_up+0x38/0x4f
     387<Apr/09 11:04 pm>[2979144.047728]  [<ffffffff814fbb75>] svc_xprt_enqueue+0x1e2/0x23d
     388<Apr/09 11:04 pm>[2979144.047731]  [<ffffffff814fbc26>] svc_xprt_received+0x24/0x26
     389<Apr/09 11:04 pm>[2979144.047733]  [<ffffffff814f1a12>] svc_udp_recvfrom+0x15f/0x2f9
     390<Apr/09 11:04 pm>[2979144.047735]  [<ffffffff814fc980>] svc_recv+0x758/0x7f5
     391<Apr/09 11:04 pm>[2979144.047737]  [<ffffffff814fc1d9>] ? svc_send+0x85/0xa1
     392
     393<Apr/09 11:04 pm>[2979144.047739]  [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
     394<Apr/09 11:04 pm>[2979144.047741]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
     395<Apr/09 11:04 pm>[2979144.047743]  [<ffffffff811f0d78>] nfsd+0xa9/0x139
     396<Apr/09 11:04 pm>[2979144.047745]  [<ffffffff810546e2>] kthread+0x7d/0x85
     397<Apr/09 11:04 pm>[2979144.047747]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     398<Apr/09 11:04 pm>[2979144.047749]  [<ffffffff81054665>] ? kthread+0x0/0x85
     399<Apr/09 11:04 pm>[2979144.047751]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     400<Apr/09 11:04 pm>[2979144.047752] Code: 2e 0f 1f 84 00 00 00 00 00 48 ff c8 75 fb 48 ff c8 c9 c3 55 48 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae e8 0f 31 <89> c1 41 89 f1 66 66 90 0f ae e8 0f 31 89 c6 48 89 f0 48 29 c8
     401<Apr/09 11:04 pm>[2979144.047767] Call Trace:
     402<Apr/09 11:04 pm>[2979144.047770]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     403<Apr/09 11:04 pm>[2979144.047772]  [<ffffffff812c08fe>] __delay+0xa/0xc
     404<Apr/09 11:04 pm>[2979144.047774]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     405<Apr/09 11:04 pm>[2979144.047776]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     406<Apr/09 11:04 pm>[2979144.047778]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
     407<Apr/09 11:04 pm>[2979144.047780]  [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
     408<Apr/09 11:04 pm>[2979144.047782]  [<ffffffff81036578>] try_to_wake_up+0x2e/0x342
     409<Apr/09 11:04 pm>[2979144.047784]  [<ffffffff81036899>] default_wake_function+0xd/0xf
     410<Apr/09 11:04 pm>[2979144.047786]  [<ffffffff8102c08d>] __wake_up_common+0x46/0x76
     411<Apr/09 11:04 pm>[2979144.047788]  [<ffffffff8102faa6>] __wake_up+0x38/0x4f
     412                 <Apr/09 11:04 pm>[2979144.047790]  [<ffffffff814fbb75>] svc_xprt_enqueue+0x1e2/0x23d
     413<Apr/09 11:04 pm>[2979144.047792]  [<ffffffff814fbc26>] svc_xprt_received+0x24/0x26
     414<Apr/09 11:04 pm>[2979144.047794]  [<ffffffff814f1a12>] svc_udp_recvfrom+0x15f/0x2f9
     415
     416<Apr/09 11:04 pm>[2979144.047796]  [<ffffffff814fc980>] svc_recv+0x758/0x7f5
     417<Apr/09 11:04 pm>[2979144.047798]  [<ffffffff814fc1d9>] ? svc_send+0x85/0xa1
     418<Apr/09 11:04 pm>[2979144.047801]  [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
     419<Apr/09 11:04 pm>[2979144.047803]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
     420<Apr/09 11:04 pm>[2979144.047804]  [<ffffffff811f0d78>] nfsd+0xa9/0x139
     421<Apr/09 11:04 pm>[2979144.047806]  [<ffffffff810546e2>] kthread+0x7d/0x85
     422<Apr/09 11:04 pm>[2979144.047808]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     423<Apr/09 11:04 pm>[2979144.047811]  [<ffffffff81054665>] ? kthread+0x0/0x85
     424<Apr/09 11:04 pm>[2979144.047813]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     425<Apr/09 11:04 pm>[2979144.047814] Pid: 8541, comm: nfsd Tainted: G      D    2.6.34.4 #3
     426<Apr/09 11:04 pm>[2979144.047815] Call Trace:
     427<Apr/09 11:04 pm>[2979144.047816]  <NMI>  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
     428                 <Apr/09 11:04 pm>[2979144.047820]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
     429<Apr/09 11:04 pm>[2979144.047822]  [<ffffffff81520360>] do_nmi+0xb2/0x259
     430<Apr/09 11:04 pm>[2979144.047823]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
     431<Apr/09 11:04 pm>[2979144.047826]  [<ffffffff812c08a2>] ? delay_tsc+0x18/0x59
     432<Apr/09 11:04 pm>[2979144.047827]  <<EOE>>  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     433<Apr/09 11:04 pm>[2979144.047830]  [<ffffffff812c08fe>] __delay+0xa/0xc
     434<Apr/09 11:04 pm>[2979144.047832]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     435<Apr/09 11:04 pm>[2979144.047835]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     436<Apr/09 11:04 pm>[2979144.047836]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
     437<Apr/09 11:04 pm>[2979144.047838]  [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
     438
     439<Apr/09 11:04 pm>[2979144.047840]  [<ffffffff81036578>] try_to_wake_up+0x2e/0x342
     440<Apr/09 11:04 pm>[2979144.047843]  [<ffffffff81036899>] default_wake_function+0xd/0xf
     441<Apr/09 11:04 pm>[2979144.047845]  [<ffffffff8102c08d>] __wake_up_common+0x46/0x76
     442<Apr/09 11:04 pm>[2979144.047847]  [<ffffffff8102faa6>] __wake_up+0x38/0x4f
     443<Apr/09 11:04 pm>[2979144.047849]  [<ffffffff814fbb75>] svc_xprt_enqueue+0x1e2/0x23d
     444<Apr/09 11:04 pm>[2979144.047851]  [<ffffffff814fbc26>] svc_xprt_received+0x24/0x26
     445<Apr/09 11:04 pm>[2979144.047853]  [<ffffffff814f1a12>] svc_udp_recvfrom+0x15f/0x2f9
     446<Apr/09 11:04 pm>[2979144.047855]  [<ffffffff814fc980>] svc_recv+0x758/0x7f5
     447<Apr/09 11:04 pm>[2979144.047857]  [<ffffffff814fc1d9>] ? svc_send+0x85/0xa1
     448                 <Apr/09 11:04 pm>[2979144.047860]  [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
     449<Apr/09 11:04 pm>[2979144.047861]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
     450<Apr/09 11:04 pm>[2979144.047863]  [<ffffffff811f0d78>] nfsd+0xa9/0x139
     451<Apr/09 11:04 pm>[2979144.047865]  [<ffffffff810546e2>] kthread+0x7d/0x85
     452<Apr/09 11:04 pm>[2979144.047867]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     453<Apr/09 11:04 pm>[2979144.047870]  [<ffffffff81054665>] ? kthread+0x0/0x85
     454<Apr/09 11:04 pm>[2979144.047871]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     455<Apr/09 11:04 pm>[2979144.047873] NMI backtrace for cpu 6
     456<Apr/09 11:04 pm>[2979144.047874] CPU 6
     457<Apr/09 11:04 pm>[2979144.047875] Modules linked in: w83793 hwmon_vid autofs4 iTCO_wdt e1000e i2c_i801 i2c_core ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
     458<Apr/09 11:04 pm>[2979144.047889]
     459<Apr/09 11:04 pm>[2979144.047890] Pid: 8590, comm: nfsd Tainted: G      D    2.6.34.4 #3 S7010/empty
     460<Apr/09 11:04 pm>[2979144.047891] RIP: 0010:[<ffffffff812c08af>]  [<ffffffff812c08af>] delay_tsc+0x25/0x59
     461
     462<Apr/09 11:04 pm>[2979144.047894] RSP: 0018:ffff880655203d50  EFLAGS: 00000046
     463<Apr/09 11:04 pm>[2979144.047896] RAX: 00000000ab59e364 RBX: 0000000000000000 RCX: 00000000ab59e31c
     464<Apr/09 11:04 pm>[2979144.047897] RDX: 00000000001c399f RSI: 00000000ab59e338 RDI: 00000000000411b3
     465<Apr/09 11:04 pm>[2979144.047899] RBP: ffff880655203d58 R08: 0000000000000006 R09: 0000000000000006
     466<Apr/09 11:04 pm>[2979144.047900] R10: 000000000000000b R11: ffff880655203c18 R12: 0000000000001000
     467<Apr/09 11:04 pm>[2979144.047902] R13: 0000000000000002 R14: 0000000000000092 R15: ffff8806227e0022
     468<Apr/09 11:04 pm>[2979144.047904] FS:  0000000000000000(0000) GS:ffff880655200000(0000) knlGS:0000000000000000
     469<Apr/09 11:04 pm>[2979144.047905] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
     470<Apr/09 11:04 pm>[2979144.047907] CR2: 0000000389ca1c18 CR3: 0000000001a8a000 CR4: 00000000000006e0
     471<Apr/09 11:04 pm>[2979144.047908] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     472<Apr/09 11:04 pm>[2979144.047910] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     473<Apr/09 11:04 pm>[2979144.047911] Process nfsd (pid: 8590, threadinfo ffff880622754000, task ffff8806227ed0f0)
     474<Apr/09 11:04 pm>[2979144.047913] Stack:
     475                 <Apr/09 11:04 pm>[2979144.047913]  0000000000000000 ffff880655203d68 ffffffff812c093d ffff880655203d88
     476<Apr/09 11:04 pm>[2979144.047915] <0> ffffffff810197e9 0000000000000007 000000000000d200 ffff880655203dc8
     477<Apr/09 11:04 pm>[2979144.047918] <0> ffffffff81019c76 ffffffff81b24210 0000000000000000 ffff880655213c00
     478<Apr/09 11:04 pm>[2979144.047920] Call Trace:
     479<Apr/09 11:04 pm>[2979144.047921]  <IRQ>
     480<Apr/09 11:04 pm>[2979144.047923]  [<ffffffff812c093d>] __const_udelay+0x3d/0x3f
     481<Apr/09 11:04 pm>[2979144.047925]  [<ffffffff810197e9>] native_safe_apic_wait_icr_idle+0x31/0x48
     482                 <Apr/09 11:04 pm>[2979144.047928]  [<ffffffff81019c76>] default_send_IPI_mask_sequence_phys+0x49/0xab
     483<Apr/09 11:04 pm>[2979144.047930]  [<ffffffff8101c5d8>] physflat_send_IPI_all+0x14/0x16
     484
     485<Apr/09 11:04 pm>[2979144.047933]  [<ffffffff81019d94>] arch_trigger_all_cpu_backtrace+0x3c/0x5d
     486<Apr/09 11:04 pm>[2979144.047935]  [<ffffffff812c4ea4>] do_raw_spin_lock+0x11d/0x147
     487<Apr/09 11:04 pm>[2979144.047937]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     488<Apr/09 11:04 pm>[2979144.047939]  [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
     489<Apr/09 11:04 pm>[2979144.047941]  [<ffffffff81038588>] scheduler_tick+0x3d/0x236
     490<Apr/09 11:04 pm>[2979144.047944]  [<ffffffff810483d1>] update_process_times+0x49/0x59
     491<Apr/09 11:04 pm>[2979144.047946]  [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
     492<Apr/09 11:04 pm>[2979144.047948]  [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
     493<Apr/09 11:04 pm>[2979144.047950]  [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
     494<Apr/09 11:04 pm>[2979144.047952]  [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
     495<Apr/09 11:04 pm>[2979144.047954]  [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
     496<Apr/09 11:04 pm>[2979144.047956]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
     497<Apr/09 11:04 pm>[2979144.047958]  <EOI>
     498<Apr/09 11:04 pm>[2979144.047958] Code: fb 48 ff c8 c9 c3 55 48 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae e8 0f 31 89 c1 41 89 f1 66 66 90 0f ae e8 0f 31 <89> c6 48 89 f0 48 29 c8 48 39 f8 73 25 f3 90 65 44 8b 04 25 f0
     499<Apr/09 11:04 pm>[2979144.047974] Call Trace:
     500<Apr/09 11:04 pm>[2979144.047975]  <IRQ>  [<ffffffff812c093d>] __const_udelay+0x3d/0x3f
     501<Apr/09 11:04 pm>[2979144.047978]  [<ffffffff810197e9>] native_safe_apic_wait_icr_idle+0x31/0x48
     502<Apr/09 11:04 pm>[2979144.047981]  [<ffffffff81019c76>] default_send_IPI_mask_sequence_phys+0x49/0xab
     503<Apr/09 11:04 pm>[2979144.047984]  [<ffffffff8101c5d8>] physflat_send_IPI_all+0x14/0x16
     504<Apr/09 11:04 pm>[2979144.047986]  [<ffffffff81019d94>] arch_trigger_all_cpu_backtrace+0x3c/0x5d
     505<Apr/09 11:04 pm>[2979144.047988]  [<ffffffff812c4ea4>] do_raw_spin_lock+0x11d/0x147
     506<Apr/09 11:04 pm>[2979144.047990]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     507
     508<Apr/09 11:04 pm>[2979144.047993]  [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
     509<Apr/09 11:04 pm>[2979144.047995]  [<ffffffff81038588>] scheduler_tick+0x3d/0x236
     510<Apr/09 11:04 pm>[2979144.047997]  [<ffffffff810483d1>] update_process_times+0x49/0x59
     511<Apr/09 11:04 pm>[2979144.047999]  [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
     512<Apr/09 11:04 pm>[2979144.048001]  [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
     513<Apr/09 11:04 pm>[2979144.048003]  [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
     514<Apr/09 11:04 pm>[2979144.048005]  [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
     515<Apr/09 11:04 pm>[2979144.048007]  [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
     516<Apr/09 11:04 pm>[2979144.048009]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
     517<Apr/09 11:04 pm>[2979144.048011]  <EOI> Pid: 8590, comm: nfsd Tainted: G      D    2.6.34.4 #3
     518<Apr/09 11:04 pm>[2979144.048013] Call Trace:
     519<Apr/09 11:04 pm>[2979144.048013]  <NMI>  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
     520<Apr/09 11:04 pm>[2979144.048017]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
     521<Apr/09 11:04 pm>[2979144.048019]  [<ffffffff81520360>] do_nmi+0xb2/0x259
     522<Apr/09 11:04 pm>[2979144.048021]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
     523<Apr/09 11:04 pm>[2979144.048023]  [<ffffffff812c08af>] ? delay_tsc+0x25/0x59
     524<Apr/09 11:04 pm>[2979144.048024]  <<EOE>>  <IRQ>  [<ffffffff812c093d>] __const_udelay+0x3d/0x3f
     525<Apr/09 11:04 pm>[2979144.048028]  [<ffffffff810197e9>] native_safe_apic_wait_icr_idle+0x31/0x48
     526<Apr/09 11:04 pm>[2979144.048030]  [<ffffffff81019c76>] default_send_IPI_mask_sequence_phys+0x49/0xab
     527<Apr/09 11:04 pm>[2979144.048033]  [<ffffffff8101c5d8>] physflat_send_IPI_all+0x14/0x16
     528<Apr/09 11:04 pm>[2979144.048035]  [<ffffffff81019d94>] arch_trigger_all_cpu_backtrace+0x3c/0x5d
     529<Apr/09 11:04 pm>[2979144.048037]  [<ffffffff812c4ea4>] do_raw_spin_lock+0x11d/0x147
     530
     531<Apr/09 11:04 pm>[2979144.048040]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     532<Apr/09 11:04 pm>[2979144.048042]  [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
     533<Apr/09 11:04 pm>[2979144.048044]  [<ffffffff81038588>] scheduler_tick+0x3d/0x236
     534<Apr/09 11:04 pm>[2979144.048046]  [<ffffffff810483d1>] update_process_times+0x49/0x59
     535<Apr/09 11:04 pm>[2979144.048048]  [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
     536<Apr/09 11:04 pm>[2979144.048050]  [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
     537<Apr/09 11:04 pm>[2979144.048052]  [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
     538<Apr/09 11:04 pm>[2979144.048054]  [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
     539<Apr/09 11:04 pm>[2979144.048056]  [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
     540<Apr/09 11:04 pm>[2979144.048059]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
     541                 <Apr/09 11:04 pm>[2979144.048060]  <EOI>
     542<Apr/09 11:04 pm>[2979144.048062] NMI backtrace for cpu 7
     543<Apr/09 11:04 pm>[2979144.048063] CPU 7
     544<Apr/09 11:04 pm>[2979144.048063] Modules linked in: w83793 hwmon_vid autofs4 iTCO_wdt e1000e i2c_i801 i2c_core ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
     545<Apr/09 11:04 pm>[2979144.048077]
     546<Apr/09 11:04 pm>[2979144.048078] Pid: 8574, comm: nfsd Tainted: G      D    2.6.34.4 #3 S7010/empty
     547<Apr/09 11:04 pm>[2979144.048079] RIP: 0010:[<ffffffff812c08ad>]  [<ffffffff812c08ad>] delay_tsc+0x23/0x59
     548<Apr/09 11:04 pm>[2979144.048083] RSP: 0018:ffff880655223920  EFLAGS: 00000046
     549<Apr/09 11:04 pm>[2979144.048084] RAX: 00000000000002fc RBX: 0000000000000000 RCX: 00000000ab663114
     550<Apr/09 11:04 pm>[2979144.048086] RDX: 00000000001c399f RSI: 00000000ab663410 RDI: 0000000000000a6b
     551<Apr/09 11:04 pm>[2979144.048087] RBP: ffff880655223928 R08: 0000000000000007 R09: 0000000000000007
     552<Apr/09 11:04 pm>[2979144.048089] R10: ffff8800000b8820 R11: ffffffff81334e4e R12: ffffffff82400ac0
     553
     554<Apr/09 11:04 pm>[2979144.048090] R13: 00000000000026f3 R14: 0000000000000020 R15: 0000000000000044
     555<Apr/09 11:04 pm>[2979144.048092] FS:  0000000000000000(0000) GS:ffff880655220000(0000) knlGS:0000000000000000
     556<Apr/09 11:04 pm>[2979144.048094] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
     557<Apr/09 11:04 pm>[2979144.048095] CR2: 00007f7053196000 CR3: 0000000001a8a000 CR4: 00000000000006e0
     558<Apr/09 11:04 pm>[2979144.048097] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     559<Apr/09 11:04 pm>[2979144.048099] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     560<Apr/09 11:04 pm>[2979144.048100] Process nfsd (pid: 8574, threadinfo ffff880623fec000, task ffff8806227d3090)
     561<Apr/09 11:04 pm>[2979144.048102] Stack:
     562<Apr/09 11:04 pm>[2979144.048102]  0000000000000000 ffff880655223938 ffffffff812c093d ffff880655223968
     563<Apr/09 11:04 pm>[2979144.048104] <0> ffffffff81334786 ffffffff82400ac0 0000000000000031 ffffffff813347d5
     564<Apr/09 11:04 pm>[2979144.048107] <0> ffffffff82400ac0 ffff880655223988 ffffffff813347ec ffffffff81c3dc36
     565<Apr/09 11:04 pm>[2979144.048109] Call Trace:
     566<Apr/09 11:04 pm>[2979144.048110]  <IRQ>
     567<Apr/09 11:04 pm>[2979144.048112]  [<ffffffff812c093d>] __const_udelay+0x3d/0x3f
     568<Apr/09 11:04 pm>[2979144.048114]  [<ffffffff81334786>] wait_for_xmitr+0x43/0x92
     569                 <Apr/09 11:04 pm>[2979144.048116]  [<ffffffff813347d5>] ? serial8250_console_putchar+0x0/0x27
     570<Apr/09 11:04 pm>[2979144.048119]  [<ffffffff813347ec>] serial8250_console_putchar+0x17/0x27
     571<Apr/09 11:04 pm>[2979144.048121]  [<ffffffff813311e8>] uart_console_write+0x41/0x52
     572<Apr/09 11:04 pm>[2979144.048123]  [<ffffffff81334eb0>] serial8250_console_write+0xb6/0x108
     573<Apr/09 11:04 pm>[2979144.048126]  [<ffffffff8103b339>] __call_console_drivers+0x67/0x79
     574<Apr/09 11:04 pm>[2979144.048128]  [<ffffffff8103b3a6>] _call_console_drivers+0x5b/0x5f
     575<Apr/09 11:04 pm>[2979144.048131]  [<ffffffff8103b87b>] release_console_sem+0x121/0x1c6
     576
     577<Apr/09 11:04 pm>[2979144.048133]  [<ffffffff8103beb2>] vprintk+0x36b/0x3a0
     578<Apr/09 11:04 pm>[2979144.048135]  [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
     579<Apr/09 11:04 pm>[2979144.048138]  [<ffffffff8151c589>] printk+0x67/0x69
     580<Apr/09 11:04 pm>[2979144.048139]  [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
     581<Apr/09 11:04 pm>[2979144.048141]  [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
     582<Apr/09 11:04 pm>[2979144.048144]  [<ffffffff8151c589>] ? printk+0x67/0x69
     583<Apr/09 11:04 pm>[2979144.048146]  [<ffffffff81019eb4>] ? touch_nmi_watchdog+0x5b/0x60
     584<Apr/09 11:04 pm>[2979144.048148]  [<ffffffff810060bb>] print_trace_address+0x4e/0x57
     585<Apr/09 11:04 pm>[2979144.048150]  [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
     586                 <Apr/09 11:04 pm>[2979144.048152]  [<ffffffff810061e9>] print_context_stack+0x62/0xbd
     587<Apr/09 11:04 pm>[2979144.048154]  [<ffffffff810054e2>] dump_trace+0x23d/0x327
     588<Apr/09 11:04 pm>[2979144.048156]  [<ffffffff8100604f>] show_trace_log_lvl+0x4c/0x58
     589<Apr/09 11:04 pm>[2979144.048158]  [<ffffffff8100606b>] show_trace+0x10/0x12
     590<Apr/09 11:04 pm>[2979144.048160]  [<ffffffff8151c401>] dump_stack+0x72/0x7b
     591<Apr/09 11:04 pm>[2979144.048162]  [<ffffffff812c4e9f>] do_raw_spin_lock+0x118/0x147
     592<Apr/09 11:04 pm>[2979144.048165]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     593<Apr/09 11:04 pm>[2979144.048167]  [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
     594<Apr/09 11:04 pm>[2979144.048169]  [<ffffffff81038588>] scheduler_tick+0x3d/0x236
     595<Apr/09 11:04 pm>[2979144.048171]  [<ffffffff810483d1>] update_process_times+0x49/0x59
     596<Apr/09 11:04 pm>[2979144.048173]  [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
     597<Apr/09 11:04 pm>[2979144.048175]  [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
     598<Apr/09 11:04 pm>[2979144.048177]  [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
     599
     600<Apr/09 11:04 pm>[2979144.048179]  [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
     601<Apr/09 11:04 pm>[2979144.048181]  [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
     602<Apr/09 11:04 pm>[2979144.048184]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
     603<Apr/09 11:04 pm>[2979144.048185]  <EOI>
     604<Apr/09 11:04 pm>[2979144.048187]  [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
     605<Apr/09 11:04 pm>[2979144.048188]  [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
     606<Apr/09 11:04 pm>[2979144.048190]  [<ffffffff810d48fd>] ? get_swap_page+0x13c/0x5bb
     607<Apr/09 11:04 pm>[2979144.048193]  [<ffffffff810d17e9>] add_to_swap+0xf/0x48
     608<Apr/09 11:04 pm>[2979144.048195]  [<ffffffff810b7464>] shrink_page_list+0x214/0x68f
     609<Apr/09 11:04 pm>[2979144.048197]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     610<Apr/09 11:04 pm>[2979144.048199]  [<ffffffff810e5fb2>] ? mem_cgroup_del_lru_list+0x7b/0x8e
     611<Apr/09 11:04 pm>[2979144.048202]  [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
     612<Apr/09 11:04 pm>[2979144.048204]  [<ffffffff8144b300>] ? release_sock+0xdb/0xe4
     613<Apr/09 11:04 pm>[2979144.048207]  [<ffffffff8144b254>] ? release_sock+0x2f/0xe4
     614<Apr/09 11:04 pm>[2979144.048209]  [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
     615<Apr/09 11:04 pm>[2979144.048211]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
     616<Apr/09 11:04 pm>[2979144.048213]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
     617<Apr/09 11:04 pm>[2979144.048216]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
     618<Apr/09 11:04 pm>[2979144.048219]  [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
     619<Apr/09 11:04 pm>[2979144.048221]  [<ffffffff814fc30d>] svc_recv+0xe5/0x7f5
     620<Apr/09 11:04 pm>[2979144.048223]  [<ffffffff814fc1d9>] ? svc_send+0x85/0xa1
     621<Apr/09 11:04 pm>[2979144.048225]  [<ffffffff811f9bb1>] ? exp_readunlock+0x10/0x12
     622
     623<Apr/09 11:04 pm>[2979144.048227]  [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
     624<Apr/09 11:04 pm>[2979144.048229]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
     625<Apr/09 11:04 pm>[2979144.048231]  [<ffffffff811f0d78>] nfsd+0xa9/0x139
     626<Apr/09 11:04 pm>[2979144.048233]  [<ffffffff810546e2>] kthread+0x7d/0x85
     627<Apr/09 11:04 pm>[2979144.048235]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     628<Apr/09 11:04 pm>[2979144.048237]  [<ffffffff81054665>] ? kthread+0x0/0x85
     629<Apr/09 11:04 pm>[2979144.048239]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     630<Apr/09 11:04 pm>[2979144.048240] Code: c8 75 fb 48 ff c8 c9 c3 55 48 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae e8 0f 31 89 c1 41 89 f1 66 66 90 0f ae e8 <0f> 31 89 c6 48 89 f0 48 29 c8 48 39 f8 73 25 f3 90 65 44 8b 04
     631<Apr/09 11:04 pm>[2979144.048255] Call Trace:
     632<Apr/09 11:04 pm>[2979144.048256]  <IRQ>  [<ffffffff812c093d>] __const_udelay+0x3d/0x3f
     633<Apr/09 11:04 pm>[2979144.048260]  [<ffffffff81334786>] wait_for_xmitr+0x43/0x92
     634<Apr/09 11:04 pm>[2979144.048262]  [<ffffffff813347d5>] ? serial8250_console_putchar+0x0/0x27
     635<Apr/09 11:04 pm>[2979144.048264]  [<ffffffff813347ec>] serial8250_console_putchar+0x17/0x27
     636<Apr/09 11:04 pm>[2979144.048267]  [<ffffffff813311e8>] uart_console_write+0x41/0x52
     637<Apr/09 11:04 pm>[2979144.048269]  [<ffffffff81334eb0>] serial8250_console_write+0xb6/0x108
     638<Apr/09 11:04 pm>[2979144.048272]  [<ffffffff8103b339>] __call_console_drivers+0x67/0x79
     639<Apr/09 11:04 pm>[2979144.048274]  [<ffffffff8103b3a6>] _call_console_drivers+0x5b/0x5f
     640<Apr/09 11:04 pm>[2979144.048276]  [<ffffffff8103b87b>] release_console_sem+0x121/0x1c6
     641<Apr/09 11:04 pm>[2979144.048279]  [<ffffffff8103beb2>] vprintk+0x36b/0x3a0
     642<Apr/09 11:04 pm>[2979144.048281]  [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
     643<Apr/09 11:04 pm>[2979144.048283]  [<ffffffff8151c589>] printk+0x67/0x69
     644<Apr/09 11:04 pm>[2979144.048285]  [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
     645
     646<Apr/09 11:04 pm>[2979144.048287]  [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
     647<Apr/09 11:04 pm>[2979144.048289]  [<ffffffff8151c589>] ? printk+0x67/0x69
     648<Apr/09 11:04 pm>[2979144.048291]  [<ffffffff81019eb4>] ? touch_nmi_watchdog+0x5b/0x60
     649<Apr/09 11:04 pm>[2979144.048293]  [<ffffffff810060bb>] print_trace_address+0x4e/0x57
     650<Apr/09 11:04 pm>[2979144.048295]  [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
     651<Apr/09 11:04 pm>[2979144.048297]  [<ffffffff810061e9>] print_context_stack+0x62/0xbd
     652<Apr/09 11:04 pm>[2979144.048299]  [<ffffffff810054e2>] dump_trace+0x23d/0x327
     653<Apr/09 11:04 pm>[2979144.048302]  [<ffffffff8100604f>] show_trace_log_lvl+0x4c/0x58
     654<Apr/09 11:04 pm>[2979144.048304]  [<ffffffff8100606b>] show_trace+0x10/0x12
     655<Apr/09 11:04 pm>[2979144.048306]  [<ffffffff8151c401>] dump_stack+0x72/0x7b
     656<Apr/09 11:04 pm>[2979144.048308]  [<ffffffff812c4e9f>] do_raw_spin_lock+0x118/0x147
     657<Apr/09 11:04 pm>[2979144.048310]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     658<Apr/09 11:04 pm>[2979144.048312]  [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
     659<Apr/09 11:04 pm>[2979144.048314]  [<ffffffff81038588>] scheduler_tick+0x3d/0x236
     660<Apr/09 11:04 pm>[2979144.048316]  [<ffffffff810483d1>] update_process_times+0x49/0x59
     661<Apr/09 11:04 pm>[2979144.048318]  [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
     662<Apr/09 11:04 pm>[2979144.048320]  [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
     663<Apr/09 11:04 pm>[2979144.048322]  [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
     664<Apr/09 11:04 pm>[2979144.048324]  [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
     665<Apr/09 11:04 pm>[2979144.048326]  [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
     666<Apr/09 11:04 pm>[2979144.048329]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
     667<Apr/09 11:04 pm>[2979144.048330]  <EOI>  [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
     668
     669<Apr/09 11:04 pm>[2979144.048333]  [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
     670<Apr/09 11:04 pm>[2979144.048335]  [<ffffffff810d48fd>] ? get_swap_page+0x13c/0x5bb
     671<Apr/09 11:04 pm>[2979144.048337]  [<ffffffff810d17e9>] add_to_swap+0xf/0x48
     672<Apr/09 11:04 pm>[2979144.048339]  [<ffffffff810b7464>] shrink_page_list+0x214/0x68f
     673<Apr/09 11:04 pm>[2979144.048342]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     674<Apr/09 11:04 pm>[2979144.048344]  [<ffffffff810e5fb2>] ? mem_cgroup_del_lru_list+0x7b/0x8e
     675<Apr/09 11:04 pm>[2979144.048347]  [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
     676<Apr/09 11:04 pm>[2979144.048349]  [<ffffffff8144b300>] ? release_sock+0xdb/0xe4
     677<Apr/09 11:04 pm>[2979144.048352]  [<ffffffff8144b254>] ? release_sock+0x2f/0xe4
     678<Apr/09 11:04 pm>[2979144.048354]  [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
     679<Apr/09 11:04 pm>[2979144.048356]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
     680<Apr/09 11:04 pm>[2979144.048358]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
     681<Apr/09 11:04 pm>[2979144.048361]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
     682<Apr/09 11:04 pm>[2979144.048363]  [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
     683<Apr/09 11:04 pm>[2979144.048366]  [<ffffffff814fc30d>] svc_recv+0xe5/0x7f5
     684<Apr/09 11:04 pm>[2979144.048367]  [<ffffffff814fc1d9>] ? svc_send+0x85/0xa1
     685<Apr/09 11:04 pm>[2979144.048370]  [<ffffffff811f9bb1>] ? exp_readunlock+0x10/0x12
     686<Apr/09 11:04 pm>[2979144.048372]  [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
     687<Apr/09 11:04 pm>[2979144.048374]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
     688<Apr/09 11:04 pm>[2979144.048375]  [<ffffffff811f0d78>] nfsd+0xa9/0x139
     689<Apr/09 11:04 pm>[2979144.048377]  [<ffffffff810546e2>] kthread+0x7d/0x85
     690<Apr/09 11:04 pm>[2979144.048379]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     691
     692<Apr/09 11:04 pm>[2979144.048382]  [<ffffffff81054665>] ? kthread+0x0/0x85
     693<Apr/09 11:04 pm>[2979144.048383]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     694<Apr/09 11:04 pm>[2979144.048385] Pid: 8574, comm: nfsd Tainted: G      D    2.6.34.4 #3
     695<Apr/09 11:04 pm>[2979144.048386] Call Trace:
     696<Apr/09 11:04 pm>[2979144.048387]  <NMI>  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
     697<Apr/09 11:04 pm>[2979144.048391]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
     698<Apr/09 11:04 pm>[2979144.048392]  [<ffffffff81520360>] do_nmi+0xb2/0x259
     699<Apr/09 11:04 pm>[2979144.048394]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
     700<Apr/09 11:04 pm>[2979144.048397]  [<ffffffff81334e4e>] ? serial8250_console_write+0x54/0x108
     701<Apr/09 11:04 pm>[2979144.048399]  [<ffffffff812c08ad>] ? delay_tsc+0x23/0x59
     702<Apr/09 11:04 pm>[2979144.048400]  <<EOE>>  <IRQ>  [<ffffffff812c093d>] __const_udelay+0x3d/0x3f
     703<Apr/09 11:04 pm>[2979144.048404]  [<ffffffff81334786>] wait_for_xmitr+0x43/0x92
     704<Apr/09 11:04 pm>[2979144.048406]  [<ffffffff813347d5>] ? serial8250_console_putchar+0x0/0x27
     705<Apr/09 11:04 pm>[2979144.048409]  [<ffffffff813347ec>] serial8250_console_putchar+0x17/0x27
     706<Apr/09 11:04 pm>[2979144.048411]  [<ffffffff813311e8>] uart_console_write+0x41/0x52
     707<Apr/09 11:04 pm>[2979144.048413]  [<ffffffff81334eb0>] serial8250_console_write+0xb6/0x108
     708<Apr/09 11:04 pm>[2979144.048416]  [<ffffffff8103b339>] __call_console_drivers+0x67/0x79
     709<Apr/09 11:04 pm>[2979144.048418]  [<ffffffff8103b3a6>] _call_console_drivers+0x5b/0x5f
     710<Apr/09 11:04 pm>[2979144.048420]  [<ffffffff8103b87b>] release_console_sem+0x121/0x1c6
     711<Apr/09 11:04 pm>[2979144.048423]  [<ffffffff8103beb2>] vprintk+0x36b/0x3a0
     712<Apr/09 11:04 pm>[2979144.048425]  [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
     713<Apr/09 11:04 pm>[2979144.048427]  [<ffffffff8151c589>] printk+0x67/0x69
     714
     715<Apr/09 11:04 pm>[2979144.048429]  [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
     716<Apr/09 11:04 pm>[2979144.048431]  [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
     717<Apr/09 11:04 pm>[2979144.048433]  [<ffffffff8151c589>] ? printk+0x67/0x69
     718<Apr/09 11:04 pm>[2979144.048435]  [<ffffffff81019eb4>] ? touch_nmi_watchdog+0x5b/0x60
     719<Apr/09 11:04 pm>[2979144.048438]  [<ffffffff810060bb>] print_trace_address+0x4e/0x57
     720<Apr/09 11:04 pm>[2979144.048440]  [<ffffffff81057b81>] ? hrtimer_interrupt+0xb5/0x19f
     721<Apr/09 11:04 pm>[2979144.048442]  [<ffffffff810061e9>] print_context_stack+0x62/0xbd
     722<Apr/09 11:04 pm>[2979144.048444]  [<ffffffff810054e2>] dump_trace+0x23d/0x327
     723<Apr/09 11:04 pm>[2979144.048446]  [<ffffffff8100604f>] show_trace_log_lvl+0x4c/0x58
     724<Apr/09 11:04 pm>[2979144.048448]  [<ffffffff8100606b>] show_trace+0x10/0x12
     725<Apr/09 11:04 pm>[2979144.048450]  [<ffffffff8151c401>] dump_stack+0x72/0x7b
     726<Apr/09 11:04 pm>[2979144.048452]  [<ffffffff812c4e9f>] do_raw_spin_lock+0x118/0x147
     727<Apr/09 11:04 pm>[2979144.048454]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     728<Apr/09 11:04 pm>[2979144.048456]  [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
     729<Apr/09 11:04 pm>[2979144.048458]  [<ffffffff81038588>] scheduler_tick+0x3d/0x236
     730<Apr/09 11:04 pm>[2979144.048461]  [<ffffffff810483d1>] update_process_times+0x49/0x59
     731<Apr/09 11:04 pm>[2979144.048463]  [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
     732<Apr/09 11:04 pm>[2979144.048465]  [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
     733<Apr/09 11:04 pm>[2979144.048466]  [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
     734<Apr/09 11:04 pm>[2979144.048469]  [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
     735<Apr/09 11:04 pm>[2979144.048471]  [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
     736<Apr/09 11:04 pm>[2979144.048473]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
     737
     738<Apr/09 11:04 pm>[2979144.048474]  <EOI>  [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
     739<Apr/09 11:04 pm>[2979144.048478]  [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
     740<Apr/09 11:04 pm>[2979144.048480]  [<ffffffff810d48fd>] ? get_swap_page+0x13c/0x5bb
     741<Apr/09 11:04 pm>[2979144.048482]  [<ffffffff810d17e9>] add_to_swap+0xf/0x48
     742<Apr/09 11:04 pm>[2979144.048484]  [<ffffffff810b7464>] shrink_page_list+0x214/0x68f
     743<Apr/09 11:04 pm>[2979144.048486]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     744<Apr/09 11:04 pm>[2979144.048489]  [<ffffffff810e5fb2>] ? mem_cgroup_del_lru_list+0x7b/0x8e
     745<Apr/09 11:04 pm>[2979144.048491]  [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
     746<Apr/09 11:04 pm>[2979144.048494]  [<ffffffff8144b300>] ? release_sock+0xdb/0xe4
     747<Apr/09 11:04 pm>[2979144.048496]  [<ffffffff8144b254>] ? release_sock+0x2f/0xe4
     748                 <Apr/09 11:04 pm>[2979144.048498]  [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
     749<Apr/09 11:04 pm>[2979144.048500]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
     750<Apr/09 11:04 pm>[2979144.048503]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
     751<Apr/09 11:04 pm>[2979144.048505]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
     752<Apr/09 11:04 pm>[2979144.048508]  [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
     753<Apr/09 11:04 pm>[2979144.048510]  [<ffffffff814fc30d>] svc_recv+0xe5/0x7f5
     754<Apr/09 11:04 pm>[2979144.048512]  [<ffffffff814fc1d9>] ? svc_send+0x85/0xa1
     755<Apr/09 11:04 pm>[2979144.048514]  [<ffffffff811f9bb1>] ? exp_readunlock+0x10/0x12
     756<Apr/09 11:04 pm>[2979144.048516]  [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
     757<Apr/09 11:04 pm>[2979144.048518]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
     758<Apr/09 11:04 pm>[2979144.048519]  [<ffffffff811f0d78>] nfsd+0xa9/0x139
     759<Apr/09 11:04 pm>[2979144.048521]  [<ffffffff810546e2>] kthread+0x7d/0x85
     760
     761<Apr/09 11:04 pm>[2979144.048524]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     762<Apr/09 11:04 pm>[2979144.048526]  [<ffffffff81054665>] ? kthread+0x0/0x85
     763<Apr/09 11:04 pm>[2979144.048528]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     764<Apr/09 11:04 pm>[2979147.680342]  [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
     765<Apr/09 11:04 pm>[2979147.687277]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
     766<Apr/09 11:04 pm>[2979147.693896]  <EOI>  [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
     767<Apr/09 11:04 pm>[2979147.700832]  [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
     768<Apr/09 11:04 pm>[2979147.707094]  [<ffffffff810d48fd>] ? get_swap_page+0x13c/0x5bb
     769<Apr/09 11:04 pm>[2979147.713439]  [<ffffffff810d17e9>] add_to_swap+0xf/0x48
     770<Apr/09 11:04 pm>[2979147.719149]  [<ffffffff810b7464>] shrink_page_list+0x214/0x68f
     771<Apr/09 11:04 pm>[2979147.725582]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     772<Apr/09 11:04 pm>[2979147.732030]  [<ffffffff810e5fb2>] ? mem_cgroup_del_lru_list+0x7b/0x8e
     773<Apr/09 11:04 pm>[2979147.739095]  [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
     774<Apr/09 11:04 pm>[2979147.745084]  [<ffffffff8144b300>] ? release_sock+0xdb/0xe4
     775<Apr/09 11:04 pm>[2979147.751158]  [<ffffffff8144b254>] ? release_sock+0x2f/0xe4
     776<Apr/09 11:04 pm>[2979147.757242]  [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
     777<Apr/09 11:04 pm>[2979147.764031]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
     778<Apr/09 11:04 pm>[2979147.770357]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
     779<Apr/09 11:04 pm>[2979147.777135]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
     780<Apr/09 11:04 pm>[2979147.784114]  [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
     781<Apr/09 11:04 pm>[2979147.790637]  [<ffffffff814fc30d>] svc_recv+0xe5/0x7f5
     782<Apr/09 11:04 pm>[2979147.796242]  [<ffffffff814fc1d9>] ? svc_send+0x85/0xa1
     783
     784<Apr/09 11:04 pm>[2979147.801919]  [<ffffffff811f9bb1>] ? exp_readunlock+0x10/0x12
     785<Apr/09 11:04 pm>[2979147.808143]  [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
     786<Apr/09 11:04 pm>[2979147.814827]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
     787<Apr/09 11:04 pm>[2979147.820144]  [<ffffffff811f0d78>] nfsd+0xa9/0x139
     788<Apr/09 11:04 pm>[2979147.825362]  [<ffffffff810546e2>] kthread+0x7d/0x85
     789<Apr/09 11:04 pm>[2979147.830788]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     790<Apr/09 11:04 pm>[2979147.837317]  [<ffffffff81054665>] ? kthread+0x0/0x85
     791<Apr/09 11:04 pm>[2979147.842844]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     792<Apr/09 11:04 pm>[2979147.849545] sending NMI to all CPUs:
     793<Apr/09 11:04 pm>[2979147.853637] NMI backtrace for cpu 5
     794<Apr/09 11:04 pm>[2979147.857632] CPU 5
     795<Apr/09 11:04 pm>[2979147.859760] Modules linked in: w83793 hwmon_vid autofs4 iTCO_wdt e1000e i2c_i801 i2c_core ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
     796<Apr/09 11:04 pm>[2979147.895719]
     797<Apr/09 11:04 pm>[2979147.897633] Pid: 8541, comm: nfsd Tainted: G      D    2.6.34.4 #3 S7010/empty
     798<Apr/09 11:04 pm>[2979147.905657] RIP: 0010:[<ffffffff812c08af>]  [<ffffffff812c08af>] delay_tsc+0x25/0x59
     799<Apr/09 11:04 pm>[2979147.914295] RSP: 0018:ffff880621edfb48  EFLAGS: 00000096
     800<Apr/09 11:04 pm>[2979147.920170] RAX: 0000000008477830 RBX: 000000000cac0b19 RCX: 0000000008477814
     801<Apr/09 11:04 pm>[2979147.928106] RDX: 00000000001c39a2 RSI: 0000000000000005 RDI: 0000000000000001
     802<Apr/09 11:04 pm>[2979147.936018] RBP: ffff880621edfb50 R08: 0000000000000005 R09: 0000000000000005
     803<Apr/09 11:04 pm>[2979147.943985] R10: 0000000000000000 R11: ffff880621edf980 R12: ffff880655213c00
     804<Apr/09 11:04 pm>[2979147.951912] R13: 0000000000000000 R14: ffff8806231bb090 R15: ffff8806231bb518
     805<Apr/09 11:04 pm>[2979147.959853] FS:  0000000000000000(0000) GS:ffff8800026a0000(0000) knlGS:0000000000000000
     806
     807<Apr/09 11:04 pm>[2979147.968784] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
     808<Apr/09 11:04 pm>[2979147.975104] CR2: 00007ff62c558028 CR3: 0000000001a8a000 CR4: 00000000000006e0
     809<Apr/09 11:04 pm>[2979147.983027] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     810<Apr/09 11:04 pm>[2979147.990983] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     811<Apr/09 11:04 pm>[2979147.998941] Process nfsd (pid: 8541, threadinfo ffff880621ede000, task ffff8806231bb090)
     812<Apr/09 11:04 pm>[2979148.007896] Stack:
     813<Apr/09 11:04 pm>[2979148.010354]  ffffffff81019da9 ffff880621edfb60 ffffffff812c08fe ffff880621edfba0
     814<Apr/09 11:04 pm>[2979148.018370] <0> ffffffff812c4e65 000000009ef30658 ffff880655213c00 ffff8806230b3090
     815<Apr/09 11:04 pm>[2979148.027115] <0> 0000000000013c00 ffff880621edfc10 0000000000000005 ffff880621edfbc0
     816<Apr/09 11:04 pm>[2979148.036220] Call Trace:
     817<Apr/09 11:04 pm>[2979148.039141]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     818<Apr/09 11:04 pm>[2979148.046998]  [<ffffffff812c08fe>] __delay+0xa/0xc
     819<Apr/09 11:04 pm>[2979148.052262]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     820<Apr/09 11:04 pm>[2979148.058577]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     821<Apr/09 11:04 pm>[2979148.064641]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
     822<Apr/09 11:04 pm>[2979148.070700]  [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
     823<Apr/09 11:04 pm>[2979148.076594]  [<ffffffff81036578>] try_to_wake_up+0x2e/0x342
     824<Apr/09 11:04 pm>[2979148.082729]  [<ffffffff81036899>] default_wake_function+0xd/0xf
     825<Apr/09 11:04 pm>[2979148.089240]  [<ffffffff8102c08d>] __wake_up_common+0x46/0x76
     826<Apr/09 11:04 pm>[2979148.099950]  [<ffffffff8102faa6>] __wake_up+0x38/0x4f
     827<Apr/09 11:04 pm>[2979148.105563]  [<ffffffff814fbb75>] svc_xprt_enqueue+0x1e2/0x23d
     828<Apr/09 11:04 pm>[2979148.111972]  [<ffffffff814fbc26>] svc_xprt_received+0x24/0x26
     829
     830<Apr/09 11:04 pm>[2979148.118290]  [<ffffffff814f1a12>] svc_udp_recvfrom+0x15f/0x2f9
     831<Apr/09 11:04 pm>[2979148.124703]  [<ffffffff814fc980>] svc_recv+0x758/0x7f5
     832<Apr/09 11:04 pm>[2979148.130404]  [<ffffffff814fc1d9>] ? svc_send+0x85/0xa1
     833<Apr/09 11:04 pm>[2979148.136109]  [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
     834<Apr/09 11:04 pm>[2979148.142802]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
     835<Apr/09 11:04 pm>[2979148.148162]  [<ffffffff811f0d78>] nfsd+0xa9/0x139
     836<Apr/09 11:04 pm>[2979148.153363]  [<ffffffff810546e2>] kthread+0x7d/0x85
     837<Apr/09 11:04 pm>[2979148.158798]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     838<Apr/09 11:04 pm>[2979148.165318]  [<ffffffff81054665>] ? kthread+0x0/0x85
     839<Apr/09 11:04 pm>[2979148.170844]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     840<Apr/09 11:04 pm>[2979148.177537] Code: fb 48 ff c8 c9 c3 55 48 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae e8 0f 31 89 c1 41 89 f1 66 66 90 0f ae e8 0f 31 <89> c6 48 89 f0 48 29 c8 48 39 f8 73 25 f3 90 65 44 8b 04 25 f0
     841<Apr/09 11:04 pm>[2979148.200631] Call Trace:
     842<Apr/09 11:04 pm>[2979148.203536]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     843<Apr/09 11:04 pm>[2979148.211349]  [<ffffffff812c08fe>] __delay+0xa/0xc
     844<Apr/09 11:04 pm>[2979148.216597]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     845<Apr/09 11:04 pm>[2979148.222919]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     846<Apr/09 11:04 pm>[2979148.229001]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
     847<Apr/09 11:04 pm>[2979148.235042]  [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
     848<Apr/09 11:04 pm>[2979148.240935]  [<ffffffff81036578>] try_to_wake_up+0x2e/0x342
     849<Apr/09 11:04 pm>[2979148.247081]  [<ffffffff81036899>] default_wake_function+0xd/0xf
     850<Apr/09 11:04 pm>[2979148.253582]  [<ffffffff8102c08d>] __wake_up_common+0x46/0x76
     851<Apr/09 11:04 pm>[2979148.259830]  [<ffffffff8102faa6>] __wake_up+0x38/0x4f
     852
     853<Apr/09 11:04 pm>[2979148.265444]  [<ffffffff814fbb75>] svc_xprt_enqueue+0x1e2/0x23d
     854<Apr/09 11:04 pm>[2979148.271886]  [<ffffffff814fbc26>] svc_xprt_received+0x24/0x26
     855<Apr/09 11:04 pm>[2979148.278230]  [<ffffffff814f1a12>] svc_udp_recvfrom+0x15f/0x2f9
     856<Apr/09 11:04 pm>[2979148.284636]  [<ffffffff814fc980>] svc_recv+0x758/0x7f5
     857<Apr/09 11:04 pm>[2979148.290337]  [<ffffffff814fc1d9>] ? svc_send+0x85/0xa1
     858<Apr/09 11:04 pm>[2979148.296031]  [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
     859<Apr/09 11:04 pm>[2979148.302732]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
     860<Apr/09 11:04 pm>[2979148.308085]  [<ffffffff811f0d78>] nfsd+0xa9/0x139
     861<Apr/09 11:04 pm>[2979148.313337]  [<ffffffff810546e2>] kthread+0x7d/0x85
     862<Apr/09 11:04 pm>[2979148.318764]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     863<Apr/09 11:04 pm>[2979148.325258]  [<ffffffff81054665>] ? kthread+0x0/0x85
     864<Apr/09 11:04 pm>[2979148.330793]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     865<Apr/09 11:04 pm>[2979148.337486] Pid: 8541, comm: nfsd Tainted: G      D    2.6.34.4 #3
     866<Apr/09 11:04 pm>[2979148.344257] Call Trace:
     867<Apr/09 11:04 pm>[2979148.347165]  <NMI>  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
     868<Apr/09 11:04 pm>[2979148.353661]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
     869<Apr/09 11:04 pm>[2979148.360082]  [<ffffffff81520360>] do_nmi+0xb2/0x259
     870<Apr/09 11:04 pm>[2979148.365531]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
     871<Apr/09 11:04 pm>[2979148.370600]  [<ffffffff812c08af>] ? delay_tsc+0x25/0x59
     872<Apr/09 11:04 pm>[2979148.376381]  <<EOE>>  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     873<Apr/09 11:04 pm>[2979148.385101]  [<ffffffff812c08fe>] __delay+0xa/0xc
     874<Apr/09 11:04 pm>[2979148.390353]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     875
     876<Apr/09 11:04 pm>[2979148.396692]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     877<Apr/09 11:04 pm>[2979148.402748]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
     878<Apr/09 11:04 pm>[2979148.408798]  [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
     879<Apr/09 11:04 pm>[2979148.414655]  [<ffffffff81036578>] try_to_wake_up+0x2e/0x342
     880<Apr/09 11:04 pm>[2979148.420802]  [<ffffffff81036899>] default_wake_function+0xd/0xf
     881<Apr/09 11:04 pm>[2979148.427329]  [<ffffffff8102c08d>] __wake_up_common+0x46/0x76
     882<Apr/09 11:04 pm>[2979148.433559]  [<ffffffff8102faa6>] __wake_up+0x38/0x4f
     883<Apr/09 11:04 pm>[2979148.439183]  [<ffffffff814fbb75>] svc_xprt_enqueue+0x1e2/0x23d
     884<Apr/09 11:04 pm>[2979148.445591]  [<ffffffff814fbc26>] svc_xprt_received+0x24/0x26
     885                 <Apr/09 11:04 pm>[2979148.451900]  [<ffffffff814f1a12>] svc_udp_recvfrom+0x15f/0x2f9
     886<Apr/09 11:04 pm>[2979148.458313]  [<ffffffff814fc980>] svc_recv+0x758/0x7f5
     887<Apr/09 11:04 pm>[2979148.464016]  [<ffffffff814fc1d9>] ? svc_send+0x85/0xa1
     888<Apr/09 11:04 pm>[2979148.469708]  [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
     889<Apr/09 11:04 pm>[2979148.476377]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
     890<Apr/09 11:04 pm>[2979148.481718]  [<ffffffff811f0d78>] nfsd+0xa9/0x139
     891<Apr/09 11:04 pm>[2979148.486954]  [<ffffffff810546e2>] kthread+0x7d/0x85
     892                 <Apr/09 11:04 pm>[2979148.492364]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     893<Apr/09 11:04 pm>[2979148.498867]  [<ffffffff81054665>] ? kthread+0x0/0x85
     894<Apr/09 11:04 pm>[2979148.504411]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     895<Apr/09 11:04 pm>[2979148.511087] NMI backtrace for cpu 7
     896<Apr/09 11:04 pm>[2979148.515091] CPU 7
     897<Apr/09 11:04 pm>[2979148.517218] Modules linked in: w83793 hwmon_vid autofs4 iTCO_wdt e1000e i2c_i801 i2c_core ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
     898
     899<Apr/09 11:04 pm>[2979148.553301]
     900<Apr/09 11:04 pm>[2979148.555221] Pid: 8574, comm: nfsd Tainted: G      D    2.6.34.4 #3 S7010/empty
     901<Apr/09 11:04 pm>[2979148.563221] RIP: 0010:[<ffffffff812c08af>]  [<ffffffff812c08af>] delay_tsc+0x25/0x59
     902<Apr/09 11:04 pm>[2979148.571833] RSP: 0018:ffff880655223dc0  EFLAGS: 00000046
     903<Apr/09 11:04 pm>[2979148.577716] RAX: 00000000084f9dbc RBX: 0000000000000001 RCX: 00000000084f9d44
     904<Apr/09 11:04 pm>[2979148.585644] RDX: 00000000001c39a2 RSI: 00000000084f9d90 RDI: 000000000028b0fc
     905<Apr/09 11:04 pm>[2979148.593549] RBP: ffff880655223dc8 R08: 0000000000000007 R09: 0000000000000007
     906<Apr/09 11:04 pm>[2979148.601479] R10: ffff8800000bd000 R11: 0000000000000086 R12: ffff880655233c00
     907<Apr/09 11:04 pm>[2979148.609373] R13: 0000000000000000 R14: ffff8806227d3090 R15: ffff8806227d3518
     908<Apr/09 11:04 pm>[2979148.617296] FS:  0000000000000000(0000) GS:ffff880655220000(0000) knlGS:0000000000000000
     909<Apr/09 11:04 pm>[2979148.626245] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
     910<Apr/09 11:04 pm>[2979148.632580] CR2: 00007f7053196000 CR3: 0000000001a8a000 CR4: 00000000000006e0
     911<Apr/09 11:04 pm>[2979148.640497] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     912<Apr/09 11:04 pm>[2979148.648436] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     913<Apr/09 11:04 pm>[2979148.656357] Process nfsd (pid: 8574, threadinfo ffff880623fec000, task ffff8806227d3090)
     914<Apr/09 11:04 pm>[2979148.665279] Stack:
     915                 <Apr/09 11:04 pm>[2979148.667745]  ffff8806227d3518 ffff880655223dd8 ffffffff812c093d ffff880655223df8
     916<Apr/09 11:04 pm>[2979148.675761] <0> ffffffff81019da9 000000009ef30658 000000009ef30658 ffff880655223e38
     917<Apr/09 11:04 pm>[2979148.684516] <0> ffffffff812c4ea4 000000009ef30658 ffff880655233c00 ffff880655233c00
     918<Apr/09 11:04 pm>[2979148.693662] Call Trace:
     919<Apr/09 11:04 pm>[2979148.696582]  <IRQ>
     920<Apr/09 11:04 pm>[2979148.699192]  [<ffffffff812c093d>] __const_udelay+0x3d/0x3f
     921
     922<Apr/09 11:04 pm>[2979148.705261]  [<ffffffff81019da9>] arch_trigger_all_cpu_backtrace+0x51/0x5d
     923<Apr/09 11:04 pm>[2979148.712763]  [<ffffffff812c4ea4>] do_raw_spin_lock+0x11d/0x147
     924<Apr/09 11:04 pm>[2979148.719200]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     925<Apr/09 11:04 pm>[2979148.725239]  [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
     926<Apr/09 11:04 pm>[2979148.731593]  [<ffffffff81038588>] scheduler_tick+0x3d/0x236
     927<Apr/09 11:04 pm>[2979148.737755]  [<ffffffff810483d1>] update_process_times+0x49/0x59
     928<Apr/09 11:04 pm>[2979148.744359]  [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
     929<Apr/09 11:04 pm>[2979148.750617]  [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
     930<Apr/09 11:04 pm>[2979148.756960]  [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
     931<Apr/09 11:04 pm>[2979148.763027]  [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
     932<Apr/09 11:04 pm>[2979148.769468]  [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
     933<Apr/09 11:04 pm>[2979148.776437]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
     934<Apr/09 11:04 pm>[2979148.783005]  <EOI>
     935<Apr/09 11:04 pm>[2979148.785627]  [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
     936<Apr/09 11:04 pm>[2979148.791880]  [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
     937<Apr/09 11:04 pm>[2979148.798119]  [<ffffffff810d48fd>] ? get_swap_page+0x13c/0x5bb
     938<Apr/09 11:04 pm>[2979148.804438]  [<ffffffff810d17e9>] add_to_swap+0xf/0x48
     939<Apr/09 11:04 pm>[2979148.810138]  [<ffffffff810b7464>] shrink_page_list+0x214/0x68f
     940<Apr/09 11:04 pm>[2979148.816520]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     941<Apr/09 11:04 pm>[2979148.822951]  [<ffffffff810e5fb2>] ? mem_cgroup_del_lru_list+0x7b/0x8e
     942<Apr/09 11:04 pm>[2979148.830017]  [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
     943<Apr/09 11:04 pm>[2979148.836003]  [<ffffffff8144b300>] ? release_sock+0xdb/0xe4
     944
     945<Apr/09 11:04 pm>[2979148.842061]  [<ffffffff8144b254>] ? release_sock+0x2f/0xe4
     946<Apr/09 11:04 pm>[2979148.848119]  [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
     947<Apr/09 11:04 pm>[2979148.854882]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
     948<Apr/09 11:04 pm>[2979148.861191]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
     949                 <Apr/09 11:04 pm>[2979148.867987]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
     950<Apr/09 11:04 pm>[2979148.874930]  [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
     951<Apr/09 11:04 pm>[2979148.881444]  [<ffffffff814fc30d>] svc_recv+0xe5/0x7f5
     952<Apr/09 11:04 pm>[2979148.887059]  [<ffffffff814fc1d9>] ? svc_send+0x85/0xa1
     953<Apr/09 11:04 pm>[2979148.892736]  [<ffffffff811f9bb1>] ? exp_readunlock+0x10/0x12
     954<Apr/09 11:04 pm>[2979148.898978]  [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
     955<Apr/09 11:04 pm>[2979148.905654]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
     956<Apr/09 11:04 pm>[2979148.910961]  [<ffffffff811f0d78>] nfsd+0xa9/0x139
     957<Apr/09 11:04 pm>[2979148.916214]  [<ffffffff810546e2>] kthread+0x7d/0x85
     958<Apr/09 11:04 pm>[2979148.921648]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     959<Apr/09 11:04 pm>[2979148.928169]  [<ffffffff81054665>] ? kthread+0x0/0x85
     960<Apr/09 11:04 pm>[2979148.933686]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     961<Apr/09 11:04 pm>[2979148.940337] Code: fb 48 ff c8 c9 c3 55 48 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae e8 0f 31 89 c1 41 89 f1 66 66 90 0f ae e8 0f 31 <89> c6 48 89 f0 48 29 c8 48 39 f8 73 25 f3 90 65 44 8b 04 25 f0
     962<Apr/09 11:04 pm>[2979148.963508] Call Trace:
     963<Apr/09 11:04 pm>[2979148.966436]  <IRQ>  [<ffffffff812c093d>] __const_udelay+0x3d/0x3f
     964<Apr/09 11:04 pm>[2979148.973108]  [<ffffffff81019da9>] arch_trigger_all_cpu_backtrace+0x51/0x5d
     965<Apr/09 11:04 pm>[2979148.980615]  [<ffffffff812c4ea4>] do_raw_spin_lock+0x11d/0x147
     966<Apr/09 11:04 pm>[2979148.987017]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     967
     968<Apr/09 11:04 pm>[2979148.993100]  [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
     969<Apr/09 11:04 pm>[2979148.999411]  [<ffffffff81038588>] scheduler_tick+0x3d/0x236
     970<Apr/09 11:04 pm>[2979149.005512]  [<ffffffff810483d1>] update_process_times+0x49/0x59
     971                 <Apr/09 11:04 pm>[2979149.012108]  [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
     972<Apr/09 11:04 pm>[2979149.018347]  [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
     973<Apr/09 11:04 pm>[2979149.024674]  [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
     974<Apr/09 11:04 pm>[2979149.030741]  [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
     975<Apr/09 11:04 pm>[2979149.037157]  [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
     976                 <Apr/09 11:04 pm>[2979149.044126]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
     977<Apr/09 11:04 pm>[2979149.050719]  <EOI>  [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
     978<Apr/09 11:04 pm>[2979149.057613]  [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
     979<Apr/09 11:04 pm>[2979149.063859]  [<ffffffff810d48fd>] ? get_swap_page+0x13c/0x5bb
     980<Apr/09 11:04 pm>[2979149.070203]  [<ffffffff810d17e9>] add_to_swap+0xf/0x48
     981<Apr/09 11:04 pm>[2979149.075938]  [<ffffffff810b7464>] shrink_page_list+0x214/0x68f
     982<Apr/09 11:04 pm>[2979149.082328]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     983<Apr/09 11:04 pm>[2979149.088733]  [<ffffffff810e5fb2>] ? mem_cgroup_del_lru_list+0x7b/0x8e
     984                 <Apr/09 11:04 pm>[2979149.095788]  [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
     985<Apr/09 11:04 pm>[2979149.101776]  [<ffffffff8144b300>] ? release_sock+0xdb/0xe4
     986                 <Apr/09 11:04 pm>[2979149.107851]  [<ffffffff8144b254>] ? release_sock+0x2f/0xe4
     987<Apr/09 11:04 pm>[2979149.113939]  [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
     988<Apr/09 11:04 pm>[2979149.125252]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
     989<Apr/09 11:04 pm>[2979149.131556]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
     990
     991<Apr/09 11:04 pm>[2979149.138352]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
     992<Apr/09 11:04 pm>[2979149.145314]  [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
     993<Apr/09 11:04 pm>[2979149.151819]  [<ffffffff814fc30d>] svc_recv+0xe5/0x7f5
     994<Apr/09 11:04 pm>[2979149.157451]  [<ffffffff814fc1d9>] ? svc_send+0x85/0xa1
     995<Apr/09 11:04 pm>[2979149.163144]  [<ffffffff811f9bb1>] ? exp_readunlock+0x10/0x12
     996<Apr/09 11:04 pm>[2979149.169395]  [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
     997<Apr/09 11:04 pm>[2979149.176097]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
     998<Apr/09 11:04 pm>[2979149.181438]  [<ffffffff811f0d78>] nfsd+0xa9/0x139
     999<Apr/09 11:04 pm>[2979149.186691]  [<ffffffff810546e2>] kthread+0x7d/0x85
     1000<Apr/09 11:04 pm>[2979149.192127]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     1001<Apr/09 11:04 pm>[2979149.198622]  [<ffffffff81054665>] ? kthread+0x0/0x85
     1002<Apr/09 11:04 pm>[2979149.204139]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     1003<Apr/09 11:04 pm>[2979149.210826] Pid: 8574, comm: nfsd Tainted: G      D    2.6.34.4 #3
     1004<Apr/09 11:04 pm>[2979149.217611] Call Trace:
     1005<Apr/09 11:04 pm>[2979149.220525]  <NMI>  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
     1006<Apr/09 11:04 pm>[2979149.226989]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
     1007<Apr/09 11:04 pm>[2979149.233394]  [<ffffffff81520360>] do_nmi+0xb2/0x259
     1008<Apr/09 11:04 pm>[2979149.238823]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
     1009                 <Apr/09 11:04 pm>[2979149.243893]  [<ffffffff812c08af>] ? delay_tsc+0x25/0x59
     1010<Apr/09 11:04 pm>[2979149.249666]  <<EOE>>  <IRQ>  [<ffffffff812c093d>] __const_udelay+0x3d/0x3f
     1011<Apr/09 11:04 pm>[2979149.257247]  [<ffffffff81019da9>] arch_trigger_all_cpu_backtrace+0x51/0x5d
     1012<Apr/09 11:04 pm>[2979149.264728]  [<ffffffff812c4ea4>] do_raw_spin_lock+0x11d/0x147
     1013
     1014<Apr/09 11:04 pm>[2979149.271138]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     1015<Apr/09 11:04 pm>[2979149.277168]  [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
     1016<Apr/09 11:04 pm>[2979149.283505]  [<ffffffff81038588>] scheduler_tick+0x3d/0x236
     1017<Apr/09 11:04 pm>[2979149.289675]  [<ffffffff810483d1>] update_process_times+0x49/0x59
     1018                 <Apr/09 11:04 pm>[2979149.296288]  [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
     1019<Apr/09 11:04 pm>[2979149.302528]  [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
     1020<Apr/09 11:04 pm>[2979149.308856]  [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
     1021<Apr/09 11:04 pm>[2979149.314923]  [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
     1022<Apr/09 11:04 pm>[2979149.321346]  [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
     1023<Apr/09 11:04 pm>[2979149.328315]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
     1024<Apr/09 11:04 pm>[2979149.334893]  <EOI>  [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
     1025<Apr/09 11:04 pm>[2979149.341803]  [<ffffffff810d47de>] ? get_swap_page+0x1d/0x5bb
     1026<Apr/09 11:04 pm>[2979149.348013]  [<ffffffff810d48fd>] ? get_swap_page+0x13c/0x5bb
     1027<Apr/09 11:04 pm>[2979149.354331]  [<ffffffff810d17e9>] add_to_swap+0xf/0x48
     1028<Apr/09 11:04 pm>[2979149.360040]  [<ffffffff810b7464>] shrink_page_list+0x214/0x68f
     1029<Apr/09 11:04 pm>[2979149.366482]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     1030<Apr/09 11:04 pm>[2979149.372906]  [<ffffffff810e5fb2>] ? mem_cgroup_del_lru_list+0x7b/0x8e
     1031<Apr/09 11:04 pm>[2979149.379947]  [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
     1032<Apr/09 11:04 pm>[2979149.385905]  [<ffffffff8144b300>] ? release_sock+0xdb/0xe4
     1033                 <Apr/09 11:04 pm>[2979149.391962]  [<ffffffff8144b254>] ? release_sock+0x2f/0xe4
     1034<Apr/09 11:04 pm>[2979149.398030]  [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
     1035<Apr/09 11:04 pm>[2979149.404819]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
     1036
     1037<Apr/09 11:04 pm>[2979149.411171]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
     1038<Apr/09 11:04 pm>[2979149.417975]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
     1039<Apr/09 11:04 pm>[2979149.424953]  [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
     1040<Apr/09 11:04 pm>[2979149.431460]  [<ffffffff814fc30d>] svc_recv+0xe5/0x7f5
     1041<Apr/09 11:04 pm>[2979149.437047]  [<ffffffff814fc1d9>] ? svc_send+0x85/0xa1
     1042<Apr/09 11:04 pm>[2979149.442733]  [<ffffffff811f9bb1>] ? exp_readunlock+0x10/0x12
     1043<Apr/09 11:04 pm>[2979149.448967]  [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
     1044<Apr/09 11:04 pm>[2979149.455649]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
     1045<Apr/09 11:04 pm>[2979149.460958]  [<ffffffff811f0d78>] nfsd+0xa9/0x139
     1046<Apr/09 11:04 pm>[2979149.466201]  [<ffffffff810546e2>] kthread+0x7d/0x85
     1047<Apr/09 11:04 pm>[2979149.471636]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     1048<Apr/09 11:04 pm>[2979149.478123]  [<ffffffff81054665>] ? kthread+0x0/0x85
     1049<Apr/09 11:04 pm>[2979149.483657]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     1050<Apr/09 11:04 pm>[2979149.490283] NMI backtrace for cpu 6
     1051<Apr/09 11:04 pm>[2979149.494243] CPU 6
     1052<Apr/09 11:04 pm>[2979149.496362] Modules linked in: w83793 hwmon_vid autofs4 iTCO_wdt e1000e i2c_i801 i2c_core ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
     1053<Apr/09 11:04 pm>[2979149.532444]
     1054<Apr/09 11:04 pm>[2979149.534381] Pid: 8590, comm: nfsd Tainted: G      D    2.6.34.4 #3 S7010/empty
     1055<Apr/09 11:04 pm>[2979149.542400] RIP: 0010:[<ffffffff812c08af>]  [<ffffffff812c08af>] delay_tsc+0x25/0x59
     1056<Apr/09 11:04 pm>[2979149.551003] RSP: 0018:ffff880655203de0  EFLAGS: 00000082
     1057<Apr/09 11:04 pm>[2979149.556859] RAX: 00000000084b8b04 RBX: 0000000008a39bf5 RCX: 00000000084b8ae8
     1058<Apr/09 11:04 pm>[2979149.564795] RDX: 00000000001c39a2 RSI: 0000000000000006 RDI: 0000000000000001
     1059
     1060<Apr/09 11:04 pm>[2979149.572752] RBP: ffff880655203de8 R08: 0000000000000006 R09: 0000000000000006
     1061<Apr/09 11:04 pm>[2979149.580707] R10: 000000000000000b R11: ffff880655203c18 R12: ffff880655213c00
     1062<Apr/09 11:04 pm>[2979149.588637] R13: 0000000000000000 R14: ffff8806227ed0f0 R15: ffff8806227ed578
     1063<Apr/09 11:04 pm>[2979149.596585] FS:  0000000000000000(0000) GS:ffff880655200000(0000) knlGS:0000000000000000
     1064<Apr/09 11:04 pm>[2979149.605481] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
     1065<Apr/09 11:04 pm>[2979149.611792] CR2: 0000000389ca1c18 CR3: 0000000001a8a000 CR4: 00000000000006e0
     1066<Apr/09 11:04 pm>[2979149.619718] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     1067<Apr/09 11:04 pm>[2979149.627674] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     1068<Apr/09 11:04 pm>[2979149.635588] Process nfsd (pid: 8590, threadinfo ffff880622754000, task ffff8806227ed0f0)
     1069<Apr/09 11:04 pm>[2979149.644526] Stack:
     1070<Apr/09 11:04 pm>[2979149.646993]  ffffffff81019da9 ffff880655203df8 ffffffff812c08fe ffff880655203e38
     1071<Apr/09 11:04 pm>[2979149.654991] <0> ffffffff812c4e65 000000009ef30658 ffff880655213c00 ffff880655213c00
     1072<Apr/09 11:04 pm>[2979149.663735] <0> 0000000000000006 ffff88065520eae0 ffff88065520ea70 ffff880655203e58
     1073<Apr/09 11:04 pm>[2979149.672874] Call Trace:
     1074                 <Apr/09 11:04 pm>[2979149.675795]  <IRQ>
     1075<Apr/09 11:04 pm>[2979149.678395]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     1076<Apr/09 11:04 pm>[2979149.686254]  [<ffffffff812c08fe>] __delay+0xa/0xc
     1077<Apr/09 11:04 pm>[2979149.691489]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     1078<Apr/09 11:04 pm>[2979149.697804]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     1079<Apr/09 11:04 pm>[2979149.703878]  [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
     1080<Apr/09 11:04 pm>[2979149.710188]  [<ffffffff81038588>] scheduler_tick+0x3d/0x236
     1081<Apr/09 11:04 pm>[2979149.716325]  [<ffffffff810483d1>] update_process_times+0x49/0x59
     1082
     1083<Apr/09 11:04 pm>[2979149.722938]  [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
     1084<Apr/09 11:04 pm>[2979149.729187]  [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
     1085<Apr/09 11:04 pm>[2979149.735514]  [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
     1086<Apr/09 11:04 pm>[2979149.741580]  [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
     1087<Apr/09 11:04 pm>[2979149.748012]  [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
     1088<Apr/09 11:04 pm>[2979149.754973]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
     1089<Apr/09 11:04 pm>[2979149.761558]  <EOI>
     1090<Apr/09 11:04 pm>[2979149.763773] Code: fb 48 ff c8 c9 c3 55 48 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae e8 0f 31 89 c1 41 89 f1 66 66 90 0f ae e8 0f 31 <89> c6 48 89 f0 48 29 c8 48 39 f8 73 25 f3 90 65 44 8b 04 25 f0
     1091<Apr/09 11:04 pm>[2979149.787287] Call Trace:
     1092<Apr/09 11:04 pm>[2979149.790216]  <IRQ>  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     1093<Apr/09 11:04 pm>[2979149.798755]  [<ffffffff812c08fe>] __delay+0xa/0xc
     1094<Apr/09 11:04 pm>[2979149.803987]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     1095<Apr/09 11:04 pm>[2979149.810309]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     1096<Apr/09 11:04 pm>[2979149.816367]  [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
     1097<Apr/09 11:04 pm>[2979149.822705]  [<ffffffff81038588>] scheduler_tick+0x3d/0x236
     1098<Apr/09 11:04 pm>[2979149.828848]  [<ffffffff810483d1>] update_process_times+0x49/0x59
     1099<Apr/09 11:04 pm>[2979149.835436]  [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
     1100<Apr/09 11:04 pm>[2979149.841665]  [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
     1101<Apr/09 11:04 pm>[2979149.847959]  [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
     1102<Apr/09 11:04 pm>[2979149.853990]  [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
     1103<Apr/09 11:04 pm>[2979149.860431]  [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
     1104<Apr/09 11:04 pm>[2979149.867392]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
     1105
     1106<Apr/09 11:04 pm>[2979149.873986]  <EOI> Pid: 8590, comm: nfsd Tainted: G      D    2.6.34.4 #3
     1107<Apr/09 11:04 pm>[2979149.881469] Call Trace:
     1108<Apr/09 11:04 pm>[2979149.884400]  <NMI>  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
     1109<Apr/09 11:04 pm>[2979149.890881]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
     1110<Apr/09 11:04 pm>[2979149.897310]  [<ffffffff81520360>] do_nmi+0xb2/0x259
     1111<Apr/09 11:04 pm>[2979149.902740]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
     1112<Apr/09 11:04 pm>[2979149.907809]  [<ffffffff812c08af>] ? delay_tsc+0x25/0x59
     1113<Apr/09 11:04 pm>[2979149.913610]  <<EOE>>  <IRQ>  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     1114<Apr/09 11:04 pm>[2979149.922945]  [<ffffffff812c08fe>] __delay+0xa/0xc
     1115<Apr/09 11:04 pm>[2979149.928194]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     1116<Apr/09 11:04 pm>[2979149.934534]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     1117<Apr/09 11:04 pm>[2979149.940556]  [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
     1118<Apr/09 11:04 pm>[2979149.946884]  [<ffffffff81038588>] scheduler_tick+0x3d/0x236
     1119<Apr/09 11:04 pm>[2979149.953028]  [<ffffffff810483d1>] update_process_times+0x49/0x59
     1120<Apr/09 11:04 pm>[2979149.959618]  [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
     1121<Apr/09 11:04 pm>[2979149.965830]  [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
     1122<Apr/09 11:04 pm>[2979149.972157]  [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
     1123<Apr/09 11:04 pm>[2979149.978214]  [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
     1124<Apr/09 11:04 pm>[2979149.984638]  [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
     1125<Apr/09 11:04 pm>[2979149.991582]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
     1126
     1127}}}