IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Initial Version and Version 1 of ipp054-2012-01-02-crash


Ignore:
Timestamp:
Feb 1, 2012, 12:12:45 PM (15 years ago)
Author:
Mark Huber
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • ipp054-2012-01-02-crash

    v1 v1  
     1File content:
     21973]  [<ffffffff8151cc0e>] schedule+0x25a/0x86a
     3<Feb/01 11:56 am>[4661030.797691]  [<ffffffff8151d264>] io_schedule+0x46/0x5e
     4<Feb/01 11:56 am>[4661030.803489]  [<ffffffff810ac90d>] sync_page+0x44/0x48
     5<Feb/01 11:56 am>[4661030.809095]  [<ffffffff8151d619>] __wait_on_bit_lock+0x45/0x8c
     6<Feb/01 11:56 am>[4661030.815502]  [<ffffffff810ac8c9>] ? sync_page+0x0/0x48
     7<Feb/01 11:56 am>[4661030.821203]  [<ffffffff810ac8b5>] __lock_page+0x63/0x6a
     8<Feb/01 11:56 am>[4661030.826994]  [<ffffffff81054ba4>] ? wake_bit_function+0x0/0x2a
     9<Feb/01 11:56 am>[4661030.833408]  [<ffffffff810c4f57>] handle_mm_fault+0x4c1/0x73e
     10<Feb/01 11:56 am>[4661030.839753]  [<ffffffff815227e9>] do_page_fault+0x351/0x375
     11<Feb/01 11:56 am>[4661030.845905]  [<ffffffff810348d8>] ? finish_task_switch+0x69/0xdd
     12<Feb/01 11:56 am>[4661030.852511]  [<ffffffff8103486f>] ? finish_task_switch+0x0/0xdd
     13<Feb/01 11:56 am>[4661030.859030]  [<ffffffff8151d152>] ? schedule+0x79e/0x86a
     14<Feb/01 11:56 am>[4661030.864912]  [<ffffffff8151ef6f>] ? lockdep_sys_exit_thunk+0x35/0x67
     15<Feb/01 11:56 am>[4661030.871899]  [<ffffffff81057962>] ? __run_hrtimer+0xff/0x12d
     16<Feb/01 11:56 am>[4661030.878156]  [<ffffffff8151fb5f>] page_fault+0x1f/0x30
     17<Feb/01 11:56 am>[4661030.883865] NMI backtrace for cpu 4
     18<Feb/01 11:56 am>[4661030.887865] CPU 4
     19<Feb/01 11:56 am>[4661030.889990] Modules linked in: w83793 hwmon_vid autofs4 i2c_i801 i2c_core iTCO_wdt e1000e ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
     20<Feb/01 11:56 am>[4661030.925976]
     21<Feb/01 11:56 am>[4661030.927899] Pid: 5693, comm: ppStack Tainted: G      D    2.6.34.4 #3 S7010/empty
     22<Feb/01 11:56 am>[4661030.936169] RIP: 0010:[<ffffffff812c08a2>]  [<ffffffff812c08a2>] delay_tsc+0x18/0x59
     23
     24<Feb/01 11:56 am>[4661030.944787] RSP: 0000:ffff8803edf0daf0  EFLAGS: 00000086
     25<Feb/01 11:56 am>[4661030.950670] RAX: 00000000e48ab078 RBX: 00000000244f7331 RCX: 00000000e48ab034
     26<Feb/01 11:56 am>[4661030.958631] RDX: 00000000002c28af RSI: 0000000000000004 RDI: 0000000000000001
     27<Feb/01 11:56 am>[4661030.966579] RBP: ffff8803edf0daf8 R08: 0000000000000004 R09: 0000000000000004
     28<Feb/01 11:56 am>[4661030.974518] R10: 0000000000000000 R11: ffff8803edf0d928 R12: ffff880623308460
     29<Feb/01 11:56 am>[4661030.982432] R13: 0000000000000000 R14: ffff88052a510000 R15: ffff88052a510488
     30<Feb/01 11:56 am>[4661030.990390] FS:  0000000041f46950(0063) GS:ffff880002680000(0000) knlGS:0000000000000000
     31<Feb/01 11:56 am>[4661030.999319] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
     32<Feb/01 11:56 am>[4661031.005664] CR2: 00007ff195fad000 CR3: 000000043b231000 CR4: 00000000000006e0
     33<Feb/01 11:56 am>[4661031.013615] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     34<Feb/01 11:56 am>[4661031.021555] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     35<Feb/01 11:56 am>[4661031.029501] Process ppStack (pid: 5693, threadinfo ffff8803edf0c000, task ffff88052a510000)
     36<Feb/01 11:56 am>[4661031.038711] Stack:
     37                 <Feb/01 11:56 am>[4661031.041185]  ffffffff81019da9 ffff8803edf0db08 ffffffff812c08fe ffff8803edf0db48
     38<Feb/01 11:56 am>[4661031.049157] <0> ffffffff812c4e65 000000009ef2dfac ffff880623308460 ffff880623308000
     39<Feb/01 11:56 am>[4661031.057867] <0> ffff88062341c300 ffff880623308000 0000000000000000 ffff8803edf0db68
     40<Feb/01 11:56 am>[4661031.071469] Call Trace:
     41<Feb/01 11:56 am>[4661031.074382]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     42<Feb/01 11:56 am>[4661031.082206]  [<ffffffff812c08fe>] __delay+0xa/0xc
     43<Feb/01 11:56 am>[4661031.087459]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     44<Feb/01 11:56 am>[4661031.093791]  [<ffffffff8151f115>] _raw_spin_lock_irq+0x35/0x39
     45<Feb/01 11:56 am>[4661031.100195]  [<ffffffff812aacd7>] ? __make_request+0x5e/0x445
     46
     47<Feb/01 11:56 am>[4661031.106522]  [<ffffffff812aacd7>] __make_request+0x5e/0x445
     48<Feb/01 11:56 am>[4661031.112684]  [<ffffffff812a946b>] generic_make_request+0x32a/0x39c
     49<Feb/01 11:56 am>[4661031.119464]  [<ffffffff811100ac>] ? bio_alloc_bioset+0x48/0xc0
     50<Feb/01 11:56 am>[4661031.125893]  [<ffffffff812a9591>] submit_bio+0xb4/0xbd
     51<Feb/01 11:56 am>[4661031.131610]  [<ffffffff810d127d>] ? get_swap_bio+0x31/0x77
     52<Feb/01 11:56 am>[4661031.137670]  [<ffffffff810d130a>] swap_readpage+0x47/0x4d
     53<Feb/01 11:56 am>[4661031.143642]  [<ffffffff810d161b>] read_swap_cache_async+0xb1/0xf2
     54<Feb/01 11:56 am>[4661031.150334]  [<ffffffff810d216d>] ? valid_swaphandles+0x6b/0xfd
     55<Feb/01 11:56 am>[4661031.156851]  [<ffffffff810d16b4>] swapin_readahead+0x58/0x97
     56<Feb/01 11:56 am>[4661031.163108]  [<ffffffff810c4ef1>] handle_mm_fault+0x45b/0x73e
     57<Feb/01 11:56 am>[4661031.169444]  [<ffffffff815227e9>] do_page_fault+0x351/0x375
     58<Feb/01 11:56 am>[4661031.175580]  [<ffffffff810348d8>] ? finish_task_switch+0x69/0xdd
     59<Feb/01 11:56 am>[4661031.182186]  [<ffffffff8103486f>] ? finish_task_switch+0x0/0xdd
     60<Feb/01 11:56 am>[4661031.188711]  [<ffffffff8151d152>] ? schedule+0x79e/0x86a
     61<Feb/01 11:56 am>[4661031.194604]  [<ffffffff8151ef6f>] ? lockdep_sys_exit_thunk+0x35/0x67
     62                 <Feb/01 11:56 am>[4661031.201582]  [<ffffffff81057962>] ? __run_hrtimer+0xff/0x12d
     63<Feb/01 11:56 am>[4661031.207811]  [<ffffffff8151fb5f>] page_fault+0x1f/0x30
     64                 <Feb/01 11:56 am>[4661031.213513] Code: 2e 0f 1f 84 00 00 00 00 00 48 ff c8 75 fb 48 ff c8 c9 c3 55 48 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae e8 0f 31 <89> c1 41 89 f1 66 66 90 0f ae e8 0f 31 89 c6 48 89 f0 48 29 c8
     65<Feb/01 11:56 am>[4661031.236617] Call Trace:
     66<Feb/01 11:56 am>[4661031.239532]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     67<Feb/01 11:56 am>[4661031.247371]  [<ffffffff812c08fe>] __delay+0xa/0xc
     68<Feb/01 11:56 am>[4661031.252625]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     69
     70<Feb/01 11:56 am>[4661031.258973]  [<ffffffff8151f115>] _raw_spin_lock_irq+0x35/0x39
     71<Feb/01 11:56 am>[4661031.265413]  [<ffffffff812aacd7>] ? __make_request+0x5e/0x445
     72<Feb/01 11:56 am>[4661031.271756]  [<ffffffff812aacd7>] __make_request+0x5e/0x445
     73<Feb/01 11:56 am>[4661031.277919]  [<ffffffff812a946b>] generic_make_request+0x32a/0x39c
     74<Feb/01 11:56 am>[4661031.284699]  [<ffffffff811100ac>] ? bio_alloc_bioset+0x48/0xc0
     75<Feb/01 11:56 am>[4661031.291128]  [<ffffffff812a9591>] submit_bio+0xb4/0xbd
     76<Feb/01 11:56 am>[4661031.296854]  [<ffffffff810d127d>] ? get_swap_bio+0x31/0x77
     77<Feb/01 11:56 am>[4661031.302924]  [<ffffffff810d130a>] swap_readpage+0x47/0x4d
     78<Feb/01 11:56 am>[4661031.308920]  [<ffffffff810d161b>] read_swap_cache_async+0xb1/0xf2
     79<Feb/01 11:56 am>[4661031.315608]  [<ffffffff810d216d>] ? valid_swaphandles+0x6b/0xfd
     80<Feb/01 11:56 am>[4661031.322123]  [<ffffffff810d16b4>] swapin_readahead+0x58/0x97
     81<Feb/01 11:56 am>[4661031.328361]  [<ffffffff810c4ef1>] handle_mm_fault+0x45b/0x73e
     82<Feb/01 11:56 am>[4661031.334707]  [<ffffffff815227e9>] do_page_fault+0x351/0x375
     83<Feb/01 11:56 am>[4661031.340860]  [<ffffffff810348d8>] ? finish_task_switch+0x69/0xdd
     84<Feb/01 11:56 am>[4661031.347457]  [<ffffffff8103486f>] ? finish_task_switch+0x0/0xdd
     85<Feb/01 11:56 am>[4661031.353948]  [<ffffffff8151d152>] ? schedule+0x79e/0x86a
     86<Feb/01 11:56 am>[4661031.359822]  [<ffffffff8151ef6f>] ? lockdep_sys_exit_thunk+0x35/0x67
     87<Feb/01 11:56 am>[4661031.366803]  [<ffffffff81057962>] ? __run_hrtimer+0xff/0x12d
     88<Feb/01 11:56 am>[4661031.373031]  [<ffffffff8151fb5f>] page_fault+0x1f/0x30
     89<Feb/01 11:56 am>[4661031.378715] Pid: 5693, comm: ppStack Tainted: G      D    2.6.34.4 #3
     90<Feb/01 11:56 am>[4661031.385738] Call Trace:
     91<Feb/01 11:56 am>[4661031.388675]  <NMI>  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
     92
     93<Feb/01 11:56 am>[4661031.395125]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
     94<Feb/01 11:56 am>[4661031.401562]  [<ffffffff81520360>] do_nmi+0xb2/0x259
     95<Feb/01 11:56 am>[4661031.406967]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
     96<Feb/01 11:56 am>[4661031.412063]  [<ffffffff812c08a2>] ? delay_tsc+0x18/0x59
     97<Feb/01 11:56 am>[4661031.417845]  <<EOE>>  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     98<Feb/01 11:56 am>[4661031.426583]  [<ffffffff812c08fe>] __delay+0xa/0xc
     99<Feb/01 11:56 am>[4661031.431834]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     100<Feb/01 11:56 am>[4661031.438156]  [<ffffffff8151f115>] _raw_spin_lock_irq+0x35/0x39
     101<Feb/01 11:56 am>[4661031.444587]  [<ffffffff812aacd7>] ? __make_request+0x5e/0x445
     102<Feb/01 11:56 am>[4661031.450931]  [<ffffffff812aacd7>] __make_request+0x5e/0x445
     103<Feb/01 11:56 am>[4661031.457066]  [<ffffffff812a946b>] generic_make_request+0x32a/0x39c
     104<Feb/01 11:56 am>[4661031.463854]  [<ffffffff811100ac>] ? bio_alloc_bioset+0x48/0xc0
     105<Feb/01 11:56 am>[4661031.470241]  [<ffffffff812a9591>] submit_bio+0xb4/0xbd
     106<Feb/01 11:56 am>[4661031.475941]  [<ffffffff810d127d>] ? get_swap_bio+0x31/0x77
     107<Feb/01 11:56 am>[4661031.482010]  [<ffffffff810d130a>] swap_readpage+0x47/0x4d
     108<Feb/01 11:56 am>[4661031.487998]  [<ffffffff810d161b>] read_swap_cache_async+0xb1/0xf2
     109<Feb/01 11:56 am>[4661031.494682]  [<ffffffff810d216d>] ? valid_swaphandles+0x6b/0xfd
     110<Feb/01 11:56 am>[4661031.501190]  [<ffffffff810d16b4>] swapin_readahead+0x58/0x97
     111<Feb/01 11:56 am>[4661031.507430]  [<ffffffff810c4ef1>] handle_mm_fault+0x45b/0x73e
     112<Feb/01 11:56 am>[4661031.513758]  [<ffffffff815227e9>] do_page_fault+0x351/0x375
     113<Feb/01 11:56 am>[4661031.519919]  [<ffffffff810348d8>] ? finish_task_switch+0x69/0xdd
     114<Feb/01 11:56 am>[4661031.526499]  [<ffffffff8103486f>] ? finish_task_switch+0x0/0xdd
     115
     116<Feb/01 11:56 am>[4661031.533033]  [<ffffffff8151d152>] ? schedule+0x79e/0x86a
     117<Feb/01 11:56 am>[4661031.538925]  [<ffffffff8151ef6f>] ? lockdep_sys_exit_thunk+0x35/0x67
     118<Feb/01 11:56 am>[4661031.545905]  [<ffffffff81057962>] ? __run_hrtimer+0xff/0x12d
     119<Feb/01 11:56 am>[4661031.552160]  [<ffffffff8151fb5f>] page_fault+0x1f/0x30
     120<Feb/01 11:56 am>[4661031.557860] NMI backtrace for cpu 8
     121<Feb/01 11:56 am>[4661031.561842] CPU 8
     122<Feb/01 11:56 am>[4661031.563960] Modules linked in: w83793 hwmon_vid autofs4 i2c_i801 i2c_core iTCO_wdt e1000e ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
     123<Feb/01 11:56 am>[4661031.600007]
     124<Feb/01 11:56 am>[4661031.601938] Pid: 8506, comm: nfsd Tainted: G      D    2.6.34.4 #3 S7010/empty
     125<Feb/01 11:56 am>[4661031.609983] RIP: 0010:[<ffffffff812c088a>]  [<ffffffff812c088a>] delay_tsc+0x0/0x59
     126<Feb/01 11:56 am>[4661031.618506] RSP: 0018:ffff88061b7633e8  EFLAGS: 00000002
     127<Feb/01 11:56 am>[4661031.624381] RAX: 0000000000000000 RBX: 0000000024539cc2 RCX: 00000000e49afc28
     128<Feb/01 11:56 am>[4661031.632333] RDX: 0000000000005600 RSI: 00000000e49afc44 RDI: 0000000000000001
     129<Feb/01 11:56 am>[4661031.640291] RBP: ffff88061b7633f0 R08: 0000000000000008 R09: 0000000000000008
     130<Feb/01 11:56 am>[4661031.648256] R10: 0000000000000000 R11: ffff88061b763210 R12: ffff880002673c00
     131<Feb/01 11:56 am>[4661031.656211] R13: 0000000000000000 R14: ffff88061b768000 R15: ffff88061b768488
     132<Feb/01 11:56 am>[4661031.664159] FS:  0000000000000000(0000) GS:ffff880655240000(0000) knlGS:0000000000000000
     133<Feb/01 11:56 am>[4661031.673105] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
     134<Feb/01 11:56 am>[4661031.679434] CR2: 00007ff195e94068 CR3: 0000000001a8a000 CR4: 00000000000006e0
     135<Feb/01 11:56 am>[4661031.687367] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     136<Feb/01 11:56 am>[4661031.695299] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     137<Feb/01 11:56 am>[4661031.703241] Process nfsd (pid: 8506, threadinfo ffff88061b762000, task ffff88061b768000)
     138
     139<Feb/01 11:56 am>[4661031.712125] Stack:
     140<Feb/01 11:56 am>[4661031.714599]  ffffffff812c08fe ffff88061b763430 ffffffff812c4e65 000000009ef2dfac
     141<Feb/01 11:56 am>[4661031.722625] <0> ffff880002673c00 ffff880623bb2060 0000000000013c00 ffff88061b7634a0
     142<Feb/01 11:56 am>[4661031.731370] <0> 0000000000000008 ffff88061b763450 ffffffff8151f193 ffffffff8102fb6b
     143<Feb/01 11:56 am>[4661031.740517] Call Trace:
     144<Feb/01 11:56 am>[4661031.743436]  [<ffffffff812c08fe>] ? __delay+0xa/0xc
     145<Feb/01 11:56 am>[4661031.748855]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     146<Feb/01 11:56 am>[4661031.755203]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     147<Feb/01 11:56 am>[4661031.761285]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
     148<Feb/01 11:56 am>[4661031.767335]  [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
     149<Feb/01 11:56 am>[4661031.773236]  [<ffffffff81036578>] try_to_wake_up+0x2e/0x342
     150<Feb/01 11:56 am>[4661031.779391]  [<ffffffff810368b8>] wake_up_process+0x10/0x12
     151                 <Feb/01 11:56 am>[4661031.785562]  [<ffffffff8110649a>] bdi_queue_work+0xa2/0xb9
     152<Feb/01 11:56 am>[4661031.791637]  [<ffffffff811068ae>] bdi_alloc_queue_work+0xa4/0xc4
     153<Feb/01 11:56 am>[4661031.798227]  [<ffffffff81106a1f>] wakeup_flusher_threads+0x93/0xc6
     154<Feb/01 11:56 am>[4661031.805029]  [<ffffffff811069bf>] ? wakeup_flusher_threads+0x33/0xc6
     155<Feb/01 11:56 am>[4661031.811991]  [<ffffffff810b9854>] do_try_to_free_pages+0x299/0x37a
     156<Feb/01 11:56 am>[4661031.818760]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
     157<Feb/01 11:56 am>[4661031.825076]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
     158<Feb/01 11:56 am>[4661031.831863]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
     159<Feb/01 11:56 am>[4661031.838823]  [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
     160<Feb/01 11:56 am>[4661031.845330]  [<ffffffff810acb60>] __page_cache_alloc+0x59/0x5e
     161
     162<Feb/01 11:56 am>[4661031.851752]  [<ffffffff810b4860>] __do_page_cache_readahead+0xe5/0x1fc
     163<Feb/01 11:56 am>[4661031.858897]  [<ffffffff810b47f6>] ? __do_page_cache_readahead+0x7b/0x1fc
     164<Feb/01 11:56 am>[4661031.866230]  [<ffffffff810b4993>] ra_submit+0x1c/0x20
     165<Feb/01 11:56 am>[4661031.871841]  [<ffffffff810b4cb7>] ondemand_readahead+0x243/0x256
     166<Feb/01 11:56 am>[4661031.878466]  [<ffffffff810ac48a>] ? find_get_pages_contig+0x0/0x118
     167<Feb/01 11:56 am>[4661031.885364]  [<ffffffff810b4daa>] page_cache_sync_readahead+0x38/0x3a
     168<Feb/01 11:56 am>[4661031.892430]  [<ffffffff81109bf2>] __generic_file_splice_read+0xfa/0x41c
     169<Feb/01 11:56 am>[4661031.899649]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     170<Feb/01 11:56 am>[4661031.906078]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     171<Feb/01 11:56 am>[4661031.912483]  [<ffffffff812615b1>] ? xfs_ilock+0x49/0xb2
     172<Feb/01 11:56 am>[4661031.918287]  [<ffffffff811086db>] ? spd_release_page+0x0/0x14
     173<Feb/01 11:56 am>[4661031.924624]  [<ffffffff81109f57>] generic_file_splice_read+0x43/0x74
     174<Feb/01 11:56 am>[4661031.931561]  [<ffffffff8128191a>] xfs_file_splice_read+0xde/0x125
     175<Feb/01 11:56 am>[4661031.938241]  [<ffffffff8110875d>] vfs_splice_to+0x6e/0x81
     176<Feb/01 11:56 am>[4661031.944219]  [<ffffffff81108837>] splice_direct_to_actor+0xc7/0x19c
     177<Feb/01 11:56 am>[4661031.951077]  [<ffffffff811f562c>] ? nfsd_direct_splice_actor+0x0/0x12
     178<Feb/01 11:56 am>[4661031.958107]  [<ffffffff811f54ef>] nfsd_vfs_read+0x263/0x3a0
     179<Feb/01 11:56 am>[4661031.964282]  [<ffffffff81281760>] ? xfs_file_open+0x0/0x3f
     180<Feb/01 11:56 am>[4661031.970358]  [<ffffffff811f5c4e>] nfsd_read+0xa7/0xc3
     181<Feb/01 11:56 am>[4661031.975975]  [<ffffffff811fcd79>] nfsd3_proc_read+0xfe/0x142
     182<Feb/01 11:56 am>[4661031.982217]  [<ffffffff811f0854>] nfsd_dispatch+0xe9/0x1ca
     183<Feb/01 11:56 am>[4661031.988274]  [<ffffffff814f04c5>] svc_process+0x425/0x72a
     184
     185<Feb/01 11:56 am>[4661031.994270]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
     186<Feb/01 11:56 am>[4661031.999580]  [<ffffffff811f0dc2>] nfsd+0xf3/0x139
     187<Feb/01 11:56 am>[4661032.004841]  [<ffffffff810546e2>] kthread+0x7d/0x85
     188<Feb/01 11:56 am>[4661032.010277]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     189<Feb/01 11:56 am>[4661032.016799]  [<ffffffff81054665>] ? kthread+0x0/0x85
     190<Feb/01 11:56 am>[4661032.022315]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     191<Feb/01 11:56 am>[4661032.029000] Code: 90 55 48 89 f8 48 89 e5 48 85 c0 74 19 eb 02 66 90 eb 0e 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 48 ff c8 75 fb 48 ff c8 c9 c3 <55> 48 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae
     192<Feb/01 11:56 am>[4661032.052181] Call Trace:
     193<Feb/01 11:56 am>[4661032.055094]  [<ffffffff812c08fe>] ? __delay+0xa/0xc
     194<Feb/01 11:56 am>[4661032.060520]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     195<Feb/01 11:56 am>[4661032.066840]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     196<Feb/01 11:56 am>[4661032.072924]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
     197<Feb/01 11:56 am>[4661032.078966]  [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
     198<Feb/01 11:56 am>[4661032.084822]  [<ffffffff81036578>] try_to_wake_up+0x2e/0x342
     199<Feb/01 11:56 am>[4661032.090977]  [<ffffffff810368b8>] wake_up_process+0x10/0x12
     200<Feb/01 11:56 am>[4661032.101637]  [<ffffffff8110649a>] bdi_queue_work+0xa2/0xb9
     201<Feb/01 11:56 am>[4661032.107687]  [<ffffffff811068ae>] bdi_alloc_queue_work+0xa4/0xc4
     202<Feb/01 11:56 am>[4661032.114301]  [<ffffffff81106a1f>] wakeup_flusher_threads+0x93/0xc6
     203                 <Feb/01 11:56 am>[4661032.121087]  [<ffffffff811069bf>] ? wakeup_flusher_threads+0x33/0xc6
     204<Feb/01 11:56 am>[4661032.128058]  [<ffffffff810b9854>] do_try_to_free_pages+0x299/0x37a
     205<Feb/01 11:56 am>[4661032.134852]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
     206<Feb/01 11:56 am>[4661032.141178]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
     207
     208<Feb/01 11:56 am>[4661032.147948]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
     209<Feb/01 11:56 am>[4661032.154926]  [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
     210<Feb/01 11:56 am>[4661032.161450]  [<ffffffff810acb60>] __page_cache_alloc+0x59/0x5e
     211<Feb/01 11:56 am>[4661032.167863]  [<ffffffff810b4860>] __do_page_cache_readahead+0xe5/0x1fc
     212<Feb/01 11:56 am>[4661032.175007]  [<ffffffff810b47f6>] ? __do_page_cache_readahead+0x7b/0x1fc
     213<Feb/01 11:56 am>[4661032.182340]  [<ffffffff810b4993>] ra_submit+0x1c/0x20
     214<Feb/01 11:56 am>[4661032.187966]  [<ffffffff810b4cb7>] ondemand_readahead+0x243/0x256
     215<Feb/01 11:56 am>[4661032.194558]  [<ffffffff810ac48a>] ? find_get_pages_contig+0x0/0x118
     216<Feb/01 11:56 am>[4661032.201449]  [<ffffffff810b4daa>] page_cache_sync_readahead+0x38/0x3a
     217<Feb/01 11:56 am>[4661032.208513]  [<ffffffff81109bf2>] __generic_file_splice_read+0xfa/0x41c
     218<Feb/01 11:56 am>[4661032.215749]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     219<Feb/01 11:56 am>[4661032.222170]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     220<Feb/01 11:56 am>[4661032.228609]  [<ffffffff812615b1>] ? xfs_ilock+0x49/0xb2
     221<Feb/01 11:56 am>[4661032.234406]  [<ffffffff811086db>] ? spd_release_page+0x0/0x14
     222<Feb/01 11:56 am>[4661032.240751]  [<ffffffff81109f57>] generic_file_splice_read+0x43/0x74
     223<Feb/01 11:56 am>[4661032.247713]  [<ffffffff8128191a>] xfs_file_splice_read+0xde/0x125
     224<Feb/01 11:56 am>[4661032.254395]  [<ffffffff8110875d>] vfs_splice_to+0x6e/0x81
     225<Feb/01 11:56 am>[4661032.260381]  [<ffffffff81108837>] splice_direct_to_actor+0xc7/0x19c
     226<Feb/01 11:56 am>[4661032.267248]  [<ffffffff811f562c>] ? nfsd_direct_splice_actor+0x0/0x12
     227<Feb/01 11:56 am>[4661032.274312]  [<ffffffff811f54ef>] nfsd_vfs_read+0x263/0x3a0
     228<Feb/01 11:56 am>[4661032.280445]  [<ffffffff81281760>] ? xfs_file_open+0x0/0x3f
     229<Feb/01 11:56 am>[4661032.286495]  [<ffffffff811f5c4e>] nfsd_read+0xa7/0xc3
     230
     231<Feb/01 11:56 am>[4661032.292093]  [<ffffffff811fcd79>] nfsd3_proc_read+0xfe/0x142
     232<Feb/01 11:56 am>[4661032.298326]  [<ffffffff811f0854>] nfsd_dispatch+0xe9/0x1ca
     233<Feb/01 11:56 am>[4661032.304375]  [<ffffffff814f04c5>] svc_process+0x425/0x72a
     234<Feb/01 11:56 am>[4661032.310364]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
     235<Feb/01 11:56 am>[4661032.315725]  [<ffffffff811f0dc2>] nfsd+0xf3/0x139
     236<Feb/01 11:56 am>[4661032.320977]  [<ffffffff810546e2>] kthread+0x7d/0x85
     237<Feb/01 11:56 am>[4661032.326430]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     238<Feb/01 11:56 am>[4661032.332952]  [<ffffffff81054665>] ? kthread+0x0/0x85
     239<Feb/01 11:56 am>[4661032.338460]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     240<Feb/01 11:56 am>[4661032.345162] Pid: 8506, comm: nfsd Tainted: G      D    2.6.34.4 #3
     241<Feb/01 11:56 am>[4661032.351966] Call Trace:
     242<Feb/01 11:56 am>[4661032.354884]  <NMI>  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
     243<Feb/01 11:56 am>[4661032.361320]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
     244<Feb/01 11:56 am>[4661032.367748]  [<ffffffff81520360>] do_nmi+0xb2/0x259
     245<Feb/01 11:56 am>[4661032.373206]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
     246<Feb/01 11:56 am>[4661032.378232]  [<ffffffff812c088a>] ? delay_tsc+0x0/0x59
     247<Feb/01 11:56 am>[4661032.383910]  <<EOE>>  [<ffffffff812c08fe>] ? __delay+0xa/0xc
     248<Feb/01 11:56 am>[4661032.390171]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     249<Feb/01 11:56 am>[4661032.396515]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     250<Feb/01 11:56 am>[4661032.402572]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
     251<Feb/01 11:56 am>[4661032.408656]  [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
     252<Feb/01 11:56 am>[4661032.414557]  [<ffffffff81036578>] try_to_wake_up+0x2e/0x342
     253
     254<Feb/01 11:56 am>[4661032.420687]  [<ffffffff810368b8>] wake_up_process+0x10/0x12
     255<Feb/01 11:56 am>[4661032.426865]  [<ffffffff8110649a>] bdi_queue_work+0xa2/0xb9
     256<Feb/01 11:56 am>[4661032.432932]  [<ffffffff811068ae>] bdi_alloc_queue_work+0xa4/0xc4
     257<Feb/01 11:56 am>[4661032.439546]  [<ffffffff81106a1f>] wakeup_flusher_threads+0x93/0xc6
     258<Feb/01 11:56 am>[4661032.446351]  [<ffffffff811069bf>] ? wakeup_flusher_threads+0x33/0xc6
     259<Feb/01 11:56 am>[4661032.453302]  [<ffffffff810b9854>] do_try_to_free_pages+0x299/0x37a
     260<Feb/01 11:56 am>[4661032.460088]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
     261<Feb/01 11:56 am>[4661032.466413]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
     262<Feb/01 11:56 am>[4661032.473185]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
     263<Feb/01 11:56 am>[4661032.480146]  [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
     264<Feb/01 11:56 am>[4661032.486635]  [<ffffffff810acb60>] __page_cache_alloc+0x59/0x5e
     265<Feb/01 11:56 am>[4661032.493048]  [<ffffffff810b4860>] __do_page_cache_readahead+0xe5/0x1fc
     266<Feb/01 11:56 am>[4661032.500175]  [<ffffffff810b47f6>] ? __do_page_cache_readahead+0x7b/0x1fc
     267<Feb/01 11:56 am>[4661032.507509]  [<ffffffff810b4993>] ra_submit+0x1c/0x20
     268<Feb/01 11:56 am>[4661032.513117]  [<ffffffff810b4cb7>] ondemand_readahead+0x243/0x256
     269<Feb/01 11:56 am>[4661032.519665]  [<ffffffff810ac48a>] ? find_get_pages_contig+0x0/0x118
     270<Feb/01 11:56 am>[4661032.526547]  [<ffffffff810b4daa>] page_cache_sync_readahead+0x38/0x3a
     271<Feb/01 11:56 am>[4661032.533568]  [<ffffffff81109bf2>] __generic_file_splice_read+0xfa/0x41c
     272<Feb/01 11:56 am>[4661032.540815]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     273<Feb/01 11:56 am>[4661032.547217]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     274<Feb/01 11:56 am>[4661032.553648]  [<ffffffff812615b1>] ? xfs_ilock+0x49/0xb2
     275<Feb/01 11:56 am>[4661032.559443]  [<ffffffff811086db>] ? spd_release_page+0x0/0x14
     276
     277<Feb/01 11:56 am>[4661032.565790]  [<ffffffff81109f57>] generic_file_splice_read+0x43/0x74
     278<Feb/01 11:56 am>[4661032.572768]  [<ffffffff8128191a>] xfs_file_splice_read+0xde/0x125
     279<Feb/01 11:56 am>[4661032.579450]  [<ffffffff8110875d>] vfs_splice_to+0x6e/0x81
     280                 <Feb/01 11:56 am>[4661032.585427]  [<ffffffff81108837>] splice_direct_to_actor+0xc7/0x19c
     281<Feb/01 11:56 am>[4661032.592321]  [<ffffffff811f562c>] ? nfsd_direct_splice_actor+0x0/0x12
     282<Feb/01 11:56 am>[4661032.599377]  [<ffffffff811f54ef>] nfsd_vfs_read+0x263/0x3a0
     283<Feb/01 11:56 am>[4661032.605535]  [<ffffffff81281760>] ? xfs_file_open+0x0/0x3f
     284<Feb/01 11:56 am>[4661032.611585]  [<ffffffff811f5c4e>] nfsd_read+0xa7/0xc3
     285<Feb/01 11:56 am>[4661032.617207]  [<ffffffff811fcd79>] nfsd3_proc_read+0xfe/0x142
     286<Feb/01 11:56 am>[4661032.623442]  [<ffffffff811f0854>] nfsd_dispatch+0xe9/0x1ca
     287<Feb/01 11:56 am>[4661032.629509]  [<ffffffff814f04c5>] svc_process+0x425/0x72a
     288<Feb/01 11:56 am>[4661032.635479]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
     289<Feb/01 11:56 am>[4661032.640841]  [<ffffffff811f0dc2>] nfsd+0xf3/0x139
     290<Feb/01 11:56 am>[4661032.646085]  [<ffffffff810546e2>] kthread+0x7d/0x85
     291<Feb/01 11:56 am>[4661032.651521]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     292<Feb/01 11:56 am>[4661032.658023]  [<ffffffff81054665>] ? kthread+0x0/0x85
     293<Feb/01 11:56 am>[4661032.663550]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     294<Feb/01 11:56 am>[4661032.670211] NMI backtrace for cpu 5
     295<Feb/01 11:56 am>[4661032.674206] CPU 5
     296<Feb/01 11:56 am>[4661032.676323] Modules linked in: w83793 hwmon_vid autofs4 i2c_i801 i2c_core iTCO_wdt e1000e ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
     297<Feb/01 11:56 am>[4661032.712431]
     298<Feb/01 11:56 am>[4661032.714359] Pid: 8141, comm: psphotStack Tainted: G      D    2.6.34.4 #3 S7010/empty
     299
     300<Feb/01 11:56 am>[4661032.723023] RIP: 0010:[<ffffffff812c08af>]  [<ffffffff812c08af>] delay_tsc+0x25/0x59
     301<Feb/01 11:56 am>[4661032.731642] RSP: 0018:ffff88033a5d1500  EFLAGS: 00000086
     302<Feb/01 11:56 am>[4661032.737514] RAX: 00000000e48ec350 RBX: 0000000024400779 RCX: 00000000e48ec330
     303<Feb/01 11:56 am>[4661032.745457] RDX: 00000000002c28af RSI: 0000000000000005 RDI: 0000000000000001
     304<Feb/01 11:56 am>[4661032.753406] RBP: ffff88033a5d1508 R08: 0000000000000005 R09: 0000000000000005
     305<Feb/01 11:56 am>[4661032.761318] R10: 0000000000000000 R11: ffff88033a5d1338 R12: ffff880623308460
     306<Feb/01 11:56 am>[4661032.769268] R13: 0000000000000000 R14: ffff8800634f3090 R15: ffff8800634f3518
     307<Feb/01 11:56 am>[4661032.777212] FS:  0000000042611950(0063) GS:ffff8800026a0000(0000) knlGS:0000000000000000
     308<Feb/01 11:56 am>[4661032.786136] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
     309<Feb/01 11:56 am>[4661032.792482] CR2: 00007fc17fdb1000 CR3: 00000005a4b03000 CR4: 00000000000006e0
     310<Feb/01 11:56 am>[4661032.800449] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     311<Feb/01 11:56 am>[4661032.808382] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     312<Feb/01 11:56 am>[4661032.816337] Process psphotStack (pid: 8141, threadinfo ffff88033a5d0000, task ffff8800634f3090)
     313<Feb/01 11:56 am>[4661032.825901] Stack:
     314<Feb/01 11:56 am>[4661032.828375]  ffffffff81019da9 ffff88033a5d1518 ffffffff812c08fe ffff88033a5d1558
     315<Feb/01 11:56 am>[4661032.836382] <0> ffffffff812c4e65 000000009ef2dfac ffff880623308460 ffff880623308000
     316                 <Feb/01 11:56 am>[4661032.845136] <0> ffff8800ad50f740 ffff880623308000 0000000000000001 ffff88033a5d1578
     317<Feb/01 11:56 am>[4661032.854247] Call Trace:
     318                 <Feb/01 11:56 am>[4661032.857171]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     319<Feb/01 11:56 am>[4661032.865003]  [<ffffffff812c08fe>] __delay+0xa/0xc
     320<Feb/01 11:56 am>[4661032.870266]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     321<Feb/01 11:56 am>[4661032.876561]  [<ffffffff8151f115>] _raw_spin_lock_irq+0x35/0x39
     322
     323<Feb/01 11:56 am>[4661032.883009]  [<ffffffff812aacd7>] ? __make_request+0x5e/0x445
     324<Feb/01 11:56 am>[4661032.889363]  [<ffffffff812aacd7>] __make_request+0x5e/0x445
     325<Feb/01 11:56 am>[4661032.895506]  [<ffffffff812a946b>] generic_make_request+0x32a/0x39c
     326<Feb/01 11:56 am>[4661032.902303]  [<ffffffff810b33f2>] ? test_set_page_writeback+0x53/0x10e
     327<Feb/01 11:56 am>[4661032.909438]  [<ffffffff812a9591>] submit_bio+0xb4/0xbd
     328<Feb/01 11:56 am>[4661032.915118]  [<ffffffff810ac9a9>] ? unlock_page+0x22/0x26
     329<Feb/01 11:56 am>[4661032.921100]  [<ffffffff810d139c>] swap_writepage+0x8c/0x97
     330<Feb/01 11:56 am>[4661032.927150]  [<ffffffff810b7612>] shrink_page_list+0x3c2/0x68f
     331<Feb/01 11:56 am>[4661032.933591]  [<ffffffff810e5fb2>] ? mem_cgroup_del_lru_list+0x7b/0x8e
     332<Feb/01 11:56 am>[4661032.940649]  [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
     333<Feb/01 11:56 am>[4661032.946640]  [<ffffffff8102fc7b>] ? cpuacct_charge+0x76/0x96
     334<Feb/01 11:56 am>[4661032.952882]  [<ffffffff81058774>] ? up_read+0x1e/0x36
     335<Feb/01 11:56 am>[4661032.958488]  [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
     336<Feb/01 11:56 am>[4661032.965252]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
     337<Feb/01 11:56 am>[4661032.971611]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
     338<Feb/01 11:56 am>[4661032.978416]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
     339<Feb/01 11:56 am>[4661032.985377]  [<ffffffff810da90f>] alloc_page_vma+0x148/0x14d
     340<Feb/01 11:56 am>[4661032.991606]  [<ffffffff810d15c5>] read_swap_cache_async+0x5b/0xf2
     341<Feb/01 11:56 am>[4661032.998290]  [<ffffffff810d216d>] ? valid_swaphandles+0x6b/0xfd
     342<Feb/01 11:56 am>[4661033.004781]  [<ffffffff810d16b4>] swapin_readahead+0x58/0x97
     343<Feb/01 11:56 am>[4661033.011036]  [<ffffffff810c4ef1>] handle_mm_fault+0x45b/0x73e
     344<Feb/01 11:56 am>[4661033.017382]  [<ffffffff815227e9>] do_page_fault+0x351/0x375
     345
     346<Feb/01 11:56 am>[4661033.023526]  [<ffffffff810348d8>] ? finish_task_switch+0x69/0xdd
     347<Feb/01 11:56 am>[4661033.030140]  [<ffffffff8103486f>] ? finish_task_switch+0x0/0xdd
     348<Feb/01 11:56 am>[4661033.036675]  [<ffffffff8151d152>] ? schedule+0x79e/0x86a
     349<Feb/01 11:56 am>[4661033.042549]  [<ffffffff8151ef6f>] ? lockdep_sys_exit_thunk+0x35/0x67
     350<Feb/01 11:56 am>[4661033.049513]  [<ffffffff810dce8c>] ? get_partial_node+0x1b/0x96
     351<Feb/01 11:56 am>[4661033.055908]  [<ffffffff8151fb5f>] page_fault+0x1f/0x30
     352<Feb/01 11:56 am>[4661033.061607] Code: fb 48 ff c8 c9 c3 55 48 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae e8 0f 31 89 c1 41 89 f1 66 66 90 0f ae e8 0f 31 <89> c6 48 89 f0 48 29 c8 48 39 f8 73 25 f3 90 65 44 8b 04 25 f0
     353<Feb/01 11:56 am>[4661033.084720] Call Trace:
     354<Feb/01 11:56 am>[4661033.087643]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     355<Feb/01 11:56 am>[4661033.095510]  [<ffffffff812c08fe>] __delay+0xa/0xc
     356<Feb/01 11:56 am>[4661033.100771]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     357<Feb/01 11:56 am>[4661033.107103]  [<ffffffff8151f115>] _raw_spin_lock_irq+0x35/0x39
     358<Feb/01 11:56 am>[4661033.113535]  [<ffffffff812aacd7>] ? __make_request+0x5e/0x445
     359<Feb/01 11:56 am>[4661033.119869]  [<ffffffff812aacd7>] __make_request+0x5e/0x445
     360<Feb/01 11:56 am>[4661033.126005]  [<ffffffff812a946b>] generic_make_request+0x32a/0x39c
     361<Feb/01 11:56 am>[4661033.132768]  [<ffffffff810b33f2>] ? test_set_page_writeback+0x53/0x10e
     362<Feb/01 11:56 am>[4661033.139903]  [<ffffffff812a9591>] submit_bio+0xb4/0xbd
     363                 <Feb/01 11:56 am>[4661033.145598]  [<ffffffff810ac9a9>] ? unlock_page+0x22/0x26
     364<Feb/01 11:56 am>[4661033.156079]  [<ffffffff810d139c>] swap_writepage+0x8c/0x97
     365<Feb/01 11:56 am>[4661033.162161]  [<ffffffff810b7612>] shrink_page_list+0x3c2/0x68f
     366<Feb/01 11:56 am>[4661033.168569]  [<ffffffff810e5fb2>] ? mem_cgroup_del_lru_list+0x7b/0x8e
     367<Feb/01 11:56 am>[4661033.175617]  [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
     368
     369<Feb/01 11:56 am>[4661033.181593]  [<ffffffff8102fc7b>] ? cpuacct_charge+0x76/0x96
     370<Feb/01 11:56 am>[4661033.187852]  [<ffffffff81058774>] ? up_read+0x1e/0x36
     371<Feb/01 11:56 am>[4661033.193447]  [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
     372<Feb/01 11:56 am>[4661033.200221]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
     373<Feb/01 11:56 am>[4661033.206547]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
     374<Feb/01 11:56 am>[4661033.213309]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
     375<Feb/01 11:56 am>[4661033.220262]  [<ffffffff810da90f>] alloc_page_vma+0x148/0x14d
     376<Feb/01 11:56 am>[4661033.226499]  [<ffffffff810d15c5>] read_swap_cache_async+0x5b/0xf2
     377<Feb/01 11:56 am>[4661033.233190]  [<ffffffff810d216d>] ? valid_swaphandles+0x6b/0xfd
     378<Feb/01 11:56 am>[4661033.239707]  [<ffffffff810d16b4>] swapin_readahead+0x58/0x97
     379<Feb/01 11:56 am>[4661033.245964]  [<ffffffff810c4ef1>] handle_mm_fault+0x45b/0x73e
     380<Feb/01 11:56 am>[4661033.252308]  [<ffffffff815227e9>] do_page_fault+0x351/0x375
     381<Feb/01 11:56 am>[4661033.258462]  [<ffffffff810348d8>] ? finish_task_switch+0x69/0xdd
     382<Feb/01 11:56 am>[4661033.265067]  [<ffffffff8103486f>] ? finish_task_switch+0x0/0xdd
     383<Feb/01 11:56 am>[4661033.271584]  [<ffffffff8151d152>] ? schedule+0x79e/0x86a
     384<Feb/01 11:56 am>[4661033.277475]  [<ffffffff8151ef6f>] ? lockdep_sys_exit_thunk+0x35/0x67
     385<Feb/01 11:56 am>[4661033.284431]  [<ffffffff810dce8c>] ? get_partial_node+0x1b/0x96
     386<Feb/01 11:56 am>[4661033.290868]  [<ffffffff8151fb5f>] page_fault+0x1f/0x30
     387<Feb/01 11:56 am>[4661033.296594] Pid: 8141, comm: psphotStack Tainted: G      D    2.6.34.4 #3
     388<Feb/01 11:56 am>[4661033.304009] Call Trace:
     389<Feb/01 11:56 am>[4661033.306924]  <NMI>  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
     390<Feb/01 11:56 am>[4661033.313404]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
     391
     392<Feb/01 11:56 am>[4661033.319842]  [<ffffffff81520360>] do_nmi+0xb2/0x259
     393<Feb/01 11:56 am>[4661033.325272]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
     394<Feb/01 11:56 am>[4661033.330350]  [<ffffffff812c08af>] ? delay_tsc+0x25/0x59
     395<Feb/01 11:56 am>[4661033.336116]  <<EOE>>  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     396<Feb/01 11:56 am>[4661033.344817]  [<ffffffff812c08fe>] __delay+0xa/0xc
     397<Feb/01 11:56 am>[4661033.350077]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     398<Feb/01 11:56 am>[4661033.356390]  [<ffffffff8151f115>] _raw_spin_lock_irq+0x35/0x39
     399<Feb/01 11:56 am>[4661033.362804]  [<ffffffff812aacd7>] ? __make_request+0x5e/0x445
     400<Feb/01 11:56 am>[4661033.369087]  [<ffffffff812aacd7>] __make_request+0x5e/0x445
     401<Feb/01 11:56 am>[4661033.375258]  [<ffffffff812a946b>] generic_make_request+0x32a/0x39c
     402<Feb/01 11:56 am>[4661033.382046]  [<ffffffff810b33f2>] ? test_set_page_writeback+0x53/0x10e
     403<Feb/01 11:56 am>[4661033.389198]  [<ffffffff812a9591>] submit_bio+0xb4/0xbd
     404<Feb/01 11:56 am>[4661033.394895]  [<ffffffff810ac9a9>] ? unlock_page+0x22/0x26
     405<Feb/01 11:56 am>[4661033.400850]  [<ffffffff810d139c>] swap_writepage+0x8c/0x97
     406<Feb/01 11:56 am>[4661033.406927]  [<ffffffff810b7612>] shrink_page_list+0x3c2/0x68f
     407<Feb/01 11:56 am>[4661033.413369]  [<ffffffff810e5fb2>] ? mem_cgroup_del_lru_list+0x7b/0x8e
     408<Feb/01 11:56 am>[4661033.420442]  [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
     409<Feb/01 11:56 am>[4661033.426428]  [<ffffffff8102fc7b>] ? cpuacct_charge+0x76/0x96
     410<Feb/01 11:56 am>[4661033.432652]  [<ffffffff81058774>] ? up_read+0x1e/0x36
     411<Feb/01 11:56 am>[4661033.438232]  [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
     412<Feb/01 11:56 am>[4661033.445021]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
     413<Feb/01 11:56 am>[4661033.451346]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
     414
     415<Feb/01 11:56 am>[4661033.458133]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
     416<Feb/01 11:56 am>[4661033.465079]  [<ffffffff810da90f>] alloc_page_vma+0x148/0x14d
     417<Feb/01 11:56 am>[4661033.471331]  [<ffffffff810d15c5>] read_swap_cache_async+0x5b/0xf2
     418<Feb/01 11:56 am>[4661033.478005]  [<ffffffff810d216d>] ? valid_swaphandles+0x6b/0xfd
     419<Feb/01 11:56 am>[4661033.484524]  [<ffffffff810d16b4>] swapin_readahead+0x58/0x97
     420<Feb/01 11:56 am>[4661033.490763]  [<ffffffff810c4ef1>] handle_mm_fault+0x45b/0x73e
     421                 <Feb/01 11:56 am>[4661033.497116]  [<ffffffff815227e9>] do_page_fault+0x351/0x375
     422<Feb/01 11:56 am>[4661033.503286]  [<ffffffff810348d8>] ? finish_task_switch+0x69/0xdd
     423<Feb/01 11:56 am>[4661033.509909]  [<ffffffff8103486f>] ? finish_task_switch+0x0/0xdd
     424<Feb/01 11:56 am>[4661033.516427]  [<ffffffff8151d152>] ? schedule+0x79e/0x86a
     425<Feb/01 11:56 am>[4661033.522310]  [<ffffffff8151ef6f>] ? lockdep_sys_exit_thunk+0x35/0x67
     426<Feb/01 11:56 am>[4661033.529290]  [<ffffffff810dce8c>] ? get_partial_node+0x1b/0x96
     427<Feb/01 11:56 am>[4661033.535720]  [<ffffffff8151fb5f>] page_fault+0x1f/0x30
     428<Feb/01 11:56 am>[4661033.541436] NMI backtrace for cpu 7
     429<Feb/01 11:56 am>[4661033.545427] CPU 7
     430<Feb/01 11:56 am>[4661033.547545] Modules linked in: w83793 hwmon_vid autofs4 i2c_i801 i2c_core iTCO_wdt e1000e ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
     431<Feb/01 11:56 am>[4661033.583688]
     432<Feb/01 11:56 am>[4661033.585617] Pid: 0, comm: swapper Tainted: G      D    2.6.34.4 #3 S7010/empty
     433<Feb/01 11:56 am>[4661033.593636] RIP: 0010:[<ffffffff812c08b7>]  [<ffffffff812c08b7>] delay_tsc+0x2d/0x59
     434<Feb/01 11:56 am>[4661033.602273] RSP: 0018:ffff880655223b58  EFLAGS: 00000012
     435<Feb/01 11:56 am>[4661033.608165] RAX: 000000000000001c RBX: 0000000024530af5 RCX: 00000000e496e978
     436<Feb/01 11:56 am>[4661033.616101] RDX: 00000000002c28af RSI: 00000000e496e994 RDI: 0000000000000001
     437
     438<Feb/01 11:56 am>[4661033.624066] RBP: ffff880655223b60 R08: 0000000000000007 R09: 0000000000000007
     439<Feb/01 11:56 am>[4661033.632019] R10: 0000000000000000 R11: ffff880655223990 R12: ffff8806552b3c00
     440<Feb/01 11:56 am>[4661033.639970] R13: 0000000000000000 R14: ffff880c24c8c0c0 R15: ffff880c24c8c548
     441<Feb/01 11:56 am>[4661033.647874] FS:  0000000000000000(0000) GS:ffff880655220000(0000) knlGS:0000000000000000
     442<Feb/01 11:56 am>[4661033.656821] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
     443<Feb/01 11:56 am>[4661033.663140] CR2: 00007ff195e94016 CR3: 0000000001a8a000 CR4: 00000000000006e0
     444<Feb/01 11:56 am>[4661033.671100] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     445<Feb/01 11:56 am>[4661033.679040] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     446<Feb/01 11:56 am>[4661033.686978] Process swapper (pid: 0, threadinfo ffff880624d38000, task ffff880c24c8c0c0)
     447<Feb/01 11:56 am>[4661033.695848] Stack:
     448<Feb/01 11:56 am>[4661033.698316]  ffffffff81019da9 ffff880655223b70 ffffffff812c08fe ffff880655223bb0
     449<Feb/01 11:56 am>[4661033.706321] <0> ffffffff812c4e65 000000009ef2dfac ffff8806552b3c00 ffff8806552b3c00
     450<Feb/01 11:56 am>[4661033.715074] <0> ffff88065522e3f0 00000000000004aa 0000000000001000 ffff880655223bd0
     451<Feb/01 11:56 am>[4661033.724205] Call Trace:
     452<Feb/01 11:56 am>[4661033.727118]  <IRQ>
     453<Feb/01 11:56 am>[4661033.729709]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     454<Feb/01 11:56 am>[4661033.737577]  [<ffffffff812c08fe>] __delay+0xa/0xc
     455<Feb/01 11:56 am>[4661033.742821]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     456<Feb/01 11:56 am>[4661033.749169]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
     457<Feb/01 11:56 am>[4661033.755861]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76
     458<Feb/01 11:56 am>[4661033.762118]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
     459<Feb/01 11:56 am>[4661033.768201]  [<ffffffff81037672>] load_balance+0xbb6/0xf63
     460
     461<Feb/01 11:56 am>[4661033.774258]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     462<Feb/01 11:56 am>[4661033.780743]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     463<Feb/01 11:56 am>[4661033.787165]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     464                 <Feb/01 11:56 am>[4661033.793598]  [<ffffffff81047c8b>] ? run_timer_softirq+0x4d/0x30c
     465<Feb/01 11:56 am>[4661033.800192]  [<ffffffff81037af9>] rebalance_domains+0xda/0x150
     466<Feb/01 11:56 am>[4661033.806624]  [<ffffffff81037bb2>] run_rebalance_domains+0x43/0xcf
     467<Feb/01 11:56 am>[4661033.813316]  [<ffffffff8104103c>] __do_softirq+0xd8/0x194
     468<Feb/01 11:56 am>[4661033.819301]  [<ffffffff8100388c>] call_softirq+0x1c/0x28
     469<Feb/01 11:56 am>[4661033.825186]  [<ffffffff81004e7f>] do_softirq+0x33/0x68
     470<Feb/01 11:56 am>[4661033.830880]  [<ffffffff81040ec2>] irq_exit+0x36/0x75
     471<Feb/01 11:56 am>[4661033.836408]  [<ffffffff810197aa>] smp_apic_timer_interrupt+0x87/0x95
     472<Feb/01 11:56 am>[4661033.843364]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
     473<Feb/01 11:56 am>[4661033.849992]  <EOI>
     474<Feb/01 11:56 am>[4661033.852599]  [<ffffffff8151cc4a>] ? schedule+0x296/0x86a
     475<Feb/01 11:56 am>[4661033.858466]  [<ffffffff81303870>] ? acpi_idle_enter_c1+0xdd/0xf8
     476<Feb/01 11:56 am>[4661033.865040]  [<ffffffff81303851>] ? acpi_idle_enter_c1+0xbe/0xf8
     477<Feb/01 11:56 am>[4661033.871608]  [<ffffffff81430151>] cpuidle_idle_call+0x95/0xf0
     478<Feb/01 11:56 am>[4661033.877919]  [<ffffffff81001d4b>] cpu_idle+0x55/0x8d
     479<Feb/01 11:56 am>[4661033.883444]  [<ffffffff815191ca>] start_secondary+0x1ae/0x1b2
     480<Feb/01 11:56 am>[4661033.889790] Code: 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae e8 0f 31 89 c1 41 89 f1 66 66 90 0f ae e8 0f 31 89 c6 48 89 f0 48 29 c8 <48> 39 f8 73 25 f3 90 65 44 8b 04 25 f0 d1 00 00 45 39 c1 74 db
     481<Feb/01 11:56 am>[4661033.912946] Call Trace:
     482<Feb/01 11:56 am>[4661033.915849]  <IRQ>  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     483
     484<Feb/01 11:56 am>[4661033.924346]  [<ffffffff812c08fe>] __delay+0xa/0xc
     485<Feb/01 11:56 am>[4661033.929603]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     486<Feb/01 11:56 am>[4661033.935943]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
     487<Feb/01 11:56 am>[4661033.942626]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76
     488<Feb/01 11:56 am>[4661033.948857]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
     489<Feb/01 11:56 am>[4661033.954922]  [<ffffffff81037672>] load_balance+0xbb6/0xf63
     490<Feb/01 11:56 am>[4661033.961006]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     491<Feb/01 11:56 am>[4661033.967412]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     492<Feb/01 11:56 am>[4661033.973853]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     493<Feb/01 11:56 am>[4661033.980257]  [<ffffffff81047c8b>] ? run_timer_softirq+0x4d/0x30c
     494<Feb/01 11:56 am>[4661033.986880]  [<ffffffff81037af9>] rebalance_domains+0xda/0x150
     495<Feb/01 11:56 am>[4661033.993319]  [<ffffffff81037bb2>] run_rebalance_domains+0x43/0xcf
     496<Feb/01 11:56 am>[4661034.000027]  [<ffffffff8104103c>] __do_softirq+0xd8/0x194
     497<Feb/01 11:56 am>[4661034.005997]  [<ffffffff8100388c>] call_softirq+0x1c/0x28
     498<Feb/01 11:56 am>[4661034.011838]  [<ffffffff81004e7f>] do_softirq+0x33/0x68
     499<Feb/01 11:56 am>[4661034.017558]  [<ffffffff81040ec2>] irq_exit+0x36/0x75
     500<Feb/01 11:56 am>[4661034.023086]  [<ffffffff810197aa>] smp_apic_timer_interrupt+0x87/0x95
     501<Feb/01 11:56 am>[4661034.030033]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
     502<Feb/01 11:56 am>[4661034.036636]  <EOI>  [<ffffffff8151cc4a>] ? schedule+0x296/0x86a
     503<Feb/01 11:56 am>[4661034.043199]  [<ffffffff81303870>] ? acpi_idle_enter_c1+0xdd/0xf8
     504<Feb/01 11:56 am>[4661034.049793]  [<ffffffff81303851>] ? acpi_idle_enter_c1+0xbe/0xf8
     505<Feb/01 11:56 am>[4661034.056397]  [<ffffffff81430151>] cpuidle_idle_call+0x95/0xf0
     506
     507<Feb/01 11:56 am>[4661034.062707]  [<ffffffff81001d4b>] cpu_idle+0x55/0x8d
     508<Feb/01 11:56 am>[4661034.068189]  [<ffffffff815191ca>] start_secondary+0x1ae/0x1b2
     509<Feb/01 11:56 am>[4661034.074501] Pid: 0, comm: swapper Tainted: G      D    2.6.34.4 #3
     510<Feb/01 11:56 am>[4661034.081271] Call Trace:
     511<Feb/01 11:56 am>[4661034.084188]  <NMI>  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
     512<Feb/01 11:56 am>[4661034.090633]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
     513<Feb/01 11:56 am>[4661034.097071]  [<ffffffff81520360>] do_nmi+0xb2/0x259
     514<Feb/01 11:56 am>[4661034.102518]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
     515<Feb/01 11:56 am>[4661034.107597]  [<ffffffff812c08b7>] ? delay_tsc+0x2d/0x59
     516<Feb/01 11:56 am>[4661034.113379]  <<EOE>>  <IRQ>  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     517<Feb/01 11:56 am>[4661034.122776]  [<ffffffff812c08fe>] __delay+0xa/0xc
     518<Feb/01 11:56 am>[4661034.128034]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     519<Feb/01 11:56 am>[4661034.134382]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
     520<Feb/01 11:56 am>[4661034.141074]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76
     521<Feb/01 11:56 am>[4661034.147322]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
     522<Feb/01 11:56 am>[4661034.153379]  [<ffffffff81037672>] load_balance+0xbb6/0xf63
     523<Feb/01 11:56 am>[4661034.159464]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     524<Feb/01 11:56 am>[4661034.165913]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     525<Feb/01 11:56 am>[4661034.172335]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     526<Feb/01 11:56 am>[4661034.183265]  [<ffffffff81047c8b>] ? run_timer_softirq+0x4d/0x30c
     527<Feb/01 11:56 am>[4661034.189843]  [<ffffffff81037af9>] rebalance_domains+0xda/0x150
     528<Feb/01 11:56 am>[4661034.196264]  [<ffffffff81037bb2>] run_rebalance_domains+0x43/0xcf
     529
     530<Feb/01 11:56 am>[4661034.202946]  [<ffffffff8104103c>] __do_softirq+0xd8/0x194
     531<Feb/01 11:56 am>[4661034.208915]  [<ffffffff8100388c>] call_softirq+0x1c/0x28
     532<Feb/01 11:56 am>[4661034.214809]  [<ffffffff81004e7f>] do_softirq+0x33/0x68
     533<Feb/01 11:56 am>[4661034.220522]  [<ffffffff81040ec2>] irq_exit+0x36/0x75
     534<Feb/01 11:56 am>[4661034.226058]  [<ffffffff810197aa>] smp_apic_timer_interrupt+0x87/0x95
     535<Feb/01 11:56 am>[4661034.233023]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
     536<Feb/01 11:56 am>[4661034.239626]  <EOI>  [<ffffffff8151cc4a>] ? schedule+0x296/0x86a
     537<Feb/01 11:56 am>[4661034.246188]  [<ffffffff81303870>] ? acpi_idle_enter_c1+0xdd/0xf8
     538<Feb/01 11:56 am>[4661034.252792]  [<ffffffff81303851>] ? acpi_idle_enter_c1+0xbe/0xf8
     539<Feb/01 11:56 am>[4661034.259403]  [<ffffffff81430151>] cpuidle_idle_call+0x95/0xf0
     540<Feb/01 11:56 am>[4661034.265756]  [<ffffffff81001d4b>] cpu_idle+0x55/0x8d
     541<Feb/01 11:56 am>[4661034.271299]  [<ffffffff815191ca>] start_secondary+0x1ae/0x1b2
     542<Feb/01 11:56 am>[4661034.277637] NMI backtrace for cpu 1
     543                 <Feb/01 11:56 am>[4661034.281643] CPU 1
     544<Feb/01 11:56 am>[4661034.283761] Modules linked in: w83793 hwmon_vid autofs4 i2c_i801 i2c_core iTCO_wdt e1000e ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
     545<Feb/01 11:56 am>[4661034.319807]
     546<Feb/01 11:56 am>[4661034.321738] Pid: 29943, comm: pclient Tainted: G      D    2.6.34.4 #3 S7010/empty
     547<Feb/01 11:56 am>[4661034.330119] RIP: 0010:[<ffffffff812c08a4>]  [<ffffffff812c08a4>] delay_tsc+0x1a/0x59
     548<Feb/01 11:56 am>[4661034.338721] RSP: 0018:ffff880295cfd530  EFLAGS: 00000086
     549<Feb/01 11:56 am>[4661034.344588] RAX: 00000000abd8e664 RBX: 000000001fd84c96 RCX: 00000000abd8e664
     550<Feb/01 11:56 am>[4661034.352548] RDX: 00000000002c28ae RSI: 0000000000000001 RDI: 0000000000000001
     551<Feb/01 11:56 am>[4661034.360497] RBP: ffff880295cfd538 R08: 0000000000000001 R09: 0000000000000001
     552
     553<Feb/01 11:56 am>[4661034.368470] R10: 0000000000000000 R11: ffff880295cfd368 R12: ffff8806552b3c00
     554<Feb/01 11:56 am>[4661034.376409] R13: 0000000000000000 R14: ffff8806239f50f0 R15: ffff8806239f5578
     555<Feb/01 11:56 am>[4661034.384348] FS:  00007fcb6b8b76f0(0000) GS:ffff880002620000(0000) knlGS:0000000000000000
     556<Feb/01 11:56 am>[4661034.393262] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
     557<Feb/01 11:56 am>[4661034.399591] CR2: 000000000040fe90 CR3: 000000022edd0000 CR4: 00000000000006e0
     558<Feb/01 11:56 am>[4661034.407509] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     559<Feb/01 11:56 am>[4661034.415445] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     560<Feb/01 11:56 am>[4661034.423395] Process pclient (pid: 29943, threadinfo ffff880295cfc000, task ffff8806239f50f0)
     561<Feb/01 11:56 am>[4661034.432656] Stack:
     562<Feb/01 11:56 am>[4661034.435129]  ffffffff81019da9 ffff880295cfd548 ffffffff812c08fe ffff880295cfd588
     563<Feb/01 11:56 am>[4661034.443118] <0> ffffffff812c4e65 000000009ef2dfac ffff8806552b3c00 ffff8806552b3c00
     564<Feb/01 11:56 am>[4661034.451863] <0> ffff88000262e650 00000000000007fe 0000000000001000 ffff880295cfd5a8
     565<Feb/01 11:56 am>[4661034.460984] Call Trace:
     566<Feb/01 11:56 am>[4661034.463915]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     567<Feb/01 11:56 am>[4661034.471765]  [<ffffffff812c08fe>] __delay+0xa/0xc
     568<Feb/01 11:56 am>[4661034.477036]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     569<Feb/01 11:56 am>[4661034.483375]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
     570<Feb/01 11:56 am>[4661034.490067]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76
     571<Feb/01 11:56 am>[4661034.496314]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
     572<Feb/01 11:56 am>[4661034.502381]  [<ffffffff81037672>] load_balance+0xbb6/0xf63
     573<Feb/01 11:56 am>[4661034.508465]  [<ffffffff810cc693>] ? page_referenced_one+0xc9/0xe5
     574<Feb/01 11:56 am>[4661034.515139]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     575
     576<Feb/01 11:56 am>[4661034.521554]  [<ffffffff8151cc0e>] schedule+0x25a/0x86a
     577<Feb/01 11:56 am>[4661034.527228]  [<ffffffff8151e2a0>] schedule_hrtimeout_range+0xce/0x103
     578<Feb/01 11:56 am>[4661034.534262]  [<ffffffff81057842>] ? hrtimer_wakeup+0x0/0x21
     579<Feb/01 11:56 am>[4661034.540386]  [<ffffffff810583de>] ? hrtimer_start_range_ns+0xf/0x11
     580<Feb/01 11:56 am>[4661034.547261]  [<ffffffff810f87a3>] poll_schedule_timeout+0x46/0x69
     581                 <Feb/01 11:56 am>[4661034.553934]  [<ffffffff810f0b80>] ? pipe_poll+0x2e/0x8b
     582<Feb/01 11:56 am>[4661034.559729]  [<ffffffff810f945d>] do_select+0x5ad/0x5f1
     583<Feb/01 11:56 am>[4661034.565509]  [<ffffffff810f8eb0>] ? do_select+0x0/0x5f1
     584<Feb/01 11:56 am>[4661034.571299]  [<ffffffff810f9994>] ? __pollwait+0x0/0xcf
     585<Feb/01 11:56 am>[4661034.577106]  [<ffffffff810f9a63>] ? pollwake+0x0/0x59
     586<Feb/01 11:56 am>[4661034.582714]  [<ffffffff8102fc93>] ? cpuacct_charge+0x8e/0x96
     587<Feb/01 11:56 am>[4661034.588939]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
     588<Feb/01 11:56 am>[4661034.595013]  [<ffffffff8103687a>] ? try_to_wake_up+0x330/0x342
     589<Feb/01 11:56 am>[4661034.601445]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     590<Feb/01 11:56 am>[4661034.607876]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     591<Feb/01 11:56 am>[4661034.614315]  [<ffffffff810f1c3b>] ? pipe_read+0x81/0x3d6
     592<Feb/01 11:56 am>[4661034.620216]  [<ffffffff810f94de>] ? core_sys_select+0x3d/0x295
     593<Feb/01 11:56 am>[4661034.626631]  [<ffffffff810f9688>] core_sys_select+0x1e7/0x295
     594<Feb/01 11:56 am>[4661034.632949]  [<ffffffff810f94de>] ? core_sys_select+0x3d/0x295
     595<Feb/01 11:56 am>[4661034.639399]  [<ffffffff810ea38f>] ? do_sync_read+0xc7/0x10d
     596<Feb/01 11:56 am>[4661034.645568]  [<ffffffff8105cc46>] ? ktime_get_ts+0xb1/0xbe
     597<Feb/01 11:56 am>[4661034.651652]  [<ffffffff810f996c>] sys_select+0x94/0xbc
     598
     599<Feb/01 11:56 am>[4661034.657379]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
     600<Feb/01 11:56 am>[4661034.663995] Code: 1f 84 00 00 00 00 00 48 ff c8 75 fb 48 ff c8 c9 c3 55 48 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae e8 0f 31 89 c1 <41> 89 f1 66 66 90 0f ae e8 0f 31 89 c6 48 89 f0 48 29 c8 48 39
     601<Feb/01 11:56 am>[4661034.687175] Call Trace:
     602                 <Feb/01 11:56 am>[4661034.690088]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     603<Feb/01 11:56 am>[4661034.697964]  [<ffffffff812c08fe>] __delay+0xa/0xc
     604<Feb/01 11:56 am>[4661034.703218]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     605<Feb/01 11:56 am>[4661034.709566]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
     606<Feb/01 11:56 am>[4661034.716248]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76
     607<Feb/01 11:56 am>[4661034.722505]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
     608<Feb/01 11:56 am>[4661034.728572]  [<ffffffff81037672>] load_balance+0xbb6/0xf63
     609<Feb/01 11:56 am>[4661034.734656]  [<ffffffff810cc693>] ? page_referenced_one+0xc9/0xe5
     610                 <Feb/01 11:56 am>[4661034.741357]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     611<Feb/01 11:56 am>[4661034.747786]  [<ffffffff8151cc0e>] schedule+0x25a/0x86a
     612<Feb/01 11:56 am>[4661034.753496]  [<ffffffff8151e2a0>] schedule_hrtimeout_range+0xce/0x103
     613<Feb/01 11:56 am>[4661034.760564]  [<ffffffff81057842>] ? hrtimer_wakeup+0x0/0x21
     614<Feb/01 11:56 am>[4661034.766740]  [<ffffffff810583de>] ? hrtimer_start_range_ns+0xf/0x11
     615<Feb/01 11:56 am>[4661034.773573]  [<ffffffff810f87a3>] poll_schedule_timeout+0x46/0x69
     616<Feb/01 11:56 am>[4661034.780255]  [<ffffffff810f0b80>] ? pipe_poll+0x2e/0x8b
     617<Feb/01 11:56 am>[4661034.786058]  [<ffffffff810f945d>] do_select+0x5ad/0x5f1
     618<Feb/01 11:56 am>[4661034.791838]  [<ffffffff810f8eb0>] ? do_select+0x0/0x5f1
     619<Feb/01 11:56 am>[4661034.797629]  [<ffffffff810f9994>] ? __pollwait+0x0/0xcf
     620<Feb/01 11:56 am>[4661034.803426]  [<ffffffff810f9a63>] ? pollwake+0x0/0x59
     621
     622<Feb/01 11:56 am>[4661034.809008]  [<ffffffff8102fc93>] ? cpuacct_charge+0x8e/0x96
     623<Feb/01 11:56 am>[4661034.815215]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
     624<Feb/01 11:56 am>[4661034.821300]  [<ffffffff8103687a>] ? try_to_wake_up+0x330/0x342
     625<Feb/01 11:56 am>[4661034.827748]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     626<Feb/01 11:56 am>[4661034.834171]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     627<Feb/01 11:56 am>[4661034.840577]  [<ffffffff810f1c3b>] ? pipe_read+0x81/0x3d6
     628<Feb/01 11:56 am>[4661034.846460]  [<ffffffff810f94de>] ? core_sys_select+0x3d/0x295
     629<Feb/01 11:56 am>[4661034.852874]  [<ffffffff810f9688>] core_sys_select+0x1e7/0x295
     630<Feb/01 11:56 am>[4661034.859192]  [<ffffffff810f94de>] ? core_sys_select+0x3d/0x295
     631<Feb/01 11:56 am>[4661034.865598]  [<ffffffff810ea38f>] ? do_sync_read+0xc7/0x10d
     632<Feb/01 11:56 am>[4661034.871767]  [<ffffffff8105cc46>] ? ktime_get_ts+0xb1/0xbe
     633                 <Feb/01 11:56 am>[4661034.877852]  [<ffffffff810f996c>] sys_select+0x94/0xbc
     634<Feb/01 11:56 am>[4661034.883563]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
     635<Feb/01 11:56 am>[4661034.890177] Pid: 29943, comm: pclient Tainted: G      D    2.6.34.4 #3
     636<Feb/01 11:56 am>[4661034.897319] Call Trace:
     637<Feb/01 11:56 am>[4661034.900244]  <NMI>  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
     638<Feb/01 11:56 am>[4661034.906706]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
     639<Feb/01 11:56 am>[4661034.913145]  [<ffffffff81520360>] do_nmi+0xb2/0x259
     640<Feb/01 11:56 am>[4661034.918591]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
     641<Feb/01 11:56 am>[4661034.923670]  [<ffffffff812c08a4>] ? delay_tsc+0x1a/0x59
     642<Feb/01 11:56 am>[4661034.929469]  <<EOE>>  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     643<Feb/01 11:56 am>[4661034.938165]  [<ffffffff812c08fe>] __delay+0xa/0xc
     644
     645<Feb/01 11:56 am>[4661034.943414]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     646<Feb/01 11:56 am>[4661034.949736]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
     647<Feb/01 11:56 am>[4661034.956429]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76
     648<Feb/01 11:56 am>[4661034.962677]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
     649<Feb/01 11:56 am>[4661034.968751]  [<ffffffff81037672>] load_balance+0xbb6/0xf63
     650<Feb/01 11:56 am>[4661034.974837]  [<ffffffff810cc693>] ? page_referenced_one+0xc9/0xe5
     651<Feb/01 11:56 am>[4661034.981510]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     652<Feb/01 11:56 am>[4661034.987924]  [<ffffffff8151cc0e>] schedule+0x25a/0x86a
     653<Feb/01 11:56 am>[4661034.993625]  [<ffffffff8151e2a0>] schedule_hrtimeout_range+0xce/0x103
     654<Feb/01 11:56 am>[4661035.000665]  [<ffffffff81057842>] ? hrtimer_wakeup+0x0/0x21
     655<Feb/01 11:56 am>[4661035.006808]  [<ffffffff810583de>] ? hrtimer_start_range_ns+0xf/0x11
     656<Feb/01 11:56 am>[4661035.013675]  [<ffffffff810f87a3>] poll_schedule_timeout+0x46/0x69
     657<Feb/01 11:56 am>[4661035.020356]  [<ffffffff810f0b80>] ? pipe_poll+0x2e/0x8b
     658<Feb/01 11:56 am>[4661035.026151]  [<ffffffff810f945d>] do_select+0x5ad/0x5f1
     659<Feb/01 11:56 am>[4661035.031949]  [<ffffffff810f8eb0>] ? do_select+0x0/0x5f1
     660<Feb/01 11:56 am>[4661035.037757]  [<ffffffff810f9994>] ? __pollwait+0x0/0xcf
     661<Feb/01 11:56 am>[4661035.043553]  [<ffffffff810f9a63>] ? pollwake+0x0/0x59
     662<Feb/01 11:56 am>[4661035.049161]  [<ffffffff8102fc93>] ? cpuacct_charge+0x8e/0x96
     663<Feb/01 11:56 am>[4661035.055386]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
     664<Feb/01 11:56 am>[4661035.061427]  [<ffffffff8103687a>] ? try_to_wake_up+0x330/0x342
     665<Feb/01 11:56 am>[4661035.067842]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     666<Feb/01 11:56 am>[4661035.074237]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     667
     668<Feb/01 11:56 am>[4661035.080668]  [<ffffffff810f1c3b>] ? pipe_read+0x81/0x3d6
     669<Feb/01 11:56 am>[4661035.086560]  [<ffffffff810f94de>] ? core_sys_select+0x3d/0x295
     670<Feb/01 11:56 am>[4661035.092994]  [<ffffffff810f9688>] core_sys_select+0x1e7/0x295
     671<Feb/01 11:56 am>[4661035.099327]  [<ffffffff810f94de>] ? core_sys_select+0x3d/0x295
     672<Feb/01 11:56 am>[4661035.105750]  [<ffffffff810ea38f>] ? do_sync_read+0xc7/0x10d
     673<Feb/01 11:56 am>[4661035.111913]  [<ffffffff8105cc46>] ? ktime_get_ts+0xb1/0xbe
     674<Feb/01 11:56 am>[4661035.117961]  [<ffffffff810f996c>] sys_select+0x94/0xbc
     675<Feb/01 11:56 am>[4661035.123690]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
     676<Feb/01 11:56 am>[4661035.130312] NMI backtrace for cpu 10
     677<Feb/01 11:56 am>[4661035.134378] CPU 10
     678<Feb/01 11:56 am>[4661035.136574] Modules linked in: w83793 hwmon_vid autofs4 i2c_i801 i2c_core iTCO_wdt e1000e ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
     679<Feb/01 11:56 am>[4661035.172716]
     680<Feb/01 11:56 am>[4661035.174647] Pid: 5694, comm: ppStack Tainted: G      D    2.6.34.4 #3 S7010/empty
     681<Feb/01 11:56 am>[4661035.182935] RIP: 0010:[<ffffffff812c08e1>]  [<ffffffff812c08e1>] delay_tsc+0x57/0x59
     682<Feb/01 11:56 am>[4661035.191536] RSP: 0018:ffff88012129b840  EFLAGS: 00000006
     683<Feb/01 11:56 am>[4661035.201907] RAX: 000000000000001c RBX: 0000000024529cb0 RCX: 00000000e4a3215c
     684<Feb/01 11:56 am>[4661035.209852] RDX: 00000000002c28af RSI: 00000000e4a32178 RDI: 0000000000000001
     685<Feb/01 11:56 am>[4661035.217774] RBP: ffff88012129b848 R08: 000000000000000a R09: 000000000000000a
     686<Feb/01 11:56 am>[4661035.225695] R10: 0000000000000000 R11: ffff88012129b678 R12: ffff8806552b3c00
     687<Feb/01 11:56 am>[4661035.233599] R13: 0000000000000000 R14: ffff88052a5140c0 R15: ffff88052a514548
     688<Feb/01 11:56 am>[4661035.241547] FS:  0000000042747950(0063) GS:ffff880655280000(0000) knlGS:0000000000000000
     689<Feb/01 11:56 am>[4661035.250477] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
     690
     691<Feb/01 11:56 am>[4661035.256806] CR2: 00007ff195f55000 CR3: 000000043b231000 CR4: 00000000000006e0
     692<Feb/01 11:56 am>[4661035.264738] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     693<Feb/01 11:56 am>[4661035.272660] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     694<Feb/01 11:56 am>[4661035.280582] Process ppStack (pid: 5694, threadinfo ffff88012129a000, task ffff88052a5140c0)
     695<Feb/01 11:56 am>[4661035.289765] Stack:
     696<Feb/01 11:56 am>[4661035.292215]  ffffffff81019da9 ffff88012129b858 ffffffff812c08fe ffff88012129b898
     697<Feb/01 11:56 am>[4661035.300213] <0> ffffffff812c4e65 000000009ef2dfac ffff8806552b3c00 ffff8806552b3c00
     698<Feb/01 11:56 am>[4661035.308940] <0> ffff88065528e3f0 0000000000000400 0000000000001000 ffff88012129b8b8
     699<Feb/01 11:56 am>[4661035.318043] Call Trace:
     700<Feb/01 11:56 am>[4661035.320941]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     701<Feb/01 11:56 am>[4661035.328763]  [<ffffffff812c08fe>] __delay+0xa/0xc
     702<Feb/01 11:56 am>[4661035.334025]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     703<Feb/01 11:56 am>[4661035.340356]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
     704<Feb/01 11:56 am>[4661035.347065]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76
     705<Feb/01 11:56 am>[4661035.353297]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
     706<Feb/01 11:56 am>[4661035.359361]  [<ffffffff81037672>] load_balance+0xbb6/0xf63
     707<Feb/01 11:56 am>[4661035.365437]  [<ffffffff812b59f1>] ? cfq_set_request+0x313/0x36b
     708<Feb/01 11:56 am>[4661035.371948]  [<ffffffff8151cc0e>] schedule+0x25a/0x86a
     709<Feb/01 11:56 am>[4661035.377639]  [<ffffffff81066393>] futex_wait_queue_me+0xbb/0xd6
     710<Feb/01 11:56 am>[4661035.384151]  [<ffffffff810664f1>] futex_wait+0x143/0x2b1
     711<Feb/01 11:56 am>[4661035.390034]  [<ffffffff8151f67a>] ? _raw_spin_unlock+0x26/0x2b
     712<Feb/01 11:56 am>[4661035.396459]  [<ffffffff8106675f>] ? futex_wake+0x60/0xf2
     713
     714<Feb/01 11:56 am>[4661035.402342]  [<ffffffff810667e0>] ? futex_wake+0xe1/0xf2
     715<Feb/01 11:56 am>[4661035.408244]  [<ffffffff81068312>] do_futex+0x9e/0x9e3
     716<Feb/01 11:56 am>[4661035.413849]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     717<Feb/01 11:56 am>[4661035.420267]  [<ffffffff8102fc7b>] ? cpuacct_charge+0x76/0x96
     718<Feb/01 11:56 am>[4661035.426516]  [<ffffffff810348d8>] ? finish_task_switch+0x69/0xdd
     719<Feb/01 11:56 am>[4661035.433104]  [<ffffffff8151f61f>] ? _raw_spin_unlock_irq+0x26/0x2c
     720<Feb/01 11:56 am>[4661035.439881]  [<ffffffff810348d8>] ? finish_task_switch+0x69/0xdd
     721<Feb/01 11:56 am>[4661035.446476]  [<ffffffff8103486f>] ? finish_task_switch+0x0/0xdd
     722<Feb/01 11:56 am>[4661035.452976]  [<ffffffff8151d152>] ? schedule+0x79e/0x86a
     723<Feb/01 11:56 am>[4661035.458868]  [<ffffffff81068d65>] sys_futex+0x10e/0x12c
     724<Feb/01 11:56 am>[4661035.464665]  [<ffffffff8151ef6f>] ? lockdep_sys_exit_thunk+0x35/0x67
     725<Feb/01 11:56 am>[4661035.471636]  [<ffffffff8151ef6f>] ? lockdep_sys_exit_thunk+0x35/0x67
     726<Feb/01 11:56 am>[4661035.478614]  [<ffffffff8106675f>] ? futex_wake+0x60/0xf2
     727<Feb/01 11:56 am>[4661035.484486]  [<ffffffff8106675f>] ? futex_wake+0x60/0xf2
     728<Feb/01 11:56 am>[4661035.490354]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
     729<Feb/01 11:56 am>[4661035.496976] Code: c8 48 39 f8 73 25 f3 90 65 44 8b 04 25 f0 d1 00 00 45 39 c1 74 db 48 29 f1 48 01 cf 66 66 90 0f ae e8 0f 31 89 c1 45 89 c1 eb c6 <c9> c3 55 48 c7 05 71 4c 80 00 8a 08 2c 81 48 89 e5 c9 c3 55 48
     730<Feb/01 11:56 am>[4661035.520089] Call Trace:
     731<Feb/01 11:56 am>[4661035.523010]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     732<Feb/01 11:56 am>[4661035.530877]  [<ffffffff812c08fe>] __delay+0xa/0xc
     733<Feb/01 11:56 am>[4661035.536114]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     734<Feb/01 11:56 am>[4661035.542436]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
     735<Feb/01 11:56 am>[4661035.549129]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76
     736
     737<Feb/01 11:56 am>[4661035.555366]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
     738<Feb/01 11:56 am>[4661035.561424]  [<ffffffff81037672>] load_balance+0xbb6/0xf63
     739<Feb/01 11:56 am>[4661035.567508]  [<ffffffff812b59f1>] ? cfq_set_request+0x313/0x36b
     740<Feb/01 11:56 am>[4661035.574011]  [<ffffffff8151cc0e>] schedule+0x25a/0x86a
     741<Feb/01 11:56 am>[4661035.579711]  [<ffffffff81066393>] futex_wait_queue_me+0xbb/0xd6
     742<Feb/01 11:56 am>[4661035.586212]  [<ffffffff810664f1>] futex_wait+0x143/0x2b1
     743<Feb/01 11:56 am>[4661035.592088]  [<ffffffff8151f67a>] ? _raw_spin_unlock+0x26/0x2b
     744                 <Feb/01 11:56 am>[4661035.598521]  [<ffffffff8106675f>] ? futex_wake+0x60/0xf2
     745<Feb/01 11:56 am>[4661035.604412]  [<ffffffff810667e0>] ? futex_wake+0xe1/0xf2
     746<Feb/01 11:56 am>[4661035.610280]  [<ffffffff81068312>] do_futex+0x9e/0x9e3
     747<Feb/01 11:56 am>[4661035.615886]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     748<Feb/01 11:56 am>[4661035.622318]  [<ffffffff8102fc7b>] ? cpuacct_charge+0x76/0x96
     749<Feb/01 11:56 am>[4661035.628542]  [<ffffffff810348d8>] ? finish_task_switch+0x69/0xdd
     750<Feb/01 11:56 am>[4661035.635147]  [<ffffffff8151f61f>] ? _raw_spin_unlock_irq+0x26/0x2c
     751                 <Feb/01 11:56 am>[4661035.641926]  [<ffffffff810348d8>] ? finish_task_switch+0x69/0xdd
     752<Feb/01 11:56 am>[4661035.648520]  [<ffffffff8103486f>] ? finish_task_switch+0x0/0xdd
     753<Feb/01 11:56 am>[4661035.655037]  [<ffffffff8151d152>] ? schedule+0x79e/0x86a
     754<Feb/01 11:56 am>[4661035.660929]  [<ffffffff81068d65>] sys_futex+0x10e/0x12c
     755<Feb/01 11:56 am>[4661035.666710]  [<ffffffff8151ef6f>] ? lockdep_sys_exit_thunk+0x35/0x67
     756<Feb/01 11:56 am>[4661035.673689]  [<ffffffff8151ef6f>] ? lockdep_sys_exit_thunk+0x35/0x67
     757<Feb/01 11:56 am>[4661035.680632]  [<ffffffff8106675f>] ? futex_wake+0x60/0xf2
     758<Feb/01 11:56 am>[4661035.686531]  [<ffffffff8106675f>] ? futex_wake+0x60/0xf2
     759
     760<Feb/01 11:56 am>[4661035.692417]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
     761<Feb/01 11:56 am>[4661035.699024] Pid: 5694, comm: ppStack Tainted: G      D    2.6.34.4 #3
     762<Feb/01 11:56 am>[4661035.706095] Call Trace:
     763<Feb/01 11:56 am>[4661035.709020]  <NMI>  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
     764<Feb/01 11:56 am>[4661035.715499]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
     765<Feb/01 11:56 am>[4661035.721929]  [<ffffffff81520360>] do_nmi+0xb2/0x259
     766<Feb/01 11:56 am>[4661035.727368]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
     767<Feb/01 11:56 am>[4661035.732447]  [<ffffffff812c08e1>] ? delay_tsc+0x57/0x59
     768<Feb/01 11:56 am>[4661035.738246]  <<EOE>>  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     769<Feb/01 11:56 am>[4661035.746975]  [<ffffffff812c08fe>] __delay+0xa/0xc
     770<Feb/01 11:56 am>[4661035.752243]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     771<Feb/01 11:56 am>[4661035.758572]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
     772<Feb/01 11:56 am>[4661035.765258]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76
     773<Feb/01 11:56 am>[4661035.771479]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
     774<Feb/01 11:56 am>[4661035.777544]  [<ffffffff81037672>] load_balance+0xbb6/0xf63
     775<Feb/01 11:56 am>[4661035.783621]  [<ffffffff812b59f1>] ? cfq_set_request+0x313/0x36b
     776<Feb/01 11:56 am>[4661035.790131]  [<ffffffff8151cc0e>] schedule+0x25a/0x86a
     777<Feb/01 11:56 am>[4661035.795856]  [<ffffffff81066393>] futex_wait_queue_me+0xbb/0xd6
     778<Feb/01 11:56 am>[4661035.802341]  [<ffffffff810664f1>] futex_wait+0x143/0x2b1
     779<Feb/01 11:56 am>[4661035.808215]  [<ffffffff8151f67a>] ? _raw_spin_unlock+0x26/0x2b
     780<Feb/01 11:56 am>[4661035.814620]  [<ffffffff8106675f>] ? futex_wake+0x60/0xf2
     781<Feb/01 11:56 am>[4661035.820489]  [<ffffffff810667e0>] ? futex_wake+0xe1/0xf2
     782
     783<Feb/01 11:56 am>[4661035.826390]  [<ffffffff81068312>] do_futex+0x9e/0x9e3
     784<Feb/01 11:56 am>[4661035.832022]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     785<Feb/01 11:56 am>[4661035.838430]  [<ffffffff8102fc7b>] ? cpuacct_charge+0x76/0x96
     786<Feb/01 11:56 am>[4661035.844670]  [<ffffffff810348d8>] ? finish_task_switch+0x69/0xdd
     787<Feb/01 11:56 am>[4661035.851285]  [<ffffffff8151f61f>] ? _raw_spin_unlock_irq+0x26/0x2c
     788<Feb/01 11:56 am>[4661035.858054]  [<ffffffff810348d8>] ? finish_task_switch+0x69/0xdd
     789<Feb/01 11:56 am>[4661035.864674]  [<ffffffff8103486f>] ? finish_task_switch+0x0/0xdd
     790<Feb/01 11:56 am>[4661035.871165]  [<ffffffff8151d152>] ? schedule+0x79e/0x86a
     791<Feb/01 11:56 am>[4661035.877024]  [<ffffffff81068d65>] sys_futex+0x10e/0x12c
     792<Feb/01 11:56 am>[4661035.882829]  [<ffffffff8151ef6f>] ? lockdep_sys_exit_thunk+0x35/0x67
     793<Feb/01 11:56 am>[4661035.889793]  [<ffffffff8151ef6f>] ? lockdep_sys_exit_thunk+0x35/0x67
     794<Feb/01 11:56 am>[4661035.896770]  [<ffffffff8106675f>] ? futex_wake+0x60/0xf2
     795<Feb/01 11:56 am>[4661035.902660]  [<ffffffff8106675f>] ? futex_wake+0x60/0xf2
     796<Feb/01 11:56 am>[4661035.908536]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
     797<Feb/01 11:56 am>[4661035.915141] NMI backtrace for cpu 11
     798<Feb/01 11:56 am>[4661035.919217] CPU 11
     799<Feb/01 11:56 am>[4661035.921412] Modules linked in: w83793 hwmon_vid autofs4 i2c_i801 i2c_core iTCO_wdt e1000e ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c10<Feb/01 11:56 am>
     800                 <Feb/01 11:56 am>[4661035.957511]
     801<Feb/01 11:56 am>[4661035.959449] Pid: 8467, comm: nfsd Tainted: G      D    2.6.34.4 #3 S7010/empty
     802<Feb/01 11:56 am>[4661035.967493] RIP: 0010:[<ffffffff812c08ad>]  [<ffffffff812c08ad>] delay_tsc+0x23/0x59
     803<Feb/01 11:56 am>[4661035.976104] RSP: 0018:ffff88061def13e8  EFLAGS: 00000096
     804<Feb/01 11:56 am>[4661035.982006] RAX: 00000000e4a73418 RBX: 000000002453a7ea RCX: 00000000e4a73418
     805
     806<Feb/01 11:56 am>[4661035.989950] RDX: 00000000002c28af RSI: 000000000000000b RDI: 0000000000000001
     807<Feb/01 11:56 am>[4661035.997897] RBP: ffff88061def13f0 R08: 000000000000000b R09: 000000000000000b
     808<Feb/01 11:56 am>[4661036.005819] R10: 0000000000000000 R11: ffff88061def1220 R12: ffff880002673c00
     809<Feb/01 11:56 am>[4661036.013768] R13: 0000000000000000 R14: ffff88061e3d3090 R15: ffff88061e3d3518
     810<Feb/01 11:56 am>[4661036.021732] FS:  0000000000000000(0000) GS:ffff8806552a0000(0000) knlGS:0000000000000000
     811<Feb/01 11:56 am>[4661036.030662] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
     812<Feb/01 11:56 am>[4661036.036998] CR2: 00007f53a10add5c CR3: 0000000001a8a000 CR4: 00000000000006e0
     813<Feb/01 11:56 am>[4661036.044950] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     814<Feb/01 11:56 am>[4661036.052889] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     815<Feb/01 11:56 am>[4661036.060845] Process nfsd (pid: 8467, threadinfo ffff88061def0000, task ffff88061e3d3090)
     816<Feb/01 11:56 am>[4661036.069749] Stack:
     817<Feb/01 11:56 am>[4661036.072215]  ffffffff81019da9 ffff88061def1400 ffffffff812c08fe ffff88061def1440
     818<Feb/01 11:56 am>[4661036.080240] <0> ffffffff812c4e65 000000009ef2dfac ffff880002673c00 ffff880623bb2060
     819<Feb/01 11:56 am>[4661036.088951] <0> 0000000000013c00 ffff88061def14b0 000000000000000b ffff88061def1460
     820<Feb/01 11:56 am>[4661036.098055] Call Trace:
     821<Feb/01 11:56 am>[4661036.100976]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     822<Feb/01 11:56 am>[4661036.108843]  [<ffffffff812c08fe>] __delay+0xa/0xc
     823<Feb/01 11:56 am>[4661036.114106]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     824<Feb/01 11:56 am>[4661036.120428]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     825<Feb/01 11:56 am>[4661036.126511]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
     826<Feb/01 11:56 am>[4661036.132568]  [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
     827<Feb/01 11:56 am>[4661036.138419]  [<ffffffff81036578>] try_to_wake_up+0x2e/0x342
     828
     829<Feb/01 11:56 am>[4661036.144547]  [<ffffffff810368b8>] wake_up_process+0x10/0x12
     830<Feb/01 11:56 am>[4661036.150719]  [<ffffffff8110649a>] bdi_queue_work+0xa2/0xb9
     831<Feb/01 11:56 am>[4661036.156784]  [<ffffffff811068ae>] bdi_alloc_queue_work+0xa4/0xc4
     832<Feb/01 11:56 am>[4661036.167888]  [<ffffffff81106a1f>] wakeup_flusher_threads+0x93/0xc6
     833<Feb/01 11:56 am>[4661036.174649]  [<ffffffff811069bf>] ? wakeup_flusher_threads+0x33/0xc6
     834<Feb/01 11:56 am>[4661036.181600]  [<ffffffff810b9854>] do_try_to_free_pages+0x299/0x37a
     835<Feb/01 11:56 am>[4661036.188369]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
     836<Feb/01 11:56 am>[4661036.194705]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
     837<Feb/01 11:56 am>[4661036.201491]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
     838<Feb/01 11:56 am>[4661036.208426]  [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
     839<Feb/01 11:56 am>[4661036.214932]  [<ffffffff810acb60>] __page_cache_alloc+0x59/0x5e
     840<Feb/01 11:56 am>[4661036.221338]  [<ffffffff810b4860>] __do_page_cache_readahead+0xe5/0x1fc
     841<Feb/01 11:56 am>[4661036.228481]  [<ffffffff810b47f6>] ? __do_page_cache_readahead+0x7b/0x1fc
     842<Feb/01 11:56 am>[4661036.235779]  [<ffffffff812bc23c>] ? radix_tree_prev_hole+0x30/0x53
     843<Feb/01 11:56 am>[4661036.242547]  [<ffffffff810b4c40>] ondemand_readahead+0x1cc/0x256
     844<Feb/01 11:56 am>[4661036.249167]  [<ffffffff810b4ba0>] ? ondemand_readahead+0x12c/0x256
     845<Feb/01 11:56 am>[4661036.255929]  [<ffffffff810b3c7a>] ? determine_dirtyable_memory+0x15/0x27
     846<Feb/01 11:56 am>[4661036.263254]  [<ffffffff810b4daa>] page_cache_sync_readahead+0x38/0x3a
     847<Feb/01 11:56 am>[4661036.270290]  [<ffffffff81109bf2>] __generic_file_splice_read+0xfa/0x41c
     848<Feb/01 11:56 am>[4661036.277519]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     849<Feb/01 11:56 am>[4661036.283956]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     850<Feb/01 11:56 am>[4661036.290301]  [<ffffffff812615b1>] ? xfs_ilock+0x49/0xb2
     851
     852<Feb/01 11:56 am>[4661036.296080]  [<ffffffff811086db>] ? spd_release_page+0x0/0x14
     853<Feb/01 11:56 am>[4661036.302416]  [<ffffffff81109f57>] generic_file_splice_read+0x43/0x74
     854<Feb/01 11:56 am>[4661036.309396]  [<ffffffff8128191a>] xfs_file_splice_read+0xde/0x125
     855<Feb/01 11:56 am>[4661036.316094]  [<ffffffff8110875d>] vfs_splice_to+0x6e/0x81
     856                 <Feb/01 11:56 am>[4661036.322072]  [<ffffffff81108837>] splice_direct_to_actor+0xc7/0x19c
     857<Feb/01 11:56 am>[4661036.328964]  [<ffffffff811f562c>] ? nfsd_direct_splice_actor+0x0/0x12
     858<Feb/01 11:56 am>[4661036.336012]  [<ffffffff811f54ef>] nfsd_vfs_read+0x263/0x3a0
     859<Feb/01 11:56 am>[4661036.342171]  [<ffffffff81281760>] ? xfs_file_open+0x0/0x3f
     860<Feb/01 11:56 am>[4661036.348256]  [<ffffffff811f5c4e>] nfsd_read+0xa7/0xc3
     861<Feb/01 11:56 am>[4661036.353887]  [<ffffffff811fcd79>] nfsd3_proc_read+0xfe/0x142
     862<Feb/01 11:56 am>[4661036.360112]  [<ffffffff811f0854>] nfsd_dispatch+0xe9/0x1ca
     863<Feb/01 11:56 am>[4661036.366179]  [<ffffffff814f04c5>] svc_process+0x425/0x72a
     864<Feb/01 11:56 am>[4661036.372140]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
     865<Feb/01 11:56 am>[4661036.377478]  [<ffffffff811f0dc2>] nfsd+0xf3/0x139
     866<Feb/01 11:56 am>[4661036.382721]  [<ffffffff810546e2>] kthread+0x7d/0x85
     867<Feb/01 11:56 am>[4661036.388164]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     868<Feb/01 11:56 am>[4661036.394694]  [<ffffffff81054665>] ? kthread+0x0/0x85
     869<Feb/01 11:56 am>[4661036.400211]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     870<Feb/01 11:56 am>[4661036.406898] Code: c8 75 fb 48 ff c8 c9 c3 55 48 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae e8 0f 31 89 c1 41 89 f1 66 66 90 0f ae e8 <0f> 31 89 c6 48 89 f0 48 29 c8 48 39 f8 73 25 f3 90 65 44 8b 04
     871<Feb/01 11:56 am>[4661036.429990] Call Trace:
     872<Feb/01 11:56 am>[4661036.432904]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     873<Feb/01 11:56 am>[4661036.440771]  [<ffffffff812c08fe>] __delay+0xa/0xc
     874
     875<Feb/01 11:56 am>[4661036.446041]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     876<Feb/01 11:56 am>[4661036.452365]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     877<Feb/01 11:56 am>[4661036.458438]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
     878<Feb/01 11:56 am>[4661036.464480]  [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
     879<Feb/01 11:56 am>[4661036.470356]  [<ffffffff81036578>] try_to_wake_up+0x2e/0x342
     880<Feb/01 11:56 am>[4661036.476528]  [<ffffffff810368b8>] wake_up_process+0x10/0x12
     881<Feb/01 11:56 am>[4661036.482680]  [<ffffffff8110649a>] bdi_queue_work+0xa2/0xb9
     882<Feb/01 11:56 am>[4661036.488730]  [<ffffffff811068ae>] bdi_alloc_queue_work+0xa4/0xc4
     883<Feb/01 11:56 am>[4661036.495301]  [<ffffffff81106a1f>] wakeup_flusher_threads+0x93/0xc6
     884<Feb/01 11:56 am>[4661036.502097]  [<ffffffff811069bf>] ? wakeup_flusher_threads+0x33/0xc6
     885<Feb/01 11:56 am>[4661036.509074]  [<ffffffff810b9854>] do_try_to_free_pages+0x299/0x37a
     886<Feb/01 11:56 am>[4661036.515859]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
     887<Feb/01 11:56 am>[4661036.522202]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
     888<Feb/01 11:56 am>[4661036.528990]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
     889<Feb/01 11:56 am>[4661036.535968]  [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
     890<Feb/01 11:56 am>[4661036.542483]  [<ffffffff810acb60>] __page_cache_alloc+0x59/0x5e
     891<Feb/01 11:56 am>[4661036.548898]  [<ffffffff810b4860>] __do_page_cache_readahead+0xe5/0x1fc
     892<Feb/01 11:56 am>[4661036.556041]  [<ffffffff810b47f6>] ? __do_page_cache_readahead+0x7b/0x1fc
     893<Feb/01 11:56 am>[4661036.563358]  [<ffffffff812bc23c>] ? radix_tree_prev_hole+0x30/0x53
     894<Feb/01 11:56 am>[4661036.570142]  [<ffffffff810b4c40>] ondemand_readahead+0x1cc/0x256
     895<Feb/01 11:56 am>[4661036.576736]  [<ffffffff810b4ba0>] ? ondemand_readahead+0x12c/0x256
     896<Feb/01 11:56 am>[4661036.583523]  [<ffffffff810b3c7a>] ? determine_dirtyable_memory+0x15/0x27
     897
     898<Feb/01 11:56 am>[4661036.590839]  [<ffffffff810b4daa>] page_cache_sync_readahead+0x38/0x3a
     899<Feb/01 11:56 am>[4661036.597885]  [<ffffffff81109bf2>] __generic_file_splice_read+0xfa/0x41c
     900<Feb/01 11:56 am>[4661036.605131]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     901<Feb/01 11:56 am>[4661036.611551]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     902<Feb/01 11:56 am>[4661036.617974]  [<ffffffff812615b1>] ? xfs_ilock+0x49/0xb2
     903<Feb/01 11:56 am>[4661036.623769]  [<ffffffff811086db>] ? spd_release_page+0x0/0x14
     904<Feb/01 11:56 am>[4661036.630097]  [<ffffffff81109f57>] generic_file_splice_read+0x43/0x74
     905<Feb/01 11:56 am>[4661036.637069]  [<ffffffff8128191a>] xfs_file_splice_read+0xde/0x125
     906<Feb/01 11:56 am>[4661036.643757]  [<ffffffff8110875d>] vfs_splice_to+0x6e/0x81
     907                 <Feb/01 11:56 am>[4661036.649728]  [<ffffffff81108837>] splice_direct_to_actor+0xc7/0x19c
     908<Feb/01 11:56 am>[4661036.656585]  [<ffffffff811f562c>] ? nfsd_direct_splice_actor+0x0/0x12
     909<Feb/01 11:56 am>[4661036.663643]  [<ffffffff811f54ef>] nfsd_vfs_read+0x263/0x3a0
     910<Feb/01 11:56 am>[4661036.669783]  [<ffffffff81281760>] ? xfs_file_open+0x0/0x3f
     911<Feb/01 11:56 am>[4661036.675849]  [<ffffffff811f5c4e>] nfsd_read+0xa7/0xc3
     912<Feb/01 11:56 am>[4661036.681473]  [<ffffffff811fcd79>] nfsd3_proc_read+0xfe/0x142
     913<Feb/01 11:56 am>[4661036.687714]  [<ffffffff811f0854>] nfsd_dispatch+0xe9/0x1ca
     914<Feb/01 11:56 am>[4661036.693781]  [<ffffffff814f04c5>] svc_process+0x425/0x72a
     915<Feb/01 11:56 am>[4661036.699762]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
     916<Feb/01 11:56 am>[4661036.705107]  [<ffffffff811f0dc2>] nfsd+0xf3/0x139
     917<Feb/01 11:56 am>[4661036.710360]  [<ffffffff810546e2>] kthread+0x7d/0x85
     918<Feb/01 11:56 am>[4661036.715776]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     919<Feb/01 11:56 am>[4661036.722254]  [<ffffffff81054665>] ? kthread+0x0/0x85
     920
     921<Feb/01 11:56 am>[4661036.727754]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     922<Feb/01 11:56 am>[4661036.734440] Pid: 8467, comm: nfsd Tainted: G      D    2.6.34.4 #3
     923<Feb/01 11:56 am>[4661036.741226] Call Trace:
     924<Feb/01 11:56 am>[4661036.744159]  <NMI>  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
     925<Feb/01 11:56 am>[4661036.750622]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
     926<Feb/01 11:56 am>[4661036.757026]  [<ffffffff81520360>] do_nmi+0xb2/0x259
     927<Feb/01 11:56 am>[4661036.762457]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
     928<Feb/01 11:56 am>[4661036.767526]  [<ffffffff812c08ad>] ? delay_tsc+0x23/0x59
     929<Feb/01 11:56 am>[4661036.773325]  <<EOE>>  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     930<Feb/01 11:56 am>[4661036.782038]  [<ffffffff812c08fe>] __delay+0xa/0xc
     931<Feb/01 11:56 am>[4661036.787304]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     932<Feb/01 11:56 am>[4661036.793600]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     933<Feb/01 11:56 am>[4661036.799657]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
     934<Feb/01 11:56 am>[4661036.805733]  [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
     935<Feb/01 11:56 am>[4661036.811618]  [<ffffffff81036578>] try_to_wake_up+0x2e/0x342
     936<Feb/01 11:56 am>[4661036.817779]  [<ffffffff810368b8>] wake_up_process+0x10/0x12
     937<Feb/01 11:56 am>[4661036.823933]  [<ffffffff8110649a>] bdi_queue_work+0xa2/0xb9
     938<Feb/01 11:56 am>[4661036.829992]  [<ffffffff811068ae>] bdi_alloc_queue_work+0xa4/0xc4
     939<Feb/01 11:56 am>[4661036.836589]  [<ffffffff81106a1f>] wakeup_flusher_threads+0x93/0xc6
     940<Feb/01 11:56 am>[4661036.843376]  [<ffffffff811069bf>] ? wakeup_flusher_threads+0x33/0xc6
     941<Feb/01 11:56 am>[4661036.850327]  [<ffffffff810b9854>] do_try_to_free_pages+0x299/0x37a
     942<Feb/01 11:56 am>[4661036.857104]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
     943
     944<Feb/01 11:56 am>[4661036.863429]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
     945<Feb/01 11:56 am>[4661036.870201]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
     946<Feb/01 11:56 am>[4661036.877161]  [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
     947<Feb/01 11:56 am>[4661036.883669]  [<ffffffff810acb60>] __page_cache_alloc+0x59/0x5e
     948<Feb/01 11:56 am>[4661036.890064]  [<ffffffff810b4860>] __do_page_cache_readahead+0xe5/0x1fc
     949<Feb/01 11:56 am>[4661036.897191]  [<ffffffff810b47f6>] ? __do_page_cache_readahead+0x7b/0x1fc
     950<Feb/01 11:56 am>[4661036.904480]  [<ffffffff812bc23c>] ? radix_tree_prev_hole+0x30/0x53
     951<Feb/01 11:56 am>[4661036.911255]  [<ffffffff810b4c40>] ondemand_readahead+0x1cc/0x256
     952<Feb/01 11:56 am>[4661036.917868]  [<ffffffff810b4ba0>] ? ondemand_readahead+0x12c/0x256
     953<Feb/01 11:56 am>[4661036.924639]  [<ffffffff810b3c7a>] ? determine_dirtyable_memory+0x15/0x27
     954<Feb/01 11:56 am>[4661036.931945]  [<ffffffff810b4daa>] page_cache_sync_readahead+0x38/0x3a
     955<Feb/01 11:56 am>[4661036.939017]  [<ffffffff81109bf2>] __generic_file_splice_read+0xfa/0x41c
     956<Feb/01 11:56 am>[4661036.946221]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     957<Feb/01 11:56 am>[4661036.952631]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     958<Feb/01 11:56 am>[4661036.959070]  [<ffffffff812615b1>] ? xfs_ilock+0x49/0xb2
     959<Feb/01 11:56 am>[4661036.964884]  [<ffffffff811086db>] ? spd_release_page+0x0/0x14
     960<Feb/01 11:56 am>[4661036.971231]  [<ffffffff81109f57>] generic_file_splice_read+0x43/0x74
     961<Feb/01 11:56 am>[4661036.978201]  [<ffffffff8128191a>] xfs_file_splice_read+0xde/0x125
     962<Feb/01 11:56 am>[4661036.984881]  [<ffffffff8110875d>] vfs_splice_to+0x6e/0x81
     963<Feb/01 11:56 am>[4661036.990841]  [<ffffffff81108837>] splice_direct_to_actor+0xc7/0x19c
     964<Feb/01 11:56 am>[4661036.997708]  [<ffffffff811f562c>] ? nfsd_direct_splice_actor+0x0/0x12
     965<Feb/01 11:56 am>[4661037.004772]  [<ffffffff811f54ef>] nfsd_vfs_read+0x263/0x3a0
     966
     967<Feb/01 11:56 am>[4661037.010906]  [<ffffffff81281760>] ? xfs_file_open+0x0/0x3f
     968<Feb/01 11:56 am>[4661037.016956]  [<ffffffff811f5c4e>] nfsd_read+0xa7/0xc3
     969<Feb/01 11:56 am>[4661037.022588]  [<ffffffff811fcd79>] nfsd3_proc_read+0xfe/0x142
     970<Feb/01 11:56 am>[4661037.028805]  [<ffffffff811f0854>] nfsd_dispatch+0xe9/0x1ca
     971<Feb/01 11:56 am>[4661037.034879]  [<ffffffff814f04c5>] svc_process+0x425/0x72a
     972<Feb/01 11:56 am>[4661037.040851]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
     973<Feb/01 11:56 am>[4661037.046212]  [<ffffffff811f0dc2>] nfsd+0xf3/0x139
     974<Feb/01 11:56 am>[4661037.051466]  [<ffffffff810546e2>] kthread+0x7d/0x85
     975<Feb/01 11:56 am>[4661037.056882]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     976<Feb/01 11:56 am>[4661037.063361]  [<ffffffff81054665>] ? kthread+0x0/0x85
     977<Feb/01 11:56 am>[4661037.068860]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     978<Feb/01 11:56 am>[4661037.075546] NMI backtrace for cpu 6
     979<Feb/01 11:56 am>[4661037.079550] CPU 6
     980<Feb/01 11:56 am>[4661037.081651] Modules linked in: w83793 hwmon_vid autofs4 i2c_i801 i2c_core iTCO_wdt e1000e ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
     981<Feb/01 11:56 am>[4661037.117750]
     982<Feb/01 11:56 am>[4661037.119672] Pid: 0, comm: swapper Tainted: G      D    2.6.34.4 #3 S7010/empty
     983<Feb/01 11:56 am>[4661037.127716] RIP: 0010:[<ffffffff812c08af>]  [<ffffffff812c08af>] delay_tsc+0x25/0x59
     984<Feb/01 11:56 am>[4661037.136360] RSP: 0018:ffff880655203b58  EFLAGS: 00000092
     985<Feb/01 11:56 am>[4661037.142245] RAX: 00000000e492d634 RBX: 0000000024520b3e RCX: 00000000e492d618
     986<Feb/01 11:56 am>[4661037.150181] RDX: 00000000002c28af RSI: 0000000000000006 RDI: 0000000000000001
     987<Feb/01 11:56 am>[4661037.158119] RBP: ffff880655203b60 R08: 0000000000000006 R09: 0000000000000006
     988<Feb/01 11:56 am>[4661037.166041] R10: 0000000000000000 R11: ffff880655203990 R12: ffff8806552b3c00
     989
     990<Feb/01 11:56 am>[4661037.173989] R13: 0000000000000000 R14: ffff880c24c88000 R15: ffff880c24c88488
     991<Feb/01 11:56 am>[4661037.181928] FS:  0000000000000000(0000) GS:ffff880655200000(0000) knlGS:0000000000000000
     992<Feb/01 11:56 am>[4661037.190867] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
     993<Feb/01 11:56 am>[4661037.197203] CR2: 000000001d9921e0 CR3: 0000000001a8a000 CR4: 00000000000006e0
     994<Feb/01 11:56 am>[4661037.205165] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     995<Feb/01 11:56 am>[4661037.213121] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     996<Feb/01 11:56 am>[4661037.221085] Process swapper (pid: 0, threadinfo ffff880624d28000, task ffff880c24c88000)
     997<Feb/01 11:56 am>[4661037.229996] Stack:
     998<Feb/01 11:56 am>[4661037.232473]  ffffffff81019da9 ffff880655203b70 ffffffff812c08fe ffff880655203bb0
     999<Feb/01 11:56 am>[4661037.244996] <0> ffffffff812c4e65 000000009ef2dfac ffff8806552b3c00 ffff8806552b3c00
     1000<Feb/01 11:56 am>[4661037.253714] <0> ffff88065520e3f0 00000000000004aa 0000000000001000 ffff880655203bd0
     1001<Feb/01 11:56 am>[4661037.262836] Call Trace:
     1002<Feb/01 11:56 am>[4661037.265756]  <IRQ>
     1003<Feb/01 11:56 am>[4661037.268357]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     1004<Feb/01 11:56 am>[4661037.276189]  [<ffffffff812c08fe>] __delay+0xa/0xc
     1005<Feb/01 11:56 am>[4661037.281436]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     1006<Feb/01 11:56 am>[4661037.287766]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
     1007<Feb/01 11:56 am>[4661037.294449]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76
     1008<Feb/01 11:56 am>[4661037.300687]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
     1009<Feb/01 11:56 am>[4661037.306761]  [<ffffffff81037672>] load_balance+0xbb6/0xf63
     1010<Feb/01 11:56 am>[4661037.312812]  [<ffffffff81361d5e>] ? scsi_request_fn+0x39d/0x459
     1011<Feb/01 11:56 am>[4661037.319321]  [<ffffffff8151f115>] ? _raw_spin_lock_irq+0x35/0x39
     1012
     1013<Feb/01 11:56 am>[4661037.325944]  [<ffffffff81361d5e>] ? scsi_request_fn+0x39d/0x459
     1014<Feb/01 11:56 am>[4661037.332426]  [<ffffffff81047c8b>] ? run_timer_softirq+0x4d/0x30c
     1015<Feb/01 11:56 am>[4661037.339020]  [<ffffffff81037af9>] rebalance_domains+0xda/0x150
     1016<Feb/01 11:56 am>[4661037.345436]  [<ffffffff81037bb2>] run_rebalance_domains+0x43/0xcf
     1017<Feb/01 11:56 am>[4661037.352144]  [<ffffffff8104103c>] __do_softirq+0xd8/0x194
     1018<Feb/01 11:56 am>[4661037.358122]  [<ffffffff8100388c>] call_softirq+0x1c/0x28
     1019<Feb/01 11:56 am>[4661037.364023]  [<ffffffff81004e7f>] do_softirq+0x33/0x68
     1020<Feb/01 11:56 am>[4661037.369708]  [<ffffffff81040ec2>] irq_exit+0x36/0x75
     1021<Feb/01 11:56 am>[4661037.375238]  [<ffffffff810197aa>] smp_apic_timer_interrupt+0x87/0x95
     1022<Feb/01 11:56 am>[4661037.382176]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
     1023<Feb/01 11:56 am>[4661037.388795]  <EOI>
     1024<Feb/01 11:56 am>[4661037.391391]  [<ffffffff8151cc4a>] ? schedule+0x296/0x86a
     1025<Feb/01 11:56 am>[4661037.397289]  [<ffffffff81303870>] ? acpi_idle_enter_c1+0xdd/0xf8
     1026<Feb/01 11:56 am>[4661037.403911]  [<ffffffff81303851>] ? acpi_idle_enter_c1+0xbe/0xf8
     1027<Feb/01 11:56 am>[4661037.410533]  [<ffffffff81430151>] cpuidle_idle_call+0x95/0xf0
     1028<Feb/01 11:56 am>[4661037.416841]  [<ffffffff81001d4b>] cpu_idle+0x55/0x8d
     1029<Feb/01 11:56 am>[4661037.422351]  [<ffffffff815191ca>] start_secondary+0x1ae/0x1b2
     1030<Feb/01 11:56 am>[4661037.428680] Code: fb 48 ff c8 c9 c3 55 48 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae e8 0f 31 89 c1 41 89 f1 66 66 90 0f ae e8 0f 31 <89> c6 48 89 f0 48 29 c8 48 39 f8 73 25 f3 90 65 44 8b 04 25 f0
     1031<Feb/01 11:56 am>[4661037.451861] Call Trace:
     1032<Feb/01 11:56 am>[4661037.454773]  <IRQ>  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     1033<Feb/01 11:56 am>[4661037.463296]  [<ffffffff812c08fe>] __delay+0xa/0xc
     1034<Feb/01 11:56 am>[4661037.468537]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     1035
     1036<Feb/01 11:56 am>[4661037.474824]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
     1037<Feb/01 11:56 am>[4661037.481535]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76
     1038<Feb/01 11:56 am>[4661037.487764]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
     1039<Feb/01 11:56 am>[4661037.493830]  [<ffffffff81037672>] load_balance+0xbb6/0xf63
     1040<Feb/01 11:56 am>[4661037.499889]  [<ffffffff81361d5e>] ? scsi_request_fn+0x39d/0x459
     1041<Feb/01 11:56 am>[4661037.506408]  [<ffffffff8151f115>] ? _raw_spin_lock_irq+0x35/0x39
     1042<Feb/01 11:56 am>[4661037.513004]  [<ffffffff81361d5e>] ? scsi_request_fn+0x39d/0x459
     1043<Feb/01 11:56 am>[4661037.519531]  [<ffffffff81047c8b>] ? run_timer_softirq+0x4d/0x30c
     1044                 <Feb/01 11:56 am>[4661037.526126]  [<ffffffff81037af9>] rebalance_domains+0xda/0x150
     1045<Feb/01 11:56 am>[4661037.532539]  [<ffffffff81037bb2>] run_rebalance_domains+0x43/0xcf
     1046<Feb/01 11:56 am>[4661037.539221]  [<ffffffff8104103c>] __do_softirq+0xd8/0x194
     1047<Feb/01 11:56 am>[4661037.545207]  [<ffffffff8100388c>] call_softirq+0x1c/0x28
     1048<Feb/01 11:56 am>[4661037.551092]  [<ffffffff81004e7f>] do_softirq+0x33/0x68
     1049<Feb/01 11:56 am>[4661037.556794]  [<ffffffff81040ec2>] irq_exit+0x36/0x75
     1050<Feb/01 11:56 am>[4661037.562298]  [<ffffffff810197aa>] smp_apic_timer_interrupt+0x87/0x95
     1051<Feb/01 11:56 am>[4661037.569244]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
     1052<Feb/01 11:56 am>[4661037.575848]  <EOI>  [<ffffffff8151cc4a>] ? schedule+0x296/0x86a
     1053<Feb/01 11:56 am>[4661037.582419]  [<ffffffff81303870>] ? acpi_idle_enter_c1+0xdd/0xf8
     1054<Feb/01 11:56 am>[4661037.589013]  [<ffffffff81303851>] ? acpi_idle_enter_c1+0xbe/0xf8
     1055<Feb/01 11:56 am>[4661037.595634]  [<ffffffff81430151>] cpuidle_idle_call+0x95/0xf0
     1056<Feb/01 11:56 am>[4661037.601951]  [<ffffffff81001d4b>] cpu_idle+0x55/0x8d
     1057<Feb/01 11:56 am>[4661037.607445]  [<ffffffff815191ca>] start_secondary+0x1ae/0x1b2
     1058
     1059<Feb/01 11:56 am>[4661037.613791] Pid: 0, comm: swapper Tainted: G      D    2.6.34.4 #3
     1060<Feb/01 11:56 am>[4661037.620578] Call Trace:
     1061<Feb/01 11:56 am>[4661037.623503]  <NMI>  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
     1062                 <Feb/01 11:56 am>[4661037.629965]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
     1063<Feb/01 11:56 am>[4661037.636394]  [<ffffffff81520360>] do_nmi+0xb2/0x259
     1064<Feb/01 11:56 am>[4661037.641808]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
     1065<Feb/01 11:56 am>[4661037.646895]  [<ffffffff812c08af>] ? delay_tsc+0x25/0x59
     1066<Feb/01 11:56 am>[4661037.652704]  <<EOE>>  <IRQ>  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     1067<Feb/01 11:56 am>[4661037.662091]  [<ffffffff812c08fe>] __delay+0xa/0xc
     1068<Feb/01 11:56 am>[4661037.667357]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     1069<Feb/01 11:56 am>[4661037.673637]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
     1070<Feb/01 11:56 am>[4661037.680338]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76
     1071<Feb/01 11:56 am>[4661037.686592]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
     1072<Feb/01 11:56 am>[4661037.692668]  [<ffffffff81037672>] load_balance+0xbb6/0xf63
     1073<Feb/01 11:56 am>[4661037.698726]  [<ffffffff81361d5e>] ? scsi_request_fn+0x39d/0x459
     1074<Feb/01 11:56 am>[4661037.705245]  [<ffffffff8151f115>] ? _raw_spin_lock_irq+0x35/0x39
     1075<Feb/01 11:56 am>[4661037.711860]  [<ffffffff81361d5e>] ? scsi_request_fn+0x39d/0x459
     1076<Feb/01 11:56 am>[4661037.718350]  [<ffffffff81047c8b>] ? run_timer_softirq+0x4d/0x30c
     1077<Feb/01 11:56 am>[4661037.724945]  [<ffffffff81037af9>] rebalance_domains+0xda/0x150
     1078<Feb/01 11:56 am>[4661037.731359]  [<ffffffff81037bb2>] run_rebalance_domains+0x43/0xcf
     1079<Feb/01 11:56 am>[4661037.738058]  [<ffffffff8104103c>] __do_softirq+0xd8/0x194
     1080<Feb/01 11:56 am>[4661037.744027]  [<ffffffff8100388c>] call_softirq+0x1c/0x28
     1081
     1082<Feb/01 11:56 am>[4661037.749912]  [<ffffffff81004e7f>] do_softirq+0x33/0x68
     1083<Feb/01 11:56 am>[4661037.755615]  [<ffffffff81040ec2>] irq_exit+0x36/0x75
     1084<Feb/01 11:56 am>[4661037.761143]  [<ffffffff810197aa>] smp_apic_timer_interrupt+0x87/0x95
     1085<Feb/01 11:56 am>[4661037.768091]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
     1086<Feb/01 11:56 am>[4661037.774650]  <EOI>  [<ffffffff8151cc4a>] ? schedule+0x296/0x86a
     1087<Feb/01 11:56 am>[4661037.781205]  [<ffffffff81303870>] ? acpi_idle_enter_c1+0xdd/0xf8
     1088<Feb/01 11:56 am>[4661037.787781]  [<ffffffff81303851>] ? acpi_idle_enter_c1+0xbe/0xf8
     1089                 <Feb/01 11:56 am>[4661037.794385]  [<ffffffff81430151>] cpuidle_idle_call+0x95/0xf0
     1090<Feb/01 11:56 am>[4661037.800728]  [<ffffffff81001d4b>] cpu_idle+0x55/0x8d
     1091<Feb/01 11:56 am>[4661037.806256]  [<ffffffff815191ca>] start_secondary+0x1ae/0x1b2