IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Initial Version and Version 1 of ipp058-20120102-crash.log


Ignore:
Timestamp:
Feb 1, 2012, 4:14:08 PM (14 years ago)
Author:
Mark Huber
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • ipp058-20120102-crash.log

    v1 v1  
     1{{{
     2
     3File content:
     4520272]  <<EOE>>  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     5<Feb/01 03:14 pm>[2789481.520276]  [<ffffffff812c08fe>] __delay+0xa/0xc
     6<Feb/01 03:14 pm>[2789481.520278]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     7<Feb/01 03:14 pm>[2789481.520280]  [<ffffffff8151f115>] _raw_spin_lock_irq+0x35/0x39
     8<Feb/01 03:14 pm>[2789481.520282]  [<ffffffff812aaa39>] ? get_request_wait+0x155/0x192
     9<Feb/01 03:14 pm>[2789481.520284]  [<ffffffff812aaa39>] get_request_wait+0x155/0x192
     10<Feb/01 03:14 pm>[2789481.520287]  [<ffffffff81054b6c>] ? autoremove_wake_function+0x0/0x38
     11<Feb/01 03:14 pm>[2789481.520289]  [<ffffffff812a3a03>] ? elv_merge+0x16a/0x1a4
     12<Feb/01 03:14 pm>[2789481.520291]  [<ffffffff812aaf95>] __make_request+0x31c/0x445
     13<Feb/01 03:14 pm>[2789481.520294]  [<ffffffff812a946b>] generic_make_request+0x32a/0x39c
     14<Feb/01 03:14 pm>[2789481.520296]  [<ffffffff810b3482>] ? test_set_page_writeback+0xe3/0x10e
     15<Feb/01 03:14 pm>[2789481.520298]  [<ffffffff812a9591>] submit_bio+0xb4/0xbd
     16<Feb/01 03:14 pm>[2789481.520300]  [<ffffffff810ac9a9>] ? unlock_page+0x22/0x26
     17<Feb/01 03:14 pm>[2789481.520302]  [<ffffffff810d139c>] swap_writepage+0x8c/0x97
     18<Feb/01 03:14 pm>[2789481.520304]  [<ffffffff810b7612>] shrink_page_list+0x3c2/0x68f
     19<Feb/01 03:14 pm>[2789481.520307]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     20<Feb/01 03:14 pm>[2789481.520309]  [<ffffffff810e5fb2>] ? mem_cgroup_del_lru_list+0x7b/0x8e
     21<Feb/01 03:14 pm>[2789481.520311]  [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
     22<Feb/01 03:14 pm>[2789481.520314]  [<ffffffff81048174>] ? try_to_del_timer_sync+0xae/0xbb
     23<Feb/01 03:14 pm>[2789481.520316]  [<ffffffff81054cb3>] ? finish_wait+0x5d/0x65
     24<Feb/01 03:14 pm>[2789481.520319]  [<ffffffff81054cb3>] ? finish_wait+0x5d/0x65
     25
     26<Feb/01 03:14 pm>[2789481.520321]  [<ffffffff81054b6c>] ? autoremove_wake_function+0x0/0x38
     27<Feb/01 03:14 pm>[2789481.520323]  [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
     28<Feb/01 03:14 pm>[2789481.520325]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
     29<Feb/01 03:14 pm>[2789481.520327]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
     30<Feb/01 03:14 pm>[2789481.520330]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
     31<Feb/01 03:14 pm>[2789481.520332]  [<ffffffff810da90f>] alloc_page_vma+0x148/0x14d
     32<Feb/01 03:14 pm>[2789481.520334]  [<ffffffff810c4d2a>] handle_mm_fault+0x294/0x73e
     33<Feb/01 03:14 pm>[2789481.520337]  [<ffffffff815227e9>] do_page_fault+0x351/0x375
     34<Feb/01 03:14 pm>[2789481.520339]  [<ffffffff810348d8>] ? finish_task_switch+0x69/0xdd
     35<Feb/01 03:14 pm>[2789481.520341]  [<ffffffff8103486f>] ? finish_task_switch+0x0/0xdd
     36<Feb/01 03:14 pm>[2789481.520343]  [<ffffffff8151d152>] ? schedule+0x79e/0x86a
     37<Feb/01 03:14 pm>[2789481.520346]  [<ffffffff8151ef6f>] ? lockdep_sys_exit_thunk+0x35/0x67
     38<Feb/01 03:14 pm>[2789481.520348]  [<ffffffff810dce21>] ? add_partial+0x1a/0x6a
     39<Feb/01 03:14 pm>[2789481.520350]  [<ffffffff8151fb5f>] page_fault+0x1f/0x30
     40<Feb/01 03:14 pm>[2789520.578555] Pid: 30947, comm: psphotStack Tainted: G      D    2.6.34.4 #3
     41                 <Feb/01 03:14 pm>[2789520.585951] Call Trace:
     42<Feb/01 03:14 pm>[2789520.588844]  [<ffffffff812c4e9f>] do_raw_spin_lock+0x118/0x147
     43<Feb/01 03:14 pm>[2789520.595184]  [<ffffffff8151f115>] _raw_spin_lock_irq+0x35/0x39
     44<Feb/01 03:14 pm>[2789520.601528]  [<ffffffff8151cae8>] ? schedule+0x134/0x86a
     45<Feb/01 03:14 pm>[2789520.607335]  [<ffffffff8151cae8>] schedule+0x134/0x86a
     46<Feb/01 03:14 pm>[2789520.612979]  [<ffffffff8151e357>] do_nanosleep+0x73/0xb3
     47<Feb/01 03:14 pm>[2789520.618806]  [<ffffffff81058477>] hrtimer_nanosleep+0x97/0x105
     48
     49<Feb/01 03:14 pm>[2789520.625168]  [<ffffffff81057842>] ? hrtimer_wakeup+0x0/0x21
     50<Feb/01 03:14 pm>[2789520.631263]  [<ffffffff810583de>] ? hrtimer_start_range_ns+0xf/0x11
     51<Feb/01 03:14 pm>[2789520.638087]  [<ffffffff81058539>] sys_nanosleep+0x54/0x6c
     52<Feb/01 03:14 pm>[2789520.643970]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
     53                 <Feb/01 03:14 pm>[2789520.650515] sending NMI to all CPUs:
     54<Feb/01 03:14 pm>[2789520.654527] NMI backtrace for cpu 0
     55<Feb/01 03:14 pm>[2789520.658486] CPU 0
     56<Feb/01 03:14 pm>[2789520.660572] Modules linked in: w83793 hwmon_vid autofs4 i2c_i801 i2c_core iTCO_wdt e1000e ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
     57<Feb/01 03:14 pm>[2789520.696404]
     58<Feb/01 03:14 pm>[2789520.698311] Pid: 8392, comm: nfsd Tainted: G      D    2.6.34.4 #3 S7010/empty
     59<Feb/01 03:14 pm>[2789520.706273] RIP: 0010:[<ffffffff812c08af>]  [<ffffffff812c08af>] delay_tsc+0x25/0x59
     60<Feb/01 03:14 pm>[2789520.714813] RSP: 0018:ffff8800026038f8  EFLAGS: 00000092
     61<Feb/01 03:14 pm>[2789520.720637] RAX: 000000005283f6bc RBX: 000000005d0c125a RCX: 000000005283f6a0
     62<Feb/01 03:14 pm>[2789520.728483] RDX: 00000000001a7f64 RSI: 0000000000000000 RDI: 0000000000000001
     63<Feb/01 03:14 pm>[2789520.736314] RBP: ffff880002603900 R08: 0000000000000000 R09: 0000000000000000
     64<Feb/01 03:14 pm>[2789520.744163] R10: 0000000000000000 R11: ffff880002603730 R12: ffff8800026b3c00
     65<Feb/01 03:14 pm>[2789520.751995] R13: 0000000000000000 R14: ffff88061df09030 R15: ffff88061df094b8
     66<Feb/01 03:14 pm>[2789520.759855] FS:  0000000000000000(0000) GS:ffff880002600000(0000) knlGS:0000000000000000
     67<Feb/01 03:14 pm>[2789520.768690] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
     68<Feb/01 03:14 pm>[2789520.774951] CR2: 00007f041d578000 CR3: 000000049cc66000 CR4: 00000000000006f0
     69<Feb/01 03:14 pm>[2789520.782794] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     70<Feb/01 03:14 pm>[2789520.790636] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     71
     72<Feb/01 03:14 pm>[2789520.798512] Process nfsd (pid: 8392, threadinfo ffff88061d80c000, task ffff88061df09030)
     73<Feb/01 03:14 pm>[2789520.807340] Stack:
     74                 <Feb/01 03:14 pm>[2789520.809791]  ffffffff81019da9 ffff880002603910 ffffffff812c08fe ffff880002603950
     75<Feb/01 03:14 pm>[2789520.817708] <0> ffffffff812c4e65 000000009ef30400 ffff8800026b3c00 ffff88011544a060
     76<Feb/01 03:14 pm>[2789520.826336] <0> ffff8800026b3c00 0000000000000002 0000000000000000 ffff880002603970
     77<Feb/01 03:14 pm>[2789520.835414] Call Trace:
     78<Feb/01 03:14 pm>[2789520.838294]  <IRQ>
     79<Feb/01 03:14 pm>[2789520.840880]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     80<Feb/01 03:14 pm>[2789520.848607]  [<ffffffff812c08fe>] __delay+0xa/0xc
     81<Feb/01 03:14 pm>[2789520.853803]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     82<Feb/01 03:14 pm>[2789520.860064]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     83<Feb/01 03:14 pm>[2789520.866054]  [<ffffffff8103666d>] ? try_to_wake_up+0x123/0x342
     84<Feb/01 03:14 pm>[2789520.872399]  [<ffffffff8103666d>] try_to_wake_up+0x123/0x342
     85<Feb/01 03:14 pm>[2789520.878573]  [<ffffffff81036899>] default_wake_function+0xd/0xf
     86<Feb/01 03:14 pm>[2789520.885028]  [<ffffffff810f9aba>] pollwake+0x57/0x59
     87<Feb/01 03:14 pm>[2789520.890472]  [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
     88<Feb/01 03:14 pm>[2789520.897055]  [<ffffffff8102c08d>] __wake_up_common+0x46/0x76
     89<Feb/01 03:14 pm>[2789520.903242]  [<ffffffff8102fa4a>] __wake_up_sync_key+0x43/0x5a
     90<Feb/01 03:14 pm>[2789520.909589]  [<ffffffff8144b7cc>] sock_def_readable+0x43/0x6f
     91<Feb/01 03:14 pm>[2789520.915838]  [<ffffffff8148e50c>] tcp_rcv_established+0x724/0xa01
     92<Feb/01 03:14 pm>[2789520.922461]  [<ffffffff81495279>] tcp_v4_do_rcv+0x31/0x1ad
     93<Feb/01 03:14 pm>[2789520.928448]  [<ffffffff814958c4>] tcp_v4_rcv+0x4cf/0x7a3
     94
     95<Feb/01 03:14 pm>[2789520.934258]  [<ffffffff8147a468>] ? ip_local_deliver_finish+0x39/0x1d0
     96<Feb/01 03:14 pm>[2789520.941334]  [<ffffffff8147a540>] ip_local_deliver_finish+0x111/0x1d0
     97<Feb/01 03:14 pm>[2789520.948300]  [<ffffffff8147a468>] ? ip_local_deliver_finish+0x39/0x1d0
     98<Feb/01 03:14 pm>[2789520.955337]  [<ffffffff8147a671>] ip_local_deliver+0x72/0x7a
     99<Feb/01 03:14 pm>[2789520.961533]  [<ffffffff8147a170>] ip_rcv_finish+0x334/0x350
     100<Feb/01 03:14 pm>[2789520.967600]  [<ffffffff8147a409>] ip_rcv+0x27d/0x2a3
     101<Feb/01 03:14 pm>[2789520.973071]  [<ffffffff814570c8>] netif_receive_skb+0x3f4/0x431
     102<Feb/01 03:14 pm>[2789520.979522]  [<ffffffff81456e72>] ? netif_receive_skb+0x19e/0x431
     103<Feb/01 03:14 pm>[2789520.986127]  [<ffffffff8145792d>] napi_skb_finish+0x24/0x3c
     104<Feb/01 03:14 pm>[2789520.992211]  [<ffffffff8145796f>] napi_gro_receive+0x2a/0x2f
     105<Feb/01 03:14 pm>[2789520.998387]  [<ffffffffa01db8dd>] e1000_receive_skb+0x5c/0x65 [e1000e]
     106<Feb/01 03:14 pm>[2789521.005399]  [<ffffffffa01dbb32>] e1000_clean_rx_irq+0x24c/0x301 [e1000e]
     107<Feb/01 03:14 pm>[2789521.012748]  [<ffffffffa01dd05a>] e1000_clean+0x75/0x22e [e1000e]
     108<Feb/01 03:14 pm>[2789521.019331]  [<ffffffff814573ba>] net_rx_action+0xb2/0x1ff
     109<Feb/01 03:14 pm>[2789521.025327]  [<ffffffff814574c5>] ? net_rx_action+0x1bd/0x1ff
     110<Feb/01 03:14 pm>[2789521.031605]  [<ffffffff8104103c>] __do_softirq+0xd8/0x194
     111                 <Feb/01 03:14 pm>[2789521.037481]  [<ffffffff8100388c>] call_softirq+0x1c/0x28
     112<Feb/01 03:14 pm>[2789521.043288]  [<ffffffff81004e7f>] do_softirq+0x33/0x68
     113<Feb/01 03:14 pm>[2789521.048934]  [<ffffffff81040ec2>] irq_exit+0x36/0x75
     114<Feb/01 03:14 pm>[2789521.054394]  [<ffffffff8100457c>] do_IRQ+0xaa/0xc1
     115<Feb/01 03:14 pm>[2789521.059684]  [<ffffffff8151f913>] ret_from_intr+0x0/0xa
     116<Feb/01 03:14 pm>[2789521.065426]  <EOI>
     117
     118<Feb/01 03:14 pm>[2789521.068020]  [<ffffffff810b5718>] ? pagevec_move_tail+0x84/0x135
     119<Feb/01 03:14 pm>[2789521.074559]  [<ffffffff812c08b1>] ? delay_tsc+0x27/0x59
     120<Feb/01 03:14 pm>[2789521.080285]  [<ffffffff812c4e65>] ? do_raw_spin_lock+0xde/0x147
     121<Feb/01 03:14 pm>[2789521.086756]  [<ffffffff812c08fe>] __delay+0xa/0xc
     122<Feb/01 03:14 pm>[2789521.091956]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     123<Feb/01 03:14 pm>[2789521.098237]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     124<Feb/01 03:14 pm>[2789521.104242]  [<ffffffff810cd2da>] ? page_lock_anon_vma+0x54/0x77
     125<Feb/01 03:14 pm>[2789521.110780]  [<ffffffff810cd2da>] page_lock_anon_vma+0x54/0x77
     126<Feb/01 03:14 pm>[2789521.117114]  [<ffffffff810cd286>] ? page_lock_anon_vma+0x0/0x77
     127<Feb/01 03:14 pm>[2789521.123590]  [<ffffffff810cd4a2>] page_referenced+0x91/0x335
     128<Feb/01 03:14 pm>[2789521.129762]  [<ffffffff810b2173>] ? __pagevec_free+0x70/0x85
     129<Feb/01 03:14 pm>[2789521.135942]  [<ffffffff810b73ef>] shrink_page_list+0x19f/0x68f
     130<Feb/01 03:14 pm>[2789521.142279]  [<ffffffff810b5718>] ? pagevec_move_tail+0x84/0x135
     131<Feb/01 03:14 pm>[2789521.148790]  [<ffffffff810e5fb2>] ? mem_cgroup_del_lru_list+0x7b/0x8e
     132<Feb/01 03:14 pm>[2789521.155775]  [<ffffffff810e6024>] ? mem_cgroup_del_lru+0x34/0x36
     133<Feb/01 03:14 pm>[2789521.162309]  [<ffffffff810b6d92>] ? isolate_pages_global+0xd2/0x23f
     134<Feb/01 03:14 pm>[2789521.169086]  [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
     135<Feb/01 03:14 pm>[2789521.175024]  [<ffffffff812734b7>] ? xfs_perag_get+0x29/0x99
     136<Feb/01 03:14 pm>[2789521.181100]  [<ffffffff81058774>] ? up_read+0x1e/0x36
     137<Feb/01 03:14 pm>[2789521.186649]  [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
     138<Feb/01 03:14 pm>[2789521.193368]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
     139<Feb/01 03:14 pm>[2789521.199643]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
     140
     141<Feb/01 03:14 pm>[2789521.206333]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
     142<Feb/01 03:14 pm>[2789521.213216]  [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
     143<Feb/01 03:14 pm>[2789521.219638]  [<ffffffff810acb60>] __page_cache_alloc+0x59/0x5e
     144<Feb/01 03:14 pm>[2789521.225972]  [<ffffffff810b4860>] __do_page_cache_readahead+0xe5/0x1fc
     145<Feb/01 03:14 pm>[2789521.233037]  [<ffffffff810b47f6>] ? __do_page_cache_readahead+0x7b/0x1fc
     146                 <Feb/01 03:14 pm>[2789521.240291]  [<ffffffff812bc23c>] ? radix_tree_prev_hole+0x30/0x53
     147<Feb/01 03:14 pm>[2789521.246988]  [<ffffffff810b4c40>] ondemand_readahead+0x1cc/0x256
     148<Feb/01 03:14 pm>[2789521.253481]  [<ffffffff810b4ba0>] ? ondemand_readahead+0x12c/0x256
     149<Feb/01 03:14 pm>[2789521.260215]  [<ffffffff810b3c7a>] ? determine_dirtyable_memory+0x15/0x27
     150<Feb/01 03:14 pm>[2789521.267445]  [<ffffffff810b4daa>] page_cache_sync_readahead+0x38/0x3a
     151<Feb/01 03:14 pm>[2789521.274385]  [<ffffffff81109bf2>] __generic_file_splice_read+0xfa/0x41c
     152<Feb/01 03:14 pm>[2789521.281544]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     153<Feb/01 03:14 pm>[2789521.287877]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     154<Feb/01 03:14 pm>[2789521.294204]  [<ffffffff812615b1>] ? xfs_ilock+0x49/0xb2
     155<Feb/01 03:14 pm>[2789521.299934]  [<ffffffff811086db>] ? spd_release_page+0x0/0x14
     156<Feb/01 03:14 pm>[2789521.306194]  [<ffffffff81109f57>] generic_file_splice_read+0x43/0x74
     157                 <Feb/01 03:14 pm>[2789521.313084]  [<ffffffff8128191a>] xfs_file_splice_read+0xde/0x125
     158<Feb/01 03:14 pm>[2789521.319698]  [<ffffffff8110875d>] vfs_splice_to+0x6e/0x81
     159<Feb/01 03:14 pm>[2789521.325609]  [<ffffffff81108837>] splice_direct_to_actor+0xc7/0x19c
     160<Feb/01 03:14 pm>[2789521.332388]  [<ffffffff811f562c>] ? nfsd_direct_splice_actor+0x0/0x12
     161<Feb/01 03:14 pm>[2789521.339360]  [<ffffffff811f54ef>] nfsd_vfs_read+0x263/0x3a0
     162<Feb/01 03:14 pm>[2789521.345422]  [<ffffffff81281760>] ? xfs_file_open+0x0/0x3f
     163
     164<Feb/01 03:14 pm>[2789521.351421]  [<ffffffff811f5c4e>] nfsd_read+0xa7/0xc3
     165<Feb/01 03:14 pm>[2789521.356962]  [<ffffffff811fcd79>] nfsd3_proc_read+0xfe/0x142
     166<Feb/01 03:14 pm>[2789521.363117]  [<ffffffff811f0854>] nfsd_dispatch+0xe9/0x1ca
     167<Feb/01 03:14 pm>[2789521.369143]  [<ffffffff814f04c5>] svc_process+0x425/0x72a
     168<Feb/01 03:14 pm>[2789521.375021]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
     169                 <Feb/01 03:14 pm>[2789521.380307]  [<ffffffff811f0dc2>] nfsd+0xf3/0x139
     170<Feb/01 03:14 pm>[2789521.385460]  [<ffffffff810546e2>] kthread+0x7d/0x85
     171<Feb/01 03:14 pm>[2789521.390828]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     172<Feb/01 03:14 pm>[2789521.397246]  [<ffffffff81054665>] ? kthread+0x0/0x85
     173<Feb/01 03:14 pm>[2789521.402723]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     174<Feb/01 03:14 pm>[2789521.409296] Code: fb 48 ff c8 c9 c3 55 48 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae e8 0f 31 89 c1 41 89 f1 66 66 90 0f ae e8 0f 31 <89> c6 48 89 f0 48 29 c8 48 39 f8 73 25 f3 90 65 44 8b 04 25 f0
     175<Feb/01 03:14 pm>[2789521.432563] Call Trace:
     176<Feb/01 03:14 pm>[2789521.435433]  <IRQ>  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     177<Feb/01 03:14 pm>[2789521.443879]  [<ffffffff812c08fe>] __delay+0xa/0xc
     178<Feb/01 03:14 pm>[2789521.449065]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     179<Feb/01 03:14 pm>[2789521.455310]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     180<Feb/01 03:14 pm>[2789521.461282]  [<ffffffff8103666d>] ? try_to_wake_up+0x123/0x342
     181<Feb/01 03:14 pm>[2789521.467612]  [<ffffffff8103666d>] try_to_wake_up+0x123/0x342
     182<Feb/01 03:14 pm>[2789521.473810]  [<ffffffff81036899>] default_wake_function+0xd/0xf
     183<Feb/01 03:14 pm>[2789521.480249]  [<ffffffff810f9aba>] pollwake+0x57/0x59
     184<Feb/01 03:14 pm>[2789521.485708]  [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
     185<Feb/01 03:14 pm>[2789521.492316]  [<ffffffff8102c08d>] __wake_up_common+0x46/0x76
     186
     187<Feb/01 03:14 pm>[2789521.498463]  [<ffffffff8102fa4a>] __wake_up_sync_key+0x43/0x5a
     188<Feb/01 03:14 pm>[2789521.504800]  [<ffffffff8144b7cc>] sock_def_readable+0x43/0x6f
     189<Feb/01 03:14 pm>[2789521.515570]  [<ffffffff8148e50c>] tcp_rcv_established+0x724/0xa01
     190<Feb/01 03:14 pm>[2789521.522174]  [<ffffffff81495279>] tcp_v4_do_rcv+0x31/0x1ad
     191<Feb/01 03:14 pm>[2789521.528161]  [<ffffffff814958c4>] tcp_v4_rcv+0x4cf/0x7a3
     192<Feb/01 03:14 pm>[2789521.533990]  [<ffffffff8147a468>] ? ip_local_deliver_finish+0x39/0x1d0
     193<Feb/01 03:14 pm>[2789521.541064]  [<ffffffff8147a540>] ip_local_deliver_finish+0x111/0x1d0
     194<Feb/01 03:14 pm>[2789521.548040]  [<ffffffff8147a468>] ? ip_local_deliver_finish+0x39/0x1d0
     195<Feb/01 03:14 pm>[2789521.555093]  [<ffffffff8147a671>] ip_local_deliver+0x72/0x7a
     196<Feb/01 03:14 pm>[2789521.561237]  [<ffffffff8147a170>] ip_rcv_finish+0x334/0x350
     197                 <Feb/01 03:14 pm>[2789521.567332]  [<ffffffff8147a409>] ip_rcv+0x27d/0x2a3
     198<Feb/01 03:14 pm>[2789521.572784]  [<ffffffff814570c8>] netif_receive_skb+0x3f4/0x431
     199<Feb/01 03:14 pm>[2789521.579201]  [<ffffffff81456e72>] ? netif_receive_skb+0x19e/0x431
     200<Feb/01 03:14 pm>[2789521.585807]  [<ffffffff8145792d>] napi_skb_finish+0x24/0x3c
     201<Feb/01 03:14 pm>[2789521.591916]  [<ffffffff8145796f>] napi_gro_receive+0x2a/0x2f
     202<Feb/01 03:14 pm>[2789521.598127]  [<ffffffffa01db8dd>] e1000_receive_skb+0x5c/0x65 [e1000e]
     203<Feb/01 03:14 pm>[2789521.605200]  [<ffffffffa01dbb32>] e1000_clean_rx_irq+0x24c/0x301 [e1000e]
     204<Feb/01 03:14 pm>[2789521.612532]  [<ffffffffa01dd05a>] e1000_clean+0x75/0x22e [e1000e]
     205<Feb/01 03:14 pm>[2789521.619149]  [<ffffffff814573ba>] net_rx_action+0xb2/0x1ff
     206<Feb/01 03:14 pm>[2789521.625110]  [<ffffffff814574c5>] ? net_rx_action+0x1bd/0x1ff
     207<Feb/01 03:14 pm>[2789521.631396]  [<ffffffff8104103c>] __do_softirq+0xd8/0x194
     208<Feb/01 03:14 pm>[2789521.637273]  [<ffffffff8100388c>] call_softirq+0x1c/0x28
     209
     210<Feb/01 03:14 pm>[2789521.643072]  [<ffffffff81004e7f>] do_softirq+0x33/0x68
     211<Feb/01 03:14 pm>[2789521.648709]  [<ffffffff81040ec2>] irq_exit+0x36/0x75
     212<Feb/01 03:14 pm>[2789521.654143]  [<ffffffff8100457c>] do_IRQ+0xaa/0xc1
     213<Feb/01 03:14 pm>[2789521.659400]  [<ffffffff8151f913>] ret_from_intr+0x0/0xa
     214<Feb/01 03:14 pm>[2789521.665123]  <EOI>  [<ffffffff810b5718>] ? pagevec_move_tail+0x84/0x135
     215<Feb/01 03:14 pm>[2789521.672296]  [<ffffffff812c08b1>] ? delay_tsc+0x27/0x59
     216<Feb/01 03:14 pm>[2789521.678023]  [<ffffffff812c4e65>] ? do_raw_spin_lock+0xde/0x147
     217<Feb/01 03:14 pm>[2789521.684465]  [<ffffffff812c08fe>] __delay+0xa/0xc
     218<Feb/01 03:14 pm>[2789521.689663]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     219                 <Feb/01 03:14 pm>[2789521.695927]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     220<Feb/01 03:14 pm>[2789521.701917]  [<ffffffff810cd2da>] ? page_lock_anon_vma+0x54/0x77
     221<Feb/01 03:14 pm>[2789521.708417]  [<ffffffff810cd2da>] page_lock_anon_vma+0x54/0x77
     222<Feb/01 03:14 pm>[2789521.714780]  [<ffffffff810cd286>] ? page_lock_anon_vma+0x0/0x77
     223<Feb/01 03:14 pm>[2789521.721221]  [<ffffffff810cd4a2>] page_referenced+0x91/0x335
     224<Feb/01 03:14 pm>[2789521.727356]  [<ffffffff810b2173>] ? __pagevec_free+0x70/0x85
     225<Feb/01 03:14 pm>[2789521.733547]  [<ffffffff810b73ef>] shrink_page_list+0x19f/0x68f
     226<Feb/01 03:14 pm>[2789521.739909]  [<ffffffff810b5718>] ? pagevec_move_tail+0x84/0x135
     227<Feb/01 03:14 pm>[2789521.746419]  [<ffffffff810e5fb2>] ? mem_cgroup_del_lru_list+0x7b/0x8e
     228<Feb/01 03:14 pm>[2789521.753414]  [<ffffffff810e6024>] ? mem_cgroup_del_lru+0x34/0x36
     229<Feb/01 03:14 pm>[2789521.759929]  [<ffffffff810b6d92>] ? isolate_pages_global+0xd2/0x23f
     230<Feb/01 03:14 pm>[2789521.766707]  [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
     231<Feb/01 03:14 pm>[2789521.772618]  [<ffffffff812734b7>] ? xfs_perag_get+0x29/0x99
     232
     233<Feb/01 03:14 pm>[2789521.778712]  [<ffffffff81058774>] ? up_read+0x1e/0x36
     234<Feb/01 03:14 pm>[2789521.784235]  [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
     235<Feb/01 03:14 pm>[2789521.790914]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
     236<Feb/01 03:14 pm>[2789521.797170]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
     237<Feb/01 03:14 pm>[2789521.803854]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
     238<Feb/01 03:14 pm>[2789521.810702]  [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
     239<Feb/01 03:14 pm>[2789521.817122]  [<ffffffff810acb60>] __page_cache_alloc+0x59/0x5e
     240<Feb/01 03:14 pm>[2789521.823440]  [<ffffffff810b4860>] __do_page_cache_readahead+0xe5/0x1fc
     241<Feb/01 03:14 pm>[2789521.830506]  [<ffffffff810b47f6>] ? __do_page_cache_readahead+0x7b/0x1fc
     242<Feb/01 03:14 pm>[2789521.837731]  [<ffffffff812bc23c>] ? radix_tree_prev_hole+0x30/0x53
     243<Feb/01 03:14 pm>[2789521.844428]  [<ffffffff810b4c40>] ondemand_readahead+0x1cc/0x256
     244<Feb/01 03:14 pm>[2789521.850928]  [<ffffffff810b4ba0>] ? ondemand_readahead+0x12c/0x256
     245<Feb/01 03:14 pm>[2789521.857631]  [<ffffffff810b3c7a>] ? determine_dirtyable_memory+0x15/0x27
     246<Feb/01 03:14 pm>[2789521.864848]  [<ffffffff810b4daa>] page_cache_sync_readahead+0x38/0x3a
     247<Feb/01 03:14 pm>[2789521.871834]  [<ffffffff81109bf2>] __generic_file_splice_read+0xfa/0x41c
     248<Feb/01 03:14 pm>[2789521.879002]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     249<Feb/01 03:14 pm>[2789521.885361]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     250<Feb/01 03:14 pm>[2789521.891738]  [<ffffffff812615b1>] ? xfs_ilock+0x49/0xb2
     251<Feb/01 03:14 pm>[2789521.897452]  [<ffffffff811086db>] ? spd_release_page+0x0/0x14
     252<Feb/01 03:14 pm>[2789521.903730]  [<ffffffff81109f57>] generic_file_splice_read+0x43/0x74
     253                 <Feb/01 03:14 pm>[2789521.910630]  [<ffffffff8128191a>] xfs_file_splice_read+0xde/0x125
     254<Feb/01 03:14 pm>[2789521.917233]  [<ffffffff8110875d>] vfs_splice_to+0x6e/0x81
     255
     256<Feb/01 03:14 pm>[2789521.923126]  [<ffffffff81108837>] splice_direct_to_actor+0xc7/0x19c
     257<Feb/01 03:14 pm>[2789521.929932]  [<ffffffff811f562c>] ? nfsd_direct_splice_actor+0x0/0x12
     258<Feb/01 03:14 pm>[2789521.936900]  [<ffffffff811f54ef>] nfsd_vfs_read+0x263/0x3a0
     259                 <Feb/01 03:14 pm>[2789521.942948]  [<ffffffff81281760>] ? xfs_file_open+0x0/0x3f
     260<Feb/01 03:14 pm>[2789521.948948]  [<ffffffff811f5c4e>] nfsd_read+0xa7/0xc3
     261<Feb/01 03:14 pm>[2789521.954487]  [<ffffffff811fcd79>] nfsd3_proc_read+0xfe/0x142
     262<Feb/01 03:14 pm>[2789521.960635]  [<ffffffff811f0854>] nfsd_dispatch+0xe9/0x1ca
     263<Feb/01 03:14 pm>[2789521.966608]  [<ffffffff814f04c5>] svc_process+0x425/0x72a
     264<Feb/01 03:14 pm>[2789521.972537]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
     265<Feb/01 03:14 pm>[2789521.977815]  [<ffffffff811f0dc2>] nfsd+0xf3/0x139
     266<Feb/01 03:14 pm>[2789521.982987]  [<ffffffff810546e2>] kthread+0x7d/0x85
     267<Feb/01 03:14 pm>[2789521.988372]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     268<Feb/01 03:14 pm>[2789521.994790]  [<ffffffff81054665>] ? kthread+0x0/0x85
     269<Feb/01 03:14 pm>[2789522.000265]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     270<Feb/01 03:14 pm>[2789522.006875] Pid: 8392, comm: nfsd Tainted: G      D    2.6.34.4 #3
     271<Feb/01 03:14 pm>[2789522.013566] Call Trace:
     272<Feb/01 03:14 pm>[2789522.016458]  <NMI>  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
     273<Feb/01 03:14 pm>[2789522.022840]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
     274<Feb/01 03:14 pm>[2789522.029167]  [<ffffffff81520360>] do_nmi+0xb2/0x259
     275<Feb/01 03:14 pm>[2789522.034560]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
     276<Feb/01 03:14 pm>[2789522.039589]  [<ffffffff812c08af>] ? delay_tsc+0x25/0x59
     277<Feb/01 03:14 pm>[2789522.045286]  <<EOE>>  <IRQ>  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     278
     279<Feb/01 03:14 pm>[2789522.054556]  [<ffffffff812c08fe>] __delay+0xa/0xc
     280<Feb/01 03:14 pm>[2789522.059740]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     281<Feb/01 03:14 pm>[2789522.065995]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     282<Feb/01 03:14 pm>[2789522.072009]  [<ffffffff8103666d>] ? try_to_wake_up+0x123/0x342
     283<Feb/01 03:14 pm>[2789522.078329]  [<ffffffff8103666d>] try_to_wake_up+0x123/0x342
     284<Feb/01 03:14 pm>[2789522.084501]  [<ffffffff81036899>] default_wake_function+0xd/0xf
     285<Feb/01 03:14 pm>[2789522.090942]  [<ffffffff810f9aba>] pollwake+0x57/0x59
     286<Feb/01 03:14 pm>[2789522.096385]  [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
     287<Feb/01 03:14 pm>[2789522.102984]  [<ffffffff8102c08d>] __wake_up_common+0x46/0x76
     288<Feb/01 03:14 pm>[2789522.109163]  [<ffffffff8102fa4a>] __wake_up_sync_key+0x43/0x5a
     289<Feb/01 03:14 pm>[2789522.115526]  [<ffffffff8144b7cc>] sock_def_readable+0x43/0x6f
     290<Feb/01 03:14 pm>[2789522.121766]  [<ffffffff8148e50c>] tcp_rcv_established+0x724/0xa01
     291<Feb/01 03:14 pm>[2789522.128364]  [<ffffffff81495279>] tcp_v4_do_rcv+0x31/0x1ad
     292<Feb/01 03:14 pm>[2789522.134378]  [<ffffffff814958c4>] tcp_v4_rcv+0x4cf/0x7a3
     293<Feb/01 03:14 pm>[2789522.140231]  [<ffffffff8147a468>] ? ip_local_deliver_finish+0x39/0x1d0
     294<Feb/01 03:14 pm>[2789522.147254]  [<ffffffff8147a540>] ip_local_deliver_finish+0x111/0x1d0
     295                 <Feb/01 03:14 pm>[2789522.154211]  [<ffffffff8147a468>] ? ip_local_deliver_finish+0x39/0x1d0
     296<Feb/01 03:14 pm>[2789522.161257]  [<ffffffff8147a671>] ip_local_deliver+0x72/0x7a
     297<Feb/01 03:14 pm>[2789522.167398]  [<ffffffff8147a170>] ip_rcv_finish+0x334/0x350
     298<Feb/01 03:14 pm>[2789522.173477]  [<ffffffff8147a409>] ip_rcv+0x27d/0x2a3
     299<Feb/01 03:14 pm>[2789522.178946]  [<ffffffff814570c8>] netif_receive_skb+0x3f4/0x431
     300<Feb/01 03:14 pm>[2789522.185380]  [<ffffffff81456e72>] ? netif_receive_skb+0x19e/0x431
     301
     302<Feb/01 03:14 pm>[2789522.191985]  [<ffffffff8145792d>] napi_skb_finish+0x24/0x3c
     303<Feb/01 03:14 pm>[2789522.198061]  [<ffffffff8145796f>] napi_gro_receive+0x2a/0x2f
     304<Feb/01 03:14 pm>[2789522.204221]  [<ffffffffa01db8dd>] e1000_receive_skb+0x5c/0x65 [e1000e]
     305<Feb/01 03:14 pm>[2789522.211295]  [<ffffffffa01dbb32>] e1000_clean_rx_irq+0x24c/0x301 [e1000e]
     306<Feb/01 03:14 pm>[2789522.218641]  [<ffffffffa01dd05a>] e1000_clean+0x75/0x22e [e1000e]
     307<Feb/01 03:14 pm>[2789522.225233]  [<ffffffff814573ba>] net_rx_action+0xb2/0x1ff
     308<Feb/01 03:14 pm>[2789522.231204]  [<ffffffff814574c5>] ? net_rx_action+0x1bd/0x1ff
     309<Feb/01 03:14 pm>[2789522.237456]  [<ffffffff8104103c>] __do_softirq+0xd8/0x194
     310<Feb/01 03:14 pm>[2789522.243376]  [<ffffffff8100388c>] call_softirq+0x1c/0x28
     311<Feb/01 03:14 pm>[2789522.249210]  [<ffffffff81004e7f>] do_softirq+0x33/0x68
     312<Feb/01 03:14 pm>[2789522.254863]  [<ffffffff81040ec2>] irq_exit+0x36/0x75
     313<Feb/01 03:14 pm>[2789522.260334]  [<ffffffff8100457c>] do_IRQ+0xaa/0xc1
     314<Feb/01 03:14 pm>[2789522.265615]  [<ffffffff8151f913>] ret_from_intr+0x0/0xa
     315<Feb/01 03:14 pm>[2789522.271346]  <EOI>  [<ffffffff810b5718>] ? pagevec_move_tail+0x84/0x135
     316<Feb/01 03:14 pm>[2789522.278573]  [<ffffffff812c08b1>] ? delay_tsc+0x27/0x59
     317<Feb/01 03:14 pm>[2789522.284281]  [<ffffffff812c4e65>] ? do_raw_spin_lock+0xde/0x147
     318<Feb/01 03:14 pm>[2789522.290722]  [<ffffffff812c08fe>] __delay+0xa/0xc
     319<Feb/01 03:14 pm>[2789522.295915]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     320<Feb/01 03:14 pm>[2789522.302150]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     321<Feb/01 03:14 pm>[2789522.308148]  [<ffffffff810cd2da>] ? page_lock_anon_vma+0x54/0x77
     322<Feb/01 03:14 pm>[2789522.314677]  [<ffffffff810cd2da>] page_lock_anon_vma+0x54/0x77
     323<Feb/01 03:14 pm>[2789522.321012]  [<ffffffff810cd286>] ? page_lock_anon_vma+0x0/0x77
     324
     325<Feb/01 03:14 pm>[2789522.327436]  [<ffffffff810cd4a2>] page_referenced+0x91/0x335
     326<Feb/01 03:14 pm>[2789522.333607]  [<ffffffff810b2173>] ? __pagevec_free+0x70/0x85
     327<Feb/01 03:14 pm>[2789522.339777]  [<ffffffff810b73ef>] shrink_page_list+0x19f/0x68f
     328<Feb/01 03:14 pm>[2789522.346124]  [<ffffffff810b5718>] ? pagevec_move_tail+0x84/0x135
     329<Feb/01 03:14 pm>[2789522.352659]  [<ffffffff810e5fb2>] ? mem_cgroup_del_lru_list+0x7b/0x8e
     330<Feb/01 03:14 pm>[2789522.359630]  [<ffffffff810e6024>] ? mem_cgroup_del_lru+0x34/0x36
     331<Feb/01 03:14 pm>[2789522.366128]  [<ffffffff810b6d92>] ? isolate_pages_global+0xd2/0x23f
     332<Feb/01 03:14 pm>[2789522.372906]  [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
     333<Feb/01 03:14 pm>[2789522.378818]  [<ffffffff812734b7>] ? xfs_perag_get+0x29/0x99
     334<Feb/01 03:14 pm>[2789522.384901]  [<ffffffff81058774>] ? up_read+0x1e/0x36
     335<Feb/01 03:14 pm>[2789522.390414]  [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
     336<Feb/01 03:14 pm>[2789522.397102]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
     337<Feb/01 03:14 pm>[2789522.403358]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
     338<Feb/01 03:14 pm>[2789522.410044]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
     339<Feb/01 03:14 pm>[2789522.416906]  [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
     340<Feb/01 03:14 pm>[2789522.423363]  [<ffffffff810acb60>] __page_cache_alloc+0x59/0x5e
     341<Feb/01 03:14 pm>[2789522.429690]  [<ffffffff810b4860>] __do_page_cache_readahead+0xe5/0x1fc
     342<Feb/01 03:14 pm>[2789522.436763]  [<ffffffff810b47f6>] ? __do_page_cache_readahead+0x7b/0x1fc
     343<Feb/01 03:14 pm>[2789522.444018]  [<ffffffff812bc23c>] ? radix_tree_prev_hole+0x30/0x53
     344<Feb/01 03:14 pm>[2789522.450733]  [<ffffffff810b4c40>] ondemand_readahead+0x1cc/0x256
     345<Feb/01 03:14 pm>[2789522.457247]  [<ffffffff810b4ba0>] ? ondemand_readahead+0x12c/0x256
     346<Feb/01 03:14 pm>[2789522.463974]  [<ffffffff810b3c7a>] ? determine_dirtyable_memory+0x15/0x27
     347
     348<Feb/01 03:14 pm>[2789522.471204]  [<ffffffff810b4daa>] page_cache_sync_readahead+0x38/0x3a
     349<Feb/01 03:14 pm>[2789522.478179]  [<ffffffff81109bf2>] __generic_file_splice_read+0xfa/0x41c
     350<Feb/01 03:14 pm>[2789522.485345]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     351<Feb/01 03:14 pm>[2789522.491654]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     352<Feb/01 03:14 pm>[2789522.497981]  [<ffffffff812615b1>] ? xfs_ilock+0x49/0xb2
     353<Feb/01 03:14 pm>[2789522.503684]  [<ffffffff811086db>] ? spd_release_page+0x0/0x14
     354<Feb/01 03:14 pm>[2789522.509935]  [<ffffffff81109f57>] generic_file_splice_read+0x43/0x74
     355<Feb/01 03:14 pm>[2789522.516836]  [<ffffffff8128191a>] xfs_file_splice_read+0xde/0x125
     356<Feb/01 03:14 pm>[2789522.523466]  [<ffffffff8110875d>] vfs_splice_to+0x6e/0x81
     357<Feb/01 03:14 pm>[2789522.529359]  [<ffffffff81108837>] splice_direct_to_actor+0xc7/0x19c
     358<Feb/01 03:14 pm>[2789522.536173]  [<ffffffff811f562c>] ? nfsd_direct_splice_actor+0x0/0x12
     359<Feb/01 03:14 pm>[2789522.543123]  [<ffffffff811f54ef>] nfsd_vfs_read+0x263/0x3a0
     360<Feb/01 03:14 pm>[2789522.549216]  [<ffffffff81281760>] ? xfs_file_open+0x0/0x3f
     361<Feb/01 03:14 pm>[2789522.555214]  [<ffffffff811f5c4e>] nfsd_read+0xa7/0xc3
     362<Feb/01 03:14 pm>[2789522.560788]  [<ffffffff811fcd79>] nfsd3_proc_read+0xfe/0x142
     363<Feb/01 03:14 pm>[2789522.566927]  [<ffffffff811f0854>] nfsd_dispatch+0xe9/0x1ca
     364<Feb/01 03:14 pm>[2789522.572944]  [<ffffffff814f04c5>] svc_process+0x425/0x72a
     365<Feb/01 03:14 pm>[2789522.578830]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
     366<Feb/01 03:14 pm>[2789522.584101]  [<ffffffff811f0dc2>] nfsd+0xf3/0x139
     367<Feb/01 03:14 pm>[2789522.589331]  [<ffffffff810546e2>] kthread+0x7d/0x85
     368<Feb/01 03:14 pm>[2789522.594681]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     369<Feb/01 03:14 pm>[2789522.601118]  [<ffffffff81054665>] ? kthread+0x0/0x85
     370
     371<Feb/01 03:14 pm>[2789522.606551]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     372<Feb/01 03:14 pm>[2789522.613184] NMI backtrace for cpu 9
     373<Feb/01 03:14 pm>[2789522.617127] CPU 9
     374<Feb/01 03:14 pm>[2789522.619214] Modules linked in: w83793 hwmon_vid autofs4 i2c_i801 i2c_core iTCO_wdt e1000e ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
     375<Feb/01 03:14 pm>[2789522.659496]
     376<Feb/01 03:14 pm>[2789522.661430] Pid: 30947, comm: psphotStack Tainted: G      D    2.6.34.4 #3 S7010/empty
     377<Feb/01 03:14 pm>[2789522.670086] RIP: 0010:[<ffffffff812c08b7>]  [<ffffffff812c08b7>] delay_tsc+0x2d/0x59
     378<Feb/01 03:14 pm>[2789522.678621] RSP: 0018:ffff88045ab05d10  EFLAGS: 00000012
     379<Feb/01 03:14 pm>[2789522.684472] RAX: 000000000000004c RBX: 0000000000000001 RCX: 00000000529c66f4
     380<Feb/01 03:14 pm>[2789522.692346] RDX: 00000000001a7f64 RSI: 00000000529c6740 RDI: 000000000028b0f8
     381<Feb/01 03:14 pm>[2789522.700210] RBP: ffff88045ab05d18 R08: 0000000000000009 R09: 0000000000000009
     382<Feb/01 03:14 pm>[2789522.708058] R10: ffff8800000b9180 R11: 0000000000000082 R12: ffff880655273c00
     383<Feb/01 03:14 pm>[2789522.715882] R13: 0000000000000000 R14: ffff88060a596120 R15: ffff88060a5965a8
     384<Feb/01 03:14 pm>[2789522.723742] FS:  00007f2059a426f0(0000) GS:ffff880655260000(0000) knlGS:0000000000000000
     385<Feb/01 03:14 pm>[2789522.732542] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
     386<Feb/01 03:14 pm>[2789522.738778] CR2: 00007f0d75254ffe CR3: 000000045ab87000 CR4: 00000000000006e0
     387<Feb/01 03:14 pm>[2789522.746631] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     388<Feb/01 03:14 pm>[2789522.754460] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     389<Feb/01 03:14 pm>[2789522.762312] Process psphotStack (pid: 30947, threadinfo ffff88045ab04000, task ffff88060a596120)
     390<Feb/01 03:14 pm>[2789522.771852] Stack:
     391<Feb/01 03:14 pm>[2789522.774300]  ffff88060a5965a8 ffff88045ab05d28 ffffffff812c093d ffff88045ab05d48
     392<Feb/01 03:14 pm>[2789522.782217] <0> ffffffff81019da9 000000009ef30400 000000009ef30400 ffff88045ab05d88
     393
     394<Feb/01 03:14 pm>[2789522.790879] <0> ffffffff812c4ea4 000000009ef30400 ffff880655273c00 0000000000000009
     395<Feb/01 03:14 pm>[2789522.799934] Call Trace:
     396<Feb/01 03:14 pm>[2789522.802796]  [<ffffffff812c093d>] __const_udelay+0x3d/0x3f
     397<Feb/01 03:14 pm>[2789522.808789]  [<ffffffff81019da9>] arch_trigger_all_cpu_backtrace+0x51/0x5d
     398<Feb/01 03:14 pm>[2789522.816219]  [<ffffffff812c4ea4>] do_raw_spin_lock+0x11d/0x147
     399<Feb/01 03:14 pm>[2789522.822561]  [<ffffffff8151f115>] _raw_spin_lock_irq+0x35/0x39
     400<Feb/01 03:14 pm>[2789522.828922]  [<ffffffff8151cae8>] ? schedule+0x134/0x86a
     401<Feb/01 03:14 pm>[2789522.834747]  [<ffffffff8151cae8>] schedule+0x134/0x86a
     402                 <Feb/01 03:14 pm>[2789522.840358]  [<ffffffff8151e357>] do_nanosleep+0x73/0xb3
     403<Feb/01 03:14 pm>[2789522.846193]  [<ffffffff81058477>] hrtimer_nanosleep+0x97/0x105
     404<Feb/01 03:14 pm>[2789522.852547]  [<ffffffff81057842>] ? hrtimer_wakeup+0x0/0x21
     405<Feb/01 03:14 pm>[2789522.858643]  [<ffffffff810583de>] ? hrtimer_start_range_ns+0xf/0x11
     406<Feb/01 03:14 pm>[2789522.865438]  [<ffffffff81058539>] sys_nanosleep+0x54/0x6c
     407<Feb/01 03:14 pm>[2789522.871339]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
     408<Feb/01 03:14 pm>[2789522.877868] Code: 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae e8 0f 31 89 c1 41 89 f1 66 66 90 0f ae e8 0f 31 89 c6 48 89 f0 48 29 c8 <48> 39 f8 73 25 f3 90 65 44 8b 04 25 f0 d1 00 00 45 39 c1 74 db
     409<Feb/01 03:14 pm>[2789522.901066] Call Trace:
     410<Feb/01 03:14 pm>[2789522.903928]  [<ffffffff812c093d>] __const_udelay+0x3d/0x3f
     411<Feb/01 03:14 pm>[2789522.909902]  [<ffffffff81019da9>] arch_trigger_all_cpu_backtrace+0x51/0x5d
     412<Feb/01 03:14 pm>[2789522.917350]  [<ffffffff812c4ea4>] do_raw_spin_lock+0x11d/0x147
     413<Feb/01 03:14 pm>[2789522.923693]  [<ffffffff8151f115>] _raw_spin_lock_irq+0x35/0x39
     414<Feb/01 03:14 pm>[2789522.930054]  [<ffffffff8151cae8>] ? schedule+0x134/0x86a
     415<Feb/01 03:14 pm>[2789522.935853]  [<ffffffff8151cae8>] schedule+0x134/0x86a
     416
     417<Feb/01 03:14 pm>[2789522.941514]  [<ffffffff8151e357>] do_nanosleep+0x73/0xb3
     418<Feb/01 03:14 pm>[2789522.947340]  [<ffffffff81058477>] hrtimer_nanosleep+0x97/0x105
     419<Feb/01 03:14 pm>[2789522.953697]  [<ffffffff81057842>] ? hrtimer_wakeup+0x0/0x21
     420<Feb/01 03:14 pm>[2789522.959782]  [<ffffffff810583de>] ? hrtimer_start_range_ns+0xf/0x11
     421<Feb/01 03:14 pm>[2789522.966571]  [<ffffffff81058539>] sys_nanosleep+0x54/0x6c
     422<Feb/01 03:14 pm>[2789522.972462]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
     423<Feb/01 03:14 pm>[2789522.979025] Pid: 30947, comm: psphotStack Tainted: G      D    2.6.34.4 #3
     424<Feb/01 03:14 pm>[2789522.986471] Call Trace:
     425<Feb/01 03:14 pm>[2789522.989351]  <NMI>  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
     426<Feb/01 03:14 pm>[2789522.995788]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
     427<Feb/01 03:14 pm>[2789523.002149]  [<ffffffff81520360>] do_nmi+0xb2/0x259
     428<Feb/01 03:14 pm>[2789523.007523]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
     429<Feb/01 03:14 pm>[2789523.012508]  [<ffffffff812c08b7>] ? delay_tsc+0x2d/0x59
     430<Feb/01 03:14 pm>[2789523.018250]  <<EOE>>  [<ffffffff812c093d>] __const_udelay+0x3d/0x3f
     431<Feb/01 03:14 pm>[2789523.025058]  [<ffffffff81019da9>] arch_trigger_all_cpu_backtrace+0x51/0x5d
     432                 <Feb/01 03:14 pm>[2789523.032467]  [<ffffffff812c4ea4>] do_raw_spin_lock+0x11d/0x147
     433<Feb/01 03:14 pm>[2789523.038826]  [<ffffffff8151f115>] _raw_spin_lock_irq+0x35/0x39
     434<Feb/01 03:14 pm>[2789523.045164]  [<ffffffff8151cae8>] ? schedule+0x134/0x86a
     435<Feb/01 03:14 pm>[2789523.050953]  [<ffffffff8151cae8>] schedule+0x134/0x86a
     436<Feb/01 03:14 pm>[2789523.056587]  [<ffffffff8151e357>] do_nanosleep+0x73/0xb3
     437<Feb/01 03:14 pm>[2789523.062381]  [<ffffffff81058477>] hrtimer_nanosleep+0x97/0x105
     438<Feb/01 03:14 pm>[2789523.068709]  [<ffffffff81057842>] ? hrtimer_wakeup+0x0/0x21
     439
     440<Feb/01 03:14 pm>[2789523.074796]  [<ffffffff810583de>] ? hrtimer_start_range_ns+0xf/0x11
     441<Feb/01 03:14 pm>[2789523.081584]  [<ffffffff81058539>] sys_nanosleep+0x54/0x6c
     442<Feb/01 03:14 pm>[2789523.087460]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
     443<Feb/01 03:14 pm>[2789523.093997] NMI backtrace for cpu 6
     444<Feb/01 03:14 pm>[2789523.097946] CPU 6
     445<Feb/01 03:14 pm>[2789523.100050] Modules linked in: w83793 hwmon_vid autofs4 i2c_i801 i2c_core iTCO_wdt e1000e ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
     446<Feb/01 03:14 pm>[2789523.135742]
     447<Feb/01 03:14 pm>[2789523.137643] Pid: 609, comm: kswapd1 Tainted: G      D    2.6.34.4 #3 S7010/empty
     448<Feb/01 03:14 pm>[2789523.145796] RIP: 0010:[<ffffffff812c08a0>]  [<ffffffff812c08a0>] delay_tsc+0x16/0x59
     449<Feb/01 03:14 pm>[2789523.154290] RSP: 0018:ffff880655203b38  EFLAGS: 00000092
     450<Feb/01 03:14 pm>[2789523.160104] RAX: 0000000000000000 RBX: 000000005d409918 RCX: 0000000052985480
     451<Feb/01 03:14 pm>[2789523.167942] RDX: 000000000000ac00 RSI: 0000000000000006 RDI: 0000000000000001
     452<Feb/01 03:14 pm>[2789523.175799] RBP: ffff880655203b40 R08: 0000000000000006 R09: 0000000000000006
     453<Feb/01 03:14 pm>[2789523.183675] R10: 0000000000000000 R11: ffff880655203970 R12: ffff8800026b3c00
     454<Feb/01 03:14 pm>[2789523.191541] R13: 0000000000000000 R14: ffff880623f740c0 R15: ffff880623f74548
     455<Feb/01 03:14 pm>[2789523.199399] FS:  0000000000000000(0000) GS:ffff880655200000(0000) knlGS:0000000000000000
     456<Feb/01 03:14 pm>[2789523.208230] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
     457<Feb/01 03:14 pm>[2789523.214478] CR2: 00007fbc51194c70 CR3: 0000000423990000 CR4: 00000000000006e0
     458<Feb/01 03:14 pm>[2789523.222332] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     459<Feb/01 03:14 pm>[2789523.230156] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     460<Feb/01 03:14 pm>[2789523.237989] Process kswapd1 (pid: 609, threadinfo ffff880623f6e000, task ffff880623f740c0)
     461<Feb/01 03:14 pm>[2789523.247027] Stack:
     462
     463<Feb/01 03:14 pm>[2789523.249458]  ffffffff81019da9 ffff880655203b50 ffffffff812c08fe ffff880655203b90
     464<Feb/01 03:14 pm>[2789523.257374] <0> ffffffff812c4e65 000000009ef30400 ffff8800026b3c00 ffff88000af03090
     465<Feb/01 03:14 pm>[2789523.266038] <0> 0000000000013c00 ffff880655203c00 0000000000000006 ffff880655203bb0
     466<Feb/01 03:14 pm>[2789523.275064] Call Trace:
     467<Feb/01 03:14 pm>[2789523.277925]  <IRQ>
     468<Feb/01 03:14 pm>[2789523.280505]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     469<Feb/01 03:14 pm>[2789523.288245]  [<ffffffff812c08fe>] __delay+0xa/0xc
     470<Feb/01 03:14 pm>[2789523.293452]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     471<Feb/01 03:14 pm>[2789523.299730]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     472<Feb/01 03:14 pm>[2789523.305730]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
     473<Feb/01 03:14 pm>[2789523.311737]  [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
     474<Feb/01 03:14 pm>[2789523.317554]  [<ffffffff81036578>] try_to_wake_up+0x2e/0x342
     475<Feb/01 03:14 pm>[2789523.323650]  [<ffffffff81036899>] default_wake_function+0xd/0xf
     476<Feb/01 03:14 pm>[2789523.330100]  [<ffffffff81054b7d>] autoremove_wake_function+0x11/0x38
     477<Feb/01 03:14 pm>[2789523.336983]  [<ffffffff8102c08d>] __wake_up_common+0x46/0x76
     478<Feb/01 03:14 pm>[2789523.343161]  [<ffffffff8102faa6>] __wake_up+0x38/0x4f
     479<Feb/01 03:14 pm>[2789523.348717]  [<ffffffff812a9603>] __freed_request+0x69/0x106
     480<Feb/01 03:14 pm>[2789523.354875]  [<ffffffff812a96c6>] freed_request+0x26/0x43
     481<Feb/01 03:14 pm>[2789523.360760]  [<ffffffff812a979a>] __blk_put_request+0xb7/0xc0
     482<Feb/01 03:14 pm>[2789523.367045]  [<ffffffff812a99c4>] blk_finish_request+0x221/0x230
     483<Feb/01 03:14 pm>[2789523.373556]  [<ffffffff812a9839>] ? blk_finish_request+0x96/0x230
     484<Feb/01 03:14 pm>[2789523.380169]  [<ffffffff812a9ca9>] blk_end_bidi_request+0x3a/0x56
     485
     486<Feb/01 03:14 pm>[2789523.386694]  [<ffffffff812a9cf7>] blk_end_request+0xb/0xd
     487<Feb/01 03:14 pm>[2789523.392589]  [<ffffffff813628b9>] scsi_io_completion+0x18c/0x407
     488                 <Feb/01 03:14 pm>[2789523.399143]  [<ffffffff8135c93e>] scsi_finish_command+0xad/0xb6
     489<Feb/01 03:14 pm>[2789523.405583]  [<ffffffff81362c61>] scsi_softirq_done+0x11d/0x125
     490<Feb/01 03:14 pm>[2789523.412015]  [<ffffffff8105ccb8>] ? ktime_get+0x65/0xc4
     491<Feb/01 03:14 pm>[2789523.417735]  [<ffffffff812aea39>] blk_done_softirq+0x69/0x79
     492<Feb/01 03:14 pm>[2789523.423890]  [<ffffffff8104103c>] __do_softirq+0xd8/0x194
     493<Feb/01 03:14 pm>[2789523.429846]  [<ffffffff8100388c>] call_softirq+0x1c/0x28
     494<Feb/01 03:14 pm>[2789523.435663]  [<ffffffff81004e7f>] do_softirq+0x33/0x68
     495<Feb/01 03:14 pm>[2789523.441307]  [<ffffffff81040ec2>] irq_exit+0x36/0x75
     496<Feb/01 03:14 pm>[2789523.446768]  [<ffffffff81018828>] smp_call_function_single_interrupt+0x2f/0x31
     497<Feb/01 03:14 pm>[2789523.454704]  [<ffffffff810034f3>] call_function_single_interrupt+0x13/0x20
     498<Feb/01 03:14 pm>[2789523.462093]  <EOI>
     499<Feb/01 03:14 pm>[2789523.464679]  [<ffffffff810d48fd>] ? get_swap_page+0x13c/0x5bb
     500<Feb/01 03:14 pm>[2789523.470943]  [<ffffffff8151f67a>] ? _raw_spin_unlock+0x26/0x2b
     501<Feb/01 03:14 pm>[2789523.477304]  [<ffffffff810d17e9>] add_to_swap+0xf/0x48
     502                 <Feb/01 03:14 pm>[2789523.482956]  [<ffffffff810b7464>] shrink_page_list+0x214/0x68f
     503<Feb/01 03:14 pm>[2789523.489312]  [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
     504<Feb/01 03:14 pm>[2789523.495232]  [<ffffffff8102fc7b>] ? cpuacct_charge+0x76/0x96
     505<Feb/01 03:14 pm>[2789523.501423]  [<ffffffff81058774>] ? up_read+0x1e/0x36
     506<Feb/01 03:14 pm>[2789523.506978]  [<ffffffff810b931a>] kswapd+0x61a/0x8bb
     507<Feb/01 03:14 pm>[2789523.512442]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
     508
     509<Feb/01 03:14 pm>[2789523.519136]  [<ffffffff81054b6c>] ? autoremove_wake_function+0x0/0x38
     510<Feb/01 03:14 pm>[2789523.526088]  [<ffffffff810b8d00>] ? kswapd+0x0/0x8bb
     511<Feb/01 03:14 pm>[2789523.531563]  [<ffffffff810546e2>] kthread+0x7d/0x85
     512<Feb/01 03:14 pm>[2789523.536939]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     513<Feb/01 03:14 pm>[2789523.543349]  [<ffffffff81054665>] ? kthread+0x0/0x85
     514<Feb/01 03:14 pm>[2789523.548808]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     515<Feb/01 03:14 pm>[2789523.555408] Code: 66 66 2e 0f 1f 84 00 00 00 00 00 48 ff c8 75 fb 48 ff c8 c9 c3 55 48 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae e8 <0f> 31 89 c1 41 89 f1 66 66 90 0f ae e8 0f 31 89 c6 48 89 f0 48
     516<Feb/01 03:14 pm>[2789523.578579] Call Trace:
     517<Feb/01 03:14 pm>[2789523.581441]  <IRQ>  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     518<Feb/01 03:14 pm>[2789523.594389]  [<ffffffff812c08fe>] __delay+0xa/0xc
     519<Feb/01 03:14 pm>[2789523.599568]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     520<Feb/01 03:14 pm>[2789523.605838]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     521<Feb/01 03:14 pm>[2789523.611809]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
     522<Feb/01 03:14 pm>[2789523.617818]  [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
     523<Feb/01 03:14 pm>[2789523.623618]  [<ffffffff81036578>] try_to_wake_up+0x2e/0x342
     524<Feb/01 03:14 pm>[2789523.629697]  [<ffffffff81036899>] default_wake_function+0xd/0xf
     525<Feb/01 03:14 pm>[2789523.636130]  [<ffffffff81054b7d>] autoremove_wake_function+0x11/0x38
     526<Feb/01 03:14 pm>[2789523.642968]  [<ffffffff8102c08d>] __wake_up_common+0x46/0x76
     527<Feb/01 03:14 pm>[2789523.649137]  [<ffffffff8102faa6>] __wake_up+0x38/0x4f
     528<Feb/01 03:14 pm>[2789523.654710]  [<ffffffff812a9603>] __freed_request+0x69/0x106
     529<Feb/01 03:14 pm>[2789523.660860]  [<ffffffff812a96c6>] freed_request+0x26/0x43
     530<Feb/01 03:14 pm>[2789523.666772]  [<ffffffff812a979a>] __blk_put_request+0xb7/0xc0
     531
     532<Feb/01 03:14 pm>[2789523.673023]  [<ffffffff812a99c4>] blk_finish_request+0x221/0x230
     533<Feb/01 03:14 pm>[2789523.679559]  [<ffffffff812a9839>] ? blk_finish_request+0x96/0x230
     534<Feb/01 03:14 pm>[2789523.686182]  [<ffffffff812a9ca9>] blk_end_bidi_request+0x3a/0x56
     535<Feb/01 03:14 pm>[2789523.692715]  [<ffffffff812a9cf7>] blk_end_request+0xb/0xd
     536<Feb/01 03:14 pm>[2789523.698618]  [<ffffffff813628b9>] scsi_io_completion+0x18c/0x407
     537<Feb/01 03:14 pm>[2789523.705164]  [<ffffffff8135c93e>] scsi_finish_command+0xad/0xb6
     538<Feb/01 03:14 pm>[2789523.711629]  [<ffffffff81362c61>] scsi_softirq_done+0x11d/0x125
     539<Feb/01 03:14 pm>[2789523.718078]  [<ffffffff8105ccb8>] ? ktime_get+0x65/0xc4
     540<Feb/01 03:14 pm>[2789523.723824]  [<ffffffff812aea39>] blk_done_softirq+0x69/0x79
     541<Feb/01 03:14 pm>[2789523.729997]  [<ffffffff8104103c>] __do_softirq+0xd8/0x194
     542<Feb/01 03:14 pm>[2789523.735873]  [<ffffffff8100388c>] call_softirq+0x1c/0x28
     543<Feb/01 03:14 pm>[2789523.741665]  [<ffffffff81004e7f>] do_softirq+0x33/0x68
     544<Feb/01 03:14 pm>[2789523.747302]  [<ffffffff81040ec2>] irq_exit+0x36/0x75
     545<Feb/01 03:14 pm>[2789523.752782]  [<ffffffff81018828>] smp_call_function_single_interrupt+0x2f/0x31
     546<Feb/01 03:14 pm>[2789523.760715]  [<ffffffff810034f3>] call_function_single_interrupt+0x13/0x20
     547<Feb/01 03:14 pm>[2789523.768121]  <EOI>  [<ffffffff810d48fd>] ? get_swap_page+0x13c/0x5bb
     548<Feb/01 03:14 pm>[2789523.775091]  [<ffffffff8151f67a>] ? _raw_spin_unlock+0x26/0x2b
     549<Feb/01 03:14 pm>[2789523.781423]  [<ffffffff810d17e9>] add_to_swap+0xf/0x48
     550<Feb/01 03:14 pm>[2789523.787100]  [<ffffffff810b7464>] shrink_page_list+0x214/0x68f
     551<Feb/01 03:14 pm>[2789523.793423]  [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
     552<Feb/01 03:14 pm>[2789523.799306]  [<ffffffff8102fc7b>] ? cpuacct_charge+0x76/0x96
     553<Feb/01 03:14 pm>[2789523.805462]  [<ffffffff81058774>] ? up_read+0x1e/0x36
     554
     555<Feb/01 03:14 pm>[2789523.811001]  [<ffffffff810b931a>] kswapd+0x61a/0x8bb
     556<Feb/01 03:14 pm>[2789523.816439]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
     557<Feb/01 03:14 pm>[2789523.823118]  [<ffffffff81054b6c>] ? autoremove_wake_function+0x0/0x38
     558                 <Feb/01 03:14 pm>[2789523.830083]  [<ffffffff810b8d00>] ? kswapd+0x0/0x8bb
     559<Feb/01 03:14 pm>[2789523.835535]  [<ffffffff810546e2>] kthread+0x7d/0x85
     560<Feb/01 03:14 pm>[2789523.840903]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     561<Feb/01 03:14 pm>[2789523.847347]  [<ffffffff81054665>] ? kthread+0x0/0x85
     562<Feb/01 03:14 pm>[2789523.852797]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     563<Feb/01 03:14 pm>[2789523.859422] Pid: 609, comm: kswapd1 Tainted: G      D    2.6.34.4 #3
     564<Feb/01 03:14 pm>[2789523.866288] Call Trace:
     565<Feb/01 03:14 pm>[2789523.869161]  <NMI>  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
     566<Feb/01 03:14 pm>[2789523.875537]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
     567<Feb/01 03:14 pm>[2789523.881914]  [<ffffffff81520360>] do_nmi+0xb2/0x259
     568<Feb/01 03:14 pm>[2789523.887279]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
     569<Feb/01 03:14 pm>[2789523.892318]  [<ffffffff812c08a0>] ? delay_tsc+0x16/0x59
     570<Feb/01 03:14 pm>[2789523.898059]  <<EOE>>  <IRQ>  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     571<Feb/01 03:14 pm>[2789523.907343]  [<ffffffff812c08fe>] __delay+0xa/0xc
     572<Feb/01 03:14 pm>[2789523.912538]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     573<Feb/01 03:14 pm>[2789523.918810]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     574<Feb/01 03:14 pm>[2789523.924808]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
     575<Feb/01 03:14 pm>[2789523.930805]  [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
     576<Feb/01 03:14 pm>[2789523.936598]  [<ffffffff81036578>] try_to_wake_up+0x2e/0x342
     577
     578<Feb/01 03:14 pm>[2789523.942693]  [<ffffffff81036899>] default_wake_function+0xd/0xf
     579<Feb/01 03:14 pm>[2789523.949143]  [<ffffffff81054b7d>] autoremove_wake_function+0x11/0x38
     580<Feb/01 03:14 pm>[2789523.956016]  [<ffffffff8102c08d>] __wake_up_common+0x46/0x76
     581                 <Feb/01 03:14 pm>[2789523.962177]  [<ffffffff8102faa6>] __wake_up+0x38/0x4f
     582<Feb/01 03:14 pm>[2789523.967734]  [<ffffffff812a9603>] __freed_request+0x69/0x106
     583<Feb/01 03:14 pm>[2789523.973890]  [<ffffffff812a96c6>] freed_request+0x26/0x43
     584<Feb/01 03:14 pm>[2789523.979777]  [<ffffffff812a979a>] __blk_put_request+0xb7/0xc0
     585<Feb/01 03:14 pm>[2789523.986037]  [<ffffffff812a99c4>] blk_finish_request+0x221/0x230
     586<Feb/01 03:14 pm>[2789523.992572]  [<ffffffff812a9839>] ? blk_finish_request+0x96/0x230
     587<Feb/01 03:14 pm>[2789523.999160]  [<ffffffff812a9ca9>] blk_end_bidi_request+0x3a/0x56
     588<Feb/01 03:14 pm>[2789524.005677]  [<ffffffff812a9cf7>] blk_end_request+0xb/0xd
     589<Feb/01 03:14 pm>[2789524.011597]  [<ffffffff813628b9>] scsi_io_completion+0x18c/0x407
     590<Feb/01 03:14 pm>[2789524.018125]  [<ffffffff8135c93e>] scsi_finish_command+0xad/0xb6
     591<Feb/01 03:14 pm>[2789524.024564]  [<ffffffff81362c61>] scsi_softirq_done+0x11d/0x125
     592<Feb/01 03:14 pm>[2789524.031013]  [<ffffffff8105ccb8>] ? ktime_get+0x65/0xc4
     593<Feb/01 03:14 pm>[2789524.036734]  [<ffffffff812aea39>] blk_done_softirq+0x69/0x79
     594<Feb/01 03:14 pm>[2789524.042900]  [<ffffffff8104103c>] __do_softirq+0xd8/0x194
     595<Feb/01 03:14 pm>[2789524.048777]  [<ffffffff8100388c>] call_softirq+0x1c/0x28
     596<Feb/01 03:14 pm>[2789524.054601]  [<ffffffff81004e7f>] do_softirq+0x33/0x68
     597<Feb/01 03:14 pm>[2789524.060264]  [<ffffffff81040ec2>] irq_exit+0x36/0x75
     598<Feb/01 03:14 pm>[2789524.065717]  [<ffffffff81018828>] smp_call_function_single_interrupt+0x2f/0x31
     599<Feb/01 03:14 pm>[2789524.073676]  [<ffffffff810034f3>] call_function_single_interrupt+0x13/0x20
     600
     601<Feb/01 03:14 pm>[2789524.081067]  <EOI>  [<ffffffff810d48fd>] ? get_swap_page+0x13c/0x5bb
     602<Feb/01 03:14 pm>[2789524.087983]  [<ffffffff8151f67a>] ? _raw_spin_unlock+0x26/0x2b
     603<Feb/01 03:14 pm>[2789524.094309]  [<ffffffff810d17e9>] add_to_swap+0xf/0x48
     604<Feb/01 03:14 pm>[2789524.099976]  [<ffffffff810b7464>] shrink_page_list+0x214/0x68f
     605<Feb/01 03:14 pm>[2789524.106307]  [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
     606<Feb/01 03:14 pm>[2789524.112200]  [<ffffffff8102fc7b>] ? cpuacct_charge+0x76/0x96
     607<Feb/01 03:14 pm>[2789524.118391]  [<ffffffff81058774>] ? up_read+0x1e/0x36
     608<Feb/01 03:14 pm>[2789524.123954]  [<ffffffff810b931a>] kswapd+0x61a/0x8bb
     609<Feb/01 03:14 pm>[2789524.129401]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
     610<Feb/01 03:14 pm>[2789524.136105]  [<ffffffff81054b6c>] ? autoremove_wake_function+0x0/0x38
     611<Feb/01 03:14 pm>[2789524.143099]  [<ffffffff810b8d00>] ? kswapd+0x0/0x8bb
     612<Feb/01 03:14 pm>[2789524.148548]  [<ffffffff810546e2>] kthread+0x7d/0x85
     613<Feb/01 03:14 pm>[2789524.153898]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     614<Feb/01 03:14 pm>[2789524.160333]  [<ffffffff81054665>] ? kthread+0x0/0x85
     615<Feb/01 03:14 pm>[2789524.165802]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     616<Feb/01 03:14 pm>[2789524.172402] NMI backtrace for cpu 2
     617<Feb/01 03:14 pm>[2789524.176353] CPU 2
     618<Feb/01 03:14 pm>[2789524.178447] Modules linked in: w83793 hwmon_vid autofs4 i2c_i801 i2c_core iTCO_wdt e1000e ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
     619<Feb/01 03:14 pm>[2789524.214123]
     620<Feb/01 03:14 pm>[2789524.216047] Pid: 10992, comm: ppStack Tainted: G      D    2.6.34.4 #3 S7010/empty
     621<Feb/01 03:14 pm>[2789524.224314] RIP: 0010:[<ffffffff812c08ad>]  [<ffffffff812c08ad>] delay_tsc+0x23/0x59
     622<Feb/01 03:14 pm>[2789524.232816] RSP: 0018:ffff88004fa2f520  EFLAGS: 00000082
     623
     624<Feb/01 03:14 pm>[2789524.238607] RAX: 0000000052880a50 RBX: 000000005d3eee34 RCX: 0000000052880a50
     625<Feb/01 03:14 pm>[2789524.246462] RDX: 00000000001a7f64 RSI: 0000000000000002 RDI: 0000000000000001
     626<Feb/01 03:14 pm>[2789524.254303] RBP: ffff88004fa2f528 R08: 0000000000000002 R09: 0000000000000002
     627<Feb/01 03:14 pm>[2789524.262155] R10: 0000000000000000 R11: ffff88004fa2f358 R12: ffff880623190ed8
     628<Feb/01 03:14 pm>[2789524.270000] R13: 0000000000000000 R14: ffff880616c8a060 R15: ffff880616c8a4e8
     629<Feb/01 03:14 pm>[2789524.277842] FS:  00007f4df3d496f0(0000) GS:ffff880002640000(0000) knlGS:0000000000000000
     630<Feb/01 03:14 pm>[2789524.286652] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
     631<Feb/01 03:14 pm>[2789524.292912] CR2: 00000000f73166c0 CR3: 000000001b83e000 CR4: 00000000000006e0
     632<Feb/01 03:14 pm>[2789524.300747] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     633<Feb/01 03:14 pm>[2789524.308614] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     634<Feb/01 03:14 pm>[2789524.316479] Process ppStack (pid: 10992, threadinfo ffff88004fa2e000, task ffff880616c8a060)
     635<Feb/01 03:14 pm>[2789524.325684] Stack:
     636<Feb/01 03:14 pm>[2789524.328125]  ffffffff81019da9 ffff88004fa2f538 ffffffff812c08fe ffff88004fa2f578
     637<Feb/01 03:14 pm>[2789524.336059] <0> ffffffff812c4e65 000000009ef30400 ffff880623190ed8 ffff880623190a78
     638<Feb/01 03:14 pm>[2789524.344720] <0> 0000000000000001 ffff8809d5858e40 ffff88004fa2f5c8 ffff88004fa2f598
     639<Feb/01 03:14 pm>[2789524.353758] Call Trace:
     640<Feb/01 03:14 pm>[2789524.356637]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     641<Feb/01 03:14 pm>[2789524.364407]  [<ffffffff812c08fe>] __delay+0xa/0xc
     642<Feb/01 03:14 pm>[2789524.369584]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     643<Feb/01 03:14 pm>[2789524.375829]  [<ffffffff8151f115>] _raw_spin_lock_irq+0x35/0x39
     644<Feb/01 03:14 pm>[2789524.382157]  [<ffffffff812aaa39>] ? get_request_wait+0x155/0x192
     645<Feb/01 03:14 pm>[2789524.388676]  [<ffffffff812aaa39>] get_request_wait+0x155/0x192
     646
     647<Feb/01 03:14 pm>[2789524.395031]  [<ffffffff81054b6c>] ? autoremove_wake_function+0x0/0x38
     648<Feb/01 03:14 pm>[2789524.402000]  [<ffffffff812a3a03>] ? elv_merge+0x16a/0x1a4
     649<Feb/01 03:14 pm>[2789524.407907]  [<ffffffff812aaf95>] __make_request+0x31c/0x445
     650<Feb/01 03:14 pm>[2789524.414097]  [<ffffffff812a946b>] generic_make_request+0x32a/0x39c
     651<Feb/01 03:14 pm>[2789524.420810]  [<ffffffff810b3482>] ? test_set_page_writeback+0xe3/0x10e
     652<Feb/01 03:14 pm>[2789524.427838]  [<ffffffff812a9591>] submit_bio+0xb4/0xbd
     653<Feb/01 03:14 pm>[2789524.433468]  [<ffffffff810ac9a9>] ? unlock_page+0x22/0x26
     654<Feb/01 03:14 pm>[2789524.439383]  [<ffffffff810d139c>] swap_writepage+0x8c/0x97
     655<Feb/01 03:14 pm>[2789524.445380]  [<ffffffff810b7612>] shrink_page_list+0x3c2/0x68f
     656<Feb/01 03:14 pm>[2789524.451710]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     657<Feb/01 03:14 pm>[2789524.458099]  [<ffffffff810e5fb2>] ? mem_cgroup_del_lru_list+0x7b/0x8e
     658<Feb/01 03:14 pm>[2789524.465094]  [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
     659<Feb/01 03:14 pm>[2789524.471022]  [<ffffffff81048174>] ? try_to_del_timer_sync+0xae/0xbb
     660<Feb/01 03:14 pm>[2789524.477793]  [<ffffffff81054cb3>] ? finish_wait+0x5d/0x65
     661<Feb/01 03:14 pm>[2789524.483704]  [<ffffffff81054cb3>] ? finish_wait+0x5d/0x65
     662<Feb/01 03:14 pm>[2789524.489607]  [<ffffffff81054b6c>] ? autoremove_wake_function+0x0/0x38
     663                 <Feb/01 03:14 pm>[2789524.496578]  [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
     664<Feb/01 03:14 pm>[2789524.503319]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
     665<Feb/01 03:14 pm>[2789524.509576]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
     666<Feb/01 03:14 pm>[2789524.516303]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
     667<Feb/01 03:14 pm>[2789524.523161]  [<ffffffff810da90f>] alloc_page_vma+0x148/0x14d
     668<Feb/01 03:14 pm>[2789524.529319]  [<ffffffff810c4d2a>] handle_mm_fault+0x294/0x73e
     669
     670<Feb/01 03:14 pm>[2789524.535587]  [<ffffffff815227e9>] do_page_fault+0x351/0x375
     671<Feb/01 03:14 pm>[2789524.541646]  [<ffffffff810348d8>] ? finish_task_switch+0x69/0xdd
     672                 <Feb/01 03:14 pm>[2789524.548193]  [<ffffffff8103486f>] ? finish_task_switch+0x0/0xdd
     673<Feb/01 03:14 pm>[2789524.554632]  [<ffffffff8151d152>] ? schedule+0x79e/0x86a
     674<Feb/01 03:14 pm>[2789524.560457]  [<ffffffff8151ef6f>] ? lockdep_sys_exit_thunk+0x35/0x67
     675<Feb/01 03:14 pm>[2789524.567324]  [<ffffffff810dce21>] ? add_partial+0x1a/0x6a
     676<Feb/01 03:14 pm>[2789524.573241]  [<ffffffff8151fb5f>] page_fault+0x1f/0x30
     677<Feb/01 03:14 pm>[2789524.578874] Code: c8 75 fb 48 ff c8 c9 c3 55 48 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae e8 0f 31 89 c1 41 89 f1 66 66 90 0f ae e8 <0f> 31 89 c6 48 89 f0 48 29 c8 48 39 f8 73 25 f3 90 65 44 8b 04
     678<Feb/01 03:14 pm>[2789524.602103] Call Trace:
     679<Feb/01 03:14 pm>[2789524.609501]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     680<Feb/01 03:14 pm>[2789524.617246]  [<ffffffff812c08fe>] __delay+0xa/0xc
     681<Feb/01 03:14 pm>[2789524.622458]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     682<Feb/01 03:14 pm>[2789524.628688]  [<ffffffff8151f115>] _raw_spin_lock_irq+0x35/0x39
     683<Feb/01 03:14 pm>[2789524.635059]  [<ffffffff812aaa39>] ? get_request_wait+0x155/0x192
     684<Feb/01 03:14 pm>[2789524.641560]  [<ffffffff812aaa39>] get_request_wait+0x155/0x192
     685<Feb/01 03:14 pm>[2789524.647895]  [<ffffffff81054b6c>] ? autoremove_wake_function+0x0/0x38
     686<Feb/01 03:14 pm>[2789524.654856]  [<ffffffff812a3a03>] ? elv_merge+0x16a/0x1a4
     687<Feb/01 03:14 pm>[2789524.660747]  [<ffffffff812aaf95>] __make_request+0x31c/0x445
     688<Feb/01 03:14 pm>[2789524.666930]  [<ffffffff812a946b>] generic_make_request+0x32a/0x39c
     689<Feb/01 03:14 pm>[2789524.673631]  [<ffffffff810b3482>] ? test_set_page_writeback+0xe3/0x10e
     690<Feb/01 03:14 pm>[2789524.680729]  [<ffffffff812a9591>] submit_bio+0xb4/0xbd
     691<Feb/01 03:14 pm>[2789524.686378]  [<ffffffff810ac9a9>] ? unlock_page+0x22/0x26
     692
     693<Feb/01 03:14 pm>[2789524.692264]  [<ffffffff810d139c>] swap_writepage+0x8c/0x97
     694<Feb/01 03:14 pm>[2789524.698256]  [<ffffffff810b7612>] shrink_page_list+0x3c2/0x68f
     695<Feb/01 03:14 pm>[2789524.704595]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     696<Feb/01 03:14 pm>[2789524.710929]  [<ffffffff810e5fb2>] ? mem_cgroup_del_lru_list+0x7b/0x8e
     697<Feb/01 03:14 pm>[2789524.717899]  [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
     698<Feb/01 03:14 pm>[2789524.723818]  [<ffffffff81048174>] ? try_to_del_timer_sync+0xae/0xbb
     699<Feb/01 03:14 pm>[2789524.730633]  [<ffffffff81054cb3>] ? finish_wait+0x5d/0x65
     700<Feb/01 03:14 pm>[2789524.736542]  [<ffffffff81054cb3>] ? finish_wait+0x5d/0x65
     701<Feb/01 03:14 pm>[2789524.742464]  [<ffffffff81054b6c>] ? autoremove_wake_function+0x0/0x38
     702<Feb/01 03:14 pm>[2789524.749417]  [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
     703                 <Feb/01 03:14 pm>[2789524.756114]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
     704<Feb/01 03:14 pm>[2789524.762365]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
     705<Feb/01 03:14 pm>[2789524.769065]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
     706                 <Feb/01 03:14 pm>[2789524.775965]  [<ffffffff810da90f>] alloc_page_vma+0x148/0x14d
     707<Feb/01 03:14 pm>[2789524.782143]  [<ffffffff810c4d2a>] handle_mm_fault+0x294/0x73e
     708<Feb/01 03:14 pm>[2789524.788409]  [<ffffffff815227e9>] do_page_fault+0x351/0x375
     709<Feb/01 03:14 pm>[2789524.794513]  [<ffffffff810348d8>] ? finish_task_switch+0x69/0xdd
     710<Feb/01 03:14 pm>[2789524.801040]  [<ffffffff8103486f>] ? finish_task_switch+0x0/0xdd
     711<Feb/01 03:14 pm>[2789524.807471]  [<ffffffff8151d152>] ? schedule+0x79e/0x86a
     712<Feb/01 03:14 pm>[2789524.813295]  [<ffffffff8151ef6f>] ? lockdep_sys_exit_thunk+0x35/0x67
     713<Feb/01 03:14 pm>[2789524.820164]  [<ffffffff810dce21>] ? add_partial+0x1a/0x6a
     714<Feb/01 03:14 pm>[2789524.826065]  [<ffffffff8151fb5f>] page_fault+0x1f/0x30
     715
     716<Feb/01 03:14 pm>[2789524.831724] Pid: 10992, comm: ppStack Tainted: G      D    2.6.34.4 #3
     717<Feb/01 03:14 pm>[2789524.838783] Call Trace:
     718<Feb/01 03:14 pm>[2789524.841680]  <NMI>  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
     719<Feb/01 03:14 pm>[2789524.848049]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
     720<Feb/01 03:14 pm>[2789524.854393]  [<ffffffff81520360>] do_nmi+0xb2/0x259
     721<Feb/01 03:14 pm>[2789524.859792]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
     722<Feb/01 03:14 pm>[2789524.864797]  [<ffffffff812c08ad>] ? delay_tsc+0x23/0x59
     723<Feb/01 03:14 pm>[2789524.870503]  <<EOE>>  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     724<Feb/01 03:14 pm>[2789524.879132]  [<ffffffff812c08fe>] __delay+0xa/0xc
     725<Feb/01 03:14 pm>[2789524.884341]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     726<Feb/01 03:14 pm>[2789524.890622]  [<ffffffff8151f115>] _raw_spin_lock_irq+0x35/0x39
     727<Feb/01 03:14 pm>[2789524.896976]  [<ffffffff812aaa39>] ? get_request_wait+0x155/0x192
     728<Feb/01 03:14 pm>[2789524.903487]  [<ffffffff812aaa39>] get_request_wait+0x155/0x192
     729<Feb/01 03:14 pm>[2789524.909839]  [<ffffffff81054b6c>] ? autoremove_wake_function+0x0/0x38
     730<Feb/01 03:14 pm>[2789524.916798]  [<ffffffff812a3a03>] ? elv_merge+0x16a/0x1a4
     731<Feb/01 03:14 pm>[2789524.922708]  [<ffffffff812aaf95>] __make_request+0x31c/0x445
     732<Feb/01 03:14 pm>[2789524.928890]  [<ffffffff812a946b>] generic_make_request+0x32a/0x39c
     733<Feb/01 03:14 pm>[2789524.935607]  [<ffffffff810b3482>] ? test_set_page_writeback+0xe3/0x10e
     734<Feb/01 03:14 pm>[2789524.942647]  [<ffffffff812a9591>] submit_bio+0xb4/0xbd
     735<Feb/01 03:14 pm>[2789524.948303]  [<ffffffff810ac9a9>] ? unlock_page+0x22/0x26
     736<Feb/01 03:14 pm>[2789524.954234]  [<ffffffff810d139c>] swap_writepage+0x8c/0x97
     737<Feb/01 03:14 pm>[2789524.960251]  [<ffffffff810b7612>] shrink_page_list+0x3c2/0x68f
     738
     739<Feb/01 03:14 pm>[2789524.966587]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     740<Feb/01 03:14 pm>[2789524.972918]  [<ffffffff810e5fb2>] ? mem_cgroup_del_lru_list+0x7b/0x8e
     741<Feb/01 03:14 pm>[2789524.979896]  [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
     742<Feb/01 03:14 pm>[2789524.985794]  [<ffffffff81048174>] ? try_to_del_timer_sync+0xae/0xbb
     743<Feb/01 03:14 pm>[2789524.992593]  [<ffffffff81054cb3>] ? finish_wait+0x5d/0x65
     744<Feb/01 03:14 pm>[2789524.998503]  [<ffffffff81054cb3>] ? finish_wait+0x5d/0x65
     745<Feb/01 03:14 pm>[2789525.004399]  [<ffffffff81054b6c>] ? autoremove_wake_function+0x0/0x38
     746<Feb/01 03:14 pm>[2789525.011377]  [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
     747<Feb/01 03:14 pm>[2789525.018075]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
     748<Feb/01 03:14 pm>[2789525.024361]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
     749<Feb/01 03:14 pm>[2789525.031068]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
     750<Feb/01 03:14 pm>[2789525.037933]  [<ffffffff810da90f>] alloc_page_vma+0x148/0x14d
     751<Feb/01 03:14 pm>[2789525.044086]  [<ffffffff810c4d2a>] handle_mm_fault+0x294/0x73e
     752<Feb/01 03:14 pm>[2789525.050335]  [<ffffffff815227e9>] do_page_fault+0x351/0x375
     753<Feb/01 03:14 pm>[2789525.056394]  [<ffffffff810348d8>] ? finish_task_switch+0x69/0xdd
     754<Feb/01 03:14 pm>[2789525.062924]  [<ffffffff8103486f>] ? finish_task_switch+0x0/0xdd
     755<Feb/01 03:14 pm>[2789525.069370]  [<ffffffff8151d152>] ? schedule+0x79e/0x86a
     756<Feb/01 03:14 pm>[2789525.075194]  [<ffffffff8151ef6f>] ? lockdep_sys_exit_thunk+0x35/0x67
     757<Feb/01 03:14 pm>[2789525.082107]  [<ffffffff810dce21>] ? add_partial+0x1a/0x6a
     758<Feb/01 03:14 pm>[2789525.088014]  [<ffffffff8151fb5f>] page_fault+0x1f/0x30
     759<Feb/01 03:14 pm>[2789525.093616] NMI backtrace for cpu 3
     760<Feb/01 03:14 pm>[2789525.097560] CPU 3
     761
     762<Feb/01 03:14 pm>[2789525.099637] Modules linked in: w83793 hwmon_vid autofs4 i2c_i801 i2c_core iTCO_wdt e1000e ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
     763<Feb/01 03:14 pm>[2789525.135455]
     764<Feb/01 03:14 pm>[2789525.137370] Pid: 12798, comm: pswarp Tainted: G      D    2.6.34.4 #3 S7010/empty
     765<Feb/01 03:14 pm>[2789525.145576] RIP: 0010:[<ffffffff812c08a2>]  [<ffffffff812c08a2>] delay_tsc+0x18/0x59
     766<Feb/01 03:14 pm>[2789525.154111] RSP: 0018:ffff880516a59520  EFLAGS: 00000082
     767<Feb/01 03:14 pm>[2789525.159928] RAX: 00000000528c1ce4 RBX: 000000005d327f3c RCX: 00000000528c1ca0
     768<Feb/01 03:14 pm>[2789525.167781] RDX: 00000000001a7f64 RSI: 0000000000000003 RDI: 0000000000000001
     769<Feb/01 03:14 pm>[2789525.175658] RBP: ffff880516a59528 R08: 0000000000000003 R09: 0000000000000003
     770<Feb/01 03:14 pm>[2789525.183507] R10: 0000000000000000 R11: ffff880516a59358 R12: ffff880623190ed8
     771<Feb/01 03:14 pm>[2789525.191348] R13: 0000000000000000 R14: ffff88000af040c0 R15: ffff88000af04548
     772<Feb/01 03:14 pm>[2789525.199187] FS:  00007f6b5483a6f0(0000) GS:ffff880002660000(0000) knlGS:0000000000000000
     773<Feb/01 03:14 pm>[2789525.208016] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
     774<Feb/01 03:14 pm>[2789525.214276] CR2: 000000000145d178 CR3: 0000000423990000 CR4: 00000000000006e0
     775<Feb/01 03:14 pm>[2789525.222137] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     776<Feb/01 03:14 pm>[2789525.230004] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     777<Feb/01 03:14 pm>[2789525.237853] Process pswarp (pid: 12798, threadinfo ffff880516a58000, task ffff88000af040c0)
     778<Feb/01 03:14 pm>[2789525.246892] Stack:
     779<Feb/01 03:14 pm>[2789525.249324]  ffffffff81019da9 ffff880516a59538 ffffffff812c08fe ffff880516a59578
     780<Feb/01 03:14 pm>[2789525.257242] <0> ffffffff812c4e65 000000009ef30400 ffff880623190ed8 ffff880623190a78
     781<Feb/01 03:14 pm>[2789525.265858] <0> 0000000000000001 ffff880c23816600 ffff880516a595c8 ffff880516a59598
     782<Feb/01 03:14 pm>[2789525.274914] Call Trace:
     783<Feb/01 03:14 pm>[2789525.277785]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     784
     785<Feb/01 03:14 pm>[2789525.285544]  [<ffffffff812c08fe>] __delay+0xa/0xc
     786<Feb/01 03:14 pm>[2789525.290723]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     787<Feb/01 03:14 pm>[2789525.296988]  [<ffffffff8151f115>] _raw_spin_lock_irq+0x35/0x39
     788<Feb/01 03:14 pm>[2789525.303349]  [<ffffffff812aaa39>] ? get_request_wait+0x155/0x192
     789<Feb/01 03:14 pm>[2789525.309859]  [<ffffffff812aaa39>] get_request_wait+0x155/0x192
     790<Feb/01 03:14 pm>[2789525.316238]  [<ffffffff81054b6c>] ? autoremove_wake_function+0x0/0x38
     791<Feb/01 03:14 pm>[2789525.323190]  [<ffffffff812a3a03>] ? elv_merge+0x16a/0x1a4
     792<Feb/01 03:14 pm>[2789525.329091]  [<ffffffff812aaf95>] __make_request+0x31c/0x445
     793<Feb/01 03:14 pm>[2789525.335271]  [<ffffffff812a946b>] generic_make_request+0x32a/0x39c
     794<Feb/01 03:14 pm>[2789525.341974]  [<ffffffff810b3482>] ? test_set_page_writeback+0xe3/0x10e
     795<Feb/01 03:14 pm>[2789525.349018]  [<ffffffff812a9591>] submit_bio+0xb4/0xbd
     796<Feb/01 03:14 pm>[2789525.354642]  [<ffffffff810ac9a9>] ? unlock_page+0x22/0x26
     797<Feb/01 03:14 pm>[2789525.360531]  [<ffffffff810d139c>] swap_writepage+0x8c/0x97
     798<Feb/01 03:14 pm>[2789525.366522]  [<ffffffff810b7612>] shrink_page_list+0x3c2/0x68f
     799<Feb/01 03:14 pm>[2789525.372877]  [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
     800<Feb/01 03:14 pm>[2789525.378771]  [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
     801<Feb/01 03:14 pm>[2789525.385505]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
     802<Feb/01 03:14 pm>[2789525.391781]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
     803<Feb/01 03:14 pm>[2789525.398491]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
     804<Feb/01 03:14 pm>[2789525.405347]  [<ffffffff810da90f>] alloc_page_vma+0x148/0x14d
     805<Feb/01 03:14 pm>[2789525.411541]  [<ffffffff810c4d2a>] handle_mm_fault+0x294/0x73e
     806<Feb/01 03:14 pm>[2789525.417801]  [<ffffffff815227e9>] do_page_fault+0x351/0x375
     807
     808<Feb/01 03:14 pm>[2789525.423887]  [<ffffffff810348d8>] ? finish_task_switch+0x69/0xdd
     809<Feb/01 03:14 pm>[2789525.430404]  [<ffffffff8103486f>] ? finish_task_switch+0x0/0xdd
     810<Feb/01 03:14 pm>[2789525.436836]  [<ffffffff8151d152>] ? schedule+0x79e/0x86a
     811<Feb/01 03:14 pm>[2789525.442645]  [<ffffffff8151ef6f>] ? lockdep_sys_exit_thunk+0x35/0x67
     812<Feb/01 03:14 pm>[2789525.449509]  [<ffffffff810c4dda>] ? handle_mm_fault+0x344/0x73e
     813<Feb/01 03:14 pm>[2789525.455941]  [<ffffffff8151fb5f>] page_fault+0x1f/0x30
     814<Feb/01 03:14 pm>[2789525.461582] Code: 2e 0f 1f 84 00 00 00 00 00 48 ff c8 75 fb 48 ff c8 c9 c3 55 48 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae e8 0f 31 <89> c1 41 89 f1 66 66 90 0f ae e8 0f 31 89 c6 48 89 f0 48 29 c8
     815<Feb/01 03:14 pm>[2789525.484807] Call Trace:
     816<Feb/01 03:14 pm>[2789525.487696]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     817                 <Feb/01 03:14 pm>[2789525.495466]  [<ffffffff812c08fe>] __delay+0xa/0xc
     818<Feb/01 03:14 pm>[2789525.500661]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     819<Feb/01 03:14 pm>[2789525.506906]  [<ffffffff8151f115>] _raw_spin_lock_irq+0x35/0x39
     820<Feb/01 03:14 pm>[2789525.513285]  [<ffffffff812aaa39>] ? get_request_wait+0x155/0x192
     821<Feb/01 03:14 pm>[2789525.519815]  [<ffffffff812aaa39>] get_request_wait+0x155/0x192
     822<Feb/01 03:14 pm>[2789525.526175]  [<ffffffff81054b6c>] ? autoremove_wake_function+0x0/0x38
     823<Feb/01 03:14 pm>[2789525.533146]  [<ffffffff812a3a03>] ? elv_merge+0x16a/0x1a4
     824<Feb/01 03:14 pm>[2789525.539029]  [<ffffffff812aaf95>] __make_request+0x31c/0x445
     825<Feb/01 03:14 pm>[2789525.545203]  [<ffffffff812a946b>] generic_make_request+0x32a/0x39c
     826<Feb/01 03:14 pm>[2789525.551885]  [<ffffffff810b3482>] ? test_set_page_writeback+0xe3/0x10e
     827<Feb/01 03:14 pm>[2789525.558941]  [<ffffffff812a9591>] submit_bio+0xb4/0xbd
     828<Feb/01 03:14 pm>[2789525.564562]  [<ffffffff810ac9a9>] ? unlock_page+0x22/0x26
     829<Feb/01 03:14 pm>[2789525.570467]  [<ffffffff810d139c>] swap_writepage+0x8c/0x97
     830
     831<Feb/01 03:14 pm>[2789525.576441]  [<ffffffff810b7612>] shrink_page_list+0x3c2/0x68f
     832<Feb/01 03:14 pm>[2789525.582771]  [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
     833<Feb/01 03:14 pm>[2789525.588664]  [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
     834<Feb/01 03:14 pm>[2789525.595367]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
     835<Feb/01 03:14 pm>[2789525.601607]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
     836<Feb/01 03:14 pm>[2789525.608325]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
     837<Feb/01 03:14 pm>[2789525.615206]  [<ffffffff810da90f>] alloc_page_vma+0x148/0x14d
     838<Feb/01 03:14 pm>[2789525.621394]  [<ffffffff810c4d2a>] handle_mm_fault+0x294/0x73e
     839<Feb/01 03:14 pm>[2789525.627643]  [<ffffffff815227e9>] do_page_fault+0x351/0x375
     840<Feb/01 03:14 pm>[2789525.633729]  [<ffffffff810348d8>] ? finish_task_switch+0x69/0xdd
     841<Feb/01 03:14 pm>[2789525.644758]  [<ffffffff8103486f>] ? finish_task_switch+0x0/0xdd
     842<Feb/01 03:14 pm>[2789525.651182]  [<ffffffff8151d152>] ? schedule+0x79e/0x86a
     843<Feb/01 03:14 pm>[2789525.656990]  [<ffffffff8151ef6f>] ? lockdep_sys_exit_thunk+0x35/0x67
     844<Feb/01 03:14 pm>[2789525.663857]  [<ffffffff810c4dda>] ? handle_mm_fault+0x344/0x73e
     845<Feb/01 03:14 pm>[2789525.670312]  [<ffffffff8151fb5f>] page_fault+0x1f/0x30
     846<Feb/01 03:14 pm>[2789525.675937] Pid: 12798, comm: pswarp Tainted: G      D    2.6.34.4 #3
     847<Feb/01 03:14 pm>[2789525.682908] Call Trace:
     848<Feb/01 03:14 pm>[2789525.685772]  <NMI>  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
     849<Feb/01 03:14 pm>[2789525.692159]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
     850<Feb/01 03:14 pm>[2789525.698494]  [<ffffffff81520360>] do_nmi+0xb2/0x259
     851<Feb/01 03:14 pm>[2789525.703850]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
     852<Feb/01 03:14 pm>[2789525.708880]  [<ffffffff812c08a2>] ? delay_tsc+0x18/0x59
     853
     854<Feb/01 03:14 pm>[2789525.714621]  <<EOE>>  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     855<Feb/01 03:14 pm>[2789525.723215]  [<ffffffff812c08fe>] __delay+0xa/0xc
     856<Feb/01 03:14 pm>[2789525.728434]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     857<Feb/01 03:14 pm>[2789525.734688]  [<ffffffff8151f115>] _raw_spin_lock_irq+0x35/0x39
     858                 <Feb/01 03:14 pm>[2789525.741024]  [<ffffffff812aaa39>] ? get_request_wait+0x155/0x192
     859<Feb/01 03:14 pm>[2789525.747524]  [<ffffffff812aaa39>] get_request_wait+0x155/0x192
     860<Feb/01 03:14 pm>[2789525.753878]  [<ffffffff81054b6c>] ? autoremove_wake_function+0x0/0x38
     861                 <Feb/01 03:14 pm>[2789525.760855]  [<ffffffff812a3a03>] ? elv_merge+0x16a/0x1a4
     862<Feb/01 03:14 pm>[2789525.766765]  [<ffffffff812aaf95>] __make_request+0x31c/0x445
     863<Feb/01 03:14 pm>[2789525.772948]  [<ffffffff812a946b>] generic_make_request+0x32a/0x39c
     864<Feb/01 03:14 pm>[2789525.779632]  [<ffffffff810b3482>] ? test_set_page_writeback+0xe3/0x10e
     865<Feb/01 03:14 pm>[2789525.786685]  [<ffffffff812a9591>] submit_bio+0xb4/0xbd
     866<Feb/01 03:14 pm>[2789525.792333]  [<ffffffff810ac9a9>] ? unlock_page+0x22/0x26
     867<Feb/01 03:14 pm>[2789525.798248]  [<ffffffff810d139c>] swap_writepage+0x8c/0x97
     868<Feb/01 03:14 pm>[2789525.804264]  [<ffffffff810b7612>] shrink_page_list+0x3c2/0x68f
     869<Feb/01 03:14 pm>[2789525.810638]  [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
     870<Feb/01 03:14 pm>[2789525.816558]  [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
     871<Feb/01 03:14 pm>[2789525.823258]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
     872<Feb/01 03:14 pm>[2789525.829499]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
     873<Feb/01 03:14 pm>[2789525.836225]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
     874<Feb/01 03:14 pm>[2789525.843108]  [<ffffffff810da90f>] alloc_page_vma+0x148/0x14d
     875<Feb/01 03:14 pm>[2789525.849285]  [<ffffffff810c4d2a>] handle_mm_fault+0x294/0x73e
     876
     877<Feb/01 03:14 pm>[2789525.855570]  [<ffffffff815227e9>] do_page_fault+0x351/0x375
     878<Feb/01 03:14 pm>[2789525.861623]  [<ffffffff810348d8>] ? finish_task_switch+0x69/0xdd
     879<Feb/01 03:14 pm>[2789525.868157]  [<ffffffff8103486f>] ? finish_task_switch+0x0/0xdd
     880<Feb/01 03:14 pm>[2789525.874589]  [<ffffffff8151d152>] ? schedule+0x79e/0x86a
     881<Feb/01 03:14 pm>[2789525.880397]  [<ffffffff8151ef6f>] ? lockdep_sys_exit_thunk+0x35/0x67
     882<Feb/01 03:14 pm>[2789525.887341]  [<ffffffff810c4dda>] ? handle_mm_fault+0x344/0x73e
     883<Feb/01 03:14 pm>[2789525.893763]  [<ffffffff8151fb5f>] page_fault+0x1f/0x30
     884<Feb/01 03:14 pm>[2789525.899422] NMI backtrace for cpu 4
     885<Feb/01 03:14 pm>[2789525.903393] CPU 4
     886<Feb/01 03:14 pm>[2789525.905486] Modules linked in: w83793 hwmon_vid autofs4 i2c_i801 i2c_core iTCO_wdt e1000e ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
     887<Feb/01 03:14 pm>[2789525.941266]
     888<Feb/01 03:14 pm>[2789525.943167] Pid: 9738, comm: pclient Tainted: G      D    2.6.34.4 #3 S7010/empty
     889<Feb/01 03:14 pm>[2789525.951356] RIP: 0010:[<ffffffff812c08af>]  [<ffffffff812c08af>] delay_tsc+0x25/0x59
     890<Feb/01 03:14 pm>[2789525.959890] RSP: 0018:ffff8806cda05b90  EFLAGS: 00000086
     891<Feb/01 03:14 pm>[2789525.965714] RAX: 0000000052902f94 RBX: 000000005d0e8a86 RCX: 0000000052902f74
     892<Feb/01 03:14 pm>[2789525.973543] RDX: 00000000001a7f64 RSI: 0000000000000004 RDI: 0000000000000001
     893<Feb/01 03:14 pm>[2789525.981375] RBP: ffff8806cda05b98 R08: 0000000000000004 R09: 0000000000000004
     894<Feb/01 03:14 pm>[2789525.989209] R10: 0000000000000000 R11: ffff8806cda059c8 R12: ffff8800026b3c00
     895<Feb/01 03:14 pm>[2789525.997055] R13: 0000000000000000 R14: ffff880c18cc8000 R15: ffff880c18cc8488
     896<Feb/01 03:14 pm>[2789526.004924] FS:  00007f6a9f7756f0(0000) GS:ffff880002680000(0000) knlGS:0000000000000000
     897<Feb/01 03:14 pm>[2789526.013761] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
     898<Feb/01 03:14 pm>[2789526.019995] CR2: 00007f1ccb460000 CR3: 0000000b8dade000 CR4: 00000000000006e0
     899
     900<Feb/01 03:14 pm>[2789526.027830] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     901<Feb/01 03:14 pm>[2789526.035672] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     902<Feb/01 03:14 pm>[2789526.043565] Process pclient (pid: 9738, threadinfo ffff8806cda04000, task ffff880c18cc8000)
     903<Feb/01 03:14 pm>[2789526.052626] Stack:
     904<Feb/01 03:14 pm>[2789526.055043]  ffffffff81019da9 ffff8806cda05ba8 ffffffff812c08fe ffff8806cda05be8
     905<Feb/01 03:14 pm>[2789526.062984] <0> ffffffff812c4e65 000000009ef30400 ffff8800026b3c00 ffff88041f942060
     906<Feb/01 03:14 pm>[2789526.071628] <0> ffff8800026b3c00 0000000000000004 0000000000000004 ffff8806cda05c08
     907<Feb/01 03:14 pm>[2789526.080641] Call Trace:
     908<Feb/01 03:14 pm>[2789526.083529]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     909<Feb/01 03:14 pm>[2789526.091280]  [<ffffffff812c08fe>] __delay+0xa/0xc
     910<Feb/01 03:14 pm>[2789526.096484]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     911<Feb/01 03:14 pm>[2789526.102749]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     912<Feb/01 03:14 pm>[2789526.108747]  [<ffffffff8103666d>] ? try_to_wake_up+0x123/0x342
     913<Feb/01 03:14 pm>[2789526.115100]  [<ffffffff8103666d>] try_to_wake_up+0x123/0x342
     914<Feb/01 03:14 pm>[2789526.121291]  [<ffffffff81036899>] default_wake_function+0xd/0xf
     915<Feb/01 03:14 pm>[2789526.127731]  [<ffffffff810f9aba>] pollwake+0x57/0x59
     916<Feb/01 03:14 pm>[2789526.133190]  [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
     917<Feb/01 03:14 pm>[2789526.139807]  [<ffffffff8102c08d>] __wake_up_common+0x46/0x76
     918<Feb/01 03:14 pm>[2789526.145978]  [<ffffffff8102fa4a>] __wake_up_sync_key+0x43/0x5a
     919<Feb/01 03:14 pm>[2789526.152307]  [<ffffffff8102fa6c>] __wake_up_sync+0xb/0xd
     920<Feb/01 03:14 pm>[2789526.158115]  [<ffffffff810f1b84>] pipe_write+0x4de/0x514
     921<Feb/01 03:14 pm>[2789526.163922]  [<ffffffff810ea282>] do_sync_write+0xc7/0x10d
     922
     923<Feb/01 03:14 pm>[2789526.169905]  [<ffffffff8151f61f>] ? _raw_spin_unlock_irq+0x26/0x2c
     924<Feb/01 03:14 pm>[2789526.176583]  [<ffffffff8104aa72>] ? do_sigaction+0x158/0x170
     925<Feb/01 03:14 pm>[2789526.182761]  [<ffffffff812937db>] ? security_file_permission+0x11/0x13
     926<Feb/01 03:14 pm>[2789526.189817]  [<ffffffff810eaba1>] vfs_write+0xae/0x14a
     927<Feb/01 03:14 pm>[2789526.195448]  [<ffffffff810ead01>] sys_write+0x47/0x6e
     928<Feb/01 03:14 pm>[2789526.201016]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
     929<Feb/01 03:14 pm>[2789526.207535] Code: fb 48 ff c8 c9 c3 55 48 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae e8 0f 31 89 c1 41 89 f1 66 66 90 0f ae e8 0f 31 <89> c6 48 89 f0 48 29 c8 48 39 f8 73 25 f3 90 65 44 8b 04 25 f0
     930<Feb/01 03:14 pm>[2789526.230812] Call Trace:
     931<Feb/01 03:14 pm>[2789526.233683]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     932<Feb/01 03:14 pm>[2789526.241435]  [<ffffffff812c08fe>] __delay+0xa/0xc
     933<Feb/01 03:14 pm>[2789526.246640]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     934<Feb/01 03:14 pm>[2789526.252910]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     935<Feb/01 03:14 pm>[2789526.258901]  [<ffffffff8103666d>] ? try_to_wake_up+0x123/0x342
     936<Feb/01 03:14 pm>[2789526.265255]  [<ffffffff8103666d>] try_to_wake_up+0x123/0x342
     937<Feb/01 03:14 pm>[2789526.271453]  [<ffffffff81036899>] default_wake_function+0xd/0xf
     938<Feb/01 03:14 pm>[2789526.277867]  [<ffffffff810f9aba>] pollwake+0x57/0x59
     939<Feb/01 03:14 pm>[2789526.283329]  [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
     940<Feb/01 03:14 pm>[2789526.289944]  [<ffffffff8102c08d>] __wake_up_common+0x46/0x76
     941<Feb/01 03:14 pm>[2789526.296106]  [<ffffffff8102fa4a>] __wake_up_sync_key+0x43/0x5a
     942<Feb/01 03:14 pm>[2789526.302479]  [<ffffffff8102fa6c>] __wake_up_sync+0xb/0xd
     943<Feb/01 03:14 pm>[2789526.308303]  [<ffffffff810f1b84>] pipe_write+0x4de/0x514
     944<Feb/01 03:14 pm>[2789526.314120]  [<ffffffff810ea282>] do_sync_write+0xc7/0x10d
     945
     946<Feb/01 03:14 pm>[2789526.320094]  [<ffffffff8151f61f>] ? _raw_spin_unlock_irq+0x26/0x2c
     947<Feb/01 03:14 pm>[2789526.326769]  [<ffffffff8104aa72>] ? do_sigaction+0x158/0x170
     948<Feb/01 03:14 pm>[2789526.332922]  [<ffffffff812937db>] ? security_file_permission+0x11/0x13
     949<Feb/01 03:14 pm>[2789526.339971]  [<ffffffff810eaba1>] vfs_write+0xae/0x14a
     950<Feb/01 03:14 pm>[2789526.345610]  [<ffffffff810ead01>] sys_write+0x47/0x6e
     951<Feb/01 03:14 pm>[2789526.351135]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
     952<Feb/01 03:14 pm>[2789526.357647] Pid: 9738, comm: pclient Tainted: G      D    2.6.34.4 #3
     953<Feb/01 03:14 pm>[2789526.364606] Call Trace:
     954<Feb/01 03:14 pm>[2789526.367497]  <NMI>  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
     955<Feb/01 03:14 pm>[2789526.373899]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
     956<Feb/01 03:14 pm>[2789526.380276]  [<ffffffff81520360>] do_nmi+0xb2/0x259
     957<Feb/01 03:14 pm>[2789526.385641]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
     958<Feb/01 03:14 pm>[2789526.390630]  [<ffffffff812c08af>] ? delay_tsc+0x25/0x59
     959<Feb/01 03:14 pm>[2789526.396360]  <<EOE>>  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     960<Feb/01 03:14 pm>[2789526.404939]  [<ffffffff812c08fe>] __delay+0xa/0xc
     961<Feb/01 03:14 pm>[2789526.410131]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     962<Feb/01 03:14 pm>[2789526.416392]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     963<Feb/01 03:14 pm>[2789526.422383]  [<ffffffff8103666d>] ? try_to_wake_up+0x123/0x342
     964<Feb/01 03:14 pm>[2789526.428729]  [<ffffffff8103666d>] try_to_wake_up+0x123/0x342
     965<Feb/01 03:14 pm>[2789526.434883]  [<ffffffff81036899>] default_wake_function+0xd/0xf
     966<Feb/01 03:14 pm>[2789526.441297]  [<ffffffff810f9aba>] pollwake+0x57/0x59
     967<Feb/01 03:14 pm>[2789526.446768]  [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
     968
     969<Feb/01 03:14 pm>[2789526.453365]  [<ffffffff8102c08d>] __wake_up_common+0x46/0x76
     970<Feb/01 03:14 pm>[2789526.459529]  [<ffffffff8102fa4a>] __wake_up_sync_key+0x43/0x5a
     971<Feb/01 03:14 pm>[2789526.465875]  [<ffffffff8102fa6c>] __wake_up_sync+0xb/0xd
     972<Feb/01 03:14 pm>[2789526.471699]  [<ffffffff810f1b84>] pipe_write+0x4de/0x514
     973<Feb/01 03:14 pm>[2789526.477517]  [<ffffffff810ea282>] do_sync_write+0xc7/0x10d
     974<Feb/01 03:14 pm>[2789526.483499]  [<ffffffff8151f61f>] ? _raw_spin_unlock_irq+0x26/0x2c
     975<Feb/01 03:14 pm>[2789526.490174]  [<ffffffff8104aa72>] ? do_sigaction+0x158/0x170
     976<Feb/01 03:14 pm>[2789526.496328]  [<ffffffff812937db>] ? security_file_permission+0x11/0x13
     977<Feb/01 03:14 pm>[2789526.503368]  [<ffffffff810eaba1>] vfs_write+0xae/0x14a
     978<Feb/01 03:14 pm>[2789526.509015]  [<ffffffff810ead01>] sys_write+0x47/0x6e
     979<Feb/01 03:14 pm>[2789526.514557]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
     980<Feb/01 03:14 pm>[2789526.521078] NMI backtrace for cpu 5
     981<Feb/01 03:14 pm>[2789526.525045] CPU 5
     982<Feb/01 03:14 pm>[2789526.527131] Modules linked in: w83793 hwmon_vid autofs4 i2c_i801 i2c_core iTCO_wdt e1000e ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
     983<Feb/01 03:14 pm>[2789526.562870]
     984<Feb/01 03:14 pm>[2789526.564769] Pid: 8398, comm: nfsd Tainted: G      D    2.6.34.4 #3 S7010/empty
     985<Feb/01 03:14 pm>[2789526.572715] RIP: 0010:[<ffffffff812c08a2>]  [<ffffffff812c08a2>] delay_tsc+0x18/0x59
     986<Feb/01 03:14 pm>[2789526.581197] RSP: 0018:ffff8800026a3ce0  EFLAGS: 00000082
     987<Feb/01 03:14 pm>[2789526.587006] RAX: 0000000052944250 RBX: 000000005917697c RCX: 000000005294420c
     988<Feb/01 03:14 pm>[2789526.594842] RDX: 00000000001a7f64 RSI: 0000000000000005 RDI: 0000000000000001
     989<Feb/01 03:14 pm>[2789526.602684] RBP: ffff8800026a3ce8 R08: 0000000000000005 R09: 0000000000000005
     990<Feb/01 03:14 pm>[2789526.610517] R10: 0000000000000000 R11: ffff8800026a3b18 R12: ffff8800026b3c00
     991
     992<Feb/01 03:14 pm>[2789526.618356] R13: 0000000000000000 R14: ffff88061d648000 R15: ffff88061d648488
     993<Feb/01 03:14 pm>[2789526.630724] FS:  0000000000000000(0000) GS:ffff8800026a0000(0000) knlGS:0000000000000000
     994<Feb/01 03:14 pm>[2789526.639535] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
     995<Feb/01 03:14 pm>[2789526.645782] CR2: fffffffb89ca1b58 CR3: 00000002176fb000 CR4: 00000000000006e0
     996<Feb/01 03:14 pm>[2789526.653599] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     997<Feb/01 03:14 pm>[2789526.661474] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     998<Feb/01 03:14 pm>[2789526.669298] Process nfsd (pid: 8398, threadinfo ffff88061d646000, task ffff88061d648000)
     999<Feb/01 03:14 pm>[2789526.678155] Stack:
     1000<Feb/01 03:14 pm>[2789526.680578]  ffffffff81019da9 ffff8800026a3cf8 ffffffff812c08fe ffff8800026a3d38
     1001<Feb/01 03:14 pm>[2789526.688475] <0> ffffffff812c4e65 000000009ef30400 ffff8800026b3c00 ffff880c1c760000
     1002<Feb/01 03:14 pm>[2789526.697105] <0> ffff8800026b3c00 0000000000000002 0000000000000005 ffff8800026a3d58
     1003<Feb/01 03:14 pm>[2789526.706096] Call Trace:
     1004<Feb/01 03:14 pm>[2789526.708968]  <IRQ>
     1005<Feb/01 03:14 pm>[2789526.711535]  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     1006<Feb/01 03:14 pm>[2789526.719273]  [<ffffffff812c08fe>] __delay+0xa/0xc
     1007<Feb/01 03:14 pm>[2789526.724441]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     1008<Feb/01 03:14 pm>[2789526.730677]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     1009<Feb/01 03:14 pm>[2789526.736658]  [<ffffffff8103666d>] ? try_to_wake_up+0x123/0x342
     1010                 <Feb/01 03:14 pm>[2789526.743013]  [<ffffffff8103666d>] try_to_wake_up+0x123/0x342
     1011<Feb/01 03:14 pm>[2789526.749166]  [<ffffffff81036899>] default_wake_function+0xd/0xf
     1012<Feb/01 03:14 pm>[2789526.755582]  [<ffffffff810f9aba>] pollwake+0x57/0x59
     1013<Feb/01 03:14 pm>[2789526.761061]  [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
     1014
     1015<Feb/01 03:14 pm>[2789526.767687]  [<ffffffff8102c08d>] __wake_up_common+0x46/0x76
     1016<Feb/01 03:14 pm>[2789526.773848]  [<ffffffff8102faa6>] __wake_up+0x38/0x4f
     1017<Feb/01 03:14 pm>[2789526.779396]  [<ffffffff8103b959>] printk_tick+0x39/0x3b
     1018<Feb/01 03:14 pm>[2789526.785128]  [<ffffffff810483c7>] update_process_times+0x3f/0x59
     1019<Feb/01 03:14 pm>[2789526.791639]  [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
     1020<Feb/01 03:14 pm>[2789526.797826]  [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
     1021<Feb/01 03:14 pm>[2789526.804084]  [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
     1022<Feb/01 03:14 pm>[2789526.810075]  [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
     1023<Feb/01 03:14 pm>[2789526.816405]  [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
     1024<Feb/01 03:14 pm>[2789526.823270]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
     1025<Feb/01 03:14 pm>[2789526.829785]  <EOI>
     1026<Feb/01 03:14 pm>[2789526.831986] Code: 2e 0f 1f 84 00 00 00 00 00 48 ff c8 75 fb 48 ff c8 c9 c3 55 48 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae e8 0f 31 <89> c1 41 89 f1 66 66 90 0f ae e8 0f 31 89 c6 48 89 f0 48 29 c8
     1027<Feb/01 03:14 pm>[2789526.855595] Call Trace:
     1028<Feb/01 03:14 pm>[2789526.858483]  <IRQ>  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     1029<Feb/01 03:14 pm>[2789526.866912]  [<ffffffff812c08fe>] __delay+0xa/0xc
     1030<Feb/01 03:14 pm>[2789526.872114]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     1031<Feb/01 03:14 pm>[2789526.878368]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     1032<Feb/01 03:14 pm>[2789526.884358]  [<ffffffff8103666d>] ? try_to_wake_up+0x123/0x342
     1033<Feb/01 03:14 pm>[2789526.890685]  [<ffffffff8103666d>] try_to_wake_up+0x123/0x342
     1034<Feb/01 03:14 pm>[2789526.896841]  [<ffffffff81036899>] default_wake_function+0xd/0xf
     1035<Feb/01 03:14 pm>[2789526.903272]  [<ffffffff810f9aba>] pollwake+0x57/0x59
     1036<Feb/01 03:14 pm>[2789526.908732]  [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
     1037
     1038<Feb/01 03:14 pm>[2789526.915333]  [<ffffffff8102c08d>] __wake_up_common+0x46/0x76
     1039<Feb/01 03:14 pm>[2789526.921520]  [<ffffffff8102faa6>] __wake_up+0x38/0x4f
     1040<Feb/01 03:14 pm>[2789526.927059]  [<ffffffff8103b959>] printk_tick+0x39/0x3b
     1041<Feb/01 03:14 pm>[2789526.932774]  [<ffffffff810483c7>] update_process_times+0x3f/0x59
     1042<Feb/01 03:14 pm>[2789526.939310]  [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
     1043<Feb/01 03:14 pm>[2789526.945474]  [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
     1044<Feb/01 03:14 pm>[2789526.951715]  [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
     1045<Feb/01 03:14 pm>[2789526.957679]  [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
     1046<Feb/01 03:14 pm>[2789526.964016]  [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
     1047<Feb/01 03:14 pm>[2789526.970883]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
     1048<Feb/01 03:14 pm>[2789526.977379]  <EOI> Pid: 8398, comm: nfsd Tainted: G      D    2.6.34.4 #3
     1049<Feb/01 03:14 pm>[2789526.984741] Call Trace:
     1050<Feb/01 03:14 pm>[2789526.987620]  <NMI>  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
     1051<Feb/01 03:14 pm>[2789526.993995]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
     1052<Feb/01 03:14 pm>[2789527.000348]  [<ffffffff81520360>] do_nmi+0xb2/0x259
     1053<Feb/01 03:14 pm>[2789527.005740]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
     1054<Feb/01 03:14 pm>[2789527.010770]  [<ffffffff812c08a2>] ? delay_tsc+0x18/0x59
     1055<Feb/01 03:14 pm>[2789527.016503]  <<EOE>>  <IRQ>  [<ffffffff81019da9>] ? arch_trigger_all_cpu_backtrace+0x51/0x5d
     1056<Feb/01 03:14 pm>[2789527.025754]  [<ffffffff812c08fe>] __delay+0xa/0xc
     1057<Feb/01 03:14 pm>[2789527.030954]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     1058<Feb/01 03:14 pm>[2789527.037200]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     1059<Feb/01 03:14 pm>[2789527.043155]  [<ffffffff8103666d>] ? try_to_wake_up+0x123/0x342
     1060
     1061<Feb/01 03:14 pm>[2789527.049493]  [<ffffffff8103666d>] try_to_wake_up+0x123/0x342
     1062<Feb/01 03:14 pm>[2789527.055682]  [<ffffffff81036899>] default_wake_function+0xd/0xf
     1063<Feb/01 03:14 pm>[2789527.062122]  [<ffffffff810f9aba>] pollwake+0x57/0x59
     1064<Feb/01 03:14 pm>[2789527.067573]  [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
     1065<Feb/01 03:14 pm>[2789527.074199]  [<ffffffff8102c08d>] __wake_up_common+0x46/0x76
     1066<Feb/01 03:14 pm>[2789527.080379]  [<ffffffff8102faa6>] __wake_up+0x38/0x4f
     1067<Feb/01 03:14 pm>[2789527.085936]  [<ffffffff8103b959>] printk_tick+0x39/0x3b
     1068<Feb/01 03:14 pm>[2789527.091675]  [<ffffffff810483c7>] update_process_times+0x3f/0x59
     1069<Feb/01 03:14 pm>[2789527.098178]  [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
     1070<Feb/01 03:14 pm>[2789527.104341]  [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
     1071<Feb/01 03:14 pm>[2789527.110583]  [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
     1072<Feb/01 03:14 pm>[2789527.116555]  [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
     1073<Feb/01 03:14 pm>[2789527.122884]  [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
     1074<Feb/01 03:14 pm>[2789527.129751]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
     1075
     1076}}}