IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Initial Version and Version 1 of Ipp061-crash-20120920


Ignore:
Timestamp:
Sep 20, 2012, 8:40:02 PM (14 years ago)
Author:
Mark Huber
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • Ipp061-crash-20120920

    v1 v1  
     1From console display
     2{{{
     3<Sep/20 08:12 pm>ipp061 login: [1308286.027478] BUG: unable to handle kernel paging request at 000000004c96b900
     4<Sep/20 08:12 pm>[1308286.035173] IP: [<ffffffff8103186b>] enqueue_task_fair+0x2f6/0x332
     5<Sep/20 08:12 pm>[1308286.041891] PGD 0
     6<Sep/20 08:12 pm>[1308286.044321] Thread overran stack, or stack corrupted
     7<Sep/20 08:12 pm>[1308286.049755] Oops: 0000 [#1] SMP
     8<Sep/20 08:12 pm>[1308286.053463] last sysfs file: /sys/devices/pci0000:00/0000:00:1f.3/i2c-0/0-002f/temp6_alarm
     9<Sep/20 08:12 pm>[1308286.062457] CPU 6
     10<Sep/20 08:12 pm>[1308286.064544] Modules linked in: w83793 hwmon_vid autofs4 i2c_i801 i2c_core iTCO_wdt e1000e ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
     11<Sep/20 08:12 pm>[1308286.099149]
     12<Sep/20 08:12 pm>[1308286.101027] Pid: 8429, comm: nfsd Not tainted 2.6.34.4 #3 S7010/empty
     13<Sep/20 08:12 pm>[1308286.107994] RIP: 0010:[<ffffffff8103186b>]  [<ffffffff8103186b>] enqueue_task_fair+0x2f6/0x332
     14<Sep/20 08:12 pm>[1308286.117355] RSP: 0018:ffff8806196c4180  EFLAGS: 00010046
     15<Sep/20 08:12 pm>[1308286.123147] RAX: ffff8806196c4000 RBX: ffff88061c219030 RCX: 00000000195c9030
     16<Sep/20 08:12 pm>[1308286.130953] RDX: 0000000000013c00 RSI: ffff88064e213cb0 RDI: ffff88061c219078
     17<Sep/20 08:12 pm>[1308286.138790] RBP: ffff8806196c41c0 R08: 0000000000000000 R09: 0002d260e5493086
     18<Sep/20 08:12 pm>[1308286.146632] R10: ffffffff81035927 R11: ffffffff81a9daa0 R12: ffff88064e213c88
     19<Sep/20 08:12 pm>[1308286.154443] R13: 0000000000000000 R14: ffff88064e213c00 R15: ffff88064e213c00
     20<Sep/20 08:12 pm>[1308286.162296] FS:  0000000000000000(0000) GS:ffff88064e200000(0000) knlGS:0000000000000000
     21<Sep/20 08:12 pm>[1308286.171105] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
     22<Sep/20 08:12 pm>[1308286.177327] CR2: 000000004c96b900 CR3: 0000000001a8a000 CR4: 00000000000006e0
     23<Sep/20 08:12 pm>[1308286.185140] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     24
     25<Sep/20 08:12 pm>[1308286.192992] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     26<Sep/20 08:12 pm>[1308286.200826] Process nfsd (pid: 8429, threadinfo ffff8806196c4000, task ffff8806195c9030)
     27<Sep/20 08:12 pm>[1308286.209636] Stack:
     28<Sep/20 08:12 pm>[1308286.212041]  ffff88064e213c00 ffff88061c219068 ffff88061c219030 ffff88061c219030
     29<Sep/20 08:12 pm>[1308286.219848] <0> ffff88064e213c00 0000000000000006 000000000000080f 000000000000080f
     30<Sep/20 08:12 pm>[1308286.228316] <0> ffff8806196c41e0 ffffffff8102b548 ffff880002673c00 ffff88064e213c00
     31<Sep/20 08:12 pm>[1308286.237143] Call Trace:
     32<Sep/20 08:12 pm>[1308286.240017]  [<ffffffff8102b548>] enqueue_task+0x5e/0x69
     33<Sep/20 08:12 pm>[1308286.245804]  [<ffffffff8102b659>] activate_task+0x2a/0x32
     34<Sep/20 08:12 pm>[1308286.251689]  [<ffffffff81035666>] pull_task+0x35/0x55
     35<Sep/20 08:12 pm>[1308286.257200]  [<ffffffff81037767>] load_balance+0xcab/0xf63
     36<Sep/20 08:12 pm>[1308286.263176]  [<ffffffffa00fda25>] ? megasas_fire_cmd_gen2+0x44/0x4d [megaraid_sas]
     37<Sep/20 08:12 pm>[1308286.271487]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     38<Sep/20 08:12 pm>[1308286.277826]  [<ffffffff8151cc0e>] schedule+0x25a/0x86a
     39<Sep/20 08:12 pm>[1308286.283418]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     40<Sep/20 08:12 pm>[1308286.289765]  [<ffffffff8151d35f>] schedule_timeout+0x28/0x24c
     41<Sep/20 08:12 pm>[1308286.296014]  [<ffffffff8151c8aa>] ? wait_for_common+0xba/0x138
     42<Sep/20 08:12 pm>[1308286.302350]  [<ffffffff8151c8b2>] wait_for_common+0xc2/0x138
     43<Sep/20 08:12 pm>[1308286.308517]  [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
     44<Sep/20 08:12 pm>[1308286.315129]  [<ffffffff81280d3e>] ? _xfs_buf_read+0x3f/0x44
     45<Sep/20 08:12 pm>[1308286.321206]  [<ffffffff81277887>] ? xfs_trans_read_buf+0x238/0x396
     46<Sep/20 08:12 pm>[1308286.327873]  [<ffffffff8151c9b2>] wait_for_completion+0x18/0x1a
     47
     48<Sep/20 08:12 pm>[1308286.334269]  [<ffffffff8127f236>] xfs_buf_iowait+0x7e/0xba
     49<Sep/20 08:12 pm>[1308286.340256]  [<ffffffff81280d3e>] _xfs_buf_read+0x3f/0x44
     50<Sep/20 08:12 pm>[1308286.346157]  [<ffffffff81281374>] xfs_buf_read+0x8a/0xc6
     51<Sep/20 08:12 pm>[1308286.351992]  [<ffffffff81277887>] xfs_trans_read_buf+0x238/0x396
     52<Sep/20 08:12 pm>[1308286.358514]  [<ffffffff8124e176>] xfs_btree_read_buf_block+0x4d/0x91
     53<Sep/20 08:12 pm>[1308286.365385]  [<ffffffff8124dfa6>] ? xfs_btree_check_sblock+0x6a/0x11e
     54<Sep/20 08:12 pm>[1308286.372346]  [<ffffffff8124f1dc>] xfs_btree_lookup_get_block+0x88/0xae
     55<Sep/20 08:12 pm>[1308286.379366]  [<ffffffff8124f7ca>] xfs_btree_lookup+0x138/0x3dc
     56<Sep/20 08:12 pm>[1308286.385699]  [<ffffffff81239f17>] xfs_alloc_lookup_eq+0x14/0x16
     57<Sep/20 08:12 pm>[1308286.392121]  [<ffffffff8123a00f>] xfs_alloc_fixup_trees+0x<Sep/20 08:12 pm>
     58                 <Sep/20 08:12 pm>[1308286.398805]  [<ffffffff8123cd84>] ? xfs_allocbt_init_cursor+0x38/0x96
     59<Sep/20 08:12 pm>[1308286.405756]  [<ffffffff8123bbcd>] xfs_alloc_ag_vextent_near+0x382/0xa30
     60<Sep/20 08:12 pm>[1308286.412881]  [<ffffffff812734b7>] ? xfs_perag_get+0x29/0x99
     61<Sep/20 08:12 pm>[1308286.418978]  [<ffffffff8123c2a9>] xfs_alloc_ag_vextent+0x2e/0xf5
     62<Sep/20 08:12 pm>[1308286.425498]  [<ffffffff8123cb1b>] xfs_alloc_vextent+0x318/0x4ad
     63<Sep/20 08:12 pm>[1308286.431902]  [<ffffffff812488c1>] xfs_bmap_btalloc+0x3c2/0x576
     64<Sep/20 08:12 pm>[1308286.438212]  [<ffffffff81248a98>] xfs_bmap_alloc+0x23/0x25
     65<Sep/20 08:12 pm>[1308286.444183]  [<ffffffff8124add5>] xfs_bmapi+0x7b5/0xe73
     66<Sep/20 08:12 pm>[1308286.449911]  [<ffffffff8126cbeb>] ? xlog_grant_log_space+0x3e0/0x484
     67<Sep/20 08:12 pm>[1308286.456787]  [<ffffffff8105861d>] ? down_write_nested+0x2a/0x4a
     68<Sep/20 08:12 pm>[1308286.463201]  [<ffffffff81267363>] xfs_iomap_write_allocate+0x227/0x353
     69<Sep/20 08:12 pm>[1308286.470230]  [<ffffffff81267eac>] xfs_iomap+0x318/0x39c
     70
     71<Sep/20 08:12 pm>[1308286.475921]  [<ffffffff8127d1e5>] xfs_map_blocks+0x27/0x2b
     72<Sep/20 08:12 pm>[1308286.481867]  [<ffffffff8127e160>] xfs_page_state_convert+0x2d8/0x5b2
     73<Sep/20 08:12 pm>[1308286.488734]  [<ffffffff81048174>] ? try_to_del_timer_sync+0xae/0xbb
     74<Sep/20 08:12 pm>[1308286.495511]  [<ffffffff810480a2>] ? lock_timer_base+0x26/0x4a
     75<Sep/20 08:12 pm>[1308286.501757]  [<ffffffff8127e71f>] xfs_vm_writepage+0xd9/0x113
     76<Sep/20 08:12 pm>[1308286.507975]  [<ffffffff810b7612>] shrink_page_list+0x3c2/0x68f
     77<Sep/20 08:12 pm>[1308286.514298]  [<ffffffff81048181>] ? del_timer_sync+0x0/0x67
     78<Sep/20 08:12 pm>[1308286.520359]  [<ffffffff81054b6c>] ? autoremove_wake_functi<Sep/20 08:12 pm>
     79                 <Sep/20 08:12 pm>[1308286.527310]  [<ffffffff810b83e3>] shrink_zone+0x864/0xb86
     80<Sep/20 08:12 pm>[1308286.533194]  [<ffffffff8128892e>] ? xfs_reclaim_inode_shrink+0x11b/0x126
     81<Sep/20 08:12 pm>[1308286.540443]  [<ffffffff812734b7>] ? xfs_perag_get+0x29/0x99
     82<Sep/20 08:12 pm>[1308286.546504]  [<ffffffff810b8943>] ? shrink_slab+0x136/0x148
     83<Sep/20 08:12 pm>[1308286.552562]  [<ffffffff81058774>] ? up_read+0x1e/0x36
     84<Sep/20 08:12 pm>[1308286.558109]  [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
     85<Sep/20 08:12 pm>[1308286.564803]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
     86<Sep/20 08:12 pm>[1308286.571043]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
     87<Sep/20 08:12 pm>[1308286.577720]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
     88<Sep/20 08:12 pm>[1308286.584585]  [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
     89<Sep/20 08:12 pm>[1308286.590994]  [<ffffffff810df0c8>] new_slab+0x57/0x26b
     90<Sep/20 08:12 pm>[1308286.596503]  [<ffffffff810df4f8>] __slab_alloc+0x21c/0x45f
     91<Sep/20 08:12 pm>[1308286.602502]  [<ffffffff8127c99f>] ? kmem_zone_alloc+0x67/0xaf
     92<Sep/20 08:12 pm>[1308286.608744]  [<ffffffff8127c99f>] ? kmem_zone_alloc+0x67/0xaf
     93
     94<Sep/20 08:12 pm>[1308286.614993]  [<ffffffff810dfb2f>] kmem_cache_alloc+0x6a/0xd2
     95<Sep/20 08:12 pm>[1308286.621154]  [<ffffffff8127c99f>] kmem_zone_alloc+0x67/0xaf
     96<Sep/20 08:12 pm>[1308286.627213]  [<ffffffff8127c9f6>] kmem_zone_zalloc+0xf/0x30
     97<Sep/20 08:12 pm>[1308286.633295]  [<ffffffff81276720>] _xfs_trans_alloc+0x29/0x63
     98<Sep/20 08:12 pm>[1308286.643980]  [<ffffffff8127696e>] xfs_trans_alloc+0x9d/0xab
     99<Sep/20 08:12 pm>[1308286.650049]  [<ffffffff81261456>] ? xfs_iunlock+0x7e/0xb5
     100<Sep/20 08:12 pm>[1308286.655916]  [<ffffffff812671fe>] xfs_iomap_write_allocate+0xc2/0x353
     101<Sep/20 08:12 pm>[1308286.662888]  [<ffffffff81058774>] ? up_read+0x1e/0x36
     102<Sep/20 08:12 pm>[1308286.668429]  [<ffffffff81267eac>] xfs_iomap+0x318/0x39c
     103<Sep/20 08:12 pm>[1308286.674125]  [<ffffffff8127d1e5>] xfs_map_blocks+0x27/0x2b
     104<Sep/20 08:12 pm>[1308286.680106]  [<ffffffff8127e160>] xfs_page_state_convert+0x2d8/0x5b2
     105<Sep/20 08:12 pm>[1308286.686983]  [<ffffffff8127e71f>] xfs_vm_writepage+0xd9/0x113
     106<Sep/20 08:12 pm>[1308286.693221]  [<ffffffff810b32a5>] __writepage+0x15/0x3b
     107<Sep/20 08:12 pm>[1308286.698939]  [<ffffffff810b39a0>] write_cache_pages+0x1f1/0x2f9
     108<Sep/20 08:12 pm>[1308286.705381]  [<ffffffff810b3290>] ? __writepage+0x0/0x3b
     109<Sep/20 08:12 pm>[1308286.711177]  [<ffffffff8127e61d>] ? xfs_vm_writepages+0x3a/0x4e
     110<Sep/20 08:12 pm>[1308286.717602]  [<ffffffff810b3aca>] generic_writepages+0x22/0x28
     111<Sep/20 08:12 pm>[1308286.723943]  [<ffffffff8127e628>] xfs_vm_writepages+0x45/0x4e
     112<Sep/20 08:12 pm>[1308286.730202]  [<ffffffff810b3aef>] do_writepages+0x1f/0x28
     113<Sep/20 08:12 pm>[1308286.736103]  [<ffffffff810ad4b8>] __filemap_fdatawrite_range+0x51/0x59
     114<Sep/20 08:12 pm>[1308286.743144]  [<ffffffff810ad4ea>] filemap_write_and_wait_range+0x2a/0x4f
     115<Sep/20 08:12 pm>[1308286.750374]  [<ffffffff8110a11f>] vfs_fsync_range+0x5a/0xa6
     116
     117<Sep/20 08:12 pm>[1308286.756447]  [<ffffffff811f5aa2>] nfsd_commit+0x77/0xa3
     118<Sep/20 08:12 pm>[1308286.762139]  [<ffffffff811fbe15>] nfsd3_proc_commit+0xb5/0xc4
     119<Sep/20 08:12 pm>[1308286.768407]  [<ffffffff811f0854>] nfsd_dispatch+0xe9/0x1ca
     120<Sep/20 08:12 pm>[1308286.774378]  [<ffffffff814f04c5>] svc_process+0x425/0x72a
     121<Sep/20 08:12 pm>[1308286.780252]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
     122<Sep/20 08:12 pm>[1308286.785526]  [<ffffffff811f0dc2>] nfsd+0xf3/0x139
     123<Sep/20 08:12 pm>[1308286.790722]  [<ffffffff810546e2>] kthread+0x7d/0x85
     124<Sep/20 08:12 pm>[1308286.796059]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     125<Sep/20 08:12 pm>[1308286.802451]  [<ffffffff81054665>] ? kthread+0x0/0x85
     126<Sep/20 08:12 pm>[1308286.807890]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     127<Sep/20 08:12 pm>[1308286.814487] Code: 00 00 00 e8 cc b7 28 00 48 8b 55 c0 48 8b 82 e0 07 00 00 48 81 78 30 10 9c 60 81 75 3b 48 8b 40 08 48 c7 c2 00 3c 01 00 8b 48 18 <48> 8b 04 cd 80 37 b2 81 48 83 bc 02 98 00 00 00 04 77 1a 48 01
     128<Sep/20 08:12 pm>[1308286.835910] RIP  [<ffffffff8103186b>] enqueue_task_fair+0x2f6/0x332
     129<Sep/20 08:12 pm>[1308286.842695]  RSP <ffff8806196c4180>
     130<Sep/20 08:12 pm>[1308286.846634] CR2: 000000004c96b900
     131<Sep/20 08:12 pm>[1308286.850654] ---[ end trace 181732d600e91cc0 ]---
     132<Sep/20 08:12 pm>[1308286.855796] Kernel panic - not syncing: Fatal exception in interrupt
     133<Sep/20 08:12 pm>[1308286.862718] Pid: 8429, comm: nfsd Tainted: G      D    2.6.34.4 #3
     134<Sep/20 08:12 pm>[1308286.869459] Call Trace:
     135<Sep/20 08:13 pm>[1308356.029750] <B0>[1308356.030158] BUG: spinlock lockup on CPU#3, nfsd/8369, ffff880002673c00
     136<Sep/20 08:13 pm>[1308356.030161] Pid: 8369, comm: nfsd Tainted: G      D    2.6.34.4 #3
     137<Sep/20 08:13 pm>[1308356.030162] Call Trace:
     138<Sep/20 08:13 pm>[1308356.030164]  <IRQ>  [<ffffffff812c4e9f>] do_raw_spin_lock+0x118/0x147
     139
     140<Sep/20 08:13 pm>[1308356.030173]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     141<Sep/20 08:13 pm>[1308356.030177]  [<ffffffff81038588>] ? scheduler_tick+0x3d/0x236
     142<Sep/20 08:13 pm>[1308356.030179]  [<ffffffff81038588>] scheduler_tick+0x3d/0x236
     143<Sep/20 08:13 pm>[1308356.030183]  [<ffffffff810483d1>] update_process_times+0x49/0x59
     144<Sep/20 08:13 pm>[1308356.030186]  [<ffffffff810610b4>] tick_sched_timer+0x73/0x9d
     145<Sep/20 08:13 pm>[1308356.030188]  [<ffffffff81061041>] ? tick_sched_timer+0x0/0x9d
     146<Sep/20 08:13 pm>[1308356.030191]  [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
     147<Sep/20 08:13 pm>[1308356.030193]  [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
     148<Sep/20 08:13 pm>[1308356.030197]  [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
     149<Sep/20 08:13 pm>[1308356.030201]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
     150<Sep/20 08:13 pm>[1308356.030202]  <EOI>  [<ffffffff812734b7>] ? xfs_perag_get+0x29/0x99
     151<Sep/20 08:13 pm>[1308356.030208]  [<ffffffff812734b7>] ? xfs_perag_get+0x29/0x99
     152<Sep/20 08:13 pm>[1308356.030210]  [<ffffffff812c4e3d>] ? do_raw_spin_lock+0xb6/0x147
     153<Sep/20 08:13 pm>[1308356.030213]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
     154<Sep/20 08:13 pm>[1308356.030215]  [<ffffffff812734b7>] ? xfs_perag_get+0x29/0x99
     155<Sep/20 08:13 pm>[1308356.030217]  [<ffffffff812734b7>] xfs_perag_get+0x29/0x99
     156<Sep/20 08:13 pm>[1308356.030220]  [<ffffffff8128878f>] xfs_inode_ag_iterator+0x4a/0xce
     157<Sep/20 08:13 pm>[1308356.030223]  [<ffffffff81287ae3>] ? xfs_reclaim_inode+0x0/0x216
     158<Sep/20 08:13 pm>[1308356.030225]  [<ffffffff81288874>] xfs_reclaim_inode_shrink+0x61/0x126
     159<Sep/20 08:13 pm>[1308356.030229]  [<ffffffff810b88e5>] shrink_slab+0xd8/0x148
     160<Sep/20 08:13 pm>[1308356.030231]  [<ffffffff81054b6c>] ? autoremove_wake_function+0x0/0x38
     161<Sep/20 08:13 pm>[1308356.030234]  [<ffffffff810b9801>] do_try_to_free_pages+0x246/0x37a
     162
     163<Sep/20 08:13 pm>[1308356.030236]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
     164<Sep/20 08:13 pm>[1308356.030238]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
     165<Sep/20 08:13 pm>[1308356.030241]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
     166<Sep/20 08:13 pm>[1308356.030244]  [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
     167<Sep/20 08:13 pm>[1308356.030247]  [<ffffffff814fc30d>] svc_recv+0xe5/0x7f5
     168<Sep/20 08:13 pm>[1308356.030249]  [<ffffffff814fc1d9>] ? svc_send+0x85/0xa1
     169                 <Sep/20 08:13 pm>[1308356.030252]  [<ffffffff811f9bb1>] ? exp_readunlock+0x10/0x12
     170<Sep/20 08:13 pm>[1308356.030254]  [<ffffffff8103688c>] ? default_wake_function+0x0/0xf
     171<Sep/20 08:13 pm>[1308356.030256]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
     172<Sep/20 08:13 pm>[1308356.030258]  [<ffffffff811f0d78>] nfsd+0xa9/0x139
     173<Sep/20 08:13 pm>[1308356.030260]  [<ffffffff810546e2>] kthread+0x7d/0x85
     174<Sep/20 08:13 pm>[1308356.030263]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     175<Sep/20 08:13 pm>[1308356.030265]  [<ffffffff81054665>] ? kthread+0x0/0x85
     176<Sep/20 08:13 pm>[1308356.030267]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     177<Sep/20 08:13 pm>[1308356.030268] sending NMI to all CPUs:
     178                 <Sep/20 08:13 pm>[1308356.030272] NMI backtrace for cpu 0
     179<Sep/20 08:13 pm>[1308356.030274] CPU 0
     180<Sep/20 08:13 pm>[1308356.030275] Modules linked in: w83793 hwmon_vid autofs4 i2c_i801 i2c_core iTCO_wdt e1000e ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
     181<Sep/20 08:13 pm>[1308356.030293]
     182<Sep/20 08:13 pm>[1308356.030295] Pid: 734, comm: xfsdatad/0 Tainted: G      D    2.6.34.4 #3 S7010/empty
     183<Sep/20 08:13 pm>[1308356.030297] RIP: 0010:[<ffffffff812c08a0>]  [<ffffffff812c08a0>] delay_tsc+0x16/0x59
     184<Sep/20 08:13 pm>[1308356.030302] RSP: 0018:ffff880623ae1938  EFLAGS: 00000092
     185
     186<Sep/20 08:13 pm>[1308356.030303] RAX: 0000000000000000 RBX: 000000009ef28775 RCX: 0000000036999cb8
     187<Sep/20 08:13 pm>[1308356.030304] RDX: 0000000000000100 RSI: 0000000000000000 RDI: 0000000000000001
     188<Sep/20 08:13 pm>[1308356.030306] RBP: ffff880623ae1940 R08: 0000000000000001 R09: 0000000000000000
     189<Sep/20 08:13 pm>[1308356.030307] R10: ffffffff81035915 R11: 0000000000000800 R12: ffff880002673c00
     190<Sep/20 08:13 pm>[1308356.030309] R13: 0000000000000001 R14: ffff880623ad9030 R15: ffff880623ad94b8
     191<Sep/20 08:13 pm>[1308356.030311] FS:  0000000000000000(0000) GS:ffff880002600000(0000) knlGS:0000000000000000
     192<Sep/20 08:13 pm>[1308356.030312] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
     193<Sep/20 08:13 pm>[1308356.030314] CR2: 0000000020ddbb00 CR3: 000000007d81e000 CR4: 00000000000006f0
     194<Sep/20 08:13 pm>[1308356.030315] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     195<Sep/20 08:13 pm>[1308356.030316] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     196<Sep/20 08:13 pm>[1308356.030318] Process xfsdatad/0 (pid: 734, threadinfo ffff880623ae0000, task ffff880623ad9030)
     197<Sep/20 08:13 pm>[1308356.030320] Stack:
     198<Sep/20 08:13 pm>[1308356.030320]  0000000000000046 ffff880623ae1950 ffffffff812c08fe ffff880623ae1990
     199<Sep/20 08:13 pm>[1308356.030322] <0> ffffffff812c4e65 000000009ef2f014 ffff880002673c00 ffff880002673c00
     200<Sep/20 08:13 pm>[1308356.030325] <0> ffff88000260e3f0 0000000000000400 0000000000000800 ffff880623ae19b0
     201<Sep/20 08:13 pm>[1308356.030327] Call Trace:
     202<Sep/20 08:13 pm>[1308356.030330]  [<ffffffff812c08fe>] __delay+0xa/0xc
     203<Sep/20 08:13 pm>[1308356.030332]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     204<Sep/20 08:13 pm>[1308356.030335]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
     205<Sep/20 08:13 pm>[1308356.030337]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76
     206<Sep/20 08:13 pm>[1308356.030340]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
     207<Sep/20 08:13 pm>[1308356.030342]  [<ffffffff81037672>] load_balance+0xbb6/0xf63
     208
     209<Sep/20 08:13 pm>[1308356.030344]  [<ffffffff81030875>] ? double_rq_unlock+0x1f/0x24
     210<Sep/20 08:13 pm>[1308356.030347]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     211<Sep/20 08:13 pm>[1308356.030350]  [<ffffffff8151cc0e>] schedule+0x25a/0x86a
     212<Sep/20 08:13 pm>[1308356.030353]  [<ffffffff810480a2>] ? lock_timer_base+0x26/0x4a
     213<Sep/20 08:13 pm>[1308356.030355]  [<ffffffff8151d559>] schedule_timeout+0x222/0x24c
     214<Sep/20 08:13 pm>[1308356.030357]  [<ffffffff810482a5>] ? process_timeout+0x0/0xb
     215<Sep/20 08:13 pm>[1308356.030360]  [<ffffffff8151d59c>] schedule_timeout_uninterruptible+0x19/0x1b
     216<Sep/20 08:13 pm>[1308356.030363]  [<ffffffff8127d1af>] xfs_end_io+0x7d/0x8c
     217<Sep/20 08:13 pm>[1308356.030365]  [<ffffffff81050bf4>] worker_thread+0x22c/0x332
     218<Sep/20 08:13 pm>[1308356.030367]  [<ffffffff81050b9d>] ? worker_thread+0x1d5/0x332
     219<Sep/20 08:13 pm>[1308356.030369]  [<ffffffff8127d132>] ? xfs_end_io+0x0/0x8c
     220<Sep/20 08:13 pm>[1308356.030371]  [<ffffffff81054b6c>] ? autoremove_wake_function+0x0/0x38
     221<Sep/20 08:13 pm>[1308356.030374]  [<ffffffff810509c8>] ? worker_thread+0x0/0x332
     222<Sep/20 08:13 pm>[1308356.030376]  [<ffffffff810546e2>] kthread+0x7d/0x85
     223<Sep/20 08:13 pm>[1308356.030378]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     224<Sep/20 08:13 pm>[1308356.030380]  [<ffffffff81054665>] ? kthread+0x0/0x85
     225<Sep/20 08:13 pm>[1308356.030382]  [<ffffffff81003790>] ? kernel_thread_helper+0<Sep/20 08:13 pm>
     226                 <Sep/20 08:13 pm>[1308356.030383] Code: 66 66 2e 0f 1f 84 00 00 00 00 00 48 ff c8 75 fb 48 ff c8 c9 c3 55 48 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae e8 <0f> 31 89 c1 41 89 f1 66 66 90 0f ae e8 0f 31 89 c6 48 89 f0 48
     227<Sep/20 08:13 pm>[1308356.030398] Call Trace:
     228<Sep/20 08:13 pm>[1308356.030400]  [<ffffffff812c08fe>] __delay+0xa/0xc
     229<Sep/20 08:13 pm>[1308356.030402]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     230<Sep/20 08:13 pm>[1308356.030405]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
     231
     232<Sep/20 08:13 pm>[1308356.030407]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76
     233<Sep/20 08:13 pm>[1308356.030409]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
     234<Sep/20 08:13 pm>[1308356.030411]  [<ffffffff81037672>] load_balance+0xbb6/0xf63
     235<Sep/20 08:13 pm>[1308356.030413]  [<ffffffff81030875>] ? double_rq_unlock+0x1f/0x24
     236<Sep/20 08:13 pm>[1308356.030415]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     237<Sep/20 08:13 pm>[1308356.030418]  [<ffffffff8151cc0e>] schedule+0x25a/0x86a
     238<Sep/20 08:13 pm>[1308356.030420]  [<ffffffff810480a2>] ? lock_timer_base+0x26/0x4a
     239<Sep/20 08:13 pm>[1308356.030422]  [<ffffffff8151d559>] schedule_timeout+0x222/0x24c
     240<Sep/20 08:13 pm>[1308356.030425]  [<ffffffff810482a5>] ? process_timeout+0x0/0xb
     241<Sep/20 08:13 pm>[1308356.030427]  [<ffffffff8151d59c>] schedule_timeout_uninterruptible+0x19/0x1b
     242<Sep/20 08:13 pm>[1308356.030429]  [<ffffffff8127d1af>] xfs_end_io+0x7d/0x8c
     243<Sep/20 08:13 pm>[1308356.030431]  [<ffffffff81050bf4>] worker_thread+0x22c/0x332
     244<Sep/20 08:13 pm>[1308356.030433]  [<ffffffff81050b9d>] ? worker_thread+0x1d5/0x332
     245<Sep/20 08:13 pm>[1308356.030435]  [<ffffffff8127d132>] ? xfs_end_io+0x0/0x8c
     246                 <Sep/20 08:13 pm>[1308356.030437]  [<ffffffff81054b6c>] ? autoremove_wake_function+0x0/0x38
     247<Sep/20 08:13 pm>[1308356.030440]  [<ffffffff810509c8>] ? worker_thread+0x0/0x332
     248<Sep/20 08:13 pm>[1308356.030442]  [<ffffffff810546e2>] kthread+0x7d/0x85
     249<Sep/20 08:13 pm>[1308356.030444]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     250<Sep/20 08:13 pm>[1308356.030446]  [<ffffffff81054665>] ? kthread+0x0/0x85
     251<Sep/20 08:13 pm>[1308356.030448]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     252<Sep/20 08:13 pm>[1308356.030450] Pid: 734, comm: xfsdatad/0 Tainted: G      D    2.6.34.4 #3
     253<Sep/20 08:13 pm>[1308356.030451] Call Trace:
     254
     255<Sep/20 08:13 pm>[1308356.030451]  <NMI>  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
     256<Sep/20 08:13 pm>[1308356.030456]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
     257<Sep/20 08:13 pm>[1308356.030458]  [<ffffffff81520360>] do_nmi+0xb2/0x259
     258<Sep/20 08:13 pm>[1308356.030459]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
     259<Sep/20 08:13 pm>[1308356.030462]  [<ffffffff81035915>] ? double_rq_lock+0x2b/0x76
     260<Sep/20 08:13 pm>[1308356.030464]  [<ffffffff812c08a0>] ? delay_tsc+0x16/0x59
     261<Sep/20 08:13 pm>[1308356.030465]  <<EOE>>  [<ffffffff812c08fe>] __delay+0xa/0xc
     262<Sep/20 08:13 pm>[1308356.030468]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     263<Sep/20 08:13 pm>[1308356.030470]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
     264<Sep/20 08:13 pm>[1308356.030473]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76
     265<Sep/20 08:13 pm>[1308356.030475]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
     266<Sep/20 08:13 pm>[1308356.030477]  [<ffffffff81037672>] load_balance+0xbb6/0xf63
     267<Sep/20 08:13 pm>[1308356.030479]  [<ffffffff81030875>] ? double_rq_unlock+0x1f/0x24
     268<Sep/20 08:13 pm>[1308356.030481]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     269<Sep/20 08:13 pm>[1308356.030484]  [<ffffffff8151cc0e>] schedule+0x25a/0x86a
     270<Sep/20 08:13 pm>[1308356.030486]  [<ffffffff810480a2>] ? lock_timer_base+0x26/0x4a
     271<Sep/20 08:13 pm>[1308356.030488]  [<ffffffff8151d559>] schedule_timeout+0x222/0x24c
     272<Sep/20 08:13 pm>[1308356.030490]  [<ffffffff810482a5>] ? process_timeout+0x0/0xb
     273<Sep/20 08:13 pm>[1308356.030493]  [<ffffffff8151d59c>] schedule_timeout_uninterruptible+0x19/0x1b
     274<Sep/20 08:13 pm>[1308356.030495]  [<ffffffff8127d1af>] xfs_end_io+0x7d/0x8c
     275<Sep/20 08:13 pm>[1308356.030497]  [<ffffffff81050bf4>] worker_thread+0x22c/0x332
     276<Sep/20 08:13 pm>[1308356.030499]  [<ffffffff81050b9d>] ? worker_thread+0x1d5/0x332
     277
     278<Sep/20 08:13 pm>[1308356.030501]  [<ffffffff8127d132>] ? xfs_end_io+0x0/0x8c
     279<Sep/20 08:13 pm>[1308356.030503]  [<ffffffff81054b6c>] ? autoremove_wake_function+0x0/0x38
     280<Sep/20 08:13 pm>[1308356.030505]  [<ffffffff810509c8>] ? worker_thread+0x0/0x332
     281<Sep/20 08:13 pm>[1308356.030508]  [<ffffffff810546e2>] kthread+0x7d/0x85
     282<Sep/20 08:13 pm>[1308356.030510]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     283<Sep/20 08:13 pm>[1308356.030512]  [<ffffffff81054665>] ? kthread+0x0/0x85
     284<Sep/20 08:13 pm>[1308356.030514]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     285<Sep/20 08:13 pm>[1308356.030515] NMI backtrace for cpu 2
     286<Sep/20 08:13 pm>[1308356.030516] CPU 2
     287<Sep/20 08:13 pm>[1308356.030517] Modules linked in: w83793 hwmon_vid autofs4 i2c_i801 i2c_core iTCO_wdt e1000e ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
     288<Sep/20 08:13 pm>[1308356.030531]
     289<Sep/20 08:13 pm>[1308356.030533] Pid: 8358, comm: nfsd Tainted: G      D    2.6.34.4 #3 S7010/empty
     290<Sep/20 08:13 pm>[1308356.030534] RIP: 0010:[<ffffffff8133423b>]  [<ffffffff8133423b>] io_serial_in+0x10/0x15
     291<Sep/20 08:13 pm>[1308356.030539] RSP: 0018:ffff88061dd06ed0  EFLAGS: 00000002
     292<Sep/20 08:13 pm>[1308356.030541] RAX: 0000000000000a00 RBX: 0000000000000000 RCX: 0000000000000000
     293<Sep/20 08:13 pm>[1308356.030542] RDX: 00000000000003fd RSI: 0000000000000005 RDI: ffffffff82400ac0
     294<Sep/20 08:13 pm>[1308356.030544] RBP: ffff88061dd06ed0 R08: 0000000000000002 R09: 0000000000000002
     295<Sep/20 08:13 pm>[1308356.030545] R10: 0000000000000007 R11: ffffffff81334e4e R12: ffffffff82400ac0
     296<Sep/20 08:13 pm>[1308356.030547] R13: 000000000000270e R14: 0000000000000020 R15: 000000000000004c
     297<Sep/20 08:13 pm>[1308356.030548] FS:  0000000000000000(0000) GS:ffff880002640000(0000) knlGS:0000000000000000
     298<Sep/20 08:13 pm>[1308356.030550] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
     299<Sep/20 08:13 pm>[1308356.030552] CR2: 00007fb56d825000 CR3: 0000000976986000 CR4: 00000000000006e0
     300
     301<Sep/20 08:13 pm>[1308356.030553] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     302<Sep/20 08:13 pm>[1308356.030554] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     303<Sep/20 08:13 pm>[1308356.030556] Process nfsd (pid: 8358, threadinfo ffff88061dd06000, task ffff88061dd08000)
     304<Sep/20 08:13 pm>[1308356.030557] Stack:
     305<Sep/20 08:13 pm>[1308356.030558]  ffff88061dd06f00 ffffffff81334767 ffffffff82400ac0 0000000000000030
     306<Sep/20 08:13 pm>[1308356.030560] <0> ffffffff813347d5 ffffffff82400ac0 ffff88061dd06f20 ffffffff813347ec
     307<Sep/20 08:13 pm>[1308356.030562] <0> ffffffff81c59f25 000000000000000a ffff88061dd06f60 ffffffff813311e8
     308<Sep/20 08:13 pm>[1308356.030565] Call Trace:
     309<Sep/20 08:13 pm>[1308356.030567]  [<ffffffff81334767>] wait_for_xmitr+0x24/0x92
     310<Sep/20 08:13 pm>[1308356.030570]  [<ffffffff813347d5>] ? serial8250_console_putchar+0x0/0x27
     311<Sep/20 08:13 pm>[1308356.030572]  [<ffffffff813347ec>] serial8250_console_putchar+0x17/0x27
     312<Sep/20 08:13 pm>[1308356.030575]  [<ffffffff813311e8>] uart_console_write+0x41/0x52
     313<Sep/20 08:13 pm>[1308356.030577]  [<ffffffff81334eb0>] serial8250_console_write+0xb6/0x108
     314<Sep/20 08:13 pm>[1308356.030580]  [<ffffffff8103b339>] __call_console_drivers+0x67/0x79
     315<Sep/20 08:13 pm>[1308356.030582]  [<ffffffff8103b3a6>] _call_console_drivers+0x5b/0x5f
     316<Sep/20 08:13 pm>[1308356.030585]  [<ffffffff8103b87b>] release_console_sem+0x121/0x1c6
     317<Sep/20 08:13 pm>[1308356.030587]  [<ffffffff8103beb2>] vprintk+0x36b/0x3a0
     318<Sep/20 08:13 pm>[1308356.030590]  [<ffffffff8151c589>] printk+0x67/0x69
     319<Sep/20 08:13 pm>[1308356.030593]  [<ffffffff812c08fe>] ? __delay+0xa/0xc
     320<Sep/20 08:13 pm>[1308356.030595]  [<ffffffff812c4e9a>] do_raw_spin_lock+0x113/0x147
     321<Sep/20 08:13 pm>[1308356.030598]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
     322<Sep/20 08:13 pm>[1308356.030600]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76
     323
     324<Sep/20 08:13 pm>[1308356.030602]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
     325<Sep/20 08:13 pm>[1308356.030604]  [<ffffffff81037672>] load_balance+0xbb6/0xf63
     326                 <Sep/20 08:13 pm>[1308356.030606]  [<ffffffff810b3cfb>] ? get_dirty_limits+0x22/0x258
     327<Sep/20 08:13 pm>[1308356.030609]  [<ffffffff8151cc0e>] schedule+0x25a/0x86a
     328<Sep/20 08:13 pm>[1308356.030612]  [<ffffffff810480a2>] ? lock_timer_base+0x26/0x4a
     329<Sep/20 08:13 pm>[1308356.030614]  [<ffffffff8151d559>] schedule_timeout+0x222/0x24c
     330<Sep/20 08:13 pm>[1308356.030616]  [<ffffffff810482a5>] ? process_timeout+0x0/0xb
     331<Sep/20 08:13 pm>[1308356.030619]  [<ffffffff8151c6c5>] io_schedule_timeout+0x45/0x5e
     332<Sep/20 08:13 pm>[1308356.030622]  [<ffffffff810c0abe>] congestion_wait+0x6b/0x87
     333<Sep/20 08:13 pm>[1308356.030624]  [<ffffffff81054b6c>] ? autoremove_wake_function+0x0/0x38
     334<Sep/20 08:13 pm>[1308356.030627]  [<ffffffff810b3279>] __alloc_pages_nodemask+0x65a/0x671
     335<Sep/20 08:13 pm>[1308356.030630]  [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
     336<Sep/20 08:13 pm>[1308356.030632]  [<ffffffff810acb60>] __page_cache_alloc+0x59/0x5e
     337<Sep/20 08:13 pm>[1308356.030634]  [<ffffffff810b4860>] __do_page_cache_readahead+0xe5/0x1fc
     338<Sep/20 08:13 pm>[1308356.030636]  [<ffffffff810b47f6>] ? __do_page_cache_readahead+0x7b/0x1fc
     339<Sep/20 08:13 pm>[1308356.030638]  [<ffffffff812bc23c>] ? radix_tree_prev_hole+0x30/0x53
     340<Sep/20 08:13 pm>[1308356.030640]  [<ffffffff810b4c40>] ondemand_readahead+0x1cc/0x256
     341<Sep/20 08:13 pm>[1308356.030642]  [<ffffffff810b4ba0>] ? ondemand_readahead+0x12c/0x256
     342<Sep/20 08:13 pm>[1308356.030644]  [<ffffffff810b4daa>] page_cache_sync_readahead+0x38/0x3a
     343<Sep/20 08:13 pm>[1308356.030648]  [<ffffffff81109bf2>] __generic_file_splice_read+0xfa/0x41c
     344<Sep/20 08:13 pm>[1308356.030650]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     345<Sep/20 08:13 pm>[1308356.030653]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     346
     347<Sep/20 08:13 pm>[1308356.030656]  [<ffffffff812615b1>] ? xfs_ilock+0x49/0xb2
     348<Sep/20 08:13 pm>[1308356.030659]  [<ffffffff811086db>] ? spd_release_page+0x0/0x14
     349<Sep/20 08:13 pm>[1308356.030661]  [<ffffffff81109f57>] generic_file_splice_read+0x43/0x74
     350<Sep/20 08:13 pm>[1308356.030665]  [<ffffffff8128191a>] xfs_file_splice_read+0xde/0x125
     351<Sep/20 08:13 pm>[1308356.030667]  [<ffffffff8110875d>] vfs_splice_to+0x6e/0x81
     352<Sep/20 08:13 pm>[1308356.030669]  [<ffffffff81108837>] splice_direct_to_actor+0xc7/0x19c
     353<Sep/20 08:13 pm>[1308356.030672]  [<ffffffff811f562c>] ? nfsd_direct_splice_actor+0x0/0x12
     354<Sep/20 08:13 pm>[1308356.030674]  [<ffffffff811f54ef>] nfsd_vfs_read+0x263/0x3a0
     355<Sep/20 08:13 pm>[1308356.030676]  [<ffffffff81281760>] ? xfs_file_open+0x0/0x3f
     356<Sep/20 08:13 pm>[1308356.030678]  [<ffffffff811f5c4e>] nfsd_read+0xa7/0xc3
     357<Sep/20 08:13 pm>[1308356.030681]  [<ffffffff811fcd79>] nfsd3_proc_read+0xfe/0x142
     358<Sep/20 08:13 pm>[1308356.030683]  [<ffffffff811f0854>] nfsd_dispatch+0xe9/0x1ca
     359<Sep/20 08:13 pm>[1308356.030685]  [<ffffffff814f04c5>] svc_process+0x425/0x72a
     360<Sep/20 08:13 pm>[1308356.030687]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
     361<Sep/20 08:13 pm>[1308356.030689]  [<ffffffff811f0dc2>] nfsd+0xf3/0x139
     362<Sep/20 08:13 pm>[1308356.030691]  [<ffffffff810546e2>] kthread+0x7d/0x85
     363<Sep/20 08:13 pm>[1308356.030693]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     364<Sep/20 08:13 pm>[1308356.030695]  [<ffffffff81054665>] ? kthread+0x0/0x85
     365<Sep/20 08:13 pm>[1308356.030697]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     366<Sep/20 08:13 pm>[1308356.030698] Code: 49 03 40 38 88 10 83 fe 01 77 0c be 01 00 00 00 4c 89 c7 41 ff 50 40 58 5b c9 c3 0f b6 4f 69 55 89 f2 48 89 e5 d3 e2 03 57 30 ec <0f> b6 c0 c9 c3 0f b6 4f 69 55 88 d0 48 89 e5 d3 e6 03 77 30 89
     367<Sep/20 08:13 pm>[1308356.030713] Call Trace:
     368<Sep/20 08:13 pm>[1308356.030716]  [<ffffffff81334767>] wait_for_xmitr+0x24/0x92
     369
     370<Sep/20 08:13 pm>[1308356.030718]  [<ffffffff813347d5>] ? serial8250_console_putchar+0x0/0x27
     371<Sep/20 08:13 pm>[1308356.030720]  [<ffffffff813347ec>] serial8250_console_putchar+0x17/0x27
     372<Sep/20 08:13 pm>[1308356.030722]  [<ffffffff813311e8>] uart_console_write+0x41/0x52
     373<Sep/20 08:13 pm>[1308356.030725]  [<ffffffff81334eb0>] serial8250_console_write+0xb6/0x108
     374<Sep/20 08:13 pm>[1308356.030727]  [<ffffffff8103b339>] __call_console_drivers+0x67/0x79
     375<Sep/20 08:13 pm>[1308356.030730]  [<ffffffff8103b3a6>] _call_console_drivers+0x5b/0x5f
     376<Sep/20 08:13 pm>[1308356.030732]  [<ffffffff8103b87b>] release_console_sem+0x121/0x1c6
     377<Sep/20 08:13 pm>[1308356.030734]  [<ffffffff8103beb2>] vprintk+0x36b/0x3a0
     378<Sep/20 08:13 pm>[1308356.030736]  [<ffffffff8151c589>] printk+0x67/0x69
     379<Sep/20 08:13 pm>[1308356.030739]  [<ffffffff812c08fe>] ? __delay+0xa/0xc
     380<Sep/20 08:13 pm>[1308356.030741]  [<ffffffff812c4e9a>] do_raw_spin_lock+0x113/0x147
     381<Sep/20 08:13 pm>[1308356.030743]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
     382<Sep/20 08:13 pm>[1308356.030745]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76
     383<Sep/20 08:13 pm>[1308356.030747]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
     384<Sep/20 08:13 pm>[1308356.030749]  [<ffffffff81037672>] load_balance+0xbb6/0xf63
     385<Sep/20 08:13 pm>[1308356.030751]  [<ffffffff810b3cfb>] ? get_dirty_limits+0x22/0x258
     386<Sep/20 08:13 pm>[1308356.030754]  [<ffffffff8151cc0e>] schedule+0x25a/0x86a
     387<Sep/20 08:13 pm>[1308356.030756]  [<ffffffff810480a2>] ? lock_timer_base+0x26/0x4a
     388<Sep/20 08:13 pm>[1308356.030759]  [<ffffffff8151d559>] schedule_timeout+0x222/0x24c
     389<Sep/20 08:13 pm>[1308356.030761]  [<ffffffff810482a5>] ? process_timeout+0x0/0xb
     390<Sep/20 08:13 pm>[1308356.030763]  [<ffffffff8151c6c5>] io_schedule_timeout+0x45/0x5e
     391<Sep/20 08:13 pm>[1308356.030765]  [<ffffffff810c0abe>] congestion_wait+0x6b/0x87
     392
     393<Sep/20 08:13 pm>[1308356.030767]  [<ffffffff81054b6c>] ? autoremove_wake_function+0x0/0x38
     394<Sep/20 08:13 pm>[1308356.030770]  [<ffffffff810b3279>] __alloc_pages_nodemask+0x65a/0x671
     395<Sep/20 08:13 pm>[1308356.030772]  [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
     396<Sep/20 08:13 pm>[1308356.030775]  [<ffffffff810acb60>] __page_cache_alloc+0x59/0x5e
     397<Sep/20 08:13 pm>[1308356.030776]  [<ffffffff810b4860>] __do_page_cache_readahead+0xe5/0x1fc
     398<Sep/20 08:13 pm>[1308356.030778]  [<ffffffff810b47f6>] ? __do_page_cache_readahead+0x7b/0x1fc
     399<Sep/20 08:13 pm>[1308356.030781]  [<ffffffff812bc23c>] ? radix_tree_prev_hole+0x30/0x53
     400<Sep/20 08:13 pm>[1308356.030783]  [<ffffffff810b4c40>] ondemand_readahead+0x1cc/0x256
     401<Sep/20 08:13 pm>[1308356.030784]  [<ffffffff810b4ba0>] ? ondemand_readahead+0x12c/0x256
     402<Sep/20 08:13 pm>[1308356.030786]  [<ffffffff810b4daa>] page_cache_sync_readahead+0x38/0x3a
     403<Sep/20 08:13 pm>[1308356.030789]  [<ffffffff81109bf2>] __generic_file_splice_read+0xfa/0x41c
     404<Sep/20 08:13 pm>[1308356.030791]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0<Sep/20 08:13 pm>
     405                 <Sep/20 08:13 pm>[1308356.030793]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     406<Sep/20 08:13 pm>[1308356.030796]  [<ffffffff812615b1>] ? xfs_ilock+0x49/0xb2
     407<Sep/20 08:13 pm>[1308356.030798]  [<ffffffff811086db>] ? spd_release_page+0x0/0x14
     408<Sep/20 08:13 pm>[1308356.030800]  [<ffffffff81109f57>] generic_file_splice_read+0x43/0x74
     409<Sep/20 08:13 pm>[1308356.030803]  [<ffffffff8128191a>] xfs_file_splice_read+0xd<Sep/20 08:13 pm>
     410                 <Sep/20 08:13 pm>[1308356.030805]  [<ffffffff8110875d>] vfs_splice_to+0x6e/0x81
     411<Sep/20 08:13 pm>[1308356.030807]  [<ffffffff81108837>] splice_direct_to_actor+0xc7/0x19c
     412<Sep/20 08:13 pm>[1308356.030809]  [<ffffffff811f562c>] ? nfsd_direct_splice_actor+0x0/0x12
     413<Sep/20 08:13 pm>[1308356.030811]  [<ffffffff811f54ef>] nfsd_vfs_read+0x263/0x3a0
     414<Sep/20 08:13 pm>[1308356.030813]  [<ffffffff81281760>] ? xfs_file_open+0x0/0x3f
     415
     416<Sep/20 08:13 pm>[1308356.030816]  [<ffffffff811f5c4e>] nfsd_read+0xa7/0xc3
     417<Sep/20 08:13 pm>[1308356.030818]  [<ffffffff811fcd79>] nfsd3_proc_read+0xfe/0x142
     418<Sep/20 08:13 pm>[1308356.030820]  [<ffffffff811f0854>] nfsd_dispatch+0xe9/0x1ca
     419<Sep/20 08:13 pm>[1308356.030822]  [<ffffffff814f04c5>] svc_process+0x425/0x72a
     420<Sep/20 08:13 pm>[1308356.030824]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
     421<Sep/20 08:13 pm>[1308356.030825]  [<ffffffff811f0dc2>] nfsd+0xf3/0x139
     422<Sep/20 08:13 pm>[1308356.030827]  [<ffffffff810546e2>] kthread+0x7d/0x85
     423<Sep/20 08:13 pm>[1308356.030830]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     424<Sep/20 08:13 pm>[1308356.030832]  [<ffffffff81054665>] ? kthread+0x0/0x85
     425<Sep/20 08:13 pm>[1308356.030834]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     426<Sep/20 08:13 pm>[1308356.030835] Pid: 8358, comm: nfsd Tainted: G      D    2.6.34.4 #3
     427<Sep/20 08:13 pm>[1308356.030836] Call Trace:
     428<Sep/20 08:13 pm>[1308356.030837]  <NMI>  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
     429<Sep/20 08:13 pm>[1308356.030841]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
     430<Sep/20 08:13 pm>[1308356.030843]  [<ffffffff81520360>] do_nmi+0xb2/0x259
     431<Sep/20 08:13 pm>[1308356.030845]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
     432<Sep/20 08:13 pm>[1308356.030847]  [<ffffffff81334e4e>] ? serial8250_console_write+0x54/0x108
     433<Sep/20 08:13 pm>[1308356.030849]  [<ffffffff8133423b>] ? io_serial_in+0x10/0x15
     434<Sep/20 08:13 pm>[1308356.030850]  <<EOE>>  [<ffffffff81334767>] wait_for_xmitr+0x24/0x92
     435<Sep/20 08:13 pm>[1308356.030854]  [<ffffffff813347d5>] ? serial8250_console_putchar+0x0/0x27
     436<Sep/20 08:13 pm>[1308356.030857]  [<ffffffff813347ec>] serial8250_console_putchar+0x17/0x27
     437<Sep/20 08:13 pm>[1308356.030859]  [<ffffffff813311e8>] uart_console_write+0x41/0x52
     438
     439<Sep/20 08:13 pm>[1308356.030861]  [<ffffffff81334eb0>] serial8250_console_write+0xb6/0x108
     440<Sep/20 08:13 pm>[1308356.030864]  [<ffffffff8103b339>] __call_console_drivers+0x67/0x79
     441<Sep/20 08:13 pm>[1308356.030866]  [<ffffffff8103b3a6>] _call_console_drivers+0x5b/0x5f
     442<Sep/20 08:13 pm>[1308356.030868]  [<ffffffff8103b87b>] release_console_sem+0x121/0x1c6
     443<Sep/20 08:13 pm>[1308356.030871]  [<ffffffff8103beb2>] vprintk+0x36b/0x3a0
     444<Sep/20 08:13 pm>[1308356.030873]  [<ffffffff8151c589>] printk+0x67/0x69
     445<Sep/20 08:13 pm>[1308356.030875]  [<ffffffff812c08fe>] ? __delay+0xa/0xc
     446<Sep/20 08:13 pm>[1308356.030877]  [<ffffffff812c4e9a>] do_raw_spin_lock+0x113/0x147
     447<Sep/20 08:13 pm>[1308356.030879]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
     448<Sep/20 08:13 pm>[1308356.030882]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76
     449<Sep/20 08:13 pm>[1308356.030884]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
     450<Sep/20 08:13 pm>[1308356.030886]  [<ffffffff81037672>] load_balance+0xbb6/0xf63
     451<Sep/20 08:13 pm>[1308356.030888]  [<ffffffff810b3cfb>] ? get_dirty_limits+0x22/0x258
     452<Sep/20 08:13 pm>[1308356.030891]  [<ffffffff8151cc0e>] schedule+0x25a/0x86a
     453<Sep/20 08:13 pm>[1308356.030893]  [<ffffffff810480a2>] ? lock_timer_base+0x26/0x4a
     454<Sep/20 08:13 pm>[1308356.030895]  [<ffffffff8151d559>] schedule_timeout+0x222/0x24c
     455<Sep/20 08:13 pm>[1308356.030897]  [<ffffffff810482a5>] ? process_timeout+0x0/0xb
     456<Sep/20 08:13 pm>[1308356.030900]  [<ffffffff8151c6c5>] io_schedule_timeout+0x45/0x5e
     457<Sep/20 08:13 pm>[1308356.030902]  [<ffffffff810c0abe>] congestion_wait+0x6b/0x87
     458<Sep/20 08:13 pm>[1308356.030904]  [<ffffffff81054b6c>] ? autoremove_wake_function+0x0/0x38
     459<Sep/20 08:13 pm>[1308356.030907]  [<ffffffff810b3279>] __alloc_pages_nodemask+0x65a/0x671
     460<Sep/20 08:13 pm>[1308356.030909]  [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
     461
     462<Sep/20 08:13 pm>[1308356.030911]  [<ffffffff810acb60>] __page_cache_alloc+0x59/0x5e
     463<Sep/20 08:13 pm>[1308356.030913]  [<ffffffff810b4860>] __do_page_cache_readahead+0xe5/0x1fc
     464<Sep/20 08:13 pm>[1308356.030915]  [<ffffffff810b47f6>] ? __do_page_cache_readahead+0x7b/0x1fc
     465<Sep/20 08:13 pm>[1308356.030917]  [<ffffffff812bc23c>] ? radix_tree_prev_hole+0x30/0x53
     466<Sep/20 08:13 pm>[1308356.030919]  [<ffffffff810b4c40>] ondemand_readahead+0x1cc/0x256
     467<Sep/20 08:13 pm>[1308356.030921]  [<ffffffff810b4ba0>] ? ondemand_readahead+0x12c/0x256
     468<Sep/20 08:13 pm>[1308356.030923]  [<ffffffff810b4daa>] page_cache_sync_readahead+0x38/0x3a
     469<Sep/20 08:13 pm>[1308356.030926]  [<ffffffff81109bf2>] __generic_file_splice_read+0xfa/0x41c
     470<Sep/20 08:13 pm>[1308356.030928]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     471<Sep/20 08:13 pm>[1308356.030930]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     472<Sep/20 08:13 pm>[1308356.030933]  [<ffffffff812615b1>] ? xfs_ilock+0x49/0xb2
     473<Sep/20 08:13 pm>[1308356.030935]  [<ffffffff811086db>] ? spd_release_page+0x0/0x14
     474<Sep/20 08:13 pm>[1308356.030937]  [<ffffffff81109f57>] generic_file_splice_read+0x43/0x74
     475<Sep/20 08:13 pm>[1308356.030940]  [<ffffffff8128191a>] xfs_file_splice_read+0xde/0x125
     476<Sep/20 08:13 pm>[1308356.030942]  [<ffffffff8110875d>] vfs_splice_to+0x6e/0x81
     477<Sep/20 08:13 pm>[1308356.030944]  [<ffffffff81108837>] splice_direct_to_actor+0xc7/0x19c
     478<Sep/20 08:13 pm>[1308356.030946]  [<ffffffff811f562c>] ? nfsd_direct_splice_actor+0x0/0x12
     479<Sep/20 08:13 pm>[1308356.030948]  [<ffffffff811f54ef>] nfsd_vfs_read+0x263/0x3a0
     480<Sep/20 08:13 pm>[1308356.030951]  [<ffffffff81281760>] ? xfs_file_open+0x0/0x3f
     481<Sep/20 08:13 pm>[1308356.030953]  [<ffffffff811f5c4e>] nfsd_read+0xa7/0xc3
     482<Sep/20 08:13 pm>[1308356.030955]  [<ffffffff811fcd79>] nfsd3_proc_read+0xfe/0x142
     483<Sep/20 08:13 pm>[1308356.030957]  [<ffffffff811f0854>] nfsd_dispatch+0xe9/0x1ca
     484
     485<Sep/20 08:13 pm>[1308356.030959]  [<ffffffff814f04c5>] svc_process+0x425/0x72a
     486<Sep/20 08:13 pm>[1308356.030961]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
     487<Sep/20 08:13 pm>[1308356.030963]  [<ffffffff811f0dc2>] nfsd+0xf3/0x139
     488<Sep/20 08:13 pm>[1308356.030965]  [<ffffffff810546e2>] kthread+0x7d/0x85
     489<Sep/20 08:13 pm>[1308356.030967]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     490<Sep/20 08:13 pm>[1308356.030969]  [<ffffffff81054665>] ? kthread+0x0/0x85
     491<Sep/20 08:13 pm>[1308356.030971]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     492<Sep/20 08:13 pm>[1308356.030973] NMI backtrace for cpu 1
     493<Sep/20 08:13 pm>[1308356.030974] CPU 1
     494<Sep/20 08:13 pm>[1308356.030975] Modules linked in: w83793 hwmon_vid autofs4 i2c_i801 i2c_core iTCO_wdt e1000e ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
     495<Sep/20 08:13 pm>[1308356.030990]
     496<Sep/20 08:13 pm>[1308356.030992] Pid: 1321, comm: updatedb Tainted: G      D    2.6.34.4 #3 S7010/empty
     497<Sep/20 08:13 pm>[1308356.030993] RIP: 0010:[<ffffffff812c08ad>]  [<ffffffff812c08ad>] delay_tsc+0x23/0x59
     498<Sep/20 08:13 pm>[1308356.030997] RSP: 0018:ffff880464f392a0  EFLAGS: 00000086
     499<Sep/20 08:13 pm>[1308356.030998] RAX: 00000000369db028 RBX: 000000009ef2d30a RCX: 00000000369db028
     500<Sep/20 08:13 pm>[1308356.031000] RDX: 00000000000c656e RSI: 0000000000000001 RDI: 0000000000000001
     501<Sep/20 08:13 pm>[1308356.031001] RBP: ffff880464f392a8 R08: 0000000000000001 R09: 0000000000000001
     502<Sep/20 08:13 pm>[1308356.031003] R10: ffffffff81035915 R11: 0000000117632264 R12: ffff880002673c00
     503<Sep/20 08:13 pm>[1308356.031004] R13: 0000000000000001 R14: ffff88022de16120 R15: ffff88022de165a8
     504<Sep/20 08:13 pm>[1308356.031006] FS:  00007fc4b72196f0(0000) GS:ffff880002620000(0000) knlGS:0000000000000000
     505<Sep/20 08:13 pm>[1308356.031008] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
     506<Sep/20 08:13 pm>[1308356.031009] CR2: 00007f6454d05000 CR3: 000000046f705000 CR4: 00000000000006e0
     507
     508<Sep/20 08:13 pm>[1308356.031011] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     509<Sep/20 08:13 pm>[1308356.031012] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     510<Sep/20 08:13 pm>[1308356.031014] Process updatedb (pid: 1321, threadinfo ffff880464f38000, task ffff88022de16120)
     511<Sep/20 08:13 pm>[1308356.031015] Stack:
     512<Sep/20 08:13 pm>[1308356.031016]  0000000000000046 ffff880464f392b8 ffffffff812c08fe ffff880464f392f8
     513<Sep/20 08:13 pm>[1308356.031018] <0> ffffffff812c4e65 000000009ef2f014 ffff880002673c00 ffff880002673c00
     514<Sep/20 08:13 pm>[1308356.031020] <0> ffff88000262e3f0 0000000000000400 0000000000000800 ffff880464f39318
     515<Sep/20 08:13 pm>[1308356.031023] Call Trace:
     516<Sep/20 08:13 pm>[1308356.031025]  [<ffffffff812c08fe>] __delay+0xa/0xc
     517<Sep/20 08:13 pm>[1308356.031027]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     518<Sep/20 08:13 pm>[1308356.031030]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
     519<Sep/20 08:13 pm>[1308356.031033]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76
     520<Sep/20 08:13 pm>[1308356.031035]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
     521<Sep/20 08:13 pm>[1308356.031037]  [<ffffffff81037672>] load_balance+0xbb6/0xf63
     522<Sep/20 08:13 pm>[1308356.031039]  [<ffffffff810b3cfb>] ? get_dirty_limits+0x22/0x258
     523<Sep/20 08:13 pm>[1308356.031042]  [<ffffffff8151cc0e>] schedule+0x25a/0x86a
     524<Sep/20 08:13 pm>[1308356.031045]  [<ffffffff810480a2>] ? lock_timer_base+0x26/0x4a
     525<Sep/20 08:13 pm>[1308356.031047]  [<ffffffff8151d559>] schedule_timeout+0x222/0x24c
     526<Sep/20 08:13 pm>[1308356.031049]  [<ffffffff810482a5>] ? process_timeout+0x0/0xb
     527<Sep/20 08:13 pm>[1308356.031051]  [<ffffffff8151c6c5>] io_schedule_timeout+0x45<Sep/20 08:13 pm>
     528                 <Sep/20 08:13 pm>[1308356.031054]  [<ffffffff810c0abe>] congestion_wait+0x6b/0x87
     529<Sep/20 08:13 pm>[1308356.031057]  [<ffffffff81054b6c>] ? autoremove_wake_function+0x0/0x38
     530
     531<Sep/20 08:13 pm>[1308356.031059]  [<ffffffff810b3279>] __alloc_pages_nodemask+0x65a/0x671
     532<Sep/20 08:13 pm>[1308356.031062]  [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
     533<Sep/20 08:13 pm>[1308356.031064]  [<ffffffff810acb60>] __page_cache_alloc+0x59/0x5e
     534<Sep/20 08:13 pm>[1308356.031066]  [<ffffffff810ad2d5>] find_or_create_page+0x41/0x85
     535<Sep/20 08:13 pm>[1308356.031069]  [<ffffffff81280160>] _xfs_buf_lookup_pages+0x111/0x2d6
     536<Sep/20 08:13 pm>[1308356.031072]  [<ffffffff812811df>] xfs_buf_get+0x73/0x17e
     537<Sep/20 08:13 pm>[1308356.031074]  [<ffffffff81281304>] xfs_buf_read+0x1a/0xc6
     538<Sep/20 08:13 pm>[1308356.031076]  [<ffffffff8127769d>] xfs_trans_read_buf+0x4e/0x396
     539<Sep/20 08:13 pm>[1308356.031078]  [<ffffffff81264f29>] xfs_imap_to_bp+0x44/0xfd
     540<Sep/20 08:13 pm>[1308356.031080]  [<ffffffff812661f1>] xfs_iread+0x59/0x188
     541<Sep/20 08:13 pm>[1308356.031083]  [<ffffffff81261d6e>] xfs_iget+0x380/0x57e
     542<Sep/20 08:13 pm>[1308356.031085]  [<ffffffff8127acc4>] xfs_lookup+0xac/0xdd
     543<Sep/20 08:13 pm>[1308356.031087]  [<ffffffff81284ca0>] xfs_vn_lookup+0x3f/0x7e
     544<Sep/20 08:13 pm>[1308356.031091]  [<ffffffff810f2741>] do_lookup+0xfe/0x1e2
     545<Sep/20 08:13 pm>[1308356.031093]  [<ffffffff810f4e33>] link_path_walk+0x718/0xb91
     546<Sep/20 08:13 pm>[1308356.031096]  [<ffffffff810f540a>] path_walk+0x6d/0xeb
     547<Sep/20 08:13 pm>[1308356.031098]  [<ffffffff810f55a1>] do_path_lookup+0x2a/0x87
     548<Sep/20 08:13 pm>[1308356.031100]  [<ffffffff810f5fb6>] user_path_at+0x52/0x8c
     549<Sep/20 08:13 pm>[1308356.031103]  [<ffffffff8151d8cc>] ? mutex_unlock+0x9/0xb
     550                 <Sep/20 08:13 pm>[1308356.031105]  [<ffffffff810eded3>] vfs_fstatat+0x35/0x62
     551<Sep/20 08:13 pm>[1308356.031107]  [<ffffffff810edf54>] vfs_lstat+0x19/0x1b
     552<Sep/20 08:13 pm>[1308356.031109]  [<ffffffff810edf70>] sys_newlstat+0x1a/0x3b
     553
     554<Sep/20 08:13 pm>[1308356.031111]  [<ffffffff8151ef6f>] ? lockdep_sys_exit_thunk+0x35/0x67
     555<Sep/20 08:13 pm>[1308356.031114]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
     556<Sep/20 08:13 pm>[1308356.031115] Code: c8 75 fb 48 ff c8 c9 c3 55 48 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae e8 0f 31 89 c1 41 89 f1 66 66 90 0f ae e8 <0f> 31 89 c6 48 89 f0 48 29 c8 48 39 f8 73 25 f3 90 65 44 8b 04
     557<Sep/20 08:13 pm>[1308356.031131] Call Trace:
     558<Sep/20 08:13 pm>[1308356.031133]  [<ffffffff812c08fe>] __delay+0xa/0xc
     559<Sep/20 08:13 pm>[1308356.031135]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     560<Sep/20 08:13 pm>[1308356.031137]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
     561<Sep/20 08:13 pm>[1308356.031139]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76
     562<Sep/20 08:13 pm>[1308356.031141]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
     563<Sep/20 08:13 pm>[1308356.031143]  [<ffffffff81037672>] load_balance+0xbb6/0xf63
     564<Sep/20 08:13 pm>[1308356.031145]  [<ffffffff810b3cfb>] ? get_dirty_limits+0x22/0x258
     565<Sep/20 08:13 pm>[1308356.031148]  [<ffffffff8151cc0e>] schedule+0x25a/0x86a
     566<Sep/20 08:13 pm>[1308356.031150]  [<ffffffff810480a2>] ? lock_timer_base+0x26/0x4a
     567<Sep/20 08:13 pm>[1308356.031152]  [<ffffffff8151d559>] schedule_timeout+0x222/0x24c
     568<Sep/20 08:13 pm>[1308356.031154]  [<ffffffff810482a5>] ? process_timeout+0x0/0xb
     569<Sep/20 08:13 pm>[1308356.031157]  [<ffffffff8151c6c5>] io_schedule_timeout+0x45/0x5e
     570<Sep/20 08:13 pm>[1308356.031159]  [<ffffffff810c0abe>] congestion_wait+0x6b/0x87
     571<Sep/20 08:13 pm>[1308356.031161]  [<ffffffff81054b6c>] ? autoremove_wake_function+0x0/0x38
     572<Sep/20 08:13 pm>[1308356.031163]  [<ffffffff810b3279>] __alloc_pages_nodemask+0x65a/0x671
     573<Sep/20 08:13 pm>[1308356.031166]  [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
     574<Sep/20 08:13 pm>[1308356.031168]  [<ffffffff810acb60>] __page_cache_alloc+0x59/0x5e
     575<Sep/20 08:13 pm>[1308356.031170]  [<ffffffff810ad2d5>] find_or_create_page+0x41/0x85
     576
     577<Sep/20 08:13 pm>[1308356.031173]  [<ffffffff81280160>] _xfs_buf_lookup_pages+0x111/0x2d6
     578<Sep/20 08:13 pm>[1308356.031175]  [<ffffffff812811df>] xfs_buf_get+0x73/0x17e
     579<Sep/20 08:13 pm>[1308356.031177]  [<ffffffff81281304>] xfs_buf_read+0x1a/0xc6
     580<Sep/20 08:13 pm>[1308356.031179]  [<ffffffff8127769d>] xfs_trans_read_buf+0x4e/0x396
     581<Sep/20 08:13 pm>[1308356.031181]  [<ffffffff81264f29>] xfs_imap_to_bp+0x44/0xfd
     582<Sep/20 08:13 pm>[1308356.031183]  [<ffffffff812661f1>] xfs_iread+0x59/0x188
     583<Sep/20 08:13 pm>[1308356.031185]  [<ffffffff81261d6e>] xfs_iget+0x380/0x57e
     584<Sep/20 08:13 pm>[1308356.031187]  [<ffffffff8127acc4>] xfs_lookup+0xac/0xdd
     585<Sep/20 08:13 pm>[1308356.031189]  [<ffffffff81284ca0>] xfs_vn_lookup+0x3f/0x7e
     586<Sep/20 08:13 pm>[1308356.031192]  [<ffffffff810f2741>] do_lookup+0xfe/0x1e2
     587<Sep/20 08:13 pm>[1308356.031194]  [<ffffffff810f4e33>] link_path_walk+0x718/0xb91
     588<Sep/20 08:13 pm>[1308356.031196]  [<ffffffff810f540a>] path_walk+0x6d/0xeb
     589<Sep/20 08:13 pm>[1308356.031198]  [<ffffffff810f55a1>] do_path_lookup+0x2a/0x87
     590                 <Sep/20 08:13 pm>[1308356.031201]  [<ffffffff810f5fb6>] user_path_at+0x52/0x8c
     591<Sep/20 08:13 pm>[1308356.031203]  [<ffffffff8151d8cc>] ? mutex_unlock+0x9/0xb
     592<Sep/20 08:13 pm>[1308356.031205]  [<ffffffff810eded3>] vfs_fstatat+0x35/0x62
     593<Sep/20 08:13 pm>[1308356.031207]  [<ffffffff810edf54>] vfs_lstat+0x19/0x1b
     594<Sep/20 08:13 pm>[1308356.031209]  [<ffffffff810edf70>] sys_newlstat+0x1a/0x3b
     595<Sep/20 08:13 pm>[1308356.031211]  [<ffffffff8151ef6f>] ? lockdep_sys_exit_thunk<Sep/20 08:13 pm>
     596                 <Sep/20 08:13 pm>[1308356.031214]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
     597<Sep/20 08:13 pm>[1308356.031216] Pid: 1321, comm: updatedb Tainted: G      D    2.6.34.4 #3
     598<Sep/20 08:13 pm>[1308356.031217] Call Trace:
     599
     600<Sep/20 08:13 pm>[1308356.031217]  <NMI>
     601<Sep/20 08:13 pm>[1308356.031219] BUG: spinlock lockup on CPU#0, xfsdatad/0/734, ffff880002673c00
     602<Sep/20 08:13 pm>[1308356.031221] Pid: 734, comm: xfsdatad/0 Tainted: G      D    2.6.34.4 #3
     603<Sep/20 08:13 pm>[1308356.031223]  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
     604                 <Sep/20 08:13 pm>[1308356.031224] Call Trace:
     605<Sep/20 08:13 pm>[1308356.031226]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
     606<Sep/20 08:13 pm>[1308356.031228]  [<ffffffff812c4e9f>] do_raw_spin_lock+0x118/0x147
     607<Sep/20 08:13 pm>[1308356.031230]  [<ffffffff81520360>] do_nmi+0xb2/0x259
     608<Sep/20 08:13 pm>[1308356.031233]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
     609<Sep/20 08:13 pm>[1308356.031235]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
     610<Sep/20 08:13 pm>[1308356.031237]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76
     611<Sep/20 08:13 pm>[1308356.031239]  [<ffffffff81035915>] ? double_rq_lock+0x2b/0x76
     612<Sep/20 08:13 pm>[1308356.031241]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
     613<Sep/20 08:13 pm>[1308356.031243]  [<ffffffff812c08ad>] ? delay_tsc+0x23/0x59
     614<Sep/20 08:13 pm>[1308356.031245]  [<ffffffff81037672>] load_balance+0xbb6/0xf63
     615<Sep/20 08:13 pm>[1308356.031246]  <<EOE>>  [<ffffffff81030875>] ? double_rq_unlock+0x1f/0x24
     616<Sep/20 08:13 pm>[1308356.031250]  [<ffffffff812c08fe>] __delay+0xa/0xc
     617<Sep/20 08:13 pm>[1308356.031252]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     618<Sep/20 08:13 pm>[1308356.031254]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
     619<Sep/20 08:13 pm>[1308356.031257]  [<ffffffff8151cc0e>] schedule+0x25a/0x86a
     620<Sep/20 08:13 pm>[1308356.031259]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
     621<Sep/20 08:13 pm>[1308356.031261]  [<ffffffff810480a2>] ? lock_timer_base+0x26/0x4a
     622
     623<Sep/20 08:13 pm>[1308356.031263]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76
     624<Sep/20 08:13 pm>[1308356.031266]  [<ffffffff8151d559>] schedule_timeout+0x222/0x24c
     625<Sep/20 08:13 pm>[1308356.031268]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
     626<Sep/20 08:13 pm>[1308356.031270]  [<ffffffff810482a5>] ? process_timeout+0x0/0xb
     627<Sep/20 08:13 pm>[1308356.031272]  [<ffffffff81037672>] load_balance+0xbb6/0xf63
     628<Sep/20 08:13 pm>[1308356.031275]  [<ffffffff8151d59c>] schedule_timeout_uninter<Sep/20 08:13 pm>b
     629                 <Sep/20 08:13 pm>[1308356.031276]  [<ffffffff810b3cfb>] ? get_dirty_limits+0x22/0x258
     630<Sep/20 08:13 pm>[1308356.031279]  [<ffffffff8127d1af>] xfs_end_io+0x7d/0x8c
     631<Sep/20 08:13 pm>[1308356.031281]  [<ffffffff8151cc0e>] schedule+0x25a/0x86a
     632<Sep/20 08:13 pm>[1308356.031283]  [<ffffffff81050bf4>] worker_thread+0x22c/0x33<Sep/20 08:13 pm>
     633                 <Sep/20 08:13 pm>[1308356.031285]  [<ffffffff810480a2>] ? lock_timer_base+0x26/0x4a
     634<Sep/20 08:13 pm>[1308356.031287]  [<ffffffff81050b9d>] ? worker_thread+0x1d5/0x332
     635<Sep/20 08:13 pm>[1308356.031290]  [<ffffffff8151d559>] schedule_timeout+0x222/0x24c
     636<Sep/20 08:13 pm>[1308356.031291]  [<ffffffff8127d132>] ? xfs_end_io+0x0/0x8c
     637<Sep/20 08:13 pm>[1308356.031294]  [<ffffffff810482a5>] ? process_timeout+0x0/0xb
     638<Sep/20 08:13 pm>[1308356.031296]  [<ffffffff81054b6c>] ? autoremove_wake_function+0x0/0x38
     639<Sep/20 08:13 pm>[1308356.031298]  [<ffffffff8151c6c5>] io_schedule_timeout+0x45/0x5e
     640<Sep/20 08:13 pm>[1308356.031300]  [<ffffffff810509c8>] ? worker_thread+0x0/0x332
     641<Sep/20 08:13 pm>[1308356.031302]  [<ffffffff810c0abe>] congestion_wait+0x6b/0x87
     642<Sep/20 08:13 pm>[1308356.031305]  [<ffffffff810546e2>] kthread+0x7d/0x85
     643<Sep/20 08:13 pm>[1308356.031307]  [<ffffffff81054b6c>] ? autoremove_wake_function+0x0/0x38
     644<Sep/20 08:13 pm>[1308356.031309]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     645
     646<Sep/20 08:13 pm>[1308356.031312]  [<ffffffff810b3279>] __alloc_pages_nodemask+0x65a/0x671
     647<Sep/20 08:13 pm>[1308356.031314]  [<ffffffff81054665>] ? kthread+0x0/0x85
     648<Sep/20 08:13 pm>[1308356.031316]  [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
     649<Sep/20 08:13 pm>[1308356.031318]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     650<Sep/20 08:13 pm>[1308356.031319] sending NMI to all CPUs:
     651<Sep/20 08:13 pm>[1308356.031321]  [<ffffffff810acb60>] __page_cache_alloc+0x59/0x5e
     652<Sep/20 08:13 pm>[1308356.031323]  [<ffffffff810ad2d5>] find_or_create_page+0x41/0x85
     653<Sep/20 08:13 pm>[1308356.031326]  [<ffffffff81280160>] _xfs_buf_lookup_pages+0x111/0x2d6
     654<Sep/20 08:13 pm>[1308356.031328]  [<ffffffff812811df>] xfs_buf_get+0x73/0x17e
     655<Sep/20 08:13 pm>[1308356.031330]  [<ffffffff81281304>] xfs_buf_read+0x1a/0xc6
     656<Sep/20 08:13 pm>[1308356.031332]  [<ffffffff8127769d>] xfs_trans_read_buf+0x4e/0x396
     657<Sep/20 08:13 pm>[1308356.031334]  [<ffffffff81264f29>] xfs_imap_to_bp+0x44/0xfd
     658<Sep/20 08:13 pm>[1308356.031336]  [<ffffffff812661f1>] xfs_iread+0x59/0x188
     659<Sep/20 08:13 pm>[1308356.031338]  [<ffffffff81261d6e>] xfs_iget+0x380/0x57e
     660<Sep/20 08:13 pm>[1308356.031341]  [<ffffffff8127acc4>] xfs_lookup+0xac/0xdd
     661<Sep/20 08:13 pm>[1308356.031343]  [<ffffffff81284ca0>] xfs_vn_lookup+0x3f/0x7e
     662<Sep/20 08:13 pm>[1308356.031345]  [<ffffffff810f2741>] do_lookup+0xfe/0x1e2
     663<Sep/20 08:13 pm>[1308356.031347]  [<ffffffff810f4e33>] link_path_walk+0x718/0xb91
     664<Sep/20 08:13 pm>[1308356.031349]  [<ffffffff810f540a>] path_walk+0x6d/0xeb
     665<Sep/20 08:13 pm>[1308356.031352]  [<ffffffff810f55a1>] do_path_lookup+0x2a/0x87
     666<Sep/20 08:13 pm>[1308356.031354]  [<ffffffff810f5fb6>] user_path_at+0x52/0x8c
     667<Sep/20 08:13 pm>[1308356.031356]  [<ffffffff8151d8cc>] ? mutex_unlock+0x9/0xb
     668
     669<Sep/20 08:13 pm>[1308356.031358]  [<ffffffff810eded3>] vfs_fstatat+0x35/0x62
     670<Sep/20 08:13 pm>[1308356.031360]  [<ffffffff810edf54>] vfs_lstat+0x19/0x1b
     671<Sep/20 08:13 pm>[1308356.031362]  [<ffffffff810edf70>] sys_newlstat+0x1a/0x3b
     672<Sep/20 08:13 pm>[1308356.031364]  [<ffffffff8151ef6f>] ? lockdep_sys_exit_thunk+0x35/0x67
     673<Sep/20 08:13 pm>[1308356.031367]  [<ffffffff810029ab>] system_call_fastpath+0x16/0x1b
     674<Sep/20 08:13 pm>[1308356.031368] NMI backtrace for cpu 0
     675<Sep/20 08:13 pm>[1308356.031369] CPU 0
     676<Sep/20 08:13 pm>[1308356.031370] Modules linked in: w83793 hwmon_vid autofs4 i2c_i801 i2c_core iTCO_wdt e1000e ioatdma dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
     677<Sep/20 08:13 pm>[1308356.031383]
     678<Sep/20 08:13 pm>[1308356.031384] Pid: 734, comm: xfsdatad/0 Tainted: G      D    2.6.34.4 #3 S7010/empty
     679<Sep/20 08:13 pm>[1308356.031386] RIP: 0010:[<ffffffff812c08af>]  [<ffffffff812c08af>] delay_tsc+0x25/0x59
     680<Sep/20 08:13 pm>[1308356.031388] RSP: 0018:ffff880623ae18a8  EFLAGS: 00000092
     681<Sep/20 08:13 pm>[1308356.031390] RAX: 0000000036c46518 RBX: 0000000000000000 RCX: 0000000036c464f8
     682<Sep/20 08:13 pm>[1308356.031391] RDX: 00000000000c656e RSI: 0000000000000000 RDI: 0000000000041183
     683<Sep/20 08:13 pm>[1308356.031393] RBP: ffff880623ae18b0 R08: 0000000000000020 R09: 0000000000000000
     684<Sep/20 08:13 pm>[1308356.031394] R10: 0000000000000007 R11: ffff880623ae1770 R12: 0000000000001000
     685<Sep/20 08:13 pm>[1308356.031396] R13: 0000000000000002 R14: 0000000000000086 R15: ffff880623ad0002
     686<Sep/20 08:13 pm>[1308356.031397] FS:  0000000000000000(0000) GS:ffff880002600000(0000) knlGS:0000000000000000
     687<Sep/20 08:13 pm>[1308356.031399] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
     688<Sep/20 08:13 pm>[1308356.031400] CR2: 0000000020ddbb00 CR3: 000000007d81e000 CR4: 00000000000006f0
     689<Sep/20 08:13 pm>[1308356.031402] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     690<Sep/20 08:13 pm>[1308356.031403] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     691
     692<Sep/20 08:13 pm>[1308356.031405] Process xfsdatad/0 (pid: 734, threadinfo ffff880623ae0000, task ffff880623ad9030)
     693<Sep/20 08:13 pm>[1308356.031406] Stack:
     694                 <Sep/20 08:13 pm>[1308356.031406]  ffff880623ae1600 ffff880623ae18c0 ffffffff812c093d ffff880623ae18e0
     695<Sep/20 08:13 pm>[1308356.031408] <0> ffffffff810197e9 0000000000000001 000000000000d200 ffff880623ae1920
     696<Sep/20 08:13 pm>[1308356.031411] <0> ffffffff81019c76 ffffffff81b24210 0000000000000000 ffff880002673c00
     697<Sep/20 08:13 pm>[1308356.031413] Call Trace:
     698<Sep/20 08:13 pm>[1308356.031415]  [<ffffffff812c093d>] __const_udelay+0x3d/0x3f
     699<Sep/20 08:13 pm>[1308356.031417]  [<ffffffff810197e9>] native_safe_apic_wait_icr_idle+0x31/0x48
     700<Sep/20 08:13 pm>[1308356.031420]  [<ffffffff81019c76>] default_send_IPI_mask_sequence_phys+0x49/0xab
     701<Sep/20 08:13 pm>[1308356.031422]  [<ffffffff8101c5d8>] physflat_send_IPI_all+0x14/0x16
     702<Sep/20 08:13 pm>[1308356.031425]  [<ffffffff81019d94>] arch_trigger_all_cpu_backtrace+0x3c/0x5d
     703<Sep/20 08:13 pm>[1308356.031427]  [<ffffffff812c4ea4>] do_raw_spin_lock+0x11d/0x147
     704<Sep/20 08:13 pm>[1308356.031429]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
     705<Sep/20 08:13 pm>[1308356.031431]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76
     706<Sep/20 08:13 pm>[1308356.031433]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
     707<Sep/20 08:13 pm>[1308356.031436]  [<ffffffff81037672>] load_balance+0xbb6/0xf63
     708<Sep/20 08:13 pm>[1308356.031438]  [<ffffffff81030875>] ? double_rq_unlock+0x1f/0x24
     709<Sep/20 08:13 pm>[1308356.031440]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     710<Sep/20 08:13 pm>[1308356.031443]  [<ffffffff8151cc0e>] schedule+0x25a/0x86a
     711<Sep/20 08:13 pm>[1308356.031445]  [<ffffffff810480a2>] ? lock_timer_base+0x26/0x4a
     712<Sep/20 08:13 pm>[1308356.031447]  [<ffffffff8151d559>] schedule_timeout+0x222/0x24c
     713<Sep/20 08:13 pm>[1308356.031449]  [<ffffffff810482a5>] ? process_timeout+0x0/0xb
     714
     715<Sep/20 08:13 pm>[1308356.031452]  [<ffffffff8151d59c>] schedule_timeout_uninterruptible+0x19/0x1b
     716<Sep/20 08:13 pm>[1308356.031454]  [<ffffffff8127d1af>] xfs_end_io+0x7d/0x8c
     717<Sep/20 08:13 pm>[1308356.031456]  [<ffffffff81050bf4>] worker_thread+0x22c/0x332
     718<Sep/20 08:13 pm>[1308356.031458]  [<ffffffff81050b9d>] ? worker_thread+0x1d5/0x332
     719<Sep/20 08:13 pm>[1308356.031460]  [<ffffffff8127d132>] ? xfs_end_io+0x0/0x8c
     720<Sep/20 08:13 pm>[1308356.031462]  [<ffffffff81054b6c>] ? autoremove_wake_function+0x0/0x38
     721<Sep/20 08:13 pm>[1308356.031464]  [<ffffffff810509c8>] ? worker_thread+0x0/0x332
     722<Sep/20 08:13 pm>[1308356.031466]  [<ffffffff810546e2>] kthread+0x7d/0x85
     723<Sep/20 08:13 pm>[1308356.031468]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     724<Sep/20 08:13 pm>[1308356.031471]  [<ffffffff81054665>] ? kthread+0x0/0x85
     725<Sep/20 08:13 pm>[1308356.031472]  [<ffffffff81003790>] ? kernel_thread_helper+0<Sep/20 08:13 pm>
     726                 <Sep/20 08:13 pm>[1308356.031473] Code: fb 48 ff c8 c9 c3 55 48 89 e5 48 83 ec 08 65 8b 34 25 f0 d1 00 00 66 66 90 0f ae e8 0f 31 89 c1 41 89 f1 66 66 90 0f ae e8 0f 31 <89> c6 48 89 f0 48 29 c8 48 39 f8 73 25 f3 90 65 44 8b 04 25 f0
     727<Sep/20 08:13 pm>[1308356.031489] Call Trace:
     728<Sep/20 08:13 pm>[1308356.031490]  [<ffffffff812c093d>] __const_udelay+0x3d/0x3f
     729<Sep/20 08:13 pm>[1308356.031493]  [<ffffffff810197e9>] native_safe_apic_wait_icr_idle+0x31/0x48
     730<Sep/20 08:13 pm>[1308356.031495]  [<ffffffff81019c76>] default_send_IPI_mask_sequence_phys+0x49/0xab
     731<Sep/20 08:13 pm>[1308356.031497]  [<ffffffff8101c5d8>] physflat_send_IPI_all+0x14/0x16
     732<Sep/20 08:13 pm>[1308356.031500]  [<ffffffff81019d94>] arch_trigger_all_cpu_backtrace+0x3c/0x5d
     733<Sep/20 08:13 pm>[1308356.031502]  [<ffffffff812c4ea4>] do_raw_spin_lock+0x11d/0x147
     734<Sep/20 08:13 pm>[1308356.031504]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
     735<Sep/20 08:13 pm>[1308356.031506]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76
     736<Sep/20 08:13 pm>[1308356.031508]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
     737
     738<Sep/20 08:13 pm>[1308356.031510]  [<ffffffff81037672>] load_balance+0xbb6/0xf63
     739<Sep/20 08:13 pm>[1308356.031512]  [<ffffffff81030875>] ? double_rq_unlock+0x1f/0x24
     740<Sep/20 08:13 pm>[1308356.031514]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     741<Sep/20 08:13 pm>[1308356.031517]  [<ffffffff8151cc0e>] schedule+0x25a/0x86a
     742                 <Sep/20 08:13 pm>[1308356.031519]  [<ffffffff810480a2>] ? lock_timer_base+0x26/0x4a
     743<Sep/20 08:13 pm>[1308356.031522]  [<ffffffff8151d559>] schedule_timeout+0x222/0<Sep/20 08:13 pm>
     744                 <Sep/20 08:13 pm>[1308356.031524]  [<ffffffff810482a5>] ? process_timeout+0x0/0xb
     745<Sep/20 08:13 pm>[1308356.031526]  [<ffffffff8151d59c>] schedule_timeout_uninterruptible+0x19/0x1b
     746<Sep/20 08:13 pm>[1308356.031528]  [<ffffffff8127d1af>] xfs_end_io+0x7d/0x8c
     747<Sep/20 08:13 pm>[1308356.031530]  [<ffffffff81050bf4>] worker_thread+0x22c/0x332
     748<Sep/20 08:13 pm>[1308356.031532]  [<ffffffff81050b9d>] ? worker_thread+0x1d5/0x332
     749<Sep/20 08:13 pm>[1308356.031534]  [<ffffffff8127d132>] ? xfs_end_io+0x0/0x8c
     750<Sep/20 08:13 pm>[1308356.031536]  [<ffffffff81054b6c>] ? autoremove_wake_function+0x0/0x38
     751<Sep/20 08:13 pm>[1308356.031538]  [<ffffffff810509c8>] ? worker_thread+0x0/0x332
     752<Sep/20 08:13 pm>[1308356.031541]  [<ffffffff810546e2>] kthread+0x7d/0x85
     753<Sep/20 08:13 pm>[1308356.031543]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     754<Sep/20 08:13 pm>[1308356.031545]  [<ffffffff81054665>] ? kthread+0x0/0x85
     755<Sep/20 08:13 pm>[1308356.031547]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     756<Sep/20 08:13 pm>[1308356.031548] Pid: 734, comm: xfsdatad/0 Tainted: G      D    2.6.34.4 #3
     757<Sep/20 08:13 pm>[1308356.031549] Call Trace:
     758<Sep/20 08:13 pm>[1308356.031550]  <NMI>  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
     759<Sep/20 08:13 pm>[1308356.031554]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
     760
     761<Sep/20 08:13 pm>[1308356.031556]  [<ffffffff81520360>] do_nmi+0xb2/0x259
     762                 <Sep/20 08:13 pm>[1308356.031557]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
     763<Sep/20 08:13 pm>[1308356.031559]  [<ffffffff812c08af>] ? delay_tsc+0x25/0x59
     764<Sep/20 08:13 pm>[1308356.031560]  <<EOE>>  [<ffffffff812c093d>] __const_udelay+0x3d/0x3f
     765<Sep/20 08:13 pm>[1308356.031564]  [<ffffffff810197e9>] native_safe_apic_wait_icr_idle+0x31/0x48
     766<Sep/20 08:13 pm>[1308356.031566] BUG: spinlock lockup on CPU#1, updatedb/1321, <Sep/20 08:13 pm>
     767                 <Sep/20 08:13 pm>[1308356.031568] Pid: 1321, comm: updatedb Tainted: G      D    2.6.34.4 #3
     768<Sep/20 08:13 pm>[1308356.031570]  [<ffffffff81019c76>] default_send_IPI_mask_sequence_phys+0x49/0xab
     769<Sep/20 08:13 pm>[1308356.031571] Call Trace:
     770<Sep/20 08:13 pm>[1308356.031574]  [<ffffffff8101c5d8>] physflat_send_IPI_all+0x14/0x16
     771<Sep/20 08:13 pm>[1308356.031576]  [<ffffffff812c4e9f>] do_raw_spin_lock+0x118/0x147
     772<Sep/20 08:13 pm>[1308356.031578]  [<ffffffff81019d94>] arch_trigger_all_cpu_backtrace+0x3c/0x5d
     773<Sep/20 08:13 pm>[1308356.031581]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
     774<Sep/20 08:13 pm>[1308356.031583]  [<ffffffff812c4ea4>] do_raw_spin_lock+0x11d/0x147
     775<Sep/20 08:13 pm>[1308356.031585]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76
     776<Sep/20 08:13 pm>[1308356.031588]  [<ffffffff8151f0dc>] _raw_spin_lock_nested+0x32/0x36
     777<Sep/20 08:13 pm>[1308356.031590]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
     778<Sep/20 08:13 pm>[1308356.031592]  [<ffffffff81035934>] ? double_rq_lock+0x4a/0x76
     779<Sep/20 08:13 pm>[1308356.031594]  [<ffffffff81037672>] load_balance+0xbb6/0xf63
     780<Sep/20 08:13 pm>[1308356.031597]  [<ffffffff81035934>] double_rq_lock+0x4a/0x76
     781<Sep/20 08:13 pm>[1308356.031598]  [<ffffffff810b3cfb>] ? get_dirty_limits+0x22/0x258
     782<Sep/20 08:13 pm>[1308356.031601]  [<ffffffff81037672>] load_balance+0xbb6/0xf63
     783
     784<Sep/20 08:13 pm>[1308356.031603]  [<ffffffff8151cc0e>] schedule+0x25a/0x86a
     785<Sep/20 08:13 pm>[1308356.031605]  [<ffffffff81030875>] ? double_rq_unlock+0x1f/0x24
     786<Sep/20 08:13 pm>[1308356.031607]  [<ffffffff810480a2>] ? lock_timer_base+0x26/0x4a
     787<Sep/20 08:13 pm>[1308356.031609]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
     788<Sep/20 08:13 pm>[1308356.031612]  [<ffffffff8151d559>] schedule_timeout+0x222/0x24c
     789<Sep/20 08:13 pm>[1308356.031614]  [<ffffffff8151cc0e>] schedule+0x25a/0x86a
     790<Sep/20 08:13 pm>[1308356.031616]  [<ffffffff810482a5>] ? process_timeout+0x0/0xb
     791<Sep/20 08:13 pm>[1308356.031618]  [<ffffffff810480a2>] ? lock_timer_base+0x26/0x4a
     792<Sep/20 08:13 pm>[1308356.031621]  [<ffffffff8151c6c5>] io_schedule_timeout+0x45/0x5e
     793<Sep/20 08:13 pm>[1308356.031623]  [<ffffffff8151d559>] schedule_timeout+0x222/0x24c
     794<Sep/20 08:13 pm>[1308356.031625]  [<ffffffff810c0abe>] congestion_wait+0x6b/0x87
     795<Sep/20 08:13 pm>[1308356.031627]  [<ffffffff810482a5>] ? process_timeout+0x0/0xb
     796<Sep/20 08:13 pm>[1308356.031629]  [<ffffffff81054b6c>] ? autoremove_wake_function+0x0/0x38
     797<Sep/20 08:13 pm>[1308356.031632]  [<ffffffff8151d59c>] schedule_timeout_uninterruptible+0x19/0x1b
     798<Sep/20 08:13 pm>[1308356.031634]  [<ffffffff810b3279>] __alloc_pages_nodemask+0x65a/0x671
     799<Sep/20 08:13 pm>[1308356.031636]  [<ffffffff8127d1af>] xfs_end_io+0x7d/0x8c
     800<Sep/20 08:13 pm>[1308356.031639]  [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
     801<Sep/20 08:13 pm>[1308356.031641]  [<ffffffff81050bf4>] worker_thread+0x22c/0x332
     802<Sep/20 08:13 pm>[1308356.031643]  [<ffffffff810acb60>] __page_cache_alloc+0x59/0x5e
     803<Sep/20 08:13 pm>[1308356.031645]  [<ffffffff81050b9d>] ? worker_thread+0x1d5/0x332
     804<Sep/20 08:13 pm>[1308356.031647]  [<ffffffff810ad2d5>] find_or_create_page+0x41/0x85
     805<Sep/20 08:13 pm>[1308356.031650]  [<ffffffff8127d132>] ? xfs_end_io+0x0/0x8c
     806
     807<Sep/20 08:13 pm>[1308356.031652]  [<ffffffff81280160>] _xfs_buf_lookup_pages+0x111/0x2d6
     808<Sep/20 08:13 pm>[1308356.031654]  [<ffffffff81054b6c>] ? autoremove_wake_function+0x0/0x38
     809<Sep/20 08:13 pm>[1308356.031656]  [<ffffffff812811df>] xfs_buf_get+0x73/0x17e
     810<Sep/20 08:13 pm>[1308356.031659]  [<ffffffff810509c8>] ? worker_thread+0x0/0x332
     811<Sep/20 08:13 pm>[1308356.031661]  [<ffffffff81281304>] xfs_buf_read+0x1a/0xc6
     812<Sep/20 08:13 pm>[1308356.031663]  [<ffffffff810546e2>] kthread+0x7d/0x85
     813<Sep/20 08:13 pm>[1308356.031665]  [<ffffffff8127769d>] xfs_trans_read_buf+0x4e/0x396
     814<Sep/20 08:13 pm>[1308356.031667]  [<ffffffff81264f29>] xfs_imap_to_bp+0x44/0xfd
     815<Sep/20 08:13 pm>[1308356.031669]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
     816<Sep/20 08:13 pm>[1308356.031671]  [<ffffffff812661f1>] xfs_iread+0x59/0x188
     817<Sep/20 08:13 pm>[1308356.031673]  [<ffffffff81261d6e>] xfs_iget+0x380/0x57e
     818<Sep/20 08:13 pm>[1308356.031675]  [<ffffffff81054665>] ? kthread+0x0/0x85
     819<Sep/20 08:13 pm>[1308356.031678]  [<ffffffff8127acc4>] xfs_lookup+0xac/0xdd
     820<Sep/20 08:13 pm>[1308356.031679]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
     821<Sep/20 08:13 pm>[1308356.031682] NMI backtrace for cpu 3
     822<Sep/20 08:13 pm>[1308356.031683] Uhhuh. NMI received for unknown reason 00 on CPU 1.
     823
     824}}}