IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Initial Version and Version 1 of Ippc28-crash-20140108


Ignore:
Timestamp:
Jan 10, 2015, 10:24:37 AM (12 years ago)
Author:
Mark Huber
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • Ippc28-crash-20140108

    v1 v1  
     1from console
     2{{{
     3<Jan/08 06:37 pm>ippc28 login: [446397.765597] general protection fault: 0000 [#1] SMP
     4<Jan/08 06:37 pm>[446397.768698] last sysfs file: /sys/class/i2c-adapter/i2c-0/0-002f/temp6_alarm
     5<Jan/08 06:37 pm>[446397.768698] CPU 0
     6<Jan/08 06:37 pm>[446397.768698] Modules linked in: w83793 hwmon_vid autofs4 e100 i2c_i801 i2c_core iTCO_wdt e1000e tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
     7<Jan/08 06:37 pm>[446397.768698] Pid: 1671, comm: rpciod/0 Not tainted 2.6.28-rc7-00105-gfeaf384 #4
     8<Jan/08 06:37 pm>[446397.768698] RIP: 0010:[<ffffffff802488f9>]  [<ffffffff802488f9>] run_workqueue+0x73/0x1ed
     9<Jan/08 06:37 pm>[446397.768698] RSP: 0000:ffff88063ac11e80  EFLAGS: 00010087
     10<Jan/08 06:37 pm>[446397.768698] RAX: ffff88063b3a6130 RBX: ffff880002916580 RCX: 0000000000000000
     11<Jan/08 06:37 pm>[446397.768698] RDX: 74878f32c3ab2203 RSI: ffff8800029165b0 RDI: ffff88063b3a6100
     12<Jan/08 06:37 pm>[446397.768698] RBP: ffff88063b3a6100 R08: 0000000000000001 R09: ffff880611cab1c0
     13<Jan/08 06:37 pm>[446397.768698] R10: 0000000000000000 R11: ffffffff8050a1ce R12: ffff880002916578
     14<Jan/08 06:37 pm>[446397.768698] R13: 39aea0571dafa23d R14: ffffffff808b85a0 R15: 0000000000000000
     15
     16<Jan/08 06:37 pm>[446397.768698] FS:  0000000000000000(0000) GS:ffffffff80872d80(0000) knlGS:0000000000000000
     17<Jan/08 06:37 pm>[446397.768698] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
     18<Jan/08 06:37 pm>[446397.768698] CR2: 000000000329db48 CR3: 000000042037e000 CR4: 00000000000006e0
     19<Jan/08 06:37 pm>[446397.768698] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     20<Jan/08 06:37 pm>[446397.768698] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     21<Jan/08 06:37 pm>[446397.768698] Process rpciod/0 (pid: 1671, threadinfo ffff88063ac10000, task ffff88063b389e60)
     22<Jan/08 06:37 pm>[446397.768698] Stack:
     23<Jan/08 06:37 pm>[446397.768698]  ffffffff8024891d 0000000000000046 437f5913748cf8ea c49a8803afac4212
     24<Jan/08 06:37 pm>[446397.768698]  8ce7677766760a3b ffff88063b3a6100 ffff88063cc89bf0 ffff88063b3a6100
     25<Jan/08 06:37 pm>[446397.768698]  ffff88063b3a6140 ffff88063cc89bf0 ffffffff808b5d60 ffffffff80249511
     26<Jan/08 06:37 pm>[446397.768698] Call Trace:
     27<Jan/08 06:37 pm>[446397.768698]  [<ffffffff8024891d>] ? run_workqueue+0x97/0x1ed
     28<Jan/08 06:37 pm>[446397.768698]  [<ffffffff80249511>] ? worker_thread+0xd8/0xe3
     29<Jan/08 06:37 pm>[446397.768698]  [<ffffffff8024bfda>] ? autoremove_wake_function+0x0/0x2e
     30<Jan/08 06:37 pm>[446397.768698]  [<ffffffff80249439>] ? worker_thread+0x0/0xe3
     31<Jan/08 06:37 pm>[446397.768698]  [<ffffffff8024becc>] ? kthread+0x47/0x73
     32<Jan/08 06:37 pm>[446397.768698]  [<ffffffff80234852>] ? schedule_tail+0x27/0x5f
     33<Jan/08 06:37 pm>[446397.768698]  [<ffffffff8020cd19>] ? child_rip+0xa/0x11
     34<Jan/08 06:37 pm>[446397.768698]  [<ffffffff8024be85>] ? kthread+0x0/0x73
     35<Jan/08 06:37 pm>[446397.768698]  [<ffffffff8020cd0f>] ? child_rip+0x0/0x11
     36<Jan/08 06:37 pm>[446397.768698] Code: 4c 8d 63 f8 48 8d 7c 24 10 48 8d 73 18 fc 4d 8b 6c 24 18 b9 06 00 00 00 f3 a5 4c 89 a5 80 00 00 00 48 89 ef 48 8b 43 08 48 8b 13 <48> 89 42 08 48 89 10 48 89 5b 08 48 89 1b e8 d1 ed 36 00 48 8b
     37<Jan/08 06:37 pm>[446397.768698] RIP  [<ffffffff802488f9>] run_workqueue+0x73/0x1ed
     38
     39<Jan/08 06:37 pm>[446397.768698]  RSP <ffff88063ac11e80>
     40<Jan/08 06:37 pm>[446397.768698] ---[ end trace 1ad05d85a7c9d908 ]---
     41<Jan/08 06:37 pm>[446410.602188] nfs: server ipp093 not responding, still trying
     42<Jan/08 06:38 pm>[446410.625508] BUG: spinlock lockup on CPU#5, umount.nfs/16799, ffff88063b3a6100
     43<Jan/08 06:38 pm>[446410.625508] Pid: 16799, comm: umount.nfs Tainted: G      D    2.6.28-rc7-00105-gfeaf384 #4
     44<Jan/08 06:38 pm>[446410.625508] Call Trace:
     45<Jan/08 06:38 pm>[446410.625508]  [<ffffffff803a103f>] _raw_spin_lock+0xfb/0x122
     46<Jan/08 06:38 pm>[446410.625508]  [<ffffffff8024912b>] __cancel_work_timer+0x120/0x1a3
     47<Jan/08 06:38 pm>[446410.625508]  [<ffffffff802490a1>] __cancel_work_timer+0x96/0x1a3
     48<Jan/08 06:38 pm>[446410.625508]  [<ffffffff8058a0f0>] xs_destroy+0x31/0x79
     49<Jan/08 06:38 pm>[446410.625508]  [<ffffffff80588458>] xprt_destroy+0x0/0x7c
     50<Jan/08 06:38 pm>[446410.625508]  [<ffffffff803995bf>] kref_put+0x41/0x4c
     51<Jan/08 06:38 pm>[446410.625508]  [<ffffffff80587073>] rpc_free_client+0x0/0xd1
     52<Jan/08 06:38 pm>[446410.625508]  [<ffffffff80587136>] rpc_free_client+0xc3/0xd1
     53<Jan/08 06:38 pm>[446410.625508]  [<ffffffff803995bf>] kref_put+0x41/0x4c
     54<Jan/08 06:38 pm>[446410.625508]  [<ffffffff80587144>] rpc_free_auth+0x0/0x40
     55<Jan/08 06:38 pm>[446410.625508]  [<ffffffff803995bf>] kref_put+0x41/0x4c
     56<Jan/08 06:38 pm>[446410.625508]  [<ffffffff8058724e>] rpc_shutdown_client+0xca/0xd5
     57<Jan/08 06:38 pm>[446410.625508]  [<ffffffff802423c7>] try_to_del_timer_sync+0x51/0x5a
     58<Jan/08 06:38 pm>[446410.625508]  [<ffffffff80328526>] nfs_free_client+0x79/0xc8
     59<Jan/08 06:38 pm>[446410.625508]  [<ffffffff803287a9>] nfs_free_server+0xb6/0xf4
     60<Jan/08 06:38 pm>[446410.625508]  [<ffffffff802ae0cf>] deactivate_super+0x65/0x7e
     61
     62<Jan/08 06:38 pm>[446410.625508]  [<ffffffff802c00b4>] sys_umount+0x2df/0x33e
     63<Jan/08 06:38 pm>[446410.625508]  [<ffffffff8020bd2b>] system_call_fastpath+0x16/0x1b
     64<Jan/08 06:38 pm>[446449.998489] BUG: spinlock lockup on CPU#0, swapper/0, ffff88063b3a6100
     65<Jan/08 06:38 pm>[446449.998489] Pid: 0, comm: swapper Tainted: G      D    2.6.28-rc7-00105-gfeaf384 #4
     66<Jan/08 06:38 pm>[446449.998489] Call Trace:
     67<Jan/08 06:38 pm>[446449.998489]  <IRQ>  [<ffffffff803a103f>] _raw_spin_lock+0xfb/0x122
     68<Jan/08 06:38 pm>[446449.998489]  [<ffffffff805b7758>] _spin_lock_irqsave+0x37/0x3f
     69<Jan/08 06:38 pm>[446449.998489]  [<ffffffff802492ab>] __queue_work+0x10/0x32
     70<Jan/08 06:38 pm>[446449.998489]  [<ffffffff802492ab>] __queue_work+0x10/0x32
     71<Jan/08 06:38 pm>[446449.998489]  [<ffffffff8024932c>] queue_work_on+0x37/0x40
     72<Jan/08 06:38 pm>[446449.998489]  [<ffffffff8058ce2d>] rpc_wake_up_task_queue_locked+0x261/0x2a6
     73<Jan/08 06:38 pm>[446449.998489]  [<ffffffff8058cf8e>] rpc_wake_up_queued_task+0x1d/0x2a
     74<Jan/08 06:38 pm>[446449.998489]  [<ffffffff8058bb46>] xs_udp_data_ready+0x1da/0x205
     75<Jan/08 06:38 pm>[446449.998489]  [<ffffffff8050dad6>] sock_queue_rcv_skb+0xd6/0xe2
     76<Jan/08 06:38 pm>[446449.998489]  [<ffffffff8054a1c9>] __udp_queue_rcv_skb+0x14/0xa5
     77<Jan/08 06:38 pm>[446449.998489]  [<ffffffff8054ae44>] udp_queue_rcv_skb+0x141/0x1c6
     78<Jan/08 06:38 pm>[446449.998489]  [<ffffffff8054b3fa>] __udp4_lib_rcv+0x531/0x799
     79<Jan/08 06:38 pm>[446449.998489]  [<ffffffff8020c2f6>] ret_from_intr+0x0/0xa
     80<Jan/08 06:38 pm>[446449.998489]  [<ffffffff8052e344>] ip_local_deliver+0xcb/0x164
     81<Jan/08 06:38 pm>[446449.998489]  [<ffffffff8052e2d4>] ip_local_deliver+0x5b/0x164
     82<Jan/08 06:38 pm>[446449.998489]  [<ffffffff8052e8d7>] ip_rcv+0x4fa/0x54f
     83<Jan/08 06:38 pm>[446449.998489]  [<ffffffff805164df>] netif_receive_skb+0x252/0x29b
     84
     85<Jan/08 06:38 pm>[446449.998489]  [<ffffffff80516397>] netif_receive_skb+0x10a/0x29b
     86<Jan/08 06:38 pm>[446449.998489]  [<ffffffffa016b73a>] e1000_receive_skb+0x3e/0x54 [e1000e]
     87<Jan/08 06:38 pm>[446449.998489]  [<ffffffffa016e65b>] e1000_clean_rx_irq+0x234/0x2d6 [e1000e]
     88<Jan/08 06:38 pm>[446449.998489]  [<ffffffffa016de03>] e1000_clean+0x24a/0x262 [e1000e]
     89<Jan/08 06:38 pm>[446449.998489]  [<ffffffff8051870b>] net_rx_action+0xc5/0x1fb
     90<Jan/08 06:38 pm>[446449.998489]  [<ffffffff8051869d>] net_rx_action+0x57/0x1fb
     91<Jan/08 06:38 pm>[446449.998489]  [<ffffffff8023e144>] __do_softirq+0x7a/0x13d
     92<Jan/08 06:38 pm>[446449.998489]  [<ffffffff8020d07c>] call_softirq+0x1c/0x28
     93<Jan/08 06:38 pm>[446449.998489]  [<ffffffff8020e028>] do_softirq+0x2c/0x68
     94<Jan/08 06:38 pm>[446449.998489]  [<ffffffff8023e084>] irq_exit+0x3f/0x85
     95<Jan/08 06:38 pm>[446449.998489]  [<ffffffff8020e1ae>] do_IRQ+0x14a/0x16c
     96<Jan/08 06:38 pm>[446449.998489]  [<ffffffff8020c2f6>] ret_from_intr+0x0/0xa
     97<Jan/08 06:38 pm>[446449.998489]  <EOI>  [<ffffffff803d4f96>] acpi_idle_enter_bm+0x2b0/0x31a
     98<Jan/08 06:38 pm>[446449.998489]  [<ffffffff803d4f8c>] acpi_idle_enter_bm+0x2a6/0x31a
     99<Jan/08 06:38 pm>[446449.998489]  [<ffffffff804f9763>] cpuidle_idle_call+0x7f/0xbc
     100<Jan/08 06:38 pm>[446449.998489]  [<ffffffff8020ac41>] cpu_idle+0x4a/0x6d
     101<Jan/08 06:38 pm>[446470.442790] BUG: soft lockup - CPU#11 stuck for 61s! [events/11:50]
     102<Jan/08 06:38 pm>[446470.442790] Modules linked in: w83793 hwmon_vid autofs4 e100 i2c_i801 i2c_core iTCO_wdt e1000e tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
     103<Jan/08 06:38 pm>[446470.442790] CPU 11:
     104<Jan/08 06:38 pm>[446470.442790] Modules linked in: w83793 hwmon_vid autofs4 e100 i2c_i801 i2c_core iTCO_wdt e1000e tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
     105<Jan/08 06:38 pm>[446470.442790] Pid: 50, comm: events/11 Tainted: G      D    2.6.28-rc7-00105-gfeaf384 #4
     106<Jan/08 06:38 pm>[446470.442790] RIP: 0010:[<ffffffff8025bd5d>]  [<ffffffff8025bd5d>] smp_call_function_mask+0x163/0x1d7
     107
     108<Jan/08 06:38 pm>[446470.442790] RSP: 0018:ffff88063ced9d30  EFLAGS: 00000202
     109<Jan/08 06:38 pm>[446470.442790] RAX: 0000000000000020 RBX: 000000000000000b RCX: 0000000000000020
     110<Jan/08 06:38 pm>[446470.442790] RDX: ffffffff8081a300 RSI: 0000000000000020 RDI: 0000000000000000
     111<Jan/08 06:38 pm>[446470.442790] RBP: 000000000000000b R08: 0000000000000020 R09: ffff88063ced9cf0
     112<Jan/08 06:38 pm>[446470.442790] R10: 0000000000000000 R11: ffffffff802230fe R12: 0000000000000020
     113<Jan/08 06:38 pm>[446470.442790] R13: 0000000000000020 R14: 0000000000000020 R15: ffff88063ced9cf0
     114<Jan/08 06:38 pm>[446470.442790] FS:  0000000000000000(0000) GS:ffff880c3cc4d900(0000) knlGS:0000000000000000
     115<Jan/08 06:38 pm>[446470.442790] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
     116<Jan/08 06:38 pm>[446470.442790] CR2: 000000001eea7000 CR3: 000000042037e000 CR4: 00000000000006e0
     117<Jan/08 06:38 pm>[446470.442790] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     118<Jan/08 06:38 pm>[446470.442790] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     119<Jan/08 06:38 pm>[446470.442790] Call Trace:
     120<Jan/08 06:38 pm>[446470.442790]  [<ffffffff80257376>] ? __lock_acquire+0x788/0x7f3
     121<Jan/08 06:38 pm>[446470.442790]  [<ffffffff8022ecb5>] ? update_curr+0x76/0xc9
     122<Jan/08 06:38 pm>[446470.442790]  [<ffffffff80218079>] ? mcheck_check_cpu+0x0/0x2f
     123<Jan/08 06:38 pm>[446470.442790]  [<ffffffff8023dcbb>] ? on_each_cpu+0x10/0x22
     124                 <Jan/08 06:38 pm>[446470.442790]  [<ffffffff80217997>] ? mcheck_timer+0x0/0x76
     125<Jan/08 06:38 pm>[446470.442790]  [<ffffffff802179ab>] ? mcheck_timer+0x14/0x76
     126<Jan/08 06:38 pm>[446470.442790]  [<ffffffff80248973>] ? run_workqueue+0xed/0x1ed
     127<Jan/08 06:38 pm>[446470.442790]  [<ffffffff8024891d>] ? run_workqueue+0x97/0x1ed
     128<Jan/08 06:38 pm>[446470.442790]  [<ffffffff80249511>] ? worker_thread+0xd8/0xe3
     129<Jan/08 06:38 pm>[446470.442790]  [<ffffffff8024bfda>] ? autoremove_wake_function+0x0/0x2e
     130
     131<Jan/08 06:38 pm>[446470.442790]  [<ffffffff80249439>] ? worker_thread+0x0/0xe3
     132<Jan/08 06:38 pm>[446470.442790]  [<ffffffff8024becc>] ? kthread+0x47/0x73
     133<Jan/08 06:38 pm>[446470.442790]  [<ffffffff80234852>] ? schedule_tail+0x27/0x5f
     134<Jan/08 06:38 pm>[446470.442790]  [<ffffffff8020cd19>] ? child_rip+0xa/0x11
     135<Jan/08 06:38 pm>[446470.442790]  [<ffffffff8024be85>] ? kthread+0x0/0x73
     136<Jan/08 06:38 pm>[446470.442790]  [<ffffffff8020cd0f>] ? child_rip+0x0/0x11
     137<Jan/08 06:38 pm>[446475.710229] BUG: spinlock lockup on CPU#2, pdflush/9164, ffff8806a5e05610
     138<Jan/08 06:38 pm>[446475.717530] Pid: 9164, comm: pdflush Tainted: G      D    2.6.28-rc7-00105-gfeaf384 #4
     139<Jan/08 06:38 pm>[446475.726249] Call Trace:
     140<Jan/08 06:38 pm>[446475.726253]  [<ffffffff803a103f>] _raw_spin_lock+0xfb/0x122
     141<Jan/08 06:38 pm>[446475.726256]  [<ffffffff805895ad>] xprt_prepare_transmit+0x40/0x8c
     142<Jan/08 06:38 pm>[446475.726258]  [<ffffffff805878cc>] call_transmit+0x49/0x241
     143<Jan/08 06:38 pm>[446475.726260]  [<ffffffff8058d0bf>] __rpc_execute+0x73/0x228
     144<Jan/08 06:38 pm>[446475.726262]  [<ffffffff80586f15>] rpc_run_task+0x4f/0x56
     145<Jan/08 06:38 pm>[446475.726266]  [<ffffffff80338685>] nfs_write_rpcsetup+0x200/0x220
     146<Jan/08 06:38 pm>[446475.726268]  [<ffffffff803353d8>] nfs_pageio_doio+0x2a/0x51
     147<Jan/08 06:38 pm>[446475.726271]  [<ffffffff80338bf6>] nfs_writepages+0x75/0x97
     148<Jan/08 06:38 pm>[446475.726273]  [<ffffffff80339b7e>] nfs_flush_one+0x0/0xc8
     149<Jan/08 06:38 pm>[446475.726276]  [<ffffffff802828e5>] do_writepages+0x23/0x30
     150<Jan/08 06:38 pm>[446475.726279]  [<ffffffff802c475c>] __writeback_single_inode+<Jan/08 06:38 pm>
     151                 <Jan/08 06:38 pm>[446475.726282]  [<ffffffff802c4c81>] generic_sync_sb_inodes+0x1c1/0x2b4
     152<Jan/08 06:38 pm>[446475.726284]  [<ffffffff802c4f45>] writeback_inodes+0x5c/0xa8
     153
     154<Jan/08 06:38 pm>[446475.726287]  [<ffffffff802831be>] wb_kupdate+0x9e/0x113
     155<Jan/08 06:38 pm>[446475.726289]  [<ffffffff80283582>] pdflush+0x0/0x211
     156                 <Jan/08 06:38 pm>[446475.726291]  [<ffffffff802836e2>] pdflush+0x160/0x211
     157<Jan/08 06:38 pm>[446475.726293]  [<ffffffff80283120>] wb_kupdate+0x0/0x113
     158<Jan/08 06:38 pm>[446475.726295]  [<ffffffff8024becc>] kthread+0x47/0x73
     159                 <Jan/08 06:38 pm>[446475.726299]  [<ffffffff80234852>] schedule_tail+0x27/0x5f
     160<Jan/08 06:38 pm>[446475.726301]  [<ffffffff8020cd19>] child_rip+0xa/0x11
     161<Jan/08 06:38 pm>[446475.726305]  [<ffffffff80223055>] physflat_send_IPI_mask+0x0/0x99
     162<Jan/08 06:38 pm>[446475.726307]  [<ffffffff8024be85>] kthread+0x0/0x73
     163<Jan/08 06:38 pm>[446475.726309]  [<ffffffff8020cd0f>] child_rip+0x0/0x11
     164<Jan/08 06:39 pm>[446535.939450] BUG: soft lockup - CPU#11 stuck for 61s! [event<Jan/08 06:39 pm>
     165                 <Jan/08 06:39 pm>[446535.939450] Modules linked in: w83793 hwmon_vid autofs4 e100 i2c_i801 i2c_core iTCO_wdt e1000e tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
     166<Jan/08 06:39 pm>[446535.939450] CPU 11:
     167<Jan/08 06:39 pm>[446535.939450] Modules linked in: w83793 hwmon_vid autofs4 e100 i2c_i801 i2c_core iTCO_wdt e1000e tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
     168<Jan/08 06:39 pm>[446535.939450] Pid: 50, comm: events/11 Tainted: G      D    2.6.28-rc7-00105-gfeaf384 #4
     169<Jan/08 06:39 pm>[446535.939450] RIP: 0010:[<ffffffff8025bd5d>]  [<ffffffff8025bd5d>] smp_call_function_mask+0x163/0x1d7
     170<Jan/08 06:39 pm>[446535.939450] RSP: 0018:ffff88063ced9d30  EFLAGS: 00000202
     171<Jan/08 06:39 pm>[446535.939450] RAX: 0000000000000020 RBX: 000000000000000b RCX: 0000000000000020
     172<Jan/08 06:39 pm>[446535.939450] RDX: ffffffff8081a300 RSI: 0000000000000020 RDI: 0000000000000000
     173<Jan/08 06:39 pm>[446535.939450] RBP: 000000000000000b R08: 0000000000000020 R09: ffff88063ced9cf0
     174<Jan/08 06:39 pm>[446535.939450] R10: 0000000000000000 R11: ffffffff802230fe R12: 0000000000000020
     175<Jan/08 06:39 pm>[446535.939450] R13: 0000000000000020 R14: 0000000000000020 R15: ffff88063ced9cf0
     176
     177<Jan/08 06:39 pm>[446535.939450] FS:  0000000000000000(0000) GS:ffff880c3cc4d900(0000) knlGS:0000000000000000
     178<Jan/08 06:39 pm>[446535.939450] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
     179<Jan/08 06:39 pm>[446535.939450] CR2: 000000001eea7000 CR3: 000000042037e000 CR4: 00000000000006e0
     180<Jan/08 06:39 pm>[446535.939450] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     181<Jan/08 06:39 pm>[446535.939450] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     182<Jan/08 06:39 pm>[446535.939450] Call Trace:
     183<Jan/08 06:39 pm>[446535.939450]  [<ffffffff80257376>] ? __lock_acquire+0x788/0x7f3
     184<Jan/08 06:39 pm>[446535.939450]  [<ffffffff8022ecb5>] ? update_curr+0x76/0xc9
     185<Jan/08 06:39 pm>[446535.939450]  [<ffffffff80218079>] ? mcheck_check_cpu+0x0/0x2f
     186<Jan/08 06:39 pm>[446535.939450]  [<ffffffff8023dcbb>] ? on_each_cpu+0x10/0x22
     187<Jan/08 06:39 pm>[446535.939450]  [<ffffffff80217997>] ? mcheck_timer+0x0/0x76
     188<Jan/08 06:39 pm>[446535.939450]  [<ffffffff802179ab>] ? mcheck_timer+0x14/0x76
     189<Jan/08 06:39 pm>[446535.939450]  [<ffffffff80248973>] ? run_workqueue+0xed/0x1ed
     190<Jan/08 06:39 pm>[446535.939450]  [<ffffffff8024891d>] ? run_workqueue+0x97/0x1ed
     191<Jan/08 06:39 pm>[446535.939450]  [<ffffffff80249511>] ? worker_thread+0xd8/0xe3
     192<Jan/08 06:39 pm>[446535.939450]  [<ffffffff8024bfda>] ? autoremove_wake_function+0x0/0x2e
     193<Jan/08 06:39 pm>[446535.939450]  [<ffffffff80249439>] ? worker_thread+0x0/0xe3
     194<Jan/08 06:39 pm>[446535.939450]  [<ffffffff8024becc>] ? kthread+0x47/0x73
     195<Jan/08 06:39 pm>[446535.939450]  [<ffffffff80234852>] ? schedule_tail+0x27/0x5f
     196<Jan/08 06:39 pm>[446535.939450]  [<ffffffff8020cd19>] ? child_rip+0xa/0x11
     197<Jan/08 06:39 pm>[446535.939450]  [<ffffffff8024be85>] ? kthread+0x0/0x73
     198<Jan/08 06:39 pm>[446535.939450]  [<ffffffff8020cd0f>] ? child_rip+0x0/0x11
     199
     200<Jan/08 06:39 pm>[446540.235491] BUG: soft lockup - CPU#2 stuck for 61s! [pdflush:9164]
     201<Jan/08 06:39 pm>[446540.235491] Modules linked in: w83793 hwmon_vid autofs4 e100 i2c_i801 i2c_core iTCO_wdt e1000e tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
     202<Jan/08 06:39 pm>[446540.235491] CPU 2:
     203<Jan/08 06:39 pm>[446540.235491] Modules linked in: w83793 hwmon_vid autofs4 e100 i2c_i801 i2c_core iTCO_wdt e1000e tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
     204<Jan/08 06:39 pm>[446540.235491] Pid: 9164, comm: pdflush Tainted: G      D    2.6.28-rc7-00105-gfeaf384 #4
     205<Jan/08 06:39 pm>[446540.235491] RIP: 0010:[<ffffffff8021205f>]  [<ffffffff8021205f>] native_read_tsc+0x2/0xc
     206<Jan/08 06:39 pm>[446540.235491] RSP: 0018:ffff880373fe9b78  EFLAGS: 00000212
     207<Jan/08 06:39 pm>[446540.235491] RAX: 00000000a108900f RBX: 000000008a5d4ab0 RCX: ffffffff80932380
     208<Jan/08 06:39 pm>[446540.235491] RDX: 0000000000043b40 RSI: 0000000001062560 RDI: 0000000000000001
     209<Jan/08 06:39 pm>[446540.235491] RBP: 000000008a5d4ab0 R08: ffffffff8024f65a R09: 0000000000000000
     210<Jan/08 06:39 pm>[446540.235491] R10: 0000000000000000 R11: ffffffff8039d94e R12: 00000000c77d3edd
     211<Jan/08 06:39 pm>[446540.235491] R13: 00000000c77d3edd R14: ffffffff80794266 R15: ffffffff80963300
     212<Jan/08 06:39 pm>[446540.235491] FS:  0000000000000000(0000) GS:ffff88063cc5b480(0000) knlGS:0000000000000000
     213<Jan/08 06:39 pm>[446540.235491] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
     214<Jan/08 06:39 pm>[446540.235491] CR2: 0000000041d7afe0 CR3: 0000000000201000 CR4: 00000000000006e0
     215<Jan/08 06:39 pm>[446540.235491] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     216<Jan/08 06:39 pm>[446540.235491] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     217<Jan/08 06:39 pm>[446540.235491] Call Trace:
     218<Jan/08 06:39 pm>[446540.235491]  [<ffffffff8039d963>] ? delay_tsc+0x15/0x42
     219                 <Jan/08 06:39 pm>[446540.235491]  [<ffffffff803a0ffe>] ? _raw_spin_lock+0xba/0x122
     220<Jan/08 06:39 pm>[446540.235491]  [<ffffffff805895ad>] ? xprt_prepare_transmit+0x40/0x8c
     221<Jan/08 06:39 pm>[446540.235491]  [<ffffffff805878cc>] ? call_transmit+0x49/0x241
     222
     223<Jan/08 06:39 pm>[446540.235491]  [<ffffffff8058d0bf>] ? __rpc_execute+0x73/0x228
     224<Jan/08 06:39 pm>[446540.235491]  [<ffffffff80586f15>] ? rpc_run_task+0x4f/0x56
     225<Jan/08 06:39 pm>[446540.235491]  [<ffffffff80338685>] ? nfs_write_rpcsetup+0x200/0x220
     226<Jan/08 06:39 pm>[446540.235491]  [<ffffffff803353d8>] ? nfs_pageio_doio+0x2a/0x51
     227<Jan/08 06:39 pm>[446540.235491]  [<ffffffff80338bf6>] ? nfs_writepages+0x75/0x97
     228<Jan/08 06:39 pm>[446540.235491]  [<ffffffff80339b7e>] ? nfs_flush_one+0x0/0xc8
     229                 <Jan/08 06:39 pm>[446540.235491]  [<ffffffff802828e5>] ? do_writepages+0x23/0x30
     230<Jan/08 06:39 pm>[446540.235491]  [<ffffffff802c475c>] ? __writeback_single_inode+0x150/0x2a9
     231<Jan/08 06:39 pm>[446540.235491]  [<ffffffff802c4c81>] ? generic_sync_sb_inodes+0x1c1/0x2b4
     232<Jan/08 06:39 pm>[446540.235491]  [<ffffffff802c4f45>] ? writeback_inodes+0x5c/0xa8
     233<Jan/08 06:39 pm>[446540.235491]  [<ffffffff802831be>] ? wb_kupdate+0x9e/0x113
     234<Jan/08 06:39 pm>[446540.235491]  [<ffffffff80283582>] ? pdflush+0x0/0x211
     235<Jan/08 06:39 pm>[446540.235491]  [<ffffffff802836e2>] ? pdflush+0x160/0x211
     236<Jan/08 06:39 pm>[446540.235491]  [<ffffffff80283120>] ? wb_kupdate+0x0/0x113
     237<Jan/08 06:39 pm>[446540.235491]  [<ffffffff8024becc>] ? kthread+0x47/0x73
     238<Jan/08 06:39 pm>[446540.235491]  [<ffffffff80234852>] ? schedule_tail+0x27/0x5f
     239<Jan/08 06:39 pm>[446540.235491]  [<ffffffff8020cd19>] ? child_rip+0xa/0x11
     240<Jan/08 06:39 pm>[446540.235491]  [<ffffffff80223055>] ? physflat_send_IPI_mask+0x0/0x99
     241<Jan/08 06:39 pm>[446540.235491]  [<ffffffff8024be85>] ? kthread+0x0/0x73
     242<Jan/08 06:39 pm>[446540.235491]  [<ffffffff8020cd0f>] ? child_rip+0x0/0x11
     243<Jan/08 06:40 pm>[446601.439444] BUG: soft lockup - CPU#11 stuck for 61s! [events/11:50]
     244<Jan/08 06:40 pm>[446601.439444] Modules linked in: w83793 hwmon_vid autofs4 e100 i2c_i801 i2c_core iTCO_wdt e1000e tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
     245
     246<Jan/08 06:40 pm>[446601.439444] CPU 11:
     247<Jan/08 06:40 pm>[446601.439444] Modules linked in: w83793 hwmon_vid autofs4 e100 i2c_i801 i2c_core iTCO_wdt e1000e tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
     248<Jan/08 06:40 pm>[446601.439444] Pid: 50, comm: events/11 Tainted: G      D    2.6.28-rc7-00105-gfeaf384 #4
     249<Jan/08 06:40 pm>[446601.439444] RIP: 0010:[<ffffffff8025bd5d>]  [<ffffffff8025bd5d>] smp_call_function_mask+0x163/0x1d7
     250<Jan/08 06:40 pm>[446601.439444] RSP: 0018:ffff88063ced9d30  EFLAGS: 00000202
     251<Jan/08 06:40 pm>[446601.439444] RAX: 0000000000000020 RBX: 000000000000000b RCX: 0000000000000020
     252<Jan/08 06:40 pm>[446601.439444] RDX: ffffffff8081a300 RSI: 0000000000000020 RDI: 0000000000000000
     253<Jan/08 06:40 pm>[446601.439444] RBP: 000000000000000b R08: 0000000000000020 R09: ffff88063ced9cf0
     254<Jan/08 06:40 pm>[446601.439444] R10: 0000000000000000 R11: ffffffff802230fe R12: 0000000000000020
     255<Jan/08 06:40 pm>[446601.439444] R13: 0000000000000020 R14: 0000000000000020 R15: ffff88063ced9cf0
     256<Jan/08 06:40 pm>[446601.439444] FS:  0000000000000000(0000) GS:ffff880c3cc4d900(0000) knlGS:0000000000000000
     257<Jan/08 06:40 pm>[446601.439444] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
     258<Jan/08 06:40 pm>[446601.439444] CR2: 000000001eea7000 CR3: 000000042037e000 CR4: 00000000000006e0
     259<Jan/08 06:40 pm>[446601.439444] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     260<Jan/08 06:40 pm>[446601.439444] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     261<Jan/08 06:40 pm>[446601.439444] Call Trace:
     262<Jan/08 06:40 pm>[446601.439444]  [<ffffffff80257376>] ? __lock_acquire+0x788/0x<Jan/08 06:40 pm>
     263                 <Jan/08 06:40 pm>[446601.439444]  [<ffffffff8022ecb5>] ? update_curr+0x76/0xc9
     264<Jan/08 06:40 pm>[446601.439444]  [<ffffffff80218079>] ? mcheck_check_cpu+0x0/0x2f
     265<Jan/08 06:40 pm>[446601.439444]  [<ffffffff8023dcbb>] ? on_each_cpu+0x10/0x22
     266<Jan/08 06:40 pm>[446601.439444]  [<ffffffff80217997>] ? mcheck_timer+0x0/0x76
     267<Jan/08 06:40 pm>[446601.439444]  [<ffffffff802179ab>] ? mcheck_timer+0x14/0x76
     268
     269<Jan/08 06:40 pm>[446601.439444]  [<ffffffff80248973>] ? run_workqueue+0xed/0x1ed
     270<Jan/08 06:40 pm>[446601.439444]  [<ffffffff8024891d>] ? run_workqueue+0x97/0x1ed
     271<Jan/08 06:40 pm>[446601.439444]  [<ffffffff80249511>] ? worker_thread+0xd8/0xe3
     272<Jan/08 06:40 pm>[446601.439444]  [<ffffffff8024bfda>] ? autoremove_wake_function+0x0/0x2e
     273<Jan/08 06:40 pm>[446601.439444]  [<ffffffff80249439>] ? worker_thread+0x0/0xe3
     274<Jan/08 06:40 pm>[446601.439444]  [<ffffffff8024becc>] ? kthread+0x47/0x73
     275<Jan/08 06:40 pm>[446601.439444]  [<ffffffff80234852>] ? schedule_tail+0x27/0x5f
     276<Jan/08 06:40 pm>[446601.439444]  [<ffffffff8020cd19>] ? child_rip+0xa/0x11
     277<Jan/08 06:40 pm>[446601.439444]  [<ffffffff8024be85>] ? kthread+0x0/0x73
     278<Jan/08 06:40 pm>[446601.439444]  [<ffffffff8020cd0f>] ? child_rip+0x0/0x11
     279                 <Jan/08 06:40 pm>[446605.732150] BUG: soft lockup - CPU#2 stuck for 61s! [pdflush:9164]
     280<Jan/08 06:40 pm>[446605.732150] Modules linked in: w83793 hwmon_vid autofs4 e100 i2c_i801 i2c_core iTCO_wdt e1000e tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
     281<Jan/08 06:40 pm>[446605.732150] CPU 2:
     282<Jan/08 06:40 pm>[446605.732150] Modules linked in: w83793 hwmon_vid autofs4 e100 i2c_i801 i2c_core iTCO_wdt e1000e tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
     283<Jan/08 06:40 pm>[446605.732150] Pid: 9164, comm: pdflush Tainted: G      D    2.6.28-rc7-00105-gfeaf384 #4
     284<Jan/08 06:40 pm>[446605.732150] RIP: 0010:[<ffffffff8021205f>]  [<ffffffff8021205f>] native_read_tsc+0x2/0xc
     285<Jan/08 06:40 pm>[446605.732150] RSP: 0018:ffff880373fe9b78  EFLAGS: 00000202
     286<Jan/08 06:40 pm>[446605.732150] RAX: 000000004c5eb74f RBX: 0000000090e5eb32 RCX: ffffffff80932380
     287<Jan/08 06:40 pm>[446605.732150] RDX: 0000000000043b69 RSI: 0000000001062560 RDI: 0000000000000001
     288<Jan/08 06:40 pm>[446605.732150] RBP: 0000000090e5eb32 R08: ffffffff8024f65a R09: 0000000000000000
     289<Jan/08 06:40 pm>[446605.732150] R10: 0000000000000000 R11: ffffffff8039d94e R12: 00000000c77d3edd
     290<Jan/08 06:40 pm>[446605.732150] R13: 00000000c77d3edd R14: ffffffff80794266 R15: ffffffff80963300
     291
     292<Jan/08 06:40 pm>[446605.732150] FS:  0000000000000000(0000) GS:ffff88063cc5b480(0000) knlGS:0000000000000000
     293<Jan/08 06:40 pm>[446605.732150] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
     294<Jan/08 06:40 pm>[446605.732150] CR2: 0000000041d7afe0 CR3: 0000000000201000 CR4: 00000000000006e0
     295<Jan/08 06:40 pm>[446605.732150] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     296<Jan/08 06:40 pm>[446605.732150] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     297<Jan/08 06:40 pm>[446605.732150] Call Trace:
     298<Jan/08 06:40 pm>[446605.732150]  [<ffffffff8039d963>] ? delay_tsc+0x15/0x42
     299<Jan/08 06:40 pm>[446605.732150]  [<ffffffff803a0ffe>] ? _raw_spin_lock+0xba/0x122
     300<Jan/08 06:40 pm>[446605.732150]  [<ffffffff805895ad>] ? xprt_prepare_transmit+0x40/0x8c
     301<Jan/08 06:40 pm>[446605.732150]  [<ffffffff805878cc>] ? call_transmit+0x49/0x241
     302<Jan/08 06:40 pm>[446605.732150]  [<ffffffff8058d0bf>] ? __rpc_execute+0x73/0x228
     303<Jan/08 06:40 pm>[446605.732150]  [<ffffffff80586f15>] ? rpc_run_task+0x4f/0x56
     304<Jan/08 06:40 pm>[446605.732150]  [<ffffffff80338685>] ? nfs_write_rpcsetup+0x200/0x220
     305<Jan/08 06:40 pm>[446605.732150]  [<ffffffff803353d8>] ? nfs_pageio_doio+0x2a/0x51
     306<Jan/08 06:40 pm>[446605.732150]  [<ffffffff80338bf6>] ? nfs_writepages+0x75/0x97
     307<Jan/08 06:40 pm>[446605.732150]  [<ffffffff80339b7e>] ? nfs_flush_one+0x0/0xc8
     308<Jan/08 06:40 pm>[446605.732150]  [<ffffffff802828e5>] ? do_writepages+0x23/0x30
     309<Jan/08 06:40 pm>[446605.732150]  [<ffffffff802c475c>] ? __writeback_single_inod<Jan/08 06:40 pm>
     310                 <Jan/08 06:40 pm>[446605.732150]  [<ffffffff802c4c81>] ? generic_sync_sb_inodes+0x1c1/0x2b4
     311<Jan/08 06:40 pm>[446605.732150]  [<ffffffff802c4f45>] ? writeback_inodes+0x5c/0xa8
     312<Jan/08 06:40 pm>[446605.732150]  [<ffffffff802831be>] ? wb_kupdate+0x9e/0x113
     313<Jan/08 06:40 pm>[446605.732150]  [<ffffffff80283582>] ? pdflush+0x0/0x211
     314
     315<Jan/08 06:40 pm>[446605.732150]  [<ffffffff802836e2>] ? pdflush+0x160/0x211
     316<Jan/08 06:40 pm>[446605.732150]  [<ffffffff80283120>] ? wb_kupdate+0x0/0x113
     317<Jan/08 06:40 pm>[446605.732150]  [<ffffffff8024becc>] ? kthread+0x47/0x73
     318<Jan/08 06:40 pm>[446605.732150]  [<ffffffff80234852>] ? schedule_tail+0x27/0x5f
     319<Jan/08 06:40 pm>[446605.732150]  [<ffffffff8020cd19>] ? child_rip+0xa/0x11
     320<Jan/08 06:40 pm>[446605.732150]  [<ffffffff80223055>] ? physflat_send_IPI_mask+0x0/0x99
     321<Jan/08 06:40 pm>[446605.732150]  [<ffffffff8024be85>] ? kthread+0x0/0x73
     322<Jan/08 06:40 pm>[446605.732150]  [<ffffffff8020cd0f>] ? child_rip+0x0/0x11
     323<Jan/08 06:41 pm>[446666.936104] BUG: soft lockup - CPU#11 stuck for 61s! [events/11:50]
     324<Jan/08 06:41 pm>[446666.936104] Modules linked in: w83793 hwmon_vid autofs4 e100 i2c_i801 i2c_core iTCO_wdt e1000e tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
     325<Jan/08 06:41 pm>[446666.936104] CPU 11:
     326<Jan/08 06:41 pm>[446666.936104] Modules linked in: w83793 hwmon_vid autofs4 e100 i2c_i801 i2c_core iTCO_wdt e1000e tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
     327<Jan/08 06:41 pm>[446666.936104] Pid: 50, comm: events/11 Tainted: G      D    2.6.28-rc7-00105-gfeaf384 #4
     328<Jan/08 06:41 pm>[446666.936104] RIP: 0010:[<ffffffff8025bd59>]  [<ffffffff8025bd59>] smp_call_function_mask+0x15f/0x1d7
     329<Jan/08 06:41 pm>[446666.936104] RSP: 0018:ffff88063ced9d30  EFLAGS: 00000202
     330<Jan/08 06:41 pm>[446666.936104] RAX: 0000000000000020 RBX: 000000000000000b RCX: 0000000000000020
     331<Jan/08 06:41 pm>[446666.936104] RDX: ffffffff8081a300 RSI: 0000000000000020 RDI: 0000000000000000
     332<Jan/08 06:41 pm>[446666.936104] RBP: 000000000000000b R08: 0000000000000020 R09: ffff88063ced9cf0
     333<Jan/08 06:41 pm>[446666.936104] R10: 0000000000000000 R11: ffffffff802230fe R12: 0000000000000020
     334<Jan/08 06:41 pm>[446666.936104] R13: 0000000000000020 R14: 0000000000000020 R15: ffff88063ced9cf0
     335<Jan/08 06:41 pm>[446666.936104] FS:  0000000000000000(0000) GS:ffff880c3cc4d900(0000) knlGS:0000000000000000
     336<Jan/08 06:41 pm>[446666.936104] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
     337
     338<Jan/08 06:41 pm>[446666.936104] CR2: 000000001eea7000 CR3: 000000042037e000 CR4: 00000000000006e0
     339<Jan/08 06:41 pm>[446666.936104] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     340<Jan/08 06:41 pm>[446666.936104] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     341<Jan/08 06:41 pm>[446666.936104] Call Trace:
     342<Jan/08 06:41 pm>[446666.936104]  [<ffffffff80257376>] ? __lock_acquire+0x788/0x<Jan/08 06:41 pm>
     343                 <Jan/08 06:41 pm>[446666.936104]  [<ffffffff8022ecb5>] ? update_curr+0x76/0xc9
     344<Jan/08 06:41 pm>[446666.936104]  [<ffffffff80218079>] ? mcheck_check_cpu+0x0/0x2f
     345<Jan/08 06:41 pm>[446666.936104]  [<ffffffff8023dcbb>] ? on_each_cpu+0x10/0x22
     346<Jan/08 06:41 pm>[446666.936104]  [<ffffffff80217997>] ? mcheck_timer+0x0/0x76
     347<Jan/08 06:41 pm>[446666.936104]  [<ffffffff802179ab>] ? mcheck_timer+0x14/0x76
     348<Jan/08 06:41 pm>[446666.936104]  [<ffffffff80248973>] ? run_workqueue+0xed/0x1ed
     349<Jan/08 06:41 pm>[446666.936104]  [<ffffffff8024891d>] ? run_workqueue+0x97/0x1ed
     350<Jan/08 06:41 pm>[446666.936104]  [<ffffffff80249511>] ? worker_thread+0xd8/0xe3
     351<Jan/08 06:41 pm>[446666.936104]  [<ffffffff8024bfda>] ? autoremove_wake_function+0x0/0x2e
     352<Jan/08 06:41 pm>[446666.936104]  [<ffffffff80249439>] ? worker_thread+0x0/0xe3
     353<Jan/08 06:41 pm>[446666.936104]  [<ffffffff8024becc>] ? kthread+0x47/0x73
     354<Jan/08 06:41 pm>[446666.936104]  [<ffffffff80234852>] ? schedule_tail+0x27/0x5f
     355<Jan/08 06:41 pm>[446666.936104]  [<ffffffff8020cd19>] ? child_rip+0xa/0x11
     356<Jan/08 06:41 pm>[446666.936104]  [<ffffffff8024be85>] ? kthread+0x0/0x73
     357<Jan/08 06:41 pm>[446666.936104]  [<ffffffff8020cd0f>] ? child_rip+0x0/0x11
     358<Jan/08 06:41 pm>[446671.232144] BUG: soft lockup - CPU#2 stuck for 61s! [pdflush:9164]
     359<Jan/08 06:41 pm>[446671.232144] Modules linked in: w83793 hwmon_vid autofs4 e100 i2c_i801 i2c_core iTCO_wdt e1000e tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
     360
     361<Jan/08 06:41 pm>[446671.232144] CPU 2:
     362<Jan/08 06:41 pm>[446671.232144] Modules linked in: w83793 hwmon_vid autofs4 e100 i2c_i801 i2c_core iTCO_wdt e1000e tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
     363<Jan/08 06:41 pm>[446671.232144] Pid: 9164, comm: pdflush Tainted: G      D    2.6.28-rc7-00105-gfeaf384 #4
     364<Jan/08 06:41 pm>[446671.232144] RIP: 0010:[<ffffffff8021205f>]  [<ffffffff8021205f>] native_read_tsc+0x2/0xc
     365<Jan/08 06:41 pm>[446671.232144] RSP: 0018:ffff880373fe9b78  EFLAGS: 00000202
     366<Jan/08 06:41 pm>[446671.232144] RAX: 00000000f83c83c6 RBX: 00000000f83c83a8 RCX: ffffffff80932380
     367<Jan/08 06:41 pm>[446671.232144] RDX: 0000000000043b91 RSI: 0000000001062560 RDI: 0000000000000001
     368<Jan/08 06:41 pm>[446671.232144] RBP: 00000000f83c83a8 R08: ffffffff8024f65a R09: 0000000000000000
     369<Jan/08 06:41 pm>[446671.232144] R10: 0000000000000000 R11: ffffffff8039d94e R12: 00000000c77d3edd
     370<Jan/08 06:41 pm>[446671.232144] R13: 00000000c77d3edd R14: ffffffff80794266 R15: ffffffff80963300
     371<Jan/08 06:41 pm>[446671.232144] FS:  0000000000000000(0000) GS:ffff88063cc5b480(0000) knlGS:0000000000000000
     372<Jan/08 06:41 pm>[446671.232144] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
     373<Jan/08 06:41 pm>[446671.232144] CR2: 0000000041d7afe0 CR3: 0000000000201000 CR4: 00000000000006e0
     374<Jan/08 06:41 pm>[446671.232144] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     375<Jan/08 06:41 pm>[446671.232144] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     376<Jan/08 06:41 pm>[446671.232144] Call Trace:
     377<Jan/08 06:41 pm>[446671.232144]  [<ffffffff8039d96a>] ? delay_tsc+0x1c/0x42
     378<Jan/08 06:41 pm>[446671.232144]  [<ffffffff803a0ffe>] ? _raw_spin_lock+0xba/0x122
     379<Jan/08 06:41 pm>[446671.232144]  [<ffffffff805895ad>] ? xprt_prepare_transmit+0x40/0x8c
     380<Jan/08 06:41 pm>[446671.232144]  [<ffffffff805878cc>] ? call_transmit+0x49/0x241
     381<Jan/08 06:41 pm>[446671.232144]  [<ffffffff8058d0bf>] ? __rpc_execute+0x73/0x228
     382<Jan/08 06:41 pm>[446671.232144]  [<ffffffff80586f15>] ? rpc_run_task+0x4f/0x56
     383
     384<Jan/08 06:41 pm>[446671.232144]  [<ffffffff80338685>] ? nfs_write_rpcsetup+0x200/0x220
     385<Jan/08 06:41 pm>[446671.232144]  [<ffffffff803353d8>] ? nfs_pageio_doio+0x2a/0x51
     386<Jan/08 06:41 pm>[446671.232144]  [<ffffffff80338bf6>] ? nfs_writepages+0x75/0x97
     387<Jan/08 06:41 pm>[446671.232144]  [<ffffffff80339b7e>] ? nfs_flush_one+0x0/0xc8
     388<Jan/08 06:41 pm>[446671.232144]  [<ffffffff802828e5>] ? do_writepages+0x23/0x30
     389<Jan/08 06:41 pm>[446671.232144]  [<ffffffff802c475c>] ? __writeback_single_inode+0x150/0x2a9
     390<Jan/08 06:41 pm>[446671.232144]  [<ffffffff802c4c81>] ? generic_sync_sb_inodes+0x1c1/0x2b4
     391<Jan/08 06:41 pm>[446671.232144]  [<ffffffff802c4f45>] ? writeback_inodes+0x5c/0xa8
     392<Jan/08 06:41 pm>[446671.232144]  [<ffffffff802831be>] ? wb_kupdate+0x9e/0x113
     393<Jan/08 06:41 pm>[446671.232144]  [<ffffffff80283582>] ? pdflush+0x0/0x211
     394<Jan/08 06:41 pm>[446671.232144]  [<ffffffff802836e2>] ? pdflush+0x160/0x211
     395<Jan/08 06:41 pm>[446671.232144]  [<ffffffff80283120>] ? wb_kupdate+0x0/0x113
     396<Jan/08 06:41 pm>[446671.232144]  [<ffffffff8024becc>] ? kthread+0x47/0x73
     397<Jan/08 06:41 pm>[446671.232144]  [<ffffffff80234852>] ? schedule_tail+0x27/0x5f
     398<Jan/08 06:41 pm>[446671.232144]  [<ffffffff8020cd19>] ? child_rip+0xa/0x11
     399<Jan/08 06:41 pm>[446671.232144]  [<ffffffff80223055>] ? physflat_send_IPI_mask+0x0/0x99
     400<Jan/08 06:41 pm>[446671.232144]  [<ffffffff8024be85>] ? kthread+0x0/0x73
     401<Jan/08 06:41 pm>[446671.232144]  [<ffffffff8020cd0f>] ? child_rip+0x0/0x11
     402
     403
     404}}}