IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links
wiki:ipp014-2012-03-07-crash

ipp014 became unresponsive with ramping load. The following was on the console.



<Mar/07 00:21 am>[2039834.057468] burntool[24771]: segfault at 7f4291404150 ip 000000000040acc8 sp 00007fff9941ef10 error 4 in burntool[400000+1a000]
<Mar/07 00:24 am>[2040021.686312] w83793 0-002f: set bank to 2 failed, fall back to bank 0, read reg 0x201 error
<Mar/07 00:36 am>[2040734.545943] nfs: server ipp008 not responding, still trying

<Mar/07 00:36 am>[2040741.933544] nfs: server ipp008 OK
<Mar/07 00:54 am>[2041844.945835] lockd: server ipp009 not responding, still trying
<Mar/07 00:55 am>[2041880.900253] lockd: server ipp009 OK
<Mar/07 01:00 am>[2042154.555887] lockd: server ipp009 not responding, still trying
<Mar/07 01:00 am>[2042194.559362] lockd: server ipp009 not responding, still trying
<Mar/07 01:03 am>[2042371.708094] lockd: server ipp009 OK
<Mar/07 01:03 am>[2042371.712437] lockd: server ipp009 OK
<Mar/07 01:11 am>[2042829.049117] lockd: server ipp009 not responding, still trying
<Mar/07 01:15 am>[2043069.059356] lockd: server ipp009 not responding, still trying
<Mar/07 01:16 am>[2043151.429837] lockd: server ipp009 OK
<Mar/07 01:16 am>[2043153.980385] lockd: server ipp009 OK
<Mar/07 01:27 am>[2043791.426853] burntool[3569]: segfault at 7fb990fcf150 ip 000000000040acc8 sp 00007fff98fe8ad0 error 4 in burntool[400000+1a000]
<Mar/07 01:30 am>[2043985.125630] lockd: server ipp009 not responding, still trying
<Mar/07 01:30 am>[2044011.026558] lockd: server ipp009 OK
<Mar/07 01:37 am>[2044397.882252] lockd: server ipp009 not responding, still trying
<Mar/07 01:37 am>[2044397.890795] lockd: server ipp009 OK
<Mar/07 01:44 am>[2044804.615559] nfs: server ipp009 not responding, still trying
<Mar/07 01:44 am>[2044806.812658] nfs: server ipp009 not responding, still trying
<Mar/07 01:44 am>[2044806.916741] nfs: server ipp009 OK
<Mar/07 01:44 am>[2044806.920685] nfs: server ipp009 OK
<Mar/07 01:54 am>[2045404.972359] lockd: server ipp009 not responding, still trying
<Mar/07 01:54 am>[2045444.975686] lockd: server ipp009 not responding, still trying

<Mar/07 01:58 am>[2045652.641752] lockd: server ipp009 OK
<Mar/07 01:58 am>[2045652.645778] lockd: server ipp009 OK
<Mar/07 02:05 am>[2046092.528737] lockd: server ipp009 not responding, still trying
<Mar/07 02:06 am>[2046144.050026] lockd: server ipp009 OK
<Mar/07 02:12 am>[2046489.760351] nfs: server ipp015 not responding, still trying
<Mar/07 02:12 am>[2046489.767052] nfs: server ipp015 not responding, still trying
<Mar/07 02:12 am>[2046489.773859] nfs: server ipp015 OK
<Mar/07 02:12 am>[2046489.778345] nfs: server ipp015 OK
<Mar/07 02:13 am>[2046569.385443] lockd: server ipp008 not responding, still trying
<Mar/07 02:13 am>[2046584.860384] burntool[11693]: segfault at 7f2604675150 ip 000000000040acc8 sp 00007fff0c68e180 error 4 in burntool[400000+1a000]
<Mar/07 02:13 am>[2046591.902133] lockd: server ipp009 not responding, still trying
<Mar/07 02:16 am>[2046714.437289] lockd: server ipp009 OK
<Mar/07 02:16 am>[2046742.605939] lockd: server ipp008 OK
<Mar/07 02:21 am>[2047019.472659] lockd: server ipp009 not responding, still trying
<Mar/07 02:21 am>[2047068.842140] lockd: server ipp009 OK
<Mar/07 03:09 am>[2049902.026193] burntool[17541]: segfault at 7fb860e4c150 ip 000000000040acc8 sp 00007fff68e65950 error 4 in burntool[400000+1a000]
<Mar/07 04:07 am>[2053401.048609] burntool[22815]: segfault at 7ffcc5643150 ip 000000000040acc8 sp 00007fffcd65e150 error 4 in burntool[400000+1a000]
<Mar/07 04:58 am>[2056452.322252] burntool[27240]: segfault at 7f6434b23150 ip 000000000040acc8 sp 00007fff3cb3c630 error 4 in burntool[400000+1a000]
<Mar/07 05:48 am>[2059455.129657] burntool[31353]: segfault at 7fc625234150 ip 000000000040acc8 sp 00007fff2d24ed40 error 4 in burntool[400000+1a000]
<Mar/07 06:55 am>[2063506.595390] burntool[5285]: segfault at 7f57df248150 ip 000000000040acc8 sp 00007fffe7262d50 error 4 in burntool[400000+1a000]
<Mar/07 07:48 am>[2066690.493896] burntool[9761]: segfault at 7fef540e5150 ip 000000000040acc8 sp 00007fff5c0fdbf0 error 4 in burntool[400000+1a000]
<Mar/07 08:46 am>[2070125.905056] burntool[14723]: segfault at 7f8d9fb45150 ip 000000000040acc8 sp 00007fffa7b60650 error 4 in burntool[400000+1a000]

<Mar/07 09:37 am>[2073223.362867] lockd: server ipp009 not responding, still trying
<Mar/07 09:39 am>[2073336.126094] lockd: server ipp009 OK
                 <Mar/07 09:42 am>[2073518.497426] burntool[20652]: segfault at 7fcf3d875150 ip 000000000040acc8 sp 00007fff45890380 error 4 in burntool[400000+1a000]
<Mar/07 09:55 am>[2074305.919359] lockd: server ipp009 not responding, still trying
<Mar/07 09:58 am>[2074446.655081] BUG: unable to handle kernel paging request at ffffffff800073d4
<Mar/07 09:58 am>[2074446.655894] IP: [<ffffffff80589612>] xprt_autoclose+0x19/0x4c
<Mar/07 09:58 am>[2074446.655894] PGD 203067 PUD 207063 PMD 0 
<Mar/07 09:58 am>[2074446.655894] Oops: 0000 [#1] SMP 
<Mar/07 09:58 am>[2074446.655894] last sysfs file: /sys/class/i2c-adapter/i2c-0/0-002f/temp6_alarm
<Mar/07 09:58 am>[2074446.655894] CPU 0 
<Mar/07 09:58 am>[2074446.655894] Modules linked in: w83793 hwmon_vid autofs4 i2c_i801 i2c_core iTCO_wdt e1000e tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
<Mar/07 09:58 am>[2074446.655894] Pid: 1515, comm: rpciod/0 Not tainted 2.6.28-rc7-00105-gfeaf384 #4
<Mar/07 09:58 am>[2074446.655894] RIP: 0010:[<ffffffff80589612>]  [<ffffffff80589612>] xprt_autoclose+0x19/0x4c
<Mar/07 09:58 am>[2074446.655894] RSP: 0018:ffff88033d367e60  EFLAGS: 00010246
<Mar/07 09:58 am>[2074446.655894] RAX: ffffffff80007374 RBX: ffff88010001a578 RCX: 0000000000000000
<Mar/07 09:58 am>[2074446.655894] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88010001a000
<Mar/07 09:58 am>[2074446.655894] RBP: ffff88033d365a00 R08: 0000000000000001 R09: 0000000000000000
<Mar/07 09:58 am>[2074446.655894] R10: 0000000000000000 R11: ffffffff8050a1ce R12: ffff88010001a578
<Mar/07 09:58 am>[2074446.655894] R13: ffff88010001a000 R14: ffffffff808b85a0 R15: 0000000000000000
<Mar/07 09:58 am>[2074446.655894] FS:  0000000000000000(0000) GS:ffffffff80872d80(0000) knlGS:0000000000000000
<Mar/07 09:58 am>[2074446.655894] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
<Mar/07 09:58 am>[2074446.655894] CR2: ffffffff800073d4 CR3: 000000033e52f000 CR4: 00000000000006e0

<Mar/07 09:58 am>[2074446.655894] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<Mar/07 09:58 am>[2074446.655894] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<Mar/07 09:58 am>[2074446.655894] Process rpciod/0 (pid: 1515, threadinfo ffff88033d366000, task ffff88033d1fcbf0)
<Mar/07 09:58 am>[2074446.655894] Stack:
<Mar/07 09:58 am>[2074446.655894]  ffff88033d367e90 ffff88010001a578 ffffffff805895f9 ffffffff80248973
<Mar/07 09:58 am>[2074446.655894]  ffffffff8024891d 0000000000000046 0000000000000000 0000000000000000
<Mar/07 09:58 am>[2074446.655894]  0000000000000000 ffff88033d365a00 ffff88063e443bf0 ffff88033d365a00
<Mar/07 09:58 am>[2074446.655894] Call Trace:
<Mar/07 09:58 am>[2074446.655894]  [<ffffffff805895f9>] ? xprt_autoclose+0x0/0x4c
<Mar/07 09:58 am>[2074446.655894]  [<ffffffff80248973>] ? run_workqueue+0xed/0x1ed
<Mar/07 09:58 am>[2074446.655894]  [<ffffffff8024891d>] ? run_workqueue+0x97/0x1<Mar/07 09:58 am>
                 <Mar/07 09:58 am>[2074446.655894]  [<ffffffff80249511>] ? worker_thread+0xd8/0xe3
<Mar/07 09:58 am>[2074446.655894]  [<ffffffff8024bfda>] ? autoremove_wake_function+0x0/0x2e
<Mar/07 09:58 am>[2074446.655894]  [<ffffffff80249439>] ? worker_thread+0x0/0xe3
<Mar/07 09:58 am>[2074446.655894]  [<ffffffff8024becc>] ? kthread+0x47/0x73
<Mar/07 09:58 am>[2074446.655894]  [<ffffffff80234852>] ? schedule_tail+0x27/0x5f
<Mar/07 09:58 am>[2074446.655894]  [<ffffffff8020cd19>] ? child_rip+0xa/0x11
<Mar/07 09:58 am>[2074446.655894]  [<ffffffff804fa23c>] ? menu_reflect+0x0/0x6d
<Mar/07 09:58 am>[2074446.655894]  [<ffffffff8024be85>] ? kthread+0x0/0x73
<Mar/07 09:58 am>[2074446.655894]  [<ffffffff8020cd0f>] ? child_rip+0x0/0x11
<Mar/07 09:58 am>[2074446.655894] Code: 89 f7 e8 59 dd 02 00 89 d8 5b 5d 41 5c 41 5d 41 5e c3 41 55 4c 8d af 88 fa ff ff 41 54 53 48 8b 87 90 fa ff ff 48 89 fb 4c 89 ef <ff> 50 60 f0 80 63 d0 f7 4c 8d a3 98 00 00 00 4c 89 e7 e8 43 de 
<Mar/07 09:58 am>[2074446.655894] RIP  [<ffffffff80589612>] xprt_autoclose+0x19/0x4c

<Mar/07 09:58 am>[2074446.655894]  RSP <ffff88033d367e60>
<Mar/07 09:58 am>[2074446.655894] CR2: ffffffff800073d4
<Mar/07 09:58 am>[2074446.655894] ---[ end trace 8a138b265fc38a3c ]---
<Mar/07 09:59 am>[2074508.862741] lockd: cannot unmonitor ipp065
<Mar/07 10:00 am>[2074572.829332] lockd: cannot monitor ipp049
<Mar/07 10:01 am>[2074632.832642] lockd: cannot monitor ippc28
<Mar/07 10:02 am>[2074692.835977] lockd: cannot unmonitor ippc07
<Mar/07 10:03 am>[2074752.839315] lockd: cannot unmonitor ipp059
<Mar/07 10:04 am>[2074812.842692] lockd: cannot unmonitor ippc47
<Mar/07 10:05 am>[2074872.845991] lockd: cannot unmonitor ippc02
<Mar/07 10:06 am>[2074932.849349] lockd: cannot unmonitor ippc49
<Mar/07 10:07 am>[2074979.992601] nfs: server ipp047 not responding, still trying
<Mar/07 10:07 am>[2074992.852678] lockd: cannot unmonitor ippc03
<Mar/07 10:08 am>[2075052.859339] lockd: cannot unmonitor ippc50
<Mar/07 10:09 am>[2075112.862661] lockd: cannot unmonitor ippc04
                 <Mar/07 10:10 am>[2075172.865939] lockd: cannot unmonitor ipp055
<Mar/07 10:11 am>[2075232.869312] lockd: cannot unmonitor ippc59
<Mar/07 10:12 am>[2075292.872596] lockd: cannot unmonitor ippc14
<Mar/07 10:13 am>[2075352.875919] lockd: cannot unmonitor ipp066
<Mar/07 10:14 am>[2075412.879260] lockd: cannot unmonitor ippc09
<Mar/07 10:15 am>[2075472.882577] lockd: cannot unmonitor ipp031
<Mar/07 10:16 am>[2075532.885955] lockd: cannot unmonitor ipp033

<Mar/07 10:17 am>[2075592.889281] lockd: cannot unmonitor ippc57
                 <Mar/07 10:18 am>[2075652.892636] lockd: cannot unmonitor ippc58
<Mar/07 10:19 am>[2075712.895961] lockd: cannot unmonitor ippc21
<Mar/07 10:20 am>[2075772.899213] lockd: cannot unmonitor ipp010
<Mar/07 10:21 am>[2075832.902624] lockd: cannot unmonitor ippc22
<Mar/07 10:22 am>[2075892.905865] lockd: cannot unmonitor ippc36
                 <Mar/07 10:23 am>[2075952.909192] lockd: cannot unmonitor ipp011
<Mar/07 10:24 am>[2076012.912566] lockd: cannot unmonitor ippc23
<Mar/07 10:25 am>[2076076.715871] lockd: cannot unmonitor ipp039
<Mar/07 10:26 am>[2076136.719171] lockd: cannot unmonitor ipp040
<Mar/07 10:27 am>[2076202.732481] lockd: cannot unmonitor ippc43
<Mar/07 10:28 am>[2076262.735867] lockd: cannot unmonitor ippc44
<Mar/07 10:29 am>[2076322.739173] lockd: cannot unmonitor stare01
<Mar/07 10:30 am>[2076382.742470] lockd: cannot unmonitor ippc45
<Mar/07 10:31 am>[2076442.745826] lockd: cannot unmonitor ippc46
<Mar/07 10:32 am>[2076502.749207] lockd: cannot unmonitor stare02
<Mar/07 10:33 am>[2076562.752491] lockd: cannot unmonitor ippc25
<Mar/07 10:34 am>[2076622.755774] lockd: cannot unmonitor ippc26
<Mar/07 10:35 am>[2076683.785866] lockd: cannot monitor ippc20
<Mar/07 10:36 am>[2076743.789209] lockd: cannot monitor ippc10
<Mar/07 10:37 am>[2076803.792460] lockd: cannot unmonitor ippc06
<Mar/07 10:38 am>[2076863.795773] lockd: cannot unmonitor ipp057

<Mar/07 10:39 am>[2076923.799100] lockd: cannot unmonitor ippc53
<Mar/07 10:40 am>[2076988.135760] lockd: cannot unmonitor ipp054
<Mar/07 10:41 am>[2077048.139132] lockd: cannot unmonitor ipp061
<Mar/07 10:42 am>[2077108.142416] lockd: cannot unmonitor ipp062
<Mar/07 10:43 am>[2077168.145747] lockd: cannot unmonitor ipp034
<Mar/07 10:44 am>[2077228.149083] lockd: cannot unmonitor ipp043
<Mar/07 10:45 am>[2077288.152376] lockd: cannot unmonitor ipp044
<Mar/07 10:46 am>[2077348.155761] lockd: cannot unmonitor ipp046
<Mar/07 10:47 am>[2077408.159048] lockd: cannot unmonitor ipp006
                 <Mar/07 10:48 am>[2077468.162373] lockd: cannot unmonitor ipp007
<Mar/07 10:49 am>[2077528.165702] lockd: cannot unmonitor ipp042
<Mar/07 10:50 am>[2077588.169030] lockd: cannot unmonitor ipp051
<Mar/07 10:51 am>[2077648.172356] lockd: cannot unmonitor ipp052
<Mar/07 10:52 am>[2077708.179012] lockd: cannot unmonitor ipp053
                 <Mar/07 10:53 am>[2077768.182346] lockd: cannot unmonitor ipp021
<Mar/07 10:54 am>[2077828.185683] lockd: cannot unmonitor ipp015
<Mar/07 10:55 am>[2077888.189000] lockd: cannot unmonitor ipp050
<Mar/07 10:56 am>[2077948.192358] lockd: cannot monitor ippc41
<Mar/07 10:57 am>[2078008.195721] lockd: cannot monitor ippc20
<Mar/07 10:58 am>[2078068.199012] lockd: cannot unmonitor ipp028
<Mar/07 10:59 am>[2078128.202308] lockd: cannot unmonitor ipp030
<Mar/07 11:00 am>[2078188.205639] lockd: cannot unmonitor ipp023

<Mar/07 11:01 am>[2078248.209007] lockd: cannot unmonitor ipp035
<Mar/07 11:02 am>[2078308.212341] lockd: cannot unmonitor ipp036
                 <Mar/07 11:03 am>[2078368.215659] lockd: cannot unmonitor ipp038
<Mar/07 11:04 am>[2078428.218919] lockd: cannot monitor ippc28
<Mar/07 11:05 am>[2078488.222272] lockd: cannot monitor ipp049
<Mar/07 11:06 am>[2078548.225600] lockd: cannot monitor ippc41
<Mar/07 11:07 am>[2078608.228926] lockd: cannot monitor ippc10
<Mar/07 11:08 am>[2078668.232258] lockd: cannot monitor ipp040
<Mar/07 11:09 am>[2078728.235584] lockd: cannot monitor ipp040
<Mar/07 11:10 am>[2078788.238955] lockd: cannot monitor ipp049
<Mar/07 11:11 am>[2078848.242274] lockd: cannot monitor ipp063
<Mar/07 11:12 am>[2078908.245565] lockd: cannot monitor ippc28
<Mar/07 11:13 am>[2078968.248962] lockd: cannot monitor ippc20
<Mar/07 11:14 am>[2079028.252294] lockd: cannot monitor ippc10
<Mar/07 11:15 am>[2079088.255651] lockd: cannot monitor ippc41
<Mar/07 11:16 am>[2079148.962286] lockd: cannot monitor ipp063
<Mar/07 11:17 am>[2079208.965538] lockd: cannot monitor ipp040
<Mar/07 11:18 am>[2079268.968833] lockd: cannot monitor ipp063
<Mar/07 11:19 am>[2079328.972194] lockd: cannot monitor ipp049
<Mar/07 11:20 am>[2079385.498847] w83793 0-002f: set bank to 0 failed, fall back to bank 2, read reg 0xb2 error
<Mar/07 11:20 am>[2079388.975520] lockd: cannot monitor ipp059
<Mar/07 11:21 am>[2079448.978901] lockd: cannot monitor ipp059

<Mar/07 11:22 am>[2079508.982141] lockd: cannot monitor ippc28
<Mar/07 11:23 am>[2079568.985505] lockd: cannot monitor ippc20
<Mar/07 11:24 am>[2079628.988804] lockd: cannot unmonitor ippc51
<Mar/07 11:25 am>[2079688.992157] lockd: cannot unmonitor ippc13
<Mar/07 11:26 am>[2079748.995490] lockd: cannot unmonitor ipp060
<Mar/07 11:27 am>[2079808.998807] lockd: cannot unmonitor ippc63
<Mar/07 11:28 am>[2079869.002198] lockd: cannot unmonitor ippc29
<Mar/07 11:29 am>[2079929.005463] lockd: cannot monitor ippc10
<Mar/07 11:30 am>[2079989.008832] lockd: cannot monitor ippc41
<Mar/07 11:31 am>[2080049.012123] lockd: cannot monitor ipp059
<Mar/07 11:32 am>[2080109.015525] lockd: cannot monitor ipp049
<Mar/07 11:33 am>[2080169.018844] lockd: cannot monitor ippc28
<Mar/07 11:34 am>[2080229.022098] lockd: cannot monitor ippc20
<Mar/07 11:35 am>[2080289.025501] lockd: cannot monitor ippc10
<Mar/07 11:36 am>[2080349.028796] lockd: cannot monitor ippc41

Last modified 14 years ago Last modified on Mar 7, 2012, 12:04:09 PM
Note: See TracWiki for help on using the wiki.