I was exploring a couple of <code>fault = 4</code> ('programming') errors in the stacks I'm currently processing, and found that both processes died due to memory corruption, in different places in the code. There are no known or suspected bugs in the code that would cause this. Then I noticed that they both occurred on ipp025, which also recently crashed with the do_IRQ error.

I suggest that the memory corruption I experienced is roughly equivalent to a memtest86 error, and conclude that ipp025 has bad memory.

Paul.Price 13:34, 3 November 2008 (HST)

<pre> 2008-11-03 21:27:12Z|ipp025 |I|ppStack

Time to perform rejection on image 7: 20.196237 sec

Memory Block ID: 16273299 @ 0x7f841cd084c0

Previous Block: 0x7f841cd08540 Next Block: 0x7f841cd08400 Free function: (nil) Size: 8 Reference count: 1 Persistent: No Posts: deadbeef deadbeef deadbcef Allocated in vectorAlloc at (psVector.c:78)

by Thread ID 1127573840

Memory block is corrupted; buffer overflow detected. Caught in vectorFree at (psVector.c:44).

p_psMemDecrRefCounter (psMemory.c:800) Unsafe to Continue </pre>

<pre> 2008-11-03 21:30:29Z|ipp025 |I|psModules.imcombine

Mean deviation from 98 stamps: 0.188637 +/- 0.197773

Memory Block ID: 4541300 @ 0x7f0272aba770

Previous Block: 0x7f0272aba900 Next Block: 0x7f0272aba7f0 Free function: 0x7f028e2d1210 Size: 16 Reference count: 1 Persistent: No Posts: deadbeef deadbeef deadbcef Allocated in pmSubtractionConvolve at (pmSubtraction.c:1282)

by Thread ID 139648987760368

Memory block is corrupted; buffer overflow detected. Caught in psArrayElementsFree at (psArray.c:239).

p_psMemDecrRefCounter (psMemory.c:800) Unsafe to Continue Aborting. Error stack: </pre>

<pre> mysql> select hostname, path_base from stackSumSkyfile join stackRun using(stack_id) where label = 'pap_md_nov2008' and fault = 4; +----------+-------------------------------------------------------------------------------------+ | hostname | path_base | +----------+-------------------------------------------------------------------------------------+ | ipp025 | neb://ipp010/md_demo_Nov2008//FIXNS/skycell.02874.13/FIXNS.skycell.02874.13.stk.375 | | ipp025 | neb://ipp010/md_demo_Nov2008//FIXNS/skycell.02878.62/FIXNS.skycell.02878.62.stk.434 | +----------+-------------------------------------------------------------------------------------+ 2 rows in set (0.00 sec) </pre>