ppStack occasionally runs into memory corruption in cfitsio

[bills]

I have some news about the bug that I've been trying to catch where ppStack encounters memory corruption in cfitsio
while writing out the output files. (See stack trace below)

What I think I know as of now

1. It is repeatable
2. It happens with a debug build off of the trunk as well as the current tag (so it isn't fixed).
3. It happens with multiple stack runs ( MD08: 127071 129306 129336 MD04: 117659 117660) The oldest run with this error is from mid-June.
4. In the one I've been re-running (127071) it dies during the second call to pmFPAWriteVariance when

compressing the output file for the file rule PPSTACK.OUTPUT.EXPTWT

I currently have it running single threaded with a breakpoint in pmFPAWriteVariance and a non-optimized cfitsio. Hopefully
I'll be able to step through and find something.

I'm not familiar with ppStack or cfitsio compression or how we use it so this is challenging.

These files seem to compress by a large amount so simply turning off compression as a workaround isn't a good idea.
(I decompressed one of these from another run and it grew by a factor of 30)

In summary, the bad news is the bug is more common than I had thought and it takes hours to get to the breakpoint and one fat
fingered error requires restarting.

But the good news is that it appears to be somewhat deterministic.

Sample stack trace. Sometimes it dies with 'double free or corruption' other times it says 'invalid next size' But the stack
trace and addres of the call to cfree is the same.

Stage 6: Photometry Analysis: 24.772171 sec

* glibc detected * /home/panstarrs/ipp/psconfig/ipp-20100701.lin64/bin/ppStack: double free or corruption (!prev): 0x000000000855c0c0 *
======= Backtrace: =========
/lib/libc.so.6[0x7f2eefe16b88]
/lib/libc.so.6(cfree+0x76)[0x7f2eefe18746]
/home/panstarrs/ipp/psconfigipp-20100701.lin64/lib/libcfitsio.so(imcomp_compress_tile+0x4dc)[0x7f2ef1238dfc]
/home/panstarrs/ipp/psconfigipp-20100701.lin64/lib/libcfitsio.so(fits_write_compressed_img+0xcd3)[0x7f2ef123aa43]
/home/panstarrs/ipp/psconfigipp-20100701.lin64/lib/libcfitsio.so(fits_write_compressed_img_plane+0x257)[0x7f2ef123b167]
/home/panstarrs/ipp/psconfigipp-20100701.lin64/lib/libcfitsio.so(fits_write_compressed_pixels+0x35c)[0x7f2ef123b5cc]
/home/panstarrs/ipp/psconfigipp-20100701.lin64/lib/libcfitsio.so(ffppri+0xb7)[0x7f2ef1214e07]
/home/panstarrs/ipp/psconfigipp-20100701.lin64/lib/libcfitsio.so(ffppr+0x12a)[0x7f2ef120b5aa]
/home/panstarrs/ipp/psconfigipp-20100701.lin64/lib/libpslib.so.1(psFitsInsertImageWithMask+0xaae)[0x7f2ef276015e]
/home/panstarrs/ipp/psconfigipp-20100701.lin64/lib/libpslib.so.1(psFitsWriteImageWithMask+0x96)[0x7f2ef2760236]
/home/panstarrs/ipp/psconfigipp-20100701.lin64/lib/libpslib.so.1(psFitsWriteImageCubeWithMask+0x3df)[0x7f2ef27606ff]
/home/panstarrs/ipp/psconfigipp-20100701.lin64/lib/libpsmodules.so.1[0x7f2ef4187b1f]
/home/panstarrs/ipp/psconfigipp-20100701.lin64/lib/libpsmodules.so.1[0x7f2ef4186946]
/home/panstarrs/ipp/psconfigipp-20100701.lin64/lib/libpsmodules.so.1(pmFPAWriteVariance+0x2c)[0x7f2ef4186a9c]
/home/panstarrs/ipp/psconfigipp-20100701.lin64/lib/libpsmodules.so.1[0x7f2ef4197445]
/home/panstarrs/ipp/psconfigipp-20100701.lin64/lib/libpsmodules.so.1(pmFPAfileWrite+0x406)[0x7f2ef41954f6]
/home/panstarrs/ipp/psconfigipp-20100701.lin64/lib/libpsmodules.so.1(pmFPAfileIOChecks+0x12e)[0x7f2ef4195dfe]
/home/panstarrs/ipp/psconfig/ipp-20100701.lin64/bin/ppStack[0x408c08]
/home/panstarrs/ipp/psconfig/ipp-20100701.lin64/bin/ppStack[0x4140cf]
/home/panstarrs/ipp/psconfig/ipp-20100701.lin64/bin/ppStack[0x409648]
/home/panstarrs/ipp/psconfig/ipp-20100701.lin64/bin/ppStack[0x405579]
/lib/libc.so.6(libc_start_main+0xe6)[0x7f2eefdc0486]

---

Gene suggested that we might be able to write out the affected image uncompressed and use fpack or ppImage to attempt reproduce the problem without waiting for hours and hours.

[bills]

For stack_id 127071 it does die with a debug cfitsio.
It croaks freeing the buffer cbuf when called for row 5863 of the image.

Here is the contents of the cfitio Fptr structure after abort is called.

(gdb) p *fptr->Fptr
$80 = {filehandle = 234, driver = 0, open_count = 1, filename = 0x1161390 "/data/ippc18.0/home/bills/debugppstack/gpc1/MD08.refstack.20100708/MD08/skycell.115/MD08.skycell.115.stk.127071.expwt.fits", validcode = 555, filesize = 5650560, logfilesize = 5653440, lasthdu = 1, bytepos = 5652745, io_pos = 5650560, curbuf = 12, curhdu = 1, hdutype = 2, writemode = 1, maxhdu = 1, MAXHDU = 1000, headstart = 0x327859a0, headend = 13360, nextkey = 4320, datastart = 14400, imgdim = 0, imgnaxis = {0 <repeats 99 times>}, tfield = 1, origrows = 6000, numrows = 6000, rowlength = 8, tableptr = 0x368a5470, heapstart = 48000, heapsize = 5590345, request_compress_type = 21, request_tilesize = {0, 1, 1, 0, 0, 0}, request_hcomp_scale = 0, request_hcomp_smooth = 0, compressimg = 1, zcmptype = "GZIP_1\000\000\000\000\000", compress_type = 21, zbitpix = 16, zndim = 2, znaxis = {6000, 6000, 0, 0, 0, 0}, tilesize = {6000, 1, 0, 0, 0, 0}, maxtilelen = 6000, maxelem = 9230, cn_compressed = 1, cn_uncompressed = 0, cn_zscale = 0, cn_zzero = 0, cn_zblank = 0, zscale = 0, zzero = 0, cn_bscale = 1, cn_bzero = 0, zblank = 0, rice_blocksize = 0, rice_bytepix = 0, quantize_level = 256, hcomp_scale = 0, hcomp_smooth = 0, tilerow = 0, tiledatasize = 0, tiletype = 0, tiledata = 0x0, tilenullarray = 0x0, tileanynull = 0}

[bills]

129761 fails runs into this as well.

It's interesting that all of the skycells that have the problem are g band stacks...

[bills]

While making MD04.refstack.20101102 a number of stacks failed with stack trace similar to these.