robust stats failure

[tdupuy]

Calculating either robust stdev or robust median fails for an image. psError
message was "Failed to fit a gaussian to the robust histogram." This is
annoying for two reasons:

1) apparently p_psVectorRobustStats breaks for a reasonable looking image.

2) when p_psVectorRobustStats fails, it doesn't exit it seg faults. If it
exited returning null values for the stats values I could instead ask my
program to do clipped or sample values for the stats values I need.

[gusciora@…]

I tried several cases, however I could not reproduce this problem. Is it
possible for you to either send us test data, or describe in more detail how you
produced this problem?

[tdupuy]

So far only one image has been giving me a problem; an image which seems to
otherwise be normal. It can be retreived (while logged into any po node) at:

/s04/tdupuy/cfh12k_img/f0230_c01_606365_12k_i.fits

The exact code that I used went something like this:

psStats *stats = psStatsAlloc(PS_STAT_ROBUST_MEDIAN | PS_STAT_ROBUST_STDEV);
stats->clipSigma = 2;
stats->clipIter = 5;
stats = psImageStats(stats, image, NULL, 0);

I tried varying the number of iterations and sigma with no success.

[gusciora@…]

I don't have login access to the po nodes. Can you create an attachment in
the bugzilla report? In fact, it might be easier to include the code,
makefile, and data in a tar file and attach to the bugzilla report.

[Paul Price]

The SEGV is presumably due to attempting to dereference the result of psStats,
which comes back NULL because of the errors.

I dug around a bit, and suspect that the cumulative histogram isn't correct (the
bounds are negative, when I would expect them to be positive, and the nums only
go up to 8, or so) because the limits aren't being set correctly. The image has
values mostly around ~ 1000 or so, but there are some pixels with values of
10^{7. The robust statistics was created specifically to handle these deviant
values, so it should be able to handle this image. Since this appears to be a
bug in psStats, I'm handing back to George.}

I've bzip2-ed the FITS file and placed it at
mhpcc.pan-starrs.org:/home/price/f0230_c01_606365_12k_i.fits.bz2

I will attach a short C program to reproduce the error. To build:
gcc trentBug.c pslib-config --libs --cflags -g -O0 -o trentBug

[Paul Price]

trentBug.c

[gusciora@…]

I can't find that FITS file at the URL you suggested.

[Paul Price]

It's on your computer, in my home directory. I just updated the permission on
the directory which might have prevented you reading the file.

[gusciora@…]

I can compile and run your test program with that FITS file. It appears that
the seg fault is occuring in the test program; psLib seems to be handling an
error condition reasonably (at least in your test program). Is that also true
in Trent's program?

The code does fail to fit a Gaussian with the rel9 version of psStats.c (and
exits with a memory leak, which looks to be in psLib). I'm not sure exactly why
that is. However, it produces results with the *current* CVS tree. The robust
median is about -3770 and the robust stdev is about 1820. Do those numbers
sound correct?

not psLib.

[Paul Price]

No, the SEGV is not in pslib, so don't worry about that.

I don't think that median ~ -3770 can be answer because there are very few
negative pixels.

[gusciora@…]

There's a definite bug in psStats.c. I checked it into the current code tree.
The answer I'm is about 10254 for the robust median and about 457
for the stdev.

[Paul Price]

What was the bug, and how was it fixed?

[gusciora@…]

This code was bad:

if (sigma < (2 * binSize)) {

...
psF32 medianLo = robustHistogram->bounds->data.F32[binMedian - 25];
psF32 medianHi = robustHistogram->bounds->data.F32[binMedian + 25];
...

The problem was that when we added/subtracted 25 to the binMedian, I did not
ensure that I was still within the array boundaries.

I fixed it by simply putting the appropriate PS_MIN() or PS_MAX() macro in the
array access.

[Paul Price]

Is this fixed, then?

[gusciora@…]

I think so. Please give it a try whenever you are ready.

[Paul Price]

Looks good.

[Paul Price]

Bug has been resolved.... closing.

[Paul Price]

Bugs have been fixed... closing.