ppImage Memory Explosion 2012-08

Recently several LAP exposures have had problems during chip processing. They symptom is that the memory usage by psphot becomes excessive (>20GB). This problem has so far been observed only for chips on the edge of the field of view. Many have bright features such as a super bright star just outside the FOV.

Bill has debugged this a bit and it seems that there are a number of contributing factors.

• The exposures are old, noisy, and have more common glints
• The pattern row correction generates regions of very bright apparent flux that are not visible when the correction is turned off
• A large psf (> 10 pixels) is measured
• During source detection many sources are found in the bright area
• Due to the large psf very large pixel images (some the size of the entire chip) are created blowing up the memory usage.

Many of the troublesome chips are avoided by popping a quality error if the fwhm is > 35 pixels. This work around was added to the production build on Friday 2012-08-10 but when LAP was restarted 2012-08-13 new cases emerged.

The jpeg below shows 4 views of parts of ota XY60 from o4985g0061o (2009-06-03) (chip_id 537433). On the top row are the outputs chip processing. On the left is the current default recipe (without the workaround). The upper right panel shows the results if PATTERN_ROW is set to false.

In the lower panels show the corresponding images without subtracting the measured background.

There are a number of steps that should be taken to address this problem.

• psphot should somehow limit the size of the images for a source. It makes no sense to have many sources which supposedly take up the whole OTA.
• the problem with the pattern row correction with otas at the edge of the FOV should be investigated.
• The psf measurement should apply an upper limit to the sigma values used to measure the window function. Bill has implemented this. It helps reduce this problem in some cases but not others.

Another ota which was particularly problematic was XY66. See for example chip_id 541166. In this case the wacky flux comes from a bad cell not a bright astronomical source.

How do the sources get large pixel images?

I traced through the code and found that the sources are getting allocated large pixel images in psphotKronIterate_Threaded. When the sum of pixels in the kron radius measurement == 0 the kron radius was getting set to NaN. Then it calculates the radius and finds NaN. So since r is never > NAN the sums includes all pixels in the flux measurement.

On the next kron iteration the code that is intended to avoid generating large radii for small surface brightness objects uses the huge flux to calculate a big radius. And since source->moments->Mrf is not finite it ends up using that causing the pixel images to become huge.

Basically it was a mess.

The fix is to only use the kron flux for the surface brightness cut if Mrf and the kron flux are both valid.

This has been implemented and integrated into ipp-20120802.

I also changed the code to leave the Mrf and kron flux values unchanged in the case where an invalid Mrf is measured. Maybe this will reduce the outliers in our kron radius measurements