== ppImage Memory Explosion 2012-08 ==
Recently several LAP exposures have had problems during chip processing. They symptom is that the memory usage by psphot becomes excessive (>20GB). This problem has so far been observed only for chips on the edge of the field of view. Many have bright features such as a super bright star just outside the FOV.

Bill has debugged this a bit and it seems that there are a number of contributing factors.
 * The exposures are old, noisy, and have more common glints
 * The pattern row correction generates regions of very bright apparent flux that are not visible when the correction is turned off
 * A large psf (> 10 pixels) is measured
 * During source detection many sources are found in the bright area
 * Due to the large psf very large pixel images (some the size of the entire chip) are created blowing up the memory usage.

Many of the troublesome chips are avoided by popping a quality error if the fwhm is > 35 pixels. This work around was added to the production build on Friday 2012-08-10 but when LAP was restarted 2012-08-13 new cases emerged.

The jpeg below shows 4 views of parts of ota XY60 from o4985g0061o (2009-06-03) (chip_id 537433). On the top row are the outputs chip processing. On the left is the current default recipe (without the workaround). The upper right panel shows the results if PATTERN_ROW is set to false.

In the lower panels show the corresponding images without subtracting the measured background.

There are a number of steps that should be taken to address this problem.
 * psphot should somehow limit the size of the images for a source. It makes no sense to have many sources which supposedly take up the whole OTA.
 * the problem with the pattern row correction with otas at the edge of the FOV should be investigated. 
 * The psf measurement should apply an upper limit to the sigma values used to measure the window function. Bill has implemented this. It helps reduce this problem in some cases but not others.

[[Image(o4985g0061o.jpg)]]