Changes between Initial Version and Version 1 of Magic.20081221.example

Timestamp:

Feb 24, 2009, 4:23:57 PM (17 years ago)

Author:

trac

Comment:

--

Magic.20081221.example

@@ +1 @@
+== IPP, Magic and Masking ==
+
+This set of images illustrates some of the issues the IPP+Magic
+masking of the GPC1 images as they ar processed through the science
+processing steps.
+
+[[Image(htdocs:/images/O4776g0146o.47076.cm.jpg)]]
+
+The first image is a JPEG of the entire field of M31, illustrating the
+impact of masking after the single-image analysis ('chip' analysis).
+The white regions have been masked, mostly by the static mask.  An
+extremely small number of pixels are masked because the software has
+identified them as either cosmic-rays or the saturated portions of
+stars.  The static mask was initially generated by excluding pixels
+which to exclude (a) pixels which were frequently outliers in the dark
+residuals or (b) frequently outliers in the flat residuals.  These
+were then extended by manually masking regions with either additional
+persistent glow, low CTE, or other cosmetically poor structures.  The
+total area within the 3degree unvignetted field-of-view which is
+masked by the static mask is 12.6% of the available pixels in this
+region (and 18.5% for the full GPC1).  See
+http://kiawe.ifa.hawaii.edu/IPPwiki/index.php/GPC1_Mask_2008.06. 
+
+After the individual chips are processed, the astrometric solution for
+the full mosaic is determined.  The astrometry solution and the
+astrometric reference catalog are used to generate a dynamic mask
+consisting of 3 components: circular masks on bright (saturated)
+stars, long-thin rectangles to mask the diffraction spikes, and
+rectangular regions from the bright star to the edge of the cell to
+mask the residual charge from the current readout.  
+
+The dynamic masks are merged with the initial chip-level masks applied
+to the pixels in the warp analysis.  In the warp analysis, the pixels
+are warped into the 'skycells' which have a regular and consistent
+pixel grid on the sky.  There are a few subtle issues which impact the
+total masked fraction and which make it difficult to get an accurate
+accounting of the masked regions:
+
+ * A given GPC1 exposure may be warped to multiple projection centers with overlapping skycells.
+
+ * A given GPC1 exposure may only partially fill a number of the skycells at the boundaries of the exposure.  Output skycells from the warp process which would result in less than a minimal number of pixels are skipped: these in general do not have enough stars to perform the PSF-matching needed for the stacking or difference analysis.
+
+ * The current 8-bit masks used by the IPP do not enough available bits to uniquely distinguish all possible masking effects.  Currently, the value used to identify the pixels off the edges of the existent GPC1 pixels is the same as one of the bits used for the static mask values.  
+
+ * The warping analysis involves an interpolation.  In this interpolation, masked pixels contaminate some number of their neighbors.  Pixels with more than a minimum fraction of their variance masked are themselves masked in the output image, and those with more than 0.0 but less than the previous minimum are marked with a bit to flag them as 'poor'.  Thus, the regions masked by the static and dynamic masking grow during warping.  
+
+[[Image(htdocs:/images/O4776g0146o.47076.wrp.1259.skycell.00266.07.jpg)]]
+
+The next image shows a zoom in for one of the skycells from the
+previous exposure.  The area masked by the bright star diffraction
+spikes and the other dynamic masks is clear, and in this example
+masks about 7% of the pixels.  A couple of notes about the bright star
+mask:
+
+ * This field (in M31) is somewhat more crowded at this magnitude than average, but not hugely.  The total area masked due will clearly be a function of Galactic latitude.
+
+ * The diffraction spike mask is currently rather crude.  The analysis scales the length and the width by the expected instrumental magnitude of the star.  However, we are currently limited to our synthetic photometry catalog (based on 2MASS and Tycho) and our photometry predictions are not very accurate yet.  The resulting regions are usually overly liberal (masking too much area), but for some bright stars, are too conservative (note the bright star at
+the bottom of the image with the leaking diffraction spikes).  We can probably improve the situation some with a more carefully tuned pattern (eg, triangles instead of rectangles, better measurement of the length as a function of magnitude).  
+
+ * The Air Force has agreed to let us pass back the pixels in the cores of bright stars (the circular regions), but not the diffraction spikes.  The latter very strongly resemble the satelite streaks, and so may hide real streaks under the masked regions.
+
+[[Image(htdocs:/images/Magic-O4776g0146o.47076.wrp.1259.skycell.00266.07.jpg)]]
+
+The final image shows the same skycell after the magic analysis was
+performed.  In this case, the bright star with insufficiently-masked
+diffraction spikes resulted in Magic streaks.  A second masked streak
+(also a false positive) is seen running horizontally about halfway
+through the image.
+
+For the tests in the field of M31 and the MD01, the current Magic
+analysis found a total of 20 - 90 streaks.  These are actually
+represent a smaller number of actual magic detections because of
+multiple counting.  For example, the exposure above has an increase in 
+the total masked fraction of 2% after magic.  
+
+One of the significant challenges remaining is the residual charge
+left from bright stars in the successive images.  The IPP analysis has
+a crude algorithm based on an isophotal analysis to find and mask
+these regions.  The analysis is excessively aggressive, currently
+removing too many pixels that do not actually contain residual charge.
+Paul Sydney attempted to use the the consistent position angle of
+these features to better exclude the residual charge in Hough space.
+This is particularly needed for the M31 field, since the IPP algorithm
+removes an absurd amount of area from these images, so it must be
+turned off.  This Hough-space analysis of the residual charge were not
+extremely successful, again resulting in too many false positive
+detections.  The tests to date show that we have the best results if the IPP
+residual charge analysis is used, but in the fields where we cannot
+use it, the Magic Hough-space analysis is acceptable (as seen by the
+M31 example).  We hope to gain significant further improvements by
+using a more sophisticated spatially analysis of the structures in the
+IPP, building on the analysis Tonry is developing for the camera.  
+
+Part of the difficulty for the IPP analysis of this effect is that the
+IPP is designed not to rely on a rigid sequencing of the analysis of
+the images.  Thus, the IPP would need significant modification to
+ensure the history for each pixel is available at the needed point in
+the analysis.  For future images, the Camera group is hoping to remove
+the residual charge trails using the knowledge of where bright stars
+were observed for each image.  
+
+For the STS field, the current IPP + Magic analysis is yielding a
+larger number of streaks than expected.  In some of these exposures,
+the row-by-row correlated noise structures are particularly
+significant.  These structures trigger the magic streak analysis since
+they appear as horizontal (row-direction) lines.  We only realized
+late in the Magic-assessment period that this effect was triggering
+large numbers of false positives.  A few possibilities exist for
+improvements:
+
+ * The Camera software is currently subtracting a row-by-row bias measured in the overscan regions.  This measurement corrects well the portion of each row closest to the overscan region, but does not work so well on the other side of each cell.  For the future, the Camera group has suggested reading a pre- and a post- scan region, and fitting the trend between the two ends.
+
+ * The row-by-row signal is correlated across cells in the same column.  It may be possible to measure a correction from the 8 common cells with higher fidelity and apply that correction to all 8 cells.  
+
+ * The impact of this structure is effectively a small increase in the read-noise.  However, the magic analysis is quite sensitive to the choice of the noise threshold.  It may be preferable to report the measured read noise from each cell and use that in the analysis of the full variance image, rather than a static lab value for each cell.
+
+A couple of other important points need to be made regarding masking and magic.  First, the pixels which are masked by magic are not lost : they are saved in a holding pen and may be used if either Magic or the IPP analysis is thought to have been improved.  On the other hand, the IPP can only perform magic on pixels for which a difference image can be (or has been) constructed, and in general only for a complete exposure at a time.  In the first year, this puts an important constraint on the observing, and will results in area which cannot be Magicked.  In the first year, for the Three Pi survey, we can only afford to perform the difference analysis on the pairs of images which make up a TTI and the 4-way sets of stacks.  For the initial pair-wise difference, small offsets and rotations between observations will result in regions which do not have any overlap, and thus cannot be magicked.  Some of these will be recovered after the full 4-way stack is generated (including the overlaps between tessellation centers).  Note also that the regions in the images which result in only a small fraction in the output warps cannot be magicked, and are thus going to be lost in general.