Changes between Initial Version and Version 1 of Magic.20081112.example

Timestamp:

Feb 24, 2009, 4:23:57 PM (17 years ago)

Author:

trac

Comment:

--

Magic.20081112.example

@@ +1 @@
+(by Eugene Magnier, 2008.11.12)
+
+We have had a major advance in the past few days in terms of being
+able to run the Magic streak detection and removal system.  At this
+point, we have been able to successfully run magic automatically on
+GPC1 images and have a modest false positive rate of streak detection.
+Below, I go into more detail on what we learned over the past few
+days.
+
+The caveat as of today (Nov 12) is that we have only proven the
+process on the inner 36 chips, though we should have a demonstration
+of the full camera by tomorrow (Nov 13).  The other point to note is
+that the Air Force is not yet ready to blindly trust the Magic
+processing; for the immediate future, Paul Sydney will inspect images
+by eye before they can be released to the community, and in the
+process, will attempt to build confidence in the Magic process to be
+convinced that it is working and can be relied upon.
+
+By the end of last week (Nov 7), we had the streak detection software
+generally integrated into the IPP system, but it was having trouble
+completing the first detection stage of the analysis.  Bill Sweeney
+and Paul Sydney worked to address a few issues which were causing
+troubles, mostly along the lines of assumptions about the noise model
+or bugs in one or another portion of the code.  By Saturday, the
+mechanics of the interaction were worked out, however the number of
+detected streaks was catastrophically high: essentially all image
+pixels were being masked.  
+
+At this point, it became clear that the primary difficulty was the
+large number of significant pixels in the difference images, ie,
+apparent structure in the images on which Magic was triggering and
+forming into satelite streaks.  We realized that the IPP needed to
+identify and mask a much higher fraction of the artifacts in the
+images to avoid swamping Magic with too much of a false signal.
+
+We set up a test to see if Magic would work on a sufficiently clean
+image: Paul Price manually inspected a set of images from the inner
+4 chips, masking instrumental artifacts by hand.  Bill then processed
+only the data from these 4 chips through Magic, and found a high, but
+nearly acceptable number of streaks:  There were 223 streaks in just
+the 4 chips, masking out a substantial fraction of the area of the
+chips.  
+
+We then made several improvements to reduce the number of artifacts in
+the automatic processing:
+
+# We updated the static bad-pixel mask, being much more aggressive about excluding structures caused by hot pixels and other areas of dark-current glow.  We especially concentrated on those features which resulted in long, linear artifacts.  This work was completely manual and slow, so we started with the inner chips and worked our way out. This let us test the result quickly before committing to the complete mask generation effort.  
+# We added a stage in the subtraction code to remove a 2D model of the background flux.  These background variations could be due, for example, to poorly subtracted dark current or gradients in the sky.
+# We added code to construct masks of (most of) the features caused by bright stars.  The mask is generated based on the astrometry / photometry reference catalog, and converted to pixel coordinates based on the astrometric solution.  Currently, a circular mask is drawn for bright stars (radius scaled by the brightness), a cross-hair is drawn for the diffraction spikes (also scaled by brightness), and a path is drawn along the readout direction to mask the trailed charged (width scaled by brightness).  
+# We added in a feature to detect and mask, somewhat crudely, the structures caused by the residual charge.  
+
+By Monday, we were able to generate difference images using all of
+these new mask improvements for the inner 16 chips (4x4).  For this
+portion of the camera, we now found roughly 50 streaks, compared with
+over 220 streaks for 1/4 of the area in the previous iteration.
+Tuesday, we extended the analysis to the inner 36 chips (6x6), and
+found that the total number of streaks only increased to about 100,
+consistent with the increase in area.   
+
+Paul Sydney has been tasked with examining the images by eye to
+confirm the success of Magic.  As he gains confidence in the analysis,
+he will be able to authorize the automatic release of the data.  We
+hope to receive the first set of manually inspected images Thursday,
+or possibly Friday.
+
+We will pursue two paths in parallel to increase the amount of stacked
+/ magiced data that is released to the consortium:
+
+# We will process more types of fields using only the inner 16 chips.  This will make it feasible for Paul Sydney to examine the data in a reasonable amount of time, and will allow us to provide examples of the M31 observations, the STS obervations, and one of the Medium Deep observations 
+# We will confirm that Magic can be applied to data from the full camera, and request Paul Sydney to validate at least one full GPC1 exposures.
+
+Although we cannot yet release the magic-masked image, we can show an
+example of the resulting mask.  The image below is a fraction of the
+GPC1 field of view, showing three skycells with the magic mask
+applied.  The streaks detected by magic are the long diagonal lines at
+about 20 degrees.  The small grid patterns is the gaps between cells
+and the large grid is the gaps between chips.  The cross-hairs are the
+masks of the diffraction spikes.  Note that this image has been binned
+4x4 relative to the initial skycells.
+
+The fits file may be downloaded here: [http://kiawe.ifa.hawaii.edu/eugene/downloads/magic.mask.fits Magic Mask]
+
+[[Image(htdocs:/images/Magic.mask.jpg)]]