Changes between Initial Version and Version 1 of GP1

Timestamp:

Nov 1, 2010, 9:41:01 AM (16 years ago)

Author:

chambers

Comment:

--

GP1

@@ +1 @@
+=  Gene 1 =
+
+{{{
+
+Oct 5
+Here are some notes related to the Great Reprocessing which will be
+useful for discussion today.
+
+In terms of IPP issues, I believe the following are needed before the
+reprocessing can start:
+
+1) new hardware delivered and installed (shipped late last week,
+expected in 2-3 weeks according to Freddie)
+
+2) star/galaxy/CR separation finalized.  While I was in Harvard, I got
+Eddie Schlafly up to speed on this issue.  I gave him a collection of
+SMF files in the latest format that includes all of the parameters
+which I am currently trying to use to make the star / galaxy / CR
+separation.  These exposures overlap SDSS, so he is able to use that
+as ground truth.  He is looking at the parameters I measure (Mxx, Myy,
+PSF mag, Kron mag, other aperture-like mags) and attempting to define
+the cuts in this multi-dimensional space that define stars, galaxies,
+and CRs.  He is first working to define the a-priori probability that
+a given object is consistent with a PSF.  Next, I've asked him to work
+with Shaun Cole on defining the a-posteriori probabilities (star,
+galaxy) given some other information (specifically galactic latitude
+and longitude).  He has already made some headway in better defining
+the CRs from the faint stars.  There is some subtlety here that I've
+asked him to explore: is it enough to define the probability of
+something being a star (PSF) using on the locations of stars in the
+space, or is it necessary to include some description of the shape
+distribution of the galaxies themselves.
+
+3) decisions / implications to difference image analysis from
+photfest.  The big advance over last week was that the CfA group now
+agree that ppSub SINGLE direction convolution produces results
+indistinguishable from the photpipe difference image analysis.  They
+also convinced me that, for the stack-stack diffs where we have a deep
+reference stack, it is acceptable to always choose the deep reference
+stack as the convolution target -- the occasional instances of small
+amounts of deconvolutions are acceptable given the high S/N of the
+reference stack and the over-sampling.  We also agreed that the DUAL
+convolution is at worst equivalent to an optimal SINGLE convolution
+plus an additional smoothing, and it may be the case that the DUAL
+convolution is acceptable as well.  In the near term, we are going to
+switch the MD stack-stack diff processing to use SINGLE convolution of
+the refstack, and the CfA group will start to use at least the IPP
+pixels.  We (mostly they -- Mark Huber, Micheal Wood-Vasey, and Ken
+Smith) are looking at the detections produced by the IPP, along with
+the feedback I gave them on the flags and exclusions, to see if (with
+appropriate filtering) the false positives and false negatives are
+acceptable and if the photometry is is consistent with the photpipe
+equivalent.  The quick look at the last point was positive, but they
+are doing more checks on this.
+
+As far as the Great Reprocessing goes, the main lesson & outstanding
+point from the photfest was the remaining concern that DUAL
+convolution STILL results in more smoothing than one would expect or
+hope.  Since DUAL is required for the warp-warp diffs (ie, 3pi and
+SS), it is important to continue to push on this.  Two possible
+avenues: 1) the range of kernels which we use is not as large as CfA:
+we have a set of 3 Gaussians which we scale by the FWHM, while Armin
+adds an additional large Gaussian if the FWHM gets too large.  2) the
+DUAL convolution could potentially be made to fall-over to SINGLE
+convolution if only one of the two images is convolved.  This is
+tricky to determine, but a bit of work on that may be useful.
+
+4) address recent large variations in processing throughput (perhaps
+due to nebulous mysql fragmentation?).  Serge & Roy have learned that
+running the mysql optimization process may help to speed up the
+nebulous interactions; Serge is investigating this and will also look
+into partitioning the big nebulous table.  The throughput variations
+can be a factor of 3.5!
+
+5) updates to non-linear correction, dark correction, and resulting
+mask modifcations.
+
+6) a full processing strategy (what area first, exactly what
+processing is done) and any resulting scheduling code changes (ie,
+updates to the nightly science code to manage the processing).
+
+7) astrometric and photometric reference catalog for the portion of
+the sky that is to be first processed.
+
+It would be nice, but is not required, that the 3pi static sky
+analysis be finalized, and that the IPP->PSPS interface be finished.
+It is also important to have a plan for the storage upgrades, but we
+have enough space to start.
+
+]]]