
I've been running the detrend analysis on the new gpc1 flats, and
discovered an important issue.  These chips have significant gain
variations across the devices.  

We need to have detrend.normstats run on the results of the
detResidImfiles, rather than detProcessedImfiles.  This requires
altering the processing sequence from this:

* process each of the input images (detrend.process.imfile)
* generate a stack for each chip (detrend.stack.imfile)
* use the stats from the processed chips to determine per-chip
renormalizations for the stacks. the goal in this step is to yield a
flat-field image for which a flattened input image is flat across the
entire mosaic, not just each chip.  (detrend.normstats.imfile)
* apply the normalization to each of the stacks (detrend.norm.imfile)
* generate mosaic-wide stats and jpegs for the normalized flats
(detrend.norm.exp)
* apply the normalized flats to the processed images (detrend.resid.imfile)
* generate per-exposure stats and jpegs for the residual images.
(detrend.resid.exp)

The problem is that, with a large gain variation across each chip, the
measurement of the per-chip stats from the processed images is not a
good measurement of the flattened mean value on that chip.  We really
should be measuring the statistics from the flattened images.  Paul
and I discussed the pipeline organization and have concluded that we
can re-work the detrend pipeline without too much work to achieve this
if we perform the flat-fielding before the normalization, and do the
following steps:

* process each of the input images (detrend.process.imfile)
* generate a stack for each chip (detrend.stack.imfile)
* apply the flats to the processed images (detrend.resid.imfile)
* use the stats from the residual chips to determine per-chip
renormalizations for the stacks (detrend.normstats.imfile)
* apply the normalization to each of the stacks (detrend.norm.imfile)
* generate mosaic-wide stats and jpegs for the normalized flats
(detrend.norm.exp)
* generate per-exposure stats and jpegs for the residual images,
applying the normalizations on-the-fly. (detrend.resid.exp)

--

Tasks to do:

* ppStack could do an approximate normalization on the fly (perhaps
  not needed; perhaps already done? what is the output scaling of the
  resulting flat stack?)

* dettool -toresidimfile should trigger on completed detStackedImfile
  (not detNormalizedImfile) (DONE)

* dettool -tonormalizedstat should trigger on completed
  detResidImfile, not detStackedImfile (DONE)

* detrend_norm_calc.pl needs to read the stats from detResidImfile,
  not detProcessedImfile (DONE)

* detrend.norm.imfile uses the results of detrend_norm_calc.pl as
  before (DONE)

* detrend.resid.exp applies the normalizations on-the-fly to the b1
  and b2 input images, and to the exposure-wide stats pushed into the
  db. (DONE)

* detrend.resid.exp blocks until detrend.norm.exp is done

*** the 'verify' analysis needs to be modified.  I think we can keep
    it as it was (detrend.process.imfile -> detrend.resid.imfile ->
    detrend.resid.exp) where the resid imfile is generated using the
    detNormalizedImfile rather than the detStackedImfile.  Does
    detrend.resid.exp need to be modified for 'verify' mode to *avoid*
    applying the normalization?


