Summary of MDdeeptest_pixrej_20140403

(back to MediumDeepFields summary)

For a while it has been bothersome to find junk in the MD stacks through just simple visual inspection as would not be expected in N=60 (refstack) and definitely not for N>>60 (deepstack), and they can also come and go depending on the inputs used. This seems to be at least configs needing tuning, at worst code bugs.

A related issue also to be addressed is the ratty edge of the MD stacks due to poorer edge chips and smaller number of input pixels. Possible alternatives for a solution are

• hard cut by radius field by field and filter -- difficult to implement
• combine MD+3PI to help improve by including better 3PI pixels -- non-trivial setup and unknown configs for both types of data sets
• only include pixel if N input > some minimum, similar to the SAFE (N=2) option but with a larger N minimum -- edge may vary greatly between filters
• often edge is dominated by SUSPECT mask pixels, set those to bad and reject -- potential to impact entire skycell (may be good since SUSPECT may actually be bad)
• any others options?

---

Example Junk and Bugs

Large regions of high level, bad pixels (s065i -- diffract spike chunks [1129,4256], the moth []) -- bug

img

mk

wt

num

exp

expwt

• due to NAN/0 bug from unprotected denominator being 0, setting mean and sigma to NAN (then all inputs used...)

Excess hot pixels/CR-like (s065i [4099,4651],[4752,4853]) -- bug

img

• due to bug shown after NAN/0 bug fixed, sigma sometimes being set to a negative value resulting in final mean and sigma going NAN (then all inputs used...)

Large region of low level, bad pixels (s013g []) -- bug? much can be reduced by config mods

img

mk

num

wt

• greatly reduced by lowering THRESHOLD.MASK 0.5->0.01 (may need further modifications)

Edge pomegranate-like webbing (s013g [], md09s012g extensive) -- bug? much can be reduced by config mods

img

mk

num

wt

• greatly reduced by lowering THRESHOLD.MASK 0.5->0.3 (may need further modifications, to 0.01 even better)

• md09s12g example -- similar or different case?

Misc edge junk -- add 3PI or impose a Nin miminum per pixel?

Specially skipped edge junk -- strip along skycell with fair amount of junk ignored, 2*kernel size..

---

Test Samples

All should be available via the PSS, if large or full sets are desired then can be push to the datastore (let us know)

MD04 PV2 warps -- MD04.cut2009deeptest.20131103 for manual testing, the full test runs below used all available PV2 through the end of the survey (MD.PV2)

• label: MD04.pv2_stacktest.20130903

3PI exposures overlapping with MD04 -- span Apr. 2009 to Feb. 17 2014, however any exposures before June 2010 will be excluded just like the MD04 exposures (i.e., cut2009 condition described at MD04.pv2_stacktest.20130903)

• label: MD04.test3pistack.20140217 --
• (need to add total and used numbers)

MD04+3PI overlap test with just the bug fixes to compare the edge quality -- will include additional PV2 warps through end of survey for MD04, but just 3PI though Feb. 2014

• label: MD04_3PI.cut2009teststack.20140404 -- g-band only

MD04+3PI overlap test with bug fixes and config mods to compare the edge quality -- will include same available warps

• label: MD04_3PI.cut2009teststack.20140411 -- again g-band only
• THRESHOLD.MASK 0.01
• COMBINE.REJ 2.5

MD04 with bug fixes, config mods, N>20 inputs per pixel to clean up the field edge -- can compare to MD04.cut2009deeptest.20131103 since PV2 inputs limited by date here still through end of Sept. 2013

• label: MD04.cut2009n20deeptest.20140413 -- i-band only
• THRESHOLD.MASK 0.01
• COMBINE.REJ 2.5

MD04 like above except N>30 inputs per pixel but other configs not changed as much --

• label: MD04.cut2009n30deeptest.20140416 -- i-band only
• THRESHOLD.MASK 0.3
• COMBINE.REJ 3.5

---

Configuration Mods

This should be done properly before the final version, but will take some time to do. Config params that impact the pixel rejection

COMBINE.ITER    F32     0.5             # Number of rejection iterations per input
COMBINE.REJ     F32     3.5             # Rejection threshold in combination (sigma)
COMBINE.SYS     F32     0.1             # Relative systematic error in combination
COMBINE.DISCARD F32     0.2             # Discard fraction for Olympic weighted mean

MASK.VAL        STR     MASK.VALUE,CONV.BAD,GHOST # Mask value of input bad pixels
MASK.SUSPECT    STR     SUSPECT,BURNTOOL,SPIKE,STREAK,STARCORE,CONV.POOR # Mask value of suspect pixels
MASK.BAD        STR     BLANK           # Mask value to give bad pixels
MASK.POOR       STR     CONV.POOR       # Mask value to give poor pixels

POOR.FRACTION   F32     0.25            # Maximum fraction of bad variance for poor pixels
THRESHOLD.MASK  F32     0.5             # Threshold for mask deconvolution (0..1)
IMAGE.REJ       F32     0.1             # Rejected pixel fraction threshold for rejecting entire image
VARIANCE        BOOL    TRUE            # Use variance in rejection?
SAFE            BOOL    TRUE            # Play safe when combining small number of values?


-- NINMIN -- some Nin minimum pixel threshold


Unclear how a mix of MD+3PI will be impacted by config mods... a strong reason to not make MD+3PI stacks yet.