Summary for MD.PV1 (NOT yet finalized)

• MediumDeepFields summary page -- quick links to all up to date refstacks
• MD.GR0 summary page -- summary of all reprocessing details and 2011/2012 refstacks and some deepstack samples

Processing and (re)reprocessing is defined PV1 for MD based on the date of 2012-05-16 to include improved and new detrend types (GPC1_Detrend_Documentation) and corrections for the faint-end bias photometry (ipp-20120404,r33642 tag -- details related to psphot elsewhere..). While detrends have been updated since that date, the changes are minor and should not impact the MD processing. While the latest date for the new VideoMask detrend is 2012-05-03, the date of 2012-05-16 was chosen as the day after the summary of the detrend updates had been established. Many of the details here are also taken from the MD.GR0 page, see that page for more information. In general, all the MD fields required at least partial re-reprocessing of chip to warp except MD09 for which MD.GR0 reprocessing started at the beginning of June 2012 (including the refstack and night stacks). All MD fields have refstacks created after the 2012-05-16 date. Only MD09 has a complete set of PV1 night stacks, all other fields will have a mixture of pre and post-PV1 night stacks (post-PV1 will generally be the normal nightly science processing of night stacks and not from MD.GR0).

Reprocessed Data Products

Each MD field has a variety of issues and details are summarized in each MD section of MD.GR0 or later in the relevant refstack wiki page MediumDeepFields. Label/data_group names have also slowly simplified and the appropriate MD section should be referred to if those must be be used. It is recommended to use the <stage>_id to select appropriate data products and each section below will provide the number after which the PV1 definition should hold for.

It is important to make clear that the x.PV1.x label/data_group/workdir for MD was a sub-re-reprocessing of exposures that were reprocessed or just processed before date X (now set at 5/16/2012). There have been various other re-reprocessing and reprocessing of exposures (for refstacks, GR0 late 2012 etc *) that have happened after that date. It is really the <stage>_id that needs to be used to simply select with, and so there are duplicates of exposures with valid <stage>_id.

The largest <stage>_id of such an exposure set could just be used.

duplicates were allowed because

1: didn't know what the exact date would be 2: can provide an old/new valid reprocessed exposure comparison

* even reprocessing of exposures for old format V2 exist, those were ignored and re-reprocessed for V3 warps and stacks later on. These could also be included in the duplicate case and using the largest <stage>_id in the duplicate set should exclude those if interested, however, if interested in the warp and later products the V3 format should be verified.

Re-reprocessing tried to finish out observing season for some fields since date uncertain for PV1 initially, so will be duplicate processing for the same exposure. Either should be acceptable to use, comparison of original and newer may be a helpful

Chip+Warp

Fully reprocessed but not distributed due to disk space load. These can be obtained via the Postage Stamp Server.

• MD.GR0 was a mix of synthetic/ubercal catalog calibrations except MD08, 09.
• for PV1 chip_id, warp_id >= 433987, 393977 respectively it was processed using the 20120524v0 ubercal catalog and with detrends

Camera/SMFs

Fully reprocessed and distributed

• MD.GR0 was a mix of synthetic/ubercal catalog calibrations except MD08, 09
• a possible guideline to follow, including nightly science data processing, if MD exposures with a cam_id >=412069 (actually >=411444 of some V2 destined reprocessing) is used, those will be with the ipp-20120404,r33642 tag that includes the faint-end bias fix and the ubercal catalog 20120524v0 version also used for LAP.

cam_id>=434615

Stacks

Generally, if a stack_id is >= 848641 then it was likely made from processed exposures using the 20120524v0 ubercal catalog (see night stacks where some were made using old warps if a complete set existed for MD01,02,03,05).

Night Stacks: All nights reprocessed and stacked into a night stack using the current night stack configuration used for nightly science night stacks when processed, including reference catalog

• should attempt to make stack with 2 or more input warps now
• some nights have >8 exposures, often in marginal conditions. Since marginal conditions can vary greatly, some selection should probably be made for inclusion in the night stack.
• the sample of stacks can be a mix of synthetic/ubercal catalog calibrations for the pixels but the warps going into the stack should be of the same type of calibration.

Reference Stacks (refstack): To be made prior to the start of the MD field observations for the season to support the difference image products (magic and science). The focus of MD.GR0 was on the 2011/2012 season fields starting with MD10, then 01, 02 ... through MD09. The 2012/2013 season refstacks will to replace the 2011/2012 set and have their own page linked from MediumDeepFields.

• out-of-season/anti-yearly refstacks, yearly refstack: on hold for MD.GR1 (would need to update/reprocess large numbers of exposures to do)

Deep Stacks (deeptest): Not a full production product yet. Test version exists for MD04.deeptest.20120727 (synthetic catalog, pre-PV1 warps), MD09.deeptest.20120727 (ubercal catalog 20120524v0, PV1 warps)

IQ/PSF Stack: on hold until MD.GR1 (like out-of-season, would required many updates). A sample for MD09 done but requires development/code time. Test version exists for MD02.iqtest.20120927, MD10.psfrefstack.20110814, but neither will be made from PV1 warps.

Difference Images

Originally a possible reprocessing product, but was decided not useful at this time (for future reprocessing, need to outline interests for these if any). Will need to finish the convolution choice (input stack or reference stack) and improvements to convolution before started.

However, diff_id>= will be PV1 from nightly science processing

Stack Photometry (Static Sky + Skycal)

Stack photometry (staticsky) and recalibration (skycal) is run on refstacks, deepstacks and night stacks (currently on a partial set of MD09 only). In trying to run as a uniform group, will keep a summary here, but details like specific faults, poor quality etc will be in the respective MD section when time. For example some skycells can have a clear offset fix to the zeropoint, not actually set to 25.0 in stack pixels, for some the majority of the field is off from zeropoint of 25. There appears to be upwards of 3-5% error likely in the photometry across the FPA (see MD.GR0) even in the recalibration of the stack photometry (skycal stage). A much worse example is seen in MD01.refstack.20120803.

• recalibration of the stack photometry catalogs has no method for distribution -- email for access

Re-revised planned loading into PSPS (March 2013): do the newer refstacks, so the older MD03--MD07 noted before below been have done as follows

label staticsky:  MD04.refstack.20121125.Nx20130325
data_group staticsky:  MD04.refstack.20121125.<N>x20130325 
label=data_group skycal:  MD04.refstack.20121125.Nx20130325.cal20130326

label staticsky:  MD03.refstack.20121101.Nx20130325 -- skycal label=data_group, add .cal20130326
label staticsky:  MD05.refstack.20121202.Nx20130325 -- skycal label=data_group, add .cal20130326
label staticsky:  MD06.refstack.20121221.Nx20130325 -- skycal label=data_group, add .cal20130326
label staticsky:  MD07.refstack.20130102.Nx20130325 -- skycal label=data_group, add .cal20130326

Current run planned for loading into PSPS (January 2013): stack photometry (staticsky) re-run from Summer 2012 set to incorporate improvements/fixes in the Kron code. Photometry zeropoint recalibration (skycal) re-run to incorporate various bug fixes and improved tuning (psastro running on stack catalogs). Several previous re-runs for skycal and settled on .cal20130128 version for now.

• ref stack label, data_groups: includes 5-1 filter sets now under same label, staticksky data_group will indicate <N> for # filters attempted, skycal data_group will include all under one data_group

  label staticsky:  MD09.refstack.20120831.Nx20130115 -- run on refstack data_group MD09.refstack.20120831, label for active refstacks is changed to MD<##>.refstack generically
  data_group staticsky:  MD09.refstack.20120831.<N>x20130115 -- <N> = 1,2,3,4,5 and number of filters available for the set, so 5 different data_group per MD
  label=data_group skycal:  MD09.refstack.20120831.Nx20130115.cal20130128 -- all N filter set is run under one data_group now per MD
  
  label staticsky:  MD04.refstack.20111012.Nx20130115
  data_group staticsky:  MD04.refstack.20111012.<N>x20130115 
  label=data_group skycal:  MD04.refstack.20111012.Nx20130115.cal20130128 
  
  label staticsky:  MD10.refstack.20120804.Nx20130115   (new refstack MD10.refstack.20110821, see [wiki:MD10.refstack.20120705], is deeper and replacing MD.GR0 version MD10.refstack.20110821)
  data_group staticsky: MD10.refstack.20120804.<N>x20130115  
  label=data_group skycal:  MD10.refstack.20120804.Nx20130115.cal20130128 
  
  label staticsky:  MD01.refstack.20120803.Nx20130115  (new refstack [wiki:MD01.refstack.20120803] replacing MD.GR0 version since deeper) -- skycal label=data_group, add .cal20130128 
  label staticsky:  MD02.refstack.20120927.Nx20130115  (new refstack [wiki:MD02.refstack.20120927] replacing MD.GR0 version since deeper) -- skycal label=data_group, add .cal20130128
  label staticsky:  MD03.refstack.20111010.Nx20130115 -- skycal label=data_group, add .cal20130128
  label staticsky:  MD05.refstack.20111014.Nx20130115 -- skycal label=data_group, add .cal20130128
  label staticsky:  MD06.refstack.20111122.Nx20130115 -- skycal label=data_group, add .cal20130128
  label staticsky:  MD07.refstack.20111106.Nx20130115 -- skycal label=data_group, add .cal20130128
  label staticsky:  MD08.refstack.20120422.Nx20130115 -- skycal label=data_group, add .cal20130128
  

• deep stack label, data_groups: includes 5-1 filter sets now under same label, staticsky data_group will indicate N-filter, skycal data_group all under one name

  label staticsky:  MD<##>.deeptest.20120727.Nx20130114 -- <##> = 04, 09 only made so far
  data_group staticsky:  MD<##>.deeptest.20120727.<N>x20130114 -- <N> = 1,2,3,4,5 and number of filters available for the set, so 5 different data_group per MD
  
  label=data_group skycal:  MD09.deeptest.20120727.Nx20130114.cal20130128 -- all N filter set is run under one data_group now per MD
  
  label=data_group skycal:  MD04.deeptest.20120727.Nx20130114.cal20130128 -- all N filter set is run under one data_group now per MD 
  

• night stack label, data_group: only skycal needs to be rerun on problem skycells

  -- staticsky
  label = data_group:  MD09.GR0.20120602.nightstack.1x20120926rawtest
  
  -- skycal 
  label = data_group:  MD09.GR0.20120602.nightstack.1x20120926rawtest.cal20130128 
  

Summary Table

(revised Apr. 19, 2013)

MD

chip/warp_id

Camera/SMF_id

Reference Stack

Deep Stack

Night Stack

Refstack Phot

Deep Phot

Night Phot

for PSPS

MD	Reference Stack	Deep Stack	Night Stack	Refstack Phot	Deep Phot	Night Phot	Camera/SMFs	for PSPS
01	20110820 (U)	X	done (mixed)	.20120803.Nx20130115, .cal20130128	x	x	done (mixed)	SMF+ref phot TBD
02	20110907 (U)	X	done (mixed)	.20120927.Nx20130115, .cal20130128	x	x	done (mixed)	SMF+ref phot TBD
03	20111010 (U)	X	done (mixed)	.20111010.Nx20130115, .cal20130128 (I)	x	x	done (mixed)	SMF+ref phot TBD
04	20111012 (U)	deeptest.20120727	done	.20111012.Nx20130115, .cal20130128	.Nx20130114, .cal20130128	TBD	done	SMF+ref+deep+night phot TBD (secondary)
05	20111014 (U)	X	done (mixed+full)	.20111014.Nx20130115, .cal20130128 (I)	x	x	done (mixed)	SMF+ref phot TBD
06	20111122 (U)	X	done	.20111122.Nx20130115, .cal20130128	x	x	done	SMF+ref phot TBD
07	20111106 (U)	X	done	.20111106.Nx20130115, .cal20130128	x	x	done	SMF+ref phot TBD
08	20120422	X	done	.20120422.Nx20130115, .cal20130128	x	x	done	SMF+ref phot TBD
09	20120831 (R)	deeptest.20120727	done	.20120831.Nx20130115, .cal20130128	.Nx20130114, .cal20130128	.1x20120926rawtest, .cal20130128	done	SMF+ref+deep+night phot (primary)
10	20110821 (U)	X	done	.20120804.Nx20130115, .cal20130128	x	x	done	SMF+ref phot TBD
11	not reprocessed in MD.GR0, PV1

Notes:

• TBD=To Be Decided/Done by priority, X=not planned, ?=unknown at this time
• 20YYMMDD date entries are part of the data_group name to access via the PSS or Datastore, i.e.,
  • stacks -- MD01.refstack.20110820, MD09.deeptest.20120727
  • night stacks -- a mix of label, data_group formats, see specific section
  • stack photometry -- originally the same date 20120805, but revised runs now have dates for each in the form adding for staticsky .<#>x20YYMMMDD and skycal .cal20YYMMDD.
• (U) updated -- new 2012/2013 versions exist for these refstack now (see MediumDeepFields) but not necessarily used for stack photometry (staticsky/skycal) for loading into PSPS (original MD.GR0 may be deeper and better to use). The refstack used is indicated in the Stack Photometry column. All updated refstacks of course use the ubercal catalog for calibration.
• (R) replaced -- MD09.refstack.20120603 replaced by 20120831 due to variance problems.
• (I) incomplete -- for MD03, 05 stack photometry stalled while data disks being shuffled..
• (mixed)
  • finish reprocessing all exposures to V3 for warp/night stack for MD01+02+03+05 =840+609+1286+1115 = 3850 (2-3 full days of --only-- MD processing) -- finished 12/30/13
  • (mixed) -- both camera/SMF and night stacks can have calibrations based on the synthetic catalogs (initial sample run) and ubercal catalogs (finish V3 run). See specific section for detail.
  • (mixed+full) -- MD05 -- some older and newer processing, so mix of magic and photometry catalog used and then a re-reprocessing of MD05 was done on the exposures originally processed with synthetic catalog. See MD.GR0
• stacks
  • out-of-season/anti-yearly refstacks, yearly refstack: on hold for MD.GR1 (would need to update/reprocess large numbers of exposures to do).
  • iq/psf stack on hold until MD.GR1 (like out-of-season, would required many updates) -- should do for MD04,MD09 and requires development/code time.
  • deep stack for MD04,09 for MD.GR0 only, all others on hold until MD.GR1 (many many updates would be necessary..).
• reminder that ubercal of pixels start with MD08 in the date sequence, finished processing of MD01,02,03,05 exposures into V3 will also be ubercal'd (including reprocessing of any needed to make the complete night stacks)

Questions:

• background restored warps/stacks?
• SMF loaded into PSPS limited to just the ones used in the refstack (or deepstack if available) or all available exposures Camera/SMFs?
• MD01,10 have new refstack for this season, need to decide if using for phot+PSPS (limited set of exposures reprocessed recently to make)? Or use as update test for PSPS?
• MD09.refstack.20120603 suffers from the weight image problem making an arc of lower sensitivity, but warps are still available for doing the deep stacks. 4-6 hrs/skycell to run, ~300 skycells, deepstack pantasks typically has 30 nodes, so ~2-3 days processing time, then redo? -- yes, since MD09 is a new test/comparison set with SAS -- done and date updated under refstack as MD09.refstack.20120831