Summary for MD.PV1 (NOT yet finalized)

• MediumDeepFields summary page -- quick links to all up to date refstacks
• MD.GR0 summary page -- summary of all reprocessing details and 2011/2012 refstacks and some deepstack samples

PV1 processing and (re)reprocessing is defined for MD based on the start date of 2012-05-16 to include improved and new detrend types (GPC1_Detrend_Documentation) and corrections for the faint-end bias photometry (ipp-20120404,r33642 tag -- details related to psphot elsewhere..). While detrends have been updated since that date, the changes are minor and should not impact the MD processing. Also, even though the latest date for the new VideoMask detrend is 2012-05-03, the date of 2012-05-16 was chosen as the day after the summary of the detrend updates had been established.

Many of the details here are also taken from the MD.GR0 page, see that page for more information. In general, all the MD fields required at least partial re-reprocessing of chip to warp except MD09 for which MD.GR0 reprocessing started at the beginning of June 2012 (including the refstack and night stacks). All MD fields now have refstacks created after the 2012-05-16 date. Only MD09 has a complete set of PV1 night stacks, all other fields will have a mixture of pre and post-PV1 night stacks (post-PV1 will generally be the normal nightly science processing of night stacks and not from MD.GR0).

Reprocessed Data Products

Each MD field has a variety of issues and details are summarized in each MD section of MD.GR0 or later in the relevant refstack wiki page (see links from MediumDeepFields). Label/data_group names have also slowly simplified over time and the appropriate MD section should be referred to if those must be be used. It is recommended to add the use of the <stage>_id to select appropriate data products and each section below will provide the number after which the PV1 definition should be valid for.

It is important to make clear that the x.PV1.x label/data_group/workdir for MD was a sub-re-reprocessing of exposures that were reprocessed or just processed before date X (now set at 5/16/2012). There have been various other re-reprocessing and reprocessing of exposures (for recent refstacks, GR0 late 2012 night stack completion, etc *) that have happened after that date. This means there can also be duplicate processing of exposures with valid <stage>_id. Duplicates were allowed because initially the exact date for PV1 was unclear and can also provide an old/new valid reprocessed exposure comparison (i.e., has a change in the ipp-X tag caused problems). When PV1 re-reprocessing was done, MD fields that ended their observing season around this time (i.e., MD05,06,07) were reprocessed to finish out their observing season for some fields since the date was initially uncertain for PV1. Again, either should be acceptable to use, and comparison of original and newer may be a helpful.

* also be aware of -- reprocessing of exposures for old format V2 exist, those were ignored and re-reprocessed for V3 warps and stacks later on. If interested in the warp and later products, verification of the V3 format should be done.

Chip, Warp

Fully reprocessed but not distributed due to disk space load. These can be obtained via the Postage Stamp Server

• GR0 was a mix of synthetic/ubercal catalog calibrations except MD08,09
• with PV1 chip_id, warp_id >= 433987, 393977 respectively, all processing using the 20120524v0 ubercal catalog and valid detrends

Camera/SMFs

Fully reprocessed and distributed

• GR0 was a mix of synthetic/ubercal catalog calibrations except MD08,09
• with PV1 cam_id>=434615 respectively, all processing using the 20120524v0 ubercal catalog and valid detrends

Stacks

Due to there being different types, each has its own sub-section. Generally, if a stack_id>=848641 then it was likely made from exposures processed using the 20120524v0 ubercal catalog (except some more recent night stacks were made using complete sets of old warps such as in the initially incomplete sets of MD01,02,03,05).

Night Stacks: All warps from a night (re)processed and stacked into a night stack

• although the GR0 sample of stacks can be a mix of synthetic/ubercal catalog calibrations for the warp pixels, the warps going into the stack should be of the same type of calibration
• for PV1 there is no simple, unique method for selecting other than manually creating a release listing or using the gpc1 DB stackInputSkyfile table for input warp_id

Reference Stacks (refstack): best exposure reprocessing prior to the start of the MD field observations for the season to support the difference image products. The focus of MD.GR0 was on the 2011/2012 season fields starting with MD10, then 01, 02 ... through MD09. The 2012/2013 season refstacks replace the 2011/2012 set and have their own page linked from MediumDeepFields.

• out-of-season/anti-yearly refstacks, yearly refstack: on hold for MD.GR1 (would need to update/reprocess large numbers of exposures to do)
• PV1 stack_id>= and/or label MD##.refstack should be valid, data_group of form MD##.refstack.YYYYMMDD

Deep Stacks (deeptest): Not a full production product yet. Test version exists for MD04.deeptest.20120727 (synthetic catalog, pre-PV1 warps only), MD09.deeptest.20120727 (ubercal catalog 20120524v0, PV1 warps only). Like yearly or out-of-season stacks, would required much update processing to make for other fields. Looking at updating MD04 deep stack and adding MD07 to the test set however.

(best) IQ/PSF Stack: on hold until next reprocessing round (should be done with refstacks to avoid excess processing). Test version exists for MD02.iqtest.20120927, MD10.psfrefstack.20110814, but neither are made from PV1 warps. A sample could be done for MD09 done but requires development/code time.

Difference Images

Originally a possible reprocessing product, but was decided not useful at this time (for future reprocessing will need to clearly outline interests for these, if any). Open items to address still -- finish the convolution choice (input stack or reference stack) and improvements to convolution.

• diff_id>= will be valid PV1 from nightly science processing

Stack Photometry (Static Sky + Skycal)

Stack photometry (staticsky) and recalibration (skycal) is run on refstacks, deepstacks and night stacks (currently on a partial set of MD09.GR0 only). These often get reprocessed uniformly on new stacks or when/if bugs found and fix, latest one was January 2013, and so label/data_group is more direct for identification.

• recalibration of the stack photometry catalogs has no method for distribution -- email for access

Re-revised planned loading into PSPS (March 2013) -- do for all the newer refstacks, MD09 deeptest, MD09.GR0 night stacks

• ref stack label, data_groups: includes 5-1 filter sets now under same label, staticksky data_group will indicate <N> for # filters attempted, skycal data_group will include all under one data_group

  label staticsky:  MD09.refstack.20120831.Nx20130115 -- run on refstack data_group MD09.refstack.20120831, label for active refstacks is changed to MD<##>.refstack generically
  data_group staticsky:  MD09.refstack.20120831.<N>x20130115 -- <N> = 1,2,3,4,5 and number of filters available for the set, so 5 different data_group per MD
  label=data_group skycal:  MD09.refstack.20120831.Nx20130115.cal20130128 -- all N filter set is run under one data_group now per MD
  
  label staticsky:  MD01.refstack.20120803.Nx20130115 -- skycal label=data_group, add .cal20130128 
  label staticsky:  MD02.refstack.20120927.Nx20130115 -- skycal label=data_group, add .cal20130128
  label staticsky:  MD03.refstack.20121101.Nx20130325 -- skycal label=data_group, add .cal20130326
  label staticsky:  MD04.refstack.20121125.Nx20130325 -- skycal label=data_group, add .cal20130326
  label staticsky:  MD05.refstack.20121202.Nx20130325 -- skycal label=data_group, add .cal20130326
  label staticsky:  MD06.refstack.20121221.Nx20130325 -- skycal label=data_group, add .cal20130326
  label staticsky:  MD07.refstack.20130102.Nx20130325 -- skycal label=data_group, add .cal20130326
  label staticsky:  MD08.refstack.20130401.Nx20130407 -- skycal label=data_group, add .cal20130408
  

• deep stack label, data_groups: includes 5-1 filter sets now under same label, staticsky data_group will indicate N-filter, skycal data_group all under one name -- only for MD09 at this time

  label=data_group skycal:  MD09.deeptest.20120727.Nx20130114.cal20130128 -- all N filter set is run under one data_group now per MD
  

• night stack label, data_group:

  staticsky label = data_group:  MD09.GR0.20120602.nightstack.1x20120926rawtest
  skycal label = data_group:  MD09.GR0.20120602.nightstack.1x20120926rawtest.cal20130128 
  

Summary Table by MD Field

(revised Apr. 19, 2013)

MD

chip/warp_id

Camera/SMF_id

Reference Stack

Deep Stack

Night Stack

Refstack Phot

Deep Phot

Night Phot

for PSPS

MD	Reference Stack	Deep Stack	Night Stack	Refstack Phot	Deep Phot	Night Phot	Camera/SMFs	for PSPS
01	20110820 (U)	X	done (mixed)	.20120803.Nx20130115, .cal20130128	x	x	done (mixed)	SMF+ref phot TBD
02	20110907 (U)	X	done (mixed)	.20120927.Nx20130115, .cal20130128	x	x	done (mixed)	SMF+ref phot TBD
03	20111010 (U)	X	done (mixed)	.20111010.Nx20130115, .cal20130128 (I)	x	x	done (mixed)	SMF+ref phot TBD
04	20111012 (U)	deeptest.20120727	done	.20111012.Nx20130115, .cal20130128	.Nx20130114, .cal20130128	TBD	done	SMF+ref+deep+night phot TBD (secondary)
05	20111014 (U)	X	done (mixed+full)	.20111014.Nx20130115, .cal20130128 (I)	x	x	done (mixed)	SMF+ref phot TBD
06	20111122 (U)	X	done	.20111122.Nx20130115, .cal20130128	x	x	done	SMF+ref phot TBD
07	20111106 (U)	X	done	.20111106.Nx20130115, .cal20130128	x	x	done	SMF+ref phot TBD
08	20120422	X	done	.20120422.Nx20130115, .cal20130128	x	x	done	SMF+ref phot TBD
09	20120831 (R)	deeptest.20120727	done	.20120831.Nx20130115, .cal20130128	.Nx20130114, .cal20130128	.1x20120926rawtest, .cal20130128	done	SMF+ref+deep+night phot (primary)
10	20110821 (U)	X	done	.20120804.Nx20130115, .cal20130128	x	x	done	SMF+ref phot TBD
11	not reprocessed in MD.GR0, PV1

Notes:

• TBD=To Be Decided/Done by priority, X=not planned, ?=unknown at this time
• 20YYMMDD date entries are part of the data_group name to access via the PSS or Datastore, i.e.,
  • stacks -- MD01.refstack.20110820, MD09.deeptest.20120727
  • night stacks -- a mix of label, data_group formats, see specific section
  • stack photometry -- originally the same date 20120805, but revised runs now have dates for each in the form adding for staticsky .<#>x20YYMMMDD and skycal .cal20YYMMDD.
• (U) updated -- new 2012/2013 versions exist for these refstack now (see MediumDeepFields) but not necessarily used for stack photometry (staticsky/skycal) for loading into PSPS (original MD.GR0 may be deeper and better to use). The refstack used is indicated in the Stack Photometry column. All updated refstacks of course use the ubercal catalog for calibration.
• (R) replaced -- MD09.refstack.20120603 replaced by 20120831 due to variance problems.
• (I) incomplete -- for MD03, 05 stack photometry stalled while data disks being shuffled..
• (mixed)
  • finish reprocessing all exposures to V3 for warp/night stack for MD01+02+03+05 =840+609+1286+1115 = 3850 (2-3 full days of --only-- MD processing) -- finished 12/30/13
  • (mixed) -- both camera/SMF and night stacks can have calibrations based on the synthetic catalogs (initial sample run) and ubercal catalogs (finish V3 run). See specific section for detail.
  • (mixed+full) -- MD05 -- some older and newer processing, so mix of magic and photometry catalog used and then a re-reprocessing of MD05 was done on the exposures originally processed with synthetic catalog. See MD.GR0
• stacks
  • out-of-season/anti-yearly refstacks, yearly refstack: on hold for MD.GR1 (would need to update/reprocess large numbers of exposures to do).
  • iq/psf stack on hold until MD.GR1 (like out-of-season, would required many updates) -- should do for MD04,MD09 and requires development/code time.
  • deep stack for MD04,09 for MD.GR0 only, all others on hold until MD.GR1 (many many updates would be necessary..).
• reminder that ubercal of pixels start with MD08 in the date sequence, finished processing of MD01,02,03,05 exposures into V3 will also be ubercal'd (including reprocessing of any needed to make the complete night stacks)

Questions:

• background restored warps/stacks?
• SMF loaded into PSPS limited to just the ones used in the refstack (or deepstack if available) or all available exposures Camera/SMFs?
• MD01,10 have new refstack for this season, need to decide if using for phot+PSPS (limited set of exposures reprocessed recently to make)? Or use as update test for PSPS?
• MD09.refstack.20120603 suffers from the weight image problem making an arc of lower sensitivity, but warps are still available for doing the deep stacks. 4-6 hrs/skycell to run, ~300 skycells, deepstack pantasks typically has 30 nodes, so ~2-3 days processing time, then redo? -- yes, since MD09 is a new test/comparison set with SAS -- done and date updated under refstack as MD09.refstack.20120831