= Summary for MD.PV1 = [[PageOutline]] * [wiki:MediumDeepFields] summary page -- quick links to all up to date refstacks * [wiki:MD.GR0] summary page -- summary of all reprocessing details and 2011/2012 refstacks and some deepstack samples PV1 processing and (re)reprocessing is defined for MD based on the start date of 2012-05-16 to include improved and new detrend types ([wiki:GPC1_Detrend_Documentation]) and corrections for the faint-end bias photometry (ipp-20120404,r33642 tag -- details related to psphot elsewhere..). While detrends have been updated since that date, the changes are minor and should not impact the MD (re)processing. Also, even though the latest date for the new VideoMask detrend is 2012-05-03, the date of 2012-05-16 was chosen as the day after the summary of the detrend updates had been established. Many of the details here are also taken from the [wiki:MD.GR0] page, see that page for more information. In general, all the MD fields required at least partial re-reprocessing of chip to warp except MD09 for which MD.GR0 reprocessing started at the beginning of June 2012 (including the refstack and night stacks). All MD fields now have refstacks created after the 2012-05-16 date. Only MD09 has a complete set of PV1 night stacks, all other fields will have a mixture of pre and post-PV1 night stacks (post-PV1 will generally be the normal nightly science processing of night stacks and not from MD.GR0). = Reprocessed Data Products = Each MD field has a variety of ''issues'' and details are summarized in each MD section of [wiki:MD.GR0] or later in the relevant refstack wiki page (see links from [wiki:MediumDeepFields]). Label/data_group names have also slowly simplified over time and the appropriate MD section should be referred to if those must be be used. It is recommended to add the use of the _id to select appropriate data products and each section below will provide the number after which the PV1 definition should be valid for. It is important to make clear that the x.PV1.x label/data_group/workdir for MD was a sub-re-reprocessing of exposures that were reprocessed or just processed before date X (now set at 5/16/2012). There have been various other re-reprocessing and reprocessing of exposures (for recent refstacks, GR0 late 2012 night stack completion, etc ***) that have happened after that date. This means there can also be duplicate processing of exposures with valid _id. Duplicates were allowed because initially the exact date for PV1 was unclear and can also provide an old/new valid reprocessed exposure comparison (i.e., has a change in the ipp-X tag caused problems). When PV1 re-reprocessing was done, MD fields that ended their observing season around this time (i.e., MD05,06,07) were reprocessed to finish out their observing season for some fields since the date was initially uncertain for PV1. Again, either should be acceptable to use, and ''a comparison of original and newer would probably be a helpful''. *** also be aware of -- reprocessing of exposures for old format V2 exist, those were ignored and re-reprocessed for V3 warps and stacks later on. If interested in the warp and later products, verification that the data product is of the V3 format should be done. == Chip, Warp == Fully reprocessed but not distributed due to disk space load. These can be obtained via the Postage Stamp Server * GR0 was a mix of synthetic/ubercal catalog calibrations except MD08,09 * PV1 and GR0 with chip_id, warp_id >= 457521, 416815 respectively, all processing using the 20120524v0 ubercal catalog, valid detrends and ipp-tag == Camera/SMFs == Fully reprocessed and distributed * GR0 was a mix of synthetic/ubercal catalog calibrations except MD08,09 * PV1 and GR0 with cam_id>=434615 respectively, all processing using the 20120524v0 ubercal catalog and valid detrends == Stacks == Each type of stack has its own sub-section below -- generally, if a stack_id>=923376 then it was ''likely'' made from exposures processed with PV1 requirements using the 20120524v0 ubercal catalog, detrends and ipp-tag (except some more recent night stacks were made using complete sets of old warps such as in the initially incomplete sets of MD01,02,03,05). __Night Stacks (nightstack)__: All warps from a night (re)processed and stacked into a night stack * the GR0 sample of stacks is complete but can be a mix of synthetic/ubercal catalog calibrations for the warp pixels. The warps going into the stack should be of the same type of calibration. * for PV1 there is no simple, unique method for selecting other than manually creating a release listing or using the gpc1 DB stackInputSkyfile table for input warp_id __Reference Stacks (refstack)__: best exposure reprocessing prior to the start of the MD field observations for the season to support the difference image products. The focus of MD.GR0 was on the 2011/2012 season fields starting with MD10, then 01, 02 ... through MD09. The 2012/2013 season refstacks replace the 2011/2012 set and have their own page linked from [wiki:MediumDeepFields]. * out-of-season/anti-yearly refstacks, yearly refstack: on hold for next reprocessing round (would need to update/reprocess large numbers of exposures to do) * PV1 stack_id>=923376 and/or label MD##.refstack should be valid, data_group of form MD##.refstack.YYYYMMDD __Deep Stacks (deeptest)__: Not a full production product yet. Test version exists for MD04.deeptest.20120727 (synthetic catalog, pre-PV1 warps only), MD09.deeptest.20120727 (ubercal catalog 20120524v0, PV1 warps only from pre-2012 season). Like yearly or out-of-season stacks, would required much update processing to make for other fields. Looking at updating MD04 deep stack and adding MD07 to the test set however. __best IQ/PSF Stack (none)__: on hold until next reprocessing round (should be done with refstacks to avoid excess processing). Test version exists for MD02.iqtest.20120927, MD10.psfrefstack.20110814, but neither are made from PV1 warps. A sample could be done for MD09 done but requires development/code time. == Difference Images == Originally a possible reprocessing product, but was decided not useful at this time (for future reprocessing will need to clearly outline interests for these, if any). Open items to address still -- finish the convolution choice (input stack or reference stack) and improvements to convolution. * diff_id>=240039 will be valid PV1 from nightly science processing == Stack Photometry (static sky + skycal) == Stack photometry (staticsky) and recalibration (skycal) is run on refstacks, deepstacks and night stacks (currently on a partial set of MD09.GR0 only). These often get reprocessed uniformly on new stacks or when/if bugs found and fix, latest one was January 2013, and so label/data_group is more direct for identification. * '''recalibration of the stack photometry catalogs has no method for distribution''' -- email for access Re-revised planned loading into PSPS (March 2013) -- do for all the newer refstacks, MD09 deeptest, MD09.GR0 night stacks * ref stack label, data_groups: includes 5-1 filter sets now under same label, staticksky data_group will indicate for # filters attempted, skycal data_group will include all under one data_group {{{ label staticsky: MD09.refstack.20120831.Nx20130115 -- run on refstack data_group MD09.refstack.20120831, label for active refstacks is changed to MD<##>.refstack generically data_group staticsky: MD09.refstack.20120831.x20130115 -- = 1,2,3,4,5 and number of filters available for the set, so 5 different data_group per MD label=data_group skycal: MD09.refstack.20120831.Nx20130115.cal20130128 -- all N filter set is run under one data_group now per MD label staticsky: MD01.refstack.20120803.Nx20130115 -- skycal label=data_group, add .cal20130128 label staticsky: MD02.refstack.20120927.Nx20130115 -- skycal label=data_group, add .cal20130128 label staticsky: MD03.refstack.20121101.Nx20130325 -- skycal label=data_group, add .cal20130326 label staticsky: MD04.refstack.20121125.Nx20130325 -- skycal label=data_group, add .cal20130326 label staticsky: MD05.refstack.20121202.Nx20130325 -- skycal label=data_group, add .cal20130326 label staticsky: MD06.refstack.20121221.Nx20130325 -- skycal label=data_group, add .cal20130326 label staticsky: MD07.refstack.20130102.Nx20130325 -- skycal label=data_group, add .cal20130326 label staticsky: MD08.refstack.20130401.Nx20130407 -- skycal label=data_group, add .cal20130408 label staticsky: MD09.refstack.20120831.Nx20130115 -- skycal label=data_group, add .cal20130128 }}} * deep stack label, data_groups: includes 5-1 filter sets now under same label, staticsky data_group will indicate N-filter, skycal data_group all under one name -- only for MD09 at this time {{{ label=data_group skycal: MD09.deeptest.20120727.Nx20130114.cal20130128 -- all N filter set is run under one data_group now per MD }}} * night stack label, data_group: {{{ staticsky label = data_group: MD09.GR0.20120602.nightstack.1x20120926rawtest skycal label = data_group: MD09.GR0.20120602.nightstack.1x20120926rawtest.cal20130128 }}} = PV1 Summary Table by MD Field = (revised Apr. 21, 2013) For all fields, the _id cut necessary for a processing to be PV1 * chip_id>=457521 * cam_id>=434615 * warp_id>=416815 * diff_id>=240039 (not really reprocessed, so most are just recent nightly processing data products) * stack_id>=923376 (not uniquely PV1 for the night stacks and warp_id from stackInputSkyfile should be checked) || '''MD''' || '''Reference Stack''' || '''Deep Stack''' || '''Night Stack''' || '''Refstack Phot''' || '''Deep Phot''' || '''Night Phot''' || '''for PSPS''' || || || || || || || || || || 01 || [wiki:MD01.refstack.20120803 20120803] || x || (mixed) || .Nx20130115.cal20130128 || x || x || SMF+refstack phot || || 02 || [wiki:MD02.refstack.20120927 20120927] || x || (mixed) || .Nx20130115.cal20130128 || x || x || SMF+refstack phot || || 03 || [wiki:MD03.refstack.20121101 20121101] || x || (mixed) || .Nx20130325.cal20130326 || x || x || SMF+refstack phot || || 04 || [wiki:MD04.refstack.20121125 20121125] || deepstack.YYMMDD TBD || (mixed) || .Nx20130325.cal20130326 || ''TBD'' || x || SMF+ref+deep phot (secondary) || || 05 || [wiki:MD05.refstack.20121202 20121202] || x || (mixed) || .Nx20130325.cal20130326 || x || x || SMF+refstack phot || || 06 || [wiki:MD06.refstack.20121221 20121221] || x || (mixed) || .Nx20130325.cal20130326 || x || x || SMF+refstack phot || || 07 || [wiki:MD07.refstack.20130102 20130102] || deepstack.YYMMDD TBD|| (mixed) || .Nx20130325.cal20130326 || ''TBD'' || x || SMF+ref+deep phot (secondary) || || 08 || [wiki:MD08.refstack.20130401 20130401] || x || (mixed) || .Nx20130408.cal20130408 || x || x || SMF+refstack phot || || 09 || [wiki:MD.GR0#MD09.V3referencestack 20120831] || deeptest.20120727 (TBRD) || (PV1) || .Nx20130115.cal20130128 || .Nx20130114.cal20130128 || .1x20120926rawtest.cal20130128 || SMF+ref+deep+night phot (primary) || || 10 || [wiki:MD10.refstack.20120705 20120821] || x || (mixed) || .Nx20130115.cal20130128 || x || x || SMF+refstack phot || || 11 || x || x || x || x || x || x || x || || || || || || || || || __Notes__: * TBD=To Be Decided/Done by priority, X=not planned, ?=unknown at this time, TBRD=To Be ReDone * 20YYMMDD date entries are part of the data_group name to access via the PSS or Datastore (note some may also have MD<#>.refstack.redo for replaced skycells, see specific MD field for details) * generally for stacks -- i.e., refstack for MD01 refstack.20120803, deepstack for MD09 deeptest.20120727 * refstacks -- as these are the active refstacks, they will have label MD<#>.refstack * night stacks -- a mix of label, data_group formats and cannot be grouped simply. See specific GR0 section, but trying to move to data_group format MD<#>.GR..nightstack. for reprocessed. Normal nightly processing is of form MD<#>.. Stack photometry was only run on reprocessed night stacks, so stacks from 2012 season will not be included. * stack photometry -- originally the same date, but revised runs now have dates for each in the data_group form (see section above) of adding to MD<#>.. for staticsky .<#>x20YYMMMDD and full skycal recalibration of the staticsky photometry then adding .cal20YYMMDD. So the skycal data_group will be something of the form like MD09.refstack.20120831.Nx20130115.cal20130128 in the Refstack Phot column. * (mixed) for Night Stacks -- most fields have a mix of night stack based on input warp types (not a mix going into a particular night stack) of synthetic catalog and ubercal based catalog for both the GR0 and/or the following nightly science processing night stacks. Only MD09 have a PV1 full set of night stacks currently. * Deep stack exist as PV1 only for MD09 from MD.GR0 only, MD04 needs to be redone and MD07 added (TBD) -- likely all 3 will be made with same processing tag and criteria in near future and MD09 to be replaced. * MD11 not part of the MD.GR0, PV1 reprocessing * if you find any details missing or incorrect, please let us know. __Questions__: * SMF loaded into PSPS limited to just the ones used in the refstack (or deepstack if available) or all available exposures Camera/SMFs?