| 7 | | Processing and (re)reprocessing is defined PV1 for MD based on the date of 2012-05-16 to include improved and new detrend types ([wiki:GPC1_Detrend_Documentation]) and corrections for the faint-end bias photometry (ipp-20120404,r33642 tag -- details related to psphot elsewhere..). While detrends have been updated since that date, the changes are minor and should not impact the MD processing. While the latest date for the new VideoMask detrend is 2012-05-03, the date of 2012-05-16 was chosen as the day after the summary of the detrend updates had been established. Many of the details here are also taken from the [wiki:MD.GR0] page, see that page for more information. In general, all the MD fields required at least partial re-reprocessing of chip to warp except MD09 for which MD.GR0 reprocessing started at the beginning of June 2012 (including the refstack and night stacks). All MD fields have refstacks created after the 2012-05-16 date. Only MD09 has a complete set of PV1 night stacks, all other fields will have a mixture of pre and post-PV1 night stacks (post-PV1 will generally be the normal nightly science processing of night stacks and not from MD.GR0). |
| | 7 | PV1 processing and (re)reprocessing is defined for MD based on the start date of 2012-05-16 to include improved and new detrend types ([wiki:GPC1_Detrend_Documentation]) and corrections for the faint-end bias photometry (ipp-20120404,r33642 tag -- details related to psphot elsewhere..). While detrends have been updated since that date, the changes are minor and should not impact the MD processing. Also, even though the latest date for the new VideoMask detrend is 2012-05-03, the date of 2012-05-16 was chosen as the day after the summary of the detrend updates had been established. |
| | 8 | |
| | 9 | Many of the details here are also taken from the [wiki:MD.GR0] page, see that page for more information. In general, all the MD fields required at least partial re-reprocessing of chip to warp except MD09 for which MD.GR0 reprocessing started at the beginning of June 2012 (including the refstack and night stacks). All MD fields now have refstacks created after the 2012-05-16 date. Only MD09 has a complete set of PV1 night stacks, all other fields will have a mixture of pre and post-PV1 night stacks (post-PV1 will generally be the normal nightly science processing of night stacks and not from MD.GR0). |
| 12 | | Each MD field has a variety of ''issues'' and details are summarized in each MD section of [wiki:MD.GR0] or later in the relevant refstack wiki page [wiki:MediumDeepFields]. Label/data_group names have also slowly simplified and the appropriate MD section should be referred to if those must be be used. It is recommended to use the <stage>_id to select appropriate data products and each section below will provide the number after which the PV1 definition should hold for. |
| | 14 | Each MD field has a variety of ''issues'' and details are summarized in each MD section of [wiki:MD.GR0] or later in the relevant refstack wiki page (see links from [wiki:MediumDeepFields]). Label/data_group names have also slowly simplified and the appropriate MD section should be referred to if those must be be used. It is recommended to use the <stage>_id to select appropriate data products and each section below will provide the number after which the PV1 definition should be valid for. |
| | 15 | |
| | 16 | It is important to make clear that the x.PV1.x label/data_group/workdir for MD was a sub-re-reprocessing of exposures that were reprocessed or just processed before date X (now set at 5/16/2012). There have been various other re-reprocessing and reprocessing of exposures (for recent refstacks, GR0 late 2012 night stack completion, etc ***) that have happened after that date. This means there can also be duplicate processing of exposures with valid <stage>_id. Duplicates were allowed because initially the exact date for PV1 was unclear and can also provide an old/new valid reprocessed exposure comparison (i.e., has a change in the ipp-X tag caused problems). When PV1 re-reprocessing was done, MD fields that ended their observing season around this time (i.e., MD05,06,07) were reprocessed to finish out their observing season for some fields since the date was initially uncertain for PV1. Again, either should be acceptable to use, and comparison of original and newer may be a helpful. |
| 15 | | |
| 16 | | It is important to make clear that the x.PV1.x label/data_group/workdir for MD was a sub-re-reprocessing of exposures that were reprocessed or just processed before date X (now set at 5/16/2012). There have been various other re-reprocessing and reprocessing of exposures (for refstacks, GR0 late 2012 etc ***) that have happened after that date. It is really the <stage>_id that needs to be used to simply select with, and so there are duplicates of exposures with valid <stage>_id. |
| 17 | | |
| 18 | | The largest <stage>_id of such an exposure set could just be used. |
| 19 | | |
| 20 | | duplicates were allowed because |
| 21 | | |
| 22 | | 1: didn't know what the exact date would be |
| 23 | | 2: can provide an old/new valid reprocessed exposure comparison |
| 24 | | |
| 25 | | *** even reprocessing of exposures for old format V2 exist, those were ignored and re-reprocessed for V3 warps and stacks later on. These could also be included in the duplicate case and using the largest <stage>_id in the duplicate set should exclude those if interested, however, if interested in the warp and later products the V3 format should be verified. |
| 26 | | |
| 27 | | |
| 28 | | Re-reprocessing tried to finish out observing season for some fields since date uncertain for PV1 initially, so will be duplicate processing for the same exposure. Either should be acceptable to use, comparison of original and newer may be a helpful |
| | 19 | *** even reprocessing of exposures for old format V2 exist, those were ignored and re-reprocessed for V3 warps and stacks later on. If interested in the warp and later products, the V3 format should be verified. |
| 33 | | == Chip+Warp == |
| 34 | | Fully reprocessed but not distributed due to disk space load. These can be obtained via the Postage Stamp Server. |
| 35 | | * MD.GR0 was a mix of synthetic/ubercal catalog calibrations except MD08, 09. |
| 36 | | * for PV1 chip_id, warp_id >= 433987, 393977 respectively it was processed using the 20120524v0 ubercal catalog and with detrends |
| | 24 | == Chip, Warp == |
| | 25 | Fully reprocessed but not distributed due to disk space load. These can be obtained via the Postage Stamp Server |
| | 26 | * MD.GR0 was a mix of synthetic/ubercal catalog calibrations except MD08,09 |
| | 27 | * with PV1 chip_id, warp_id >= 433987, 393977 respectively, all processing using the 20120524v0 ubercal catalog and valid detrends |
| 41 | | * MD.GR0 was a mix of synthetic/ubercal catalog calibrations except MD08, 09 |
| 42 | | * a possible guideline to follow, including nightly science data processing, if MD exposures with a cam_id >=412069 (actually >=411444 of some V2 destined reprocessing) is used, those will be with the ipp-20120404,r33642 tag that includes the faint-end bias fix and the ubercal catalog 20120524v0 version also used for LAP. |
| 43 | | |
| 44 | | cam_id>=434615 |
| 45 | | |
| 46 | | |
| 47 | | |
| | 32 | * MD.GR0 was a mix of synthetic/ubercal catalog calibrations except MD08,09 |
| | 33 | * with PV1 cam_id>=434615 respectively, all processing using the 20120524v0 ubercal catalog and valid detrends |
| 53 | | __Night Stacks__: All nights reprocessed and stacked into a night stack using the current night stack configuration used for nightly science night stacks when processed, including reference catalog |
| 54 | | * should attempt to make stack with 2 or more input warps now |
| 55 | | * some nights have >8 exposures, often in marginal conditions. Since marginal conditions can vary greatly, some selection should probably be made for inclusion in the night stack. |
| 56 | | * the sample of stacks can be a mix of synthetic/ubercal catalog calibrations for the pixels but the warps going into the stack should be of the same type of calibration. |
| | 39 | __Night Stacks__: All warps from a night (re)processed and stacked into a night stack |
| | 40 | * although the GR0 sample of stacks can be a mix of synthetic/ubercal catalog calibrations for the warp pixels, the warps going into the stack should be of the same type of calibration |
| | 41 | * for PV1 there is no simple, unique method for selecting other than manually creating a release listing or using the gpc1 DB stackInputSkyfile table for input warp_id |
| 59 | | __Reference Stacks (refstack)__: To be made prior to the start of the MD field observations for the season to support the difference image products (magic and science). The focus of MD.GR0 was on the 2011/2012 season fields starting with MD10, then 01, 02 ... through MD09. The 2012/2013 season refstacks will to replace the 2011/2012 set and have their own page linked from [wiki:MediumDeepFields]. |
| 60 | | * out-of-season/anti-yearly refstacks, yearly refstack: on hold for MD.GR1 (would need to update/reprocess large numbers of exposures to do) |
| | 47 | __Deep Stacks (deeptest)__: Not a full production product yet. Test version exists for MD04.deeptest.20120727 (synthetic catalog, pre-PV1 warps only), MD09.deeptest.20120727 (ubercal catalog 20120524v0, PV1 warps only). Like yearly or out-of-season stacks, would required much update processing to make for other fields. Looking at updating MD04 deep stack and adding MD07 to the test set however. |
| 62 | | __Deep Stacks (deeptest)__: Not a full production product yet. Test version exists for MD04.deeptest.20120727 (synthetic catalog, pre-PV1 warps), MD09.deeptest.20120727 (ubercal catalog 20120524v0, PV1 warps) |
| 63 | | |
| 64 | | |
| 65 | | __IQ/PSF Stack__: on hold until MD.GR1 (like out-of-season, would required many updates). A sample for MD09 done but requires development/code time. Test version exists for MD02.iqtest.20120927, MD10.psfrefstack.20110814, but neither will be made from PV1 warps. |
| | 49 | __(best) IQ/PSF Stack__: on hold until MD.GR1 (should be done with refstacks to avoid excess processing). Test version exists for MD02.iqtest.20120927, MD10.psfrefstack.20110814, but neither are made from PV1 warps. A sample could be done for MD09 done but requires development/code time. |
| 79 | | |
| 80 | | Re-revised planned loading into PSPS (March 2013): do the newer refstacks, so the older MD03--MD07 noted before below been have done as follows |
| 81 | | |
| 82 | | {{{ |
| 83 | | label staticsky: MD04.refstack.20121125.Nx20130325 |
| 84 | | data_group staticsky: MD04.refstack.20121125.<N>x20130325 |
| 85 | | label=data_group skycal: MD04.refstack.20121125.Nx20130325.cal20130326 |
| 86 | | |
| 87 | | label staticsky: MD03.refstack.20121101.Nx20130325 -- skycal label=data_group, add .cal20130326 |
| 88 | | label staticsky: MD05.refstack.20121202.Nx20130325 -- skycal label=data_group, add .cal20130326 |
| 89 | | label staticsky: MD06.refstack.20121221.Nx20130325 -- skycal label=data_group, add .cal20130326 |
| 90 | | label staticsky: MD07.refstack.20130102.Nx20130325 -- skycal label=data_group, add .cal20130326 |
| 91 | | }}} |
| 92 | | |
| 93 | | |
| 94 | | Current run planned for loading into PSPS (January 2013): stack photometry (staticsky) re-run from Summer 2012 set to incorporate improvements/fixes in the Kron code. Photometry zeropoint recalibration (skycal) re-run to incorporate various bug fixes and improved tuning (psastro running on stack catalogs). Several previous re-runs for skycal and settled on .cal20130128 version for now. |
| 95 | | |
| | 61 | Re-revised planned loading into PSPS (March 2013) -- do for all the newer refstacks, MD09 deeptest, MD09.GR0 night stacks |
| 102 | | label staticsky: MD04.refstack.20111012.Nx20130115 |
| 103 | | data_group staticsky: MD04.refstack.20111012.<N>x20130115 |
| 104 | | label=data_group skycal: MD04.refstack.20111012.Nx20130115.cal20130128 |
| 105 | | |
| 106 | | label staticsky: MD10.refstack.20120804.Nx20130115 (new refstack MD10.refstack.20110821, see [wiki:MD10.refstack.20120705], is deeper and replacing MD.GR0 version MD10.refstack.20110821) |
| 107 | | data_group staticsky: MD10.refstack.20120804.<N>x20130115 |
| 108 | | label=data_group skycal: MD10.refstack.20120804.Nx20130115.cal20130128 |
| 109 | | |
| 110 | | label staticsky: MD01.refstack.20120803.Nx20130115 (new refstack [wiki:MD01.refstack.20120803] replacing MD.GR0 version since deeper) -- skycal label=data_group, add .cal20130128 |
| 111 | | label staticsky: MD02.refstack.20120927.Nx20130115 (new refstack [wiki:MD02.refstack.20120927] replacing MD.GR0 version since deeper) -- skycal label=data_group, add .cal20130128 |
| 112 | | label staticsky: MD03.refstack.20111010.Nx20130115 -- skycal label=data_group, add .cal20130128 |
| 113 | | label staticsky: MD05.refstack.20111014.Nx20130115 -- skycal label=data_group, add .cal20130128 |
| 114 | | label staticsky: MD06.refstack.20111122.Nx20130115 -- skycal label=data_group, add .cal20130128 |
| 115 | | label staticsky: MD07.refstack.20111106.Nx20130115 -- skycal label=data_group, add .cal20130128 |
| 116 | | label staticsky: MD08.refstack.20120422.Nx20130115 -- skycal label=data_group, add .cal20130128 |
| | 68 | label staticsky: MD01.refstack.20120803.Nx20130115 -- skycal label=data_group, add .cal20130128 |
| | 69 | label staticsky: MD02.refstack.20120927.Nx20130115 -- skycal label=data_group, add .cal20130128 |
| | 70 | label staticsky: MD03.refstack.20121101.Nx20130325 -- skycal label=data_group, add .cal20130326 |
| | 71 | label staticsky: MD04.refstack.20121125.Nx20130325 -- skycal label=data_group, add .cal20130326 |
| | 72 | label staticsky: MD05.refstack.20121202.Nx20130325 -- skycal label=data_group, add .cal20130326 |
| | 73 | label staticsky: MD06.refstack.20121221.Nx20130325 -- skycal label=data_group, add .cal20130326 |
| | 74 | label staticsky: MD07.refstack.20130102.Nx20130325 -- skycal label=data_group, add .cal20130326 |
| | 75 | label staticsky: MD08.refstack.20130401.Nx20130407 -- skycal label=data_group, add .cal20130408 |
