Changes between Version 30 and Version 31 of MD.PV3
- Timestamp:
- Jan 9, 2015, 6:28:51 PM (12 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
MD.PV3
v30 v31 3 3 4 4 * [wiki:MediumDeepFields] summary page -- quick links to all up to date refstacks 5 * [wiki:MD.GR0] summary page -- summary of all reprocessing details and 2011/2012 refstack sand some deepstack samples5 * [wiki:MD.GR0] summary page -- summary of all reprocessing details and 2011/2012 refstack and some deepstack samples 6 6 * [wiki:MD.PV1] summary page -- the data set that PV2 update reprocessing is based on 7 7 * [wiki:MD.PV2] summary page -- update type reprocessing 8 8 9 This is a summary of the MD.PV3 full pixel reprocessing. Start date and schedule '''TBD'''. 10 11 This processing will also involve other MD-like fields, including M31, STS and CNP. Each field can be similar and different to the typical MD criteria for the stacks and diffims, summary '''TBD'''. 12 13 From Ken's talk at PS1SC@STSCI meeting in June 2014, the diffims described there may need to modifications: 9 This is a summary of the MD.PV3 full pixel reprocessing. Start date and schedule a mess with the unstable processing 12/01/2014. 10 11 This processing will also involve other MD-like fields, including CNP (M31 and STS if time allows). Each field can be similar and different to the typical MD criteria for the stacks and diffims, summary '''TBD'''. 12 13 ___General Summary Status:___ 14 * '''10/1''' start chip-warp for MD should be possible, STS+M31 sample should be possible -- new refcat responding different w/ past configs, verification necessary 15 * 11/4 -- MD PV3 tag made for processing -- ipp-mdpv3-20141104 16 * 11/07 -- MD chip-warp started with MD04 -- label: MD##.PV3.20141104 17 * 11/11 -- CNP chip-warp started -- label: CNP.PV3.20141104 18 * 11/20 -- MD04 chip-warp finished 19 * 11/23 -- CNP chip-warp finished 20 * 12/16 -- CNP deepstack started -- label: CNP.PV3.deepstack.20141104 21 * 1/2/2015 -- all MD chip-warp finished 22 * 1/6 -- MD09 deepstack started 23 * 24 25 26 ---- 27 28 == PV3 Summary Processing Products == 29 30 From Ken's talk at PS1SC@STSCI meeting in June 2014, the products to produce include 14 31 * stack: 15 * MDN -- night stack -- only made for normal MD fields? 16 * MD0 -- seasonal reference stack 17 * MDD -- deep stack 18 * MDT -- iq stack 19 * diffim: 20 * warp-stack (WS) -- (warp-ref) only. No (warp-deep) or only for CNP? (i.e., is deep and ref the same basically -- difficult w/ possible iq issues) 21 * stack-stack (SS) -- (night/stack-deep) should be (night-ref) and typically for MD/M31 with multiple exposures per night. STS has upwards 100/night, so no. 32 * MDN -- night stack -- only made for normal MD fields 33 * MD0 -- seasonal reference stack -- only where seasons exist (CNP yearly so no seasonal refstack) 34 * MDD -- deep stack -- 35 * MDT -- IQ stack -- 36 * diffim: (the diffims described there may need modifications) 37 * warp-stack (WS) -- warp-ref only. No warp-deep (deep stack PSF can be a problem for diffims) 38 * stack-stack (SS) -- night/stack-deep in talk but should be night-ref and typically for MD. M31 with multiple exposures per night may be possible. STS has upwards 100/night, so probably not. 39 22 40 23 41 || '''field''' || '''stage/data product''' || '''N job units''' || '''ave_dtime(s)''' || '''sig_dtime(s)''' || '''status''' || '''comment''' || '''label; data_group; release name''' || 24 42 || || || || || || || || 25 || MD || chip/camera/warp || ~2500k/41717/2500k || 130/610/85 || 130/500/65 || MD09,01-05 finished '''(06 in progress)''' || 41717 exposures *60 imfiles/55-60 skycells || MD<f#>.PV3.20141104; MD<f#>.<obsdate>.PV3.<process date>; (MD<f#>.PV3?) ||43 || MD || chip/camera/warp || ~2500k/41717/2500k || 130/610/85 || 130/500/65 || '''finished'' || 41717 exposures *60 imfiles/55-60 skycells || MD<f#>.PV3.20141104; MD<f#>.<obsdate>.PV3.<process date>; (MD<f#>.PV3?) || 26 44 || || || || || || || || 27 45 || || night stk || ~291k || 860 || 500 || TBD || typically 8 inputs, sometimes >20 in marginal nights, ~4849*55-60 skycells || || 28 46 || || ref stk || 15141 || 6700 || 1500 || TBD || Nin~80, 5 seasons || 29 || || deep stk || 3033 || 37.5k* || 13k || TBD || *using 8 theads, really depends on Nin ||47 || || deep stk || 3033 || 37.5k* || 13k || MD09 in progress || *using 8 theads, really depends on Nin. starting MD09 since doesn't require Jan/Jun 2010 start decision || 30 48 || || iq stk || 3325 || 2215 || 310 || TBD || edge skycells of course run faster in all stacks|| 31 49 || || || || || || || || … … 50 68 || || || || || || || || 51 69 || || ref stk || ~500? || ~6.7k? || ? || TBD || no seasons, so like 3PI WSdiff -- WS so Nin~30? || 52 || || deep stk || ~500? || ~37k? || ? || finished? || Nin~500?||70 || || deep stk || ~500? || ~37k? || ? || '''finished''' || Nin~500? Need to verify a functional deep stack since first version|| 53 71 || || iq stk || ~500? || 2200? || ? || TBD || Nin~25? || 54 72 || || || || || || || || … … 114 132 * N job units -- depend on stage/task will be exposure, imfile or skycell 115 133 134 === PV3 runtime estimates based on PV2 === 116 135 117 136 If run only on the 15 ippsXX processing nodes with high memory, 12 cores -- individual processing estimates only, some can be run ~same time like deep+iq/night/dif/staticsky/skycal possibly … … 153 172 * total ~ 110 d? very uncertain 154 173 155 156 Issues to address before starting: 157 * disk space limits 158 * config reorg, or specific PV3 named ones to create 174 ---- 175 176 == PV3 issues to address == 177 178 Issues to address, most before starting: 179 * disk space limits (turned out to also include network limits) -- 180 * config reorganization or specific PV3 named ones to create -- all configs must have PV3 to avoid confusion 159 181 * chip-camera-warp: 160 * check if refcat oddities in field problem here 182 * check if refcat oddities in field is a problem -- Eddie and Doug catalogs to compare still 183 * new refcat astrometry and zeropoint verification tests -- 184 * CNP tessellation check -- 161 185 * background restore M31 and masking for M31/STS -- test samples to be run and verified for final version 162 * CNP tessellation check163 186 * stack 164 * stack quality fault 165 * night stack config, refstack edge loss -- see diffim as well 166 * ref+deep stack input cuts -- need feedback on if 2009 and or early 2010 data is okay to use 167 * iq+ref+deep test stacks for CNP,M31,STS and to verify Nin and input cuts -- if no night stacks for these so iq Nin modification necessary?187 * stack quality fault -- sometime edge skycells fault w/ internal photometry/PSF error on the convolved stack and do not want to lose for staticsky or diffims -- 188 * night stack config, refstack edge loss -- see diffim as well -- 189 * ref+deep stack input cuts -- need feedback on if 2009 and or early 2010 data is okay to use -- 2009 is not great data so not to be used, Jan/June 2010 is still under consideration if the baffle is critical or not. 190 * iq+ref+deep test stacks for CNP,M31,STS and to verify Nin, input cuts, pixel rejection and minimum pixel limits -- 168 191 * staticsky/skycal 169 * stack photometry sky oddities 170 * check refcat oddities in the MD fields 171 * zeropt recalib in skycal improvement and characterization (y-band at least needs own config) 192 * stack photometry sky oddities -- 193 * check refcat oddities in the MD fields -- 194 * zeropt recalib in skycal improvement and characterization (y-band at least needs own config) -- 172 195 * test samples for CNP,M31,STS -- equivalent stacks in available filters vs deeper in primary filter, single filter only for STS/M31 so no extended source fits? 173 196 * diffim 174 * verify similar photometry 175 * verify similar detections/coverage (no sig loss on edges) 176 * test samples for CNP,M31,STS -- ref vs deep stack for diffim, define seasons (if appropriate for CNP) 177 * full-forced photometry -- unknown, untested still 178 179 180 ___General Summary Status:___ 181 * '''10/1''' start chip-warp for MD should be possible, STS+M31 sample should be possible 182 183 184 185 ---- 186 187 == chip -- camera -- warp == 197 * verify similar photometry to previous reductions -- 198 * verify similar detections/coverage (no sig loss on edges) -- 199 * test samples for CNP,M31,STS -- 200 * full-forced photometry -- unknown, untested still for any MD type field 201 202 ---- 203 204 205 == PV3 Processing Stages == 206 207 === chip--camera--warp === 188 208 Should be started ASAP, so that the warps are available for some early testing of the stacks and diffims. 189 209 190 Testing necessary for the M31, STS and CNP processing still .191 192 ---- 193 194 == stack==210 Testing necessary for the M31, STS and CNP processing still -- priority for CNP as last reprocessing was ~2011 211 212 ---- 213 214 === stack === 195 215 Order of operations important -- while deep stacks take 1.5 months or so to finish, night and seasonal reference stacks must be finished before diffims can be started. Lighter jobs like difference images and iq (even reference stacks) can be run along with the deep stacks for some fields. 196 216 197 === night===198 199 === seasonal reference===200 201 === deep===202 203 === iq===204 205 === photometry===206 207 ---- 208 209 210 == difference image==217 ==== night ==== 218 219 ==== seasonal reference ==== 220 221 ==== deep ==== 222 223 ==== iq ==== 224 225 ==== photometry ==== 226 227 ---- 228 229 230 === difference image === 211 231 Cannot start until both night and seasonal reference stacks finish. 212 232 … … 215 235 216 236 == Processing Notes == 217 Section for local processing details/notes -- 237 Section for local processing details/notes -- not for general users 218 238 219 239 pantasks_server nodes -- will keep current but may change due to task shuffling … … 230 250 * s4 231 251 * s5 232 * s5b252 233 253 234 254 processing loads/schedule -- … … 242 262 * MD deepstack -- 243 263 244 245 246 247 248 249 250 251 252 253 254 255 264 ---- 265 266 267 268 269 270 271 272 273 274 275
