Changes between Version 32 and Version 33 of MD.PV3
- Timestamp:
- Jan 28, 2015, 11:22:41 AM (12 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
MD.PV3
v32 v33 21 21 * 1/2/2015 -- all MD chip-warp ~finished 22 22 * 1/6 -- MD09 deepstack started 23 * 1/14 -- MD09 deepstack ~finished after multiple faults and problems due to overloaded system s24 * '''1/15 ''' all MD.PV3 processing suspended23 * 1/14 -- MD09 deepstack ~finished after multiple faults and problems due to overloaded system for LAP/3PI processing 24 * '''1/15 all MD.PV3 processing suspended''' 25 25 26 26 … … 46 46 || || night stk || ~291k || 860 || 500 || TBD || typically 8 inputs, sometimes >20 in marginal nights, ~4849*55-60 skycells || || 47 47 || || ref stk || 15141 || 6700 || 1500 || TBD || Nin~80, 5 seasons || 48 || || deep stk || 3033 || 37.5k* || 13k || MD09 in progress|| *using 8 theads, really depends on Nin. starting MD09 since doesn't require Jan/Jun 2010 start decision ||48 || || deep stk || 3033 || 37.5k* || 13k || '''MD09 finished, verifiying''' || *using 8 theads, really depends on Nin. starting MD09 since doesn't require Jan/Jun 2010 start decision || 49 49 || || iq stk || 3325 || 2215 || 310 || TBD || edge skycells of course run faster in all stacks|| 50 50 || || || || || || || || … … 69 69 || || || || || || || || 70 70 || || ref stk || ~500? || ~6.7k? || ? || TBD || no seasons, so like 3PI WSdiff -- WS so Nin~30? || 71 || || deep stk || ~500? || ~37k? || ? || '''finished ''' || Nin~500? Need to verify a functional deep stack since first version||71 || || deep stk || ~500? || ~37k? || ? || '''finished, verifying''' || Nin~500? Need to verify a functional deep stack since first version|| 72 72 || || iq stk || ~500? || 2200? || ? || TBD || Nin~25? || 73 73 || || || || || || || || … … 173 173 * total ~ 110 d? very uncertain 174 174 175 176 Revised estimate with machines possible -- 177 * MD: 178 * deep stack -- now that chip-cam-warp finished, this it the limiting process group as only certain machines can be used () 179 180 * CNP: if the deep stack verifies okay, then the WSdiff are the next large process group (full-forced phot is an unknown) 181 * 182 175 183 ---- 176 184 … … 197 205 * diffim 198 206 * verify similar photometry to previous reductions -- 199 * verify similar detections/coverage (no sig loss on edges) -- 207 * verify similar detections/coverage (no sig loss on edges) -- really a key point for reference stack creation above as well 200 208 * test samples for CNP,M31,STS -- 201 209 * full-forced photometry -- unknown, untested still for any MD type field … … 214 222 215 223 === stack === 216 Order of operations important -- while deep stacks take 1.5 months or so to finish, night and seasonal reference stacks must be finished before diffims can be started. Lighter jobs like difference images and iq (even reference stacks) can be run along with the deep stacks for some fields.224 Order of operations important -- while deep stacks take 1.5 months or so to finish, night and seasonal reference stacks must be finished before diffims can be started. Lighter jobs like difference images and iq stacks (even reference stacks) can be run along with the deep stacks for some fields. 217 225 218 226 ==== night ==== … … 230 238 231 239 === difference image === 232 Cannot start until both night and seasonal reference stacks finish. 240 241 Cannot start until night and seasonal reference stacks finish for SSdiff. 233 242 234 243 ---- … … 236 245 237 246 == Processing Notes == 247 238 248 Section for local processing details/notes -- not for general users 239 249 240 pantasks_server nodes -- will keep current but may changedue to task shuffling250 pantasks_server nodes for ippmd user -- will keep current for own sanity and may change often due to task shuffling 241 251 * ipps00 -- ~ippmd/deepstack -- high memory stacks 242 * ipps01 -- ~ippmd/stdscience -- chip-cam-warp,243 * ipps02 -- ~ippmd/distribution 244 * ipps03 -- ~ippmd/stack -- iq,night,ref252 * ipps01 -- ~ippmd/stdscience -- ---chip-cam-warp--- (finished), diffim, staticsky+skycal 253 * ipps02 -- ~ippmd/distribution -- reference and night stacks for transients group, possible skycal before loading into PSPS 254 * ipps03 -- ~ippmd/stack -- lower memory, faster running iq, night and probably reference stacks 245 255 246 256 host groups -- 247 * m0 = 248 * m1 = 249 * x0,x1 = 250 * x2 = 251 * s4 252 * s5 257 * m0 = ipps00--s04 (12 core, 144G, very efficient machines) 258 * m1 = ipps05--s14 ((12 core, 96G, very efficient machines) 259 * x0b,x1b = ippx037-x040, x041-x044 (12 core, 96G) 260 * s4 = ipp067--071 (12 core, 132G) -- data nodes and used for nightly science processing so only 10hr/day available 261 * s5 = ipp072--095 (12 core, 198G) -- data nodes and used for nightly science processing so only 10hr/day available 253 262 254 255 processing loads/schedule -- 263 data node targeting -- helps to have data going to same machine 264 * 265 * 266 * stsci main targets for LAP/3PI -- MD have less file volume to move when finished 267 * ipp067--081 are nearly full and need to have lighter allocations 268 * ipp082-095 various levels of larger free space and should have higher allocations 269 270 processing loads/schedule groups -- 256 271 * CNP chip-cam-warp -- 257 272 * CNP deepstack -- 258 * CNP iq, refstack --273 * CNP iq, refstack -- 259 274 * CNP WSdiff -- 260 275 * CNP staticsky,skycal -- 261 262 * MD chip-cam-warp -- 276 * CNP full-forced phot -- 277 278 * MD chip-cam-warp -- finished 263 279 * MD deepstack -- 264 265 ---- 266 267 268 269 270 271 272 273 274 275 276 280 * MD refstack (multiple sets for out-of-season) -- 281 * MD WSdiff -- 282 * MD iq, night stack -- 283 * MD SSdiff -- 284 * MD staticsky, skycal -- 285 * MD full-forced phot 286 287 ---- 288 289 290 291 292 293 294 295 296 297 298
