IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Version 15 and Version 16 of MD.PV3


Ignore:
Timestamp:
Sep 24, 2014, 1:42:13 PM (12 years ago)
Author:
Mark Huber
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • MD.PV3

    v15 v16  
    1212From Ken's talk at PS1SC@STSCI meeting in June 2014, the diffims described there may need to modifications:
    1313 * stack:
    14   * MDN -- night stack
     14  * MDN -- night stack -- only made for normal MD fields?
    1515  * MD0 -- seasonal reference stack
    1616  * MDD -- deep stack
    1717  * MDT -- iq stack
    1818 * diffim:
    19   * warp-stack (WS) -- (warp-ref) only, no (warp-deep) or only for CNP?
    20   * stack-stack (SS) -- must add (night-ref) but no (stack-deep)
     19  * warp-stack (WS) -- (warp-ref) only, no (warp-deep) or only for CNP (i.e., is deep and ref the same basically -- difficult w/ possible iq issues)
     20  * stack-stack (SS) -- must add (night-ref) but no (stack-deep) and only for normal MD fields?
    2121
    22 || '''field''' || '''stage''' || '''N imfile/skycells''' || '''ave_dtime(s)''' || '''sig_dtime(s)''' || '''status''' || '''comment''' || '''label/data_group, release name''' ||
     22|| '''field''' || '''stage''' || '''N job units''' || '''ave_dtime(s)''' || '''sig_dtime(s)''' || '''status''' || '''comment''' || '''label/data_group, release name''' ||
    2323|| || || || || || || ||
    2424|| MD || chip/camera/warp || ~2500k/41717/2500k || 130/610/85 || 130/500/65 || TBD || 41717 exposures *60 imfiles/55-60 skycells || MD<f#>.pv3.20141001? ||
     25|| || || || || || || ||
    2526|| || night stk || ~291k || 860 || 500 || TBD || typically 8 inputs, sometimes >20 in marginal nights, ~4849*55-60 skycells || ||
    2627|| || ref stk || 15141 || 6700 || 1500 || TBD || Nin~80, 5 seasons ||
    27 || || deep stk || 3033 || 37.5k* || 13k || TBD || *using 8 theads ||
     28|| || deep stk || 3033 || 37.5k* || 13k || TBD || *using 8 theads, really depends on Nin ||
    2829|| || iq stk || 3325 || 2215 || 310 || TBD || edge skycells of course run faster in all stacks||
     30|| || || || || || || ||
    2931|| || WSref diff || 2500k || ~200 || ~50 || TBD || done by season, same number as warps ||
    30 || || WSdeep diff || x || x || x || x || not going to do ||
     32|| || WSdeep diff || x || x || x || x || not going to do, already have WSref by season and using deep stack not a gain ||
    3133|| || SS diff || ~291k || ~200 || ~50 || TBD || done by season, same number as night stacks ||
     34|| || || || || || || ||
    3235|| || night staticsky || ~291k || 110 || 40 || TBD || same number as night stacks (no filter groups) ||
    3336|| || ref staticsky || 3025 || 4960 || 1700 || TBD || ||
    3437|| || deep staticsky || 605 || 6300 || 2550 || TBD || ||
    3538|| || iq staticsky || 665 || 4660 || 1600 || TBD || ||
     39|| || || || || || || ||
    3640|| || night skycal || ~291k || 30 || 43 || TBD || same as for staticsky since single filter ||
    3741|| || ref skycal || 15093 || 26 || 5 || TBD || ||
    3842|| || deep skycal || 3008 || 31 || 8 || TBD || ||
    3943|| || iq skycal || 3018 || 40 || 34 || TBD || ||
     44|| || || || || || || ||
    4045|| || full-forced phot || 2500k || ? || ? || TBD || ||
    4146|| || || || || || || ||
    42 || CNP || chip/camera/warp || ~162k/2695/162k || ~115/410/60 || 150/430/125 || TBD || 2695 exposures *60 imfiles/55-60 skycells || ||
    43 || || ref/deep stk || ~500? || ? || ? || TBD || Nin~ , seasons (first/last month for ref?)||
    44 || || WSref/deep diff || ~2695 || ~? || ~? || TBD || ||
    45 || || ref/deep staticsky || ~100? || ? || ? || TBD || ||
    46 || || ref/deep skycal || ~500? || ~37? || ~29? || TBD ||
    47 || || full-forced phot || ~162k || ? || ? || TBD || ||
    4847|| || || || || || || ||
    49 || M31 || chip/camera/warp || ~261/4347/261k || 129/624/62 || 190/313/30 || TBD || 4347 exposures *60 imfiles/55-60 skycells  || ||
     48|| CNP || chip/camera/warp || ~162k/2695/270k || ~115/410/60 || 150/430/125 || TBD || 2695 exposures *60 imfiles/100? skycells || ||
     49|| || ref/deep stk || ~500? || ~6.7k/37k? || ? || TBD || Nin~80/500?, no seasons? first/last month for ref stack? ||
     50|| || WSref/deep diff || ~2695 || ~200? || ? || TBD || ||
     51|| || ref/deep staticsky || ~100? || ~6300? || ? || TBD || ||
     52|| || ref/deep skycal || ~500? || ~30? || ? || TBD ||
     53|| || full-forced phot || ~270k || ? || ? || TBD || ||
     54|| || || || || || || ||
     55|| || || || || || || ||
     56|| M31 || chip/camera/warp || ~261/4347/k || 129/624/62 || 190/313/30 || TBD || 4347 exposures *60 imfiles/87-96 skycells  || ||
    5057|| || background restore ||
    51 || || ref stk || ~500? || X || x || TBD || Nin~?, 4 seasons ||
    52 || || deep stk || ?
    53 || || WSref diff || ~ || X || X || TBD || ||
     58|| || ref stk || ~500? || ? || ? || TBD || Nin~80, 4 seasons ||
     59|| || deep stk || ? || ? || ? || TBD || Nin~? by grizy, or only deep in i and others just ref? ||
     60|| || WSref diff || ~ || ~200? || ? || TBD || ||
    5461|| || ref staticsky ||
    55 || || ref skycal || ~ || ~37 || ~29 || TBD ||
     62|| || ref skycal || ~ || ~30? || ? || TBD ||
    5663|| || full-forced phot ||  || ? || ? || TBD || ||
    5764|| || || || || || || ||
     65|| || || || || || || ||
    5866|| STS || chip/camera/warp || ~996k/16606/996k || 210/1200/100 || 120/575/70 || TBD || 16606 exposures *60 imfiles/55-60 skycells || ||
    5967|| || ref stk || ~3500? || X || x || TBD || Nin~, 4 seasons ||
    … …  
    6775 * label/data_group, release name -- for gpc1 DB queries/datastore identification and specific postage-stamp server (PSS) association
    6876 * ave/sig_dtime -- roughly the dtime_script for the jobs in the gpc1 DB
    69  *
     77 * N job units -- depend on stage/task will be exposure, imfile or skycell
    7078
    7179
    72 If run only on the 15 ippsXX machines with high memory, 12 cores -- individual processing estimates only, some can be run ~same time like deep+iq/night/dif/staticsky/skycal possibly
     80If run only on the 15 ippsXX processing nodes with high memory, 12 cores -- individual processing estimates only, some can be run ~same time like deep+iq/night/dif/staticsky/skycal possibly
    7381 * MD: will be run one field at a time to limit the queued jobs
    7482  * chip/cam/warp ~6 jobs/node ~90 jobs, ~42+3+27 ~72 days
    … …  
    8694
    8795
    88 Issues to address:
    89  * disk space limiting
    90  * chip-warp:
     96Issues to address before starting:
     97 * disk space limits
     98 * config reorg, or specific PV3 named ones to create
     99 * chip-camera-warp:
    91100  * check if refcat oddities in field problem here
    92   * background restore M31 and masking for M31/STS -- test samples to be run
     101  * background restore M31 and masking for M31/STS -- test samples to be run and verified for final version
     102  * CNP tessellation check
    93103 * stack 
    94104  * stack quality fault
    … …  
    100110  * check refcat oddities in the MD fields
    101111  * zeropt recalib in skycal improvement and characterization (y-band at least needs own config)
     112  * test samples for CNP,M31,STS
    102113 * diffim
    103114  * verify similar photometry