IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Version 11 and Version 12 of ippReleaseManagement


Ignore:
Timestamp:
Jan 25, 2013, 11:12:59 AM (14 years ago)
Author:
bills
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • ippReleaseManagement

    v11 v12  
    101101in 2 new tables called "ippRelease" and "survey". Here are the proposed definitions
    102102
     103=== survey ===
    103104{{{
    104105mysql> describe survey;
    … …  
    1121133 rows in set (0.00 sec)
    113114
     115
     116}}}
     117
     118The survey table is a copy of the PSPS table Survey.
     119
     120{{{
     121
     122+----------+------+------------------------------------+
     123| surveyID | surveyName | description                                                  |
     124+----------+------------+--------------------------------------------------------------+
     125|        0 | 3PI        | PS1 3PI Survey                                               |
     126|        1 | MD01       | PS1 MD01 XMM-LSS-DXS 022224-043000                           |
     127|        2 | MD02       | PS1 MD02 CDFS/GOODS/GEMS 033224-274800                       |
     128|        3 | MD03       | PS1 MD03 IFA/Lynx 084300+444000                              |
     129|        4 | MD04       | PS1 MD04 COSMOS 100000+021200                                |
     130|        5 | MD05       | PS1 MD05 Lockman-DXS 110000+570000                           |
     131|        6 | MD06       | PS1 MD06 NGC 4258 121857+471814                              |
     132|        7 | MD07       | PS1 MD07 VISTA-Video1 140000+050000                          |
     133|        8 | MD08       | PS1 MD08 EliasN1-DXS 160000+570000                           |
     134|        9 | MD09       | PS1 MD09 Vimos4-DXS-SSA 220000+003000                        |
     135|       10 | MD10       | PS1 MD10 Deep2 233000+000000                                 |
     136|       11 | M31        | PS1 M31 Andromeda 004242+411600                              |
     137|       12 | STS1       | PS1 STS1 Outskirts of Bulge 195048+170339                    |
     138|       13 | CAL01      | PS1 Cal North Pole 000000+900000                             |
     139|       14 | CAL02      | PS1 Cal SA92 Dense Landolt Std Field 005808+004232           |
     140|       15 | CAL03      | PS1 Cal Perseus Cluster / Abell 426 031900+413300            |
     141|       16 | CAL04      | PS1 Cal SA95 Dense Landolt Std Field 065207+001710           |
     142|       17 | CAL05      | PS1 Cal Taurus dark cloud (diverse extinction) 043000+250000 |
     143|       18 | CAL06      | PS1 Cal SA98 Densle Landolt Std Field 065207+001710          |
     144|       19 | CAL07      | PS1 Cal M81-M82-HolmIX (Local Group) 095640+692200           |
     145|       20 | CAL08      | PS1 Cal Virgo Cluster (Center) 123000+130000                 |
     146|       21 | CAL09      | PS1 Cal Hercules Cluster (Center) 160508+1745+28             |
     147|       22 | CAL10      | PS1 Cal M17-M24 Region 181400+000544                         |
     148|       23 | CAL11      | PS1 Cal SA110 Dense Landolt Std Field 184207+000544          |
     149|       24 | CAL12      | PS1 Cal Kepler Mission Astrometric field 192240+443000       |
     150|       25 | CAL13      | PS1 Cal 3C454.1 (Northern field) 225033+712919               |
     151|       26 | STS2       | PS1 STS2 Hyades 004000+150000                                |
     152|       27 | STS3       | PS1 STS3 Praesepe 083000+200000                              |
     153|       28 | SSS        | PS1 Solar System Sweet Spot Survey                           |
     154+----------+------------+--------------------------------------------------------------+
     155}}}
     156
     157=== ippRelease ===
     158{{{
    114159mysql> describe ippRelease;
    115 +-------------+-------------+------+-----+---------+----------------+
    116 | Field       | Type        | Null | Key | Default | Extra          |
    117 +-------------+-------------+------+-----+---------+----------------+
    118 | rel_id      | smallint(6) | NO   | PRI | NULL    | auto_increment |
    119 | surveyID    | tinyint(8)  | YES  | MUL | NULL    |                |
    120 | releaseName | varchar(16) | YES  | MUL | NULL    |                |
    121 | state       | varchar(16) | YES  | MUL | NULL    |                |
    122 | dataRelease | tinyint(8)  | YES  |     | NULL    |                |
    123 +-------------+-------------+------+-----+---------+----------------+
    124 5 rows in set (0.00 sec)
    125 }}}
    126 
    127 The survey table is a copy of the PSPS table Survey. Example entries are
    128 
    129 {{{
    130 +----------+------+------------------------------------+
    131 | surveyID | name | description                        |
    132 +----------+------+------------------------------------+
    133 |        0 | 3PI  | PS1 3PI Survey                     |
    134 |        1 | MD01 | PS1 MD01 XMM-LSS-DXS 022224-043000 |
    135 +----------+------+------------------------------------+
     160+---------------+-------------+------+-----+---------+----------------+
     161| Field         | Type        | Null | Key | Default | Extra          |
     162+---------------+-------------+------+-----+---------+----------------+
     163| rel_id        | int(8)      | NO   | PRI | NULL    | auto_increment |
     164| surveyID      | int(8)      | YES  | MUL | NULL    |                |
     165| releaseName   | varchar(64) | YES  | MUL | NULL    |                |
     166| release_state | varchar(16) | YES  | MUL | NULL    |                |
     167| dataRelease   | int(8)      | YES  |     | NULL    |                |
     168| priority      | int(11)     | YES  |     | NULL    |                |
     169+---------------+-------------+------+-----+---------+----------------+
     1706 rows in set (0.00 sec)
    136171}}}
    137172
    138173The columns in ippRelease are
    139  * rel_id primary key for the row
    140  * surveyID for to the survey that the data in a particular release contains
    141  * releaseName - human friendly identifier for the release
     174 * rel_id - the primary key for the row
     175 * surveyID of the corresponding
     176 * releaseName - a human friendly identifier for the release
    142177 * state - current status of the release. Several values have been identified
    143178   * active - results for the release are currently available to the community
    144    * pending - release identified, construction may have begun, but the results not yet ready to be available by default
     179   * pending - release identified, construction may have begun, but the results not yet ready to be available
    145180   * archive - the release contains the results of a previous processing of the data that has been superseded.
    146181   * drop - the results of the release have been abandoned and should be ignored
    147  * dataRelease - If not null, the dataRelease value in the corresponding PSPS tables.
    148 
    149 Some examples
    150 {{{
    151 +--------+----------+-------------+---------+-------------+
    152 | rel_id | surveyID | releaseName | state   | dataRelease |
    153 +--------+----------+-------------+---------+-------------+
    154 |      1 |        0 | 3PI.nightly | active  |        NULL |
    155 |      2 |        0 | 3PI.GR0     | active  |           0 |
    156 |      3 |        0 | 3PI.GR1     | pending |           1 |
    157 |      4 |        0 | SAS.V8      | archive |           8 |
    158 |      5 |        0 | SAS.V9      | active  |           9 |
    159 +--------+----------+-------------+---------+-------------+
    160 }}}
    161 
    162 The 3PI.nightly release is active. Data is being currently added to it and is available to users who request
    163 data from that processing. Since these data are no longer being ingested into PSPS the dataRelease field is
    164 null.
    165 
    166 3PI.GR0 refers to the version of LAP processing that is currently in the PSPS database PS1_3PI. Data
    167 corresponding to that release is identified by dataRelease 0, while 3PI.GR1 refers to the future
    168 LAP release which will begin construction soon. Once that release gets under way it's state will be set
    169 to active while 3PI.GR0 will be set to "archive". XXX: insure that PSPS uses a different dataRelease ...
    170 
    171 The two SAS entries refer to PS1_SAS_08 and PS1_SAS_09.
     182 * dataRelease - If not negative, the dataRelease value in the PSPS tables corresponding to this ippRelease.
     183 * priority - An integer value giving the priority of the various releases for the survey.
     184
     185Here is an example of how this system might work for the 3pi survey. Assume that we have finished the
     186current grand reprocessing and are ready to start the next one.
     187
     188{{{
     189mysql> select * from ippRelease where releaseName like '3%' order by priority desc;
     190+--------+----------+-------------+---------------+-------------+----------+
     191| rel_id | surveyID | releaseName | release_state | dataRelease | priority |
     192+--------+----------+-------------+---------------+-------------+----------+
     193|      5 |        0 | 3PI.GR1     | active        |           1 |        2 |
     194|      1 |        0 | 3PI.nightly | active        |          -1 |        0 |
     195|      2 |        0 | 3PI.GR0     | archive       |           0 |       -1 |
     196|      6 |        0 | 3PI.GR2     | pending       |           2 |     NULL |
     197+--------+----------+-------------+---------------+-------------+----------+
     1984 rows in set (0.00 sec)
     199}}}
     200
     201There are 4 releases shown here.
     202
     203The 3PI.nightly release is active. Data is being currently added to it by the IPP nightly science
     204processing. Since these data are not being ingested into PSPS the dataRelease column is -1.
     205
     2063PI.GR1 refers to the current 3PI "grand reprocessing". This release is active. dataRelease == 1
     207
     208The 2011 grand reprocessing is named here 3PI.GR0 (LAP.ThreePi.20110809). The state for this
     209release is archive. Data corresponding to that release is identified by dataRelease == 0.
     210
     2113PI.GR2 is the next grand reprocessing dataRelease == 2. The state of this release is pending.
     212
     213The priority column is used to select the release of the data that should returned by the IPP
     214when users request a given exposure or skycell. The query above demonstrates the idea. It orders the
     215releases by descending priority. The postage stamp server and other release mechanisms
     216will use queries ordered this way.
     217
     218Data will be returned from the highest priority release. In our example this is 3PI.GR1. If the
     219given exposure or skycell has not been (successfully) processed in the highest priority release
     220data will be returned from a lower priority release.
     221
     222Note: This scheme will be used for request where the user has not explicitly listed the release of interest.
     223All IPP data will be still be available if the user can identify it.
     224
     225== Released Exposure ==
    172226
    173227Now we turn our attention to tables that record the actual contents of a release. Single frame results
    174 are represented by the table relExp (released exposure) while stacks are represented by relSkycell
    175 (released skycell). (We expect that in the future a similar table will be defined for difference stage
    176 data products).
    177 
    178 == Released Exposure ==
     228are represented by the table relExp (released exposure) while stacks are represented by relStack
     229(released stack). We expect that in the future a similar table will be defined for difference stage
     230data.
     231
     232
    179233The relExp table encapsulates the data products that were used to construct the released data
    180 for a given exposure and the results of the calibration of the exposure. The proposed table definition
     234for a given exposure and the results of the calibration(s) of the exposure. The proposed table definition
    181235is as follows
    182236
    … …  
    186240| Field      | Type             | Null | Key | Default | Extra          |
    187241+------------+------------------+------+-----+---------+----------------+
    188 | relexp_id  | bigint(20)       | NO   | PRI | NULL    | auto_increment | Primary key for the table
    189 | rel_id     | smallint(6)      | YES  | MUL | NULL    |                | The corresponding release
    190 | exp_id     | bigint(20)       | YES  |     | NULL    |                | The IPP exposure identifier
    191 | chip_id    | bigint(20)       | YES  |     | NULL    |                | id for chipRun
    192 | cam_id     | bigint(20)       | YES  |     | NULL    |                | id for camRun
    193 | warp_id    | bigint(20)       | YES  |     | NULL    |                | id for warpRun
    194 | add_id     | bigint(20)       | YES  |     | NULL    |                | id for addRun (addstar injest into DVO)
    195 | state      | varchar(16)      | YES  | MUL | NULL    |                | state for this row see below
    196 | data_state | varchar(16)      | YES  | MUL | NULL    |                | see below
     242| relexp_id  | bigint(20)       | NO   | PRI | NULL    | auto_increment |
     243| rel_id     | int(8)           | YES  | MUL | NULL    |                |
     244| exp_id     | bigint(20)       | YES  | MUL | NULL    |                |
     245| chip_id    | bigint(20)       | YES  | MUL | NULL    |                |
     246| cam_id     | bigint(20)       | YES  | MUL | NULL    |                |
     247| state      | varchar(16)      | YES  | MUL | NULL    |                |
    197248| flags      | int(10) unsigned | YES  |     | NULL    |                |
    198 | zpt_obs    | float            | YES  |     | NULL    |                | calibrated zero point and error when released
     249| zpt_obs    | float            | YES  |     | NULL    |                |
    199250| zpt_stdev  | float            | YES  |     | NULL    |                |
    200 | path_base  | varchar(255)     | YES  |     | NULL    |                | path to any supporting files
    201 | registered | datetime         | YES  |     | NULL    |                | creation time for row
    202 | time_stamp | datetime         | YES  |     | NULL    |                | time of last update to row
     251| path_base  | varchar(255)     | YES  |     | NULL    |                |
     252| registered | datetime         | YES  |     | NULL    |                |
     253| time_stamp | datetime         | YES  |     | NULL    |                |
    203254+------------+------------------+------+-----+---------+----------------+
    204 Note: this is a prototype. The actual table will use foreign key constraints where appropriate.
     25512 rows in set (0.00 sec)
    205256}}}
    206257
    … …  
    208259to construct the results for the exposure.
    209260
    210 Each time an exposure is released a new row will be added to the table.
    211 
    212 Here are a couple of examples showing the releases for one of the SAS exposures:
    213 {{{
    214 mysql> mysql> select ippRelease.releaseName, relExp.* from ippRelease join relExp using(rel_id) where exp_id = 356773;
    215 +-------------+-----------+--------+--------+---------+--------+---------+--------+----------+------------+-------+---------+-----------+-----------+---------------------+---------------------+
    216 | releaseName | relexp_id | rel_id | exp_id | chip_id | cam_id | warp_id | add_id | state    | data_state | flags | zpt_obs | zpt_stdev | path_base | registered          | time_stamp          |
    217 +-------------+-----------+--------+--------+---------+--------+---------+--------+----------+------------+-------+---------+-----------+-----------+---------------------+---------------------+
    218 | SAS.V8      |         1 |      4 | 356773 |  496671 | 473031 |  453844 | 883989 | archive  | inpsps     |     0 | 24.4298 |  0.042454 | NULL      | 2012-09-19 15:16:36 | 2012-09-19 15:16:36 |
    219 | SAS.V9      |       621 |      5 | 356773 |  496671 | 473031 |  453844 | 888750 | released | inpsps     |     0 | 24.4298 |  0.042454 | NULL      | 2012-09-19 15:18:51 | 2012-09-19 15:18:51 |
    220 +-------------+-----------+--------+--------+---------+--------+---------+--------+----------+------------+-------+---------+-----------+-----------+---------------------+---------------------+
    221 2 rows in set (0.00 sec)
    222 
    223 }}}
    224 
    225 
    226 Note that the list of ids are the same except for the add_id. This is because SAS_v9 was constructed
    227 using the SAS_v8 pixels. For single frame exposures only the DVO injest and calibration was redone. This is
    228 indicated by the change in add_id.
    229 
    230 ==== state / data_state / flags ====
    231 There are three columns related to the state of the released exposure. We haven't yet settled on which of
    232 these data are required to manage the releases. Information that we might want to record include
    233  * state - active, drop, archive (note that the release state
    234  * has the released exposure been imported into PSPS
    235  * did the calibration apply ubercal. Perhaps: if so which version?
    236  * was the data the result of a recalibration, relphot, relastro, something else ...
     261Each time an exposure is added to a release a new row will be added to this table.
     262
     263==== state ====
     264A released exposure may go through a sequence of states
     265
     266 * processed - The initial results of the camera stage for the exposure.
     267 * poor_quality - The camRun failed to find a solution for the exposure.
     268 * calibrated - The calibration information (zero point and astrometry) has been updated by a DVO calibration operation
     269 * psps - The information from the exposure has been added to the PSPS database corresponding to the release.
     270 * drop - This relExp is to be dropped for some reason.
     271
     272The full results and history of any calibration operations will be stored in files referenced by the relExp's path_base.
    237273
    238274=== Life cycle of a relExp ===
    239 ==== Nightly science ====
    240 We envision the following potential events in the history of a released exposure for the nightly science
    241 releases
    242  * created during nightly science processing once the warp has been completed before the addRun is processed. State active.
    243  * updated to include the add_id of the addRun
    244 
    245 ==== Grand Reprocessing ====
    246 For the grand reprocessing, the relExp goes through two states
    247 
    248  * row is created when addstar is performed. State set to pending.
    249  * row is updated when ipptoPSPS is run. At that time the calibration data is updated with the resulting values and the state columns are changed to reflect the PSPS export. state => active
    250 
    251 Separating the state into two steps allows the results of the re-processing to be made available prior to
    252 PSPS injest if the user requests data from the corresponding release. The data will not be the "default" for
    253 the given exposure until the calibration is complete at the time of PSPS ingest.
    254 
    255 ==== DVO Release ====
    256 This schema allows (but does not require) the contents of DVO releases to be recorded. This would allow users
    257 of the DVO catalog to refer explicitly to a given set of pixels and calibration.
    258 
    259 == Released Skycell ==
     275
     276
     277
     278== Released Stack ==
    260279The concepts for sky cells are similar to individual exposures. A row in the table refers to a particular
    261 release and (tess_id, skycell_id).
    262 
    263 Here is the proposed table definition
    264 
    265 {{{
    266 REATE TABLE relSkycell (
    267     relstack_id BIGINT AUTO_INCREMENT,
    268     rel_id      BIGINT,
    269     skycell_id  varchar(64)
    270     tess_id     varchar(64)
    271     state       VARCHAR(16),    -- current, pending, archive, drop 
    272     data_state  VARCHAR(16),    -- nightly, relphot, relastro, psps
    273     stack_id    BIGINT,         -- id of the stack (pixels)
    274     skycal_id   BIGINT,         -- id of the sky calibration run that supplied the calibration
    275                                 -- for this release.
    276                                 -- NOTE: it is the skycal_id that we save,
    277                                 -- not the stack_id
    278     add_id      BIGINT,
    279     zpt_obs     FLOAT,          -- calibrated zero point for this release of
    280                                 -- this skycell
    281     zpt_stdev   FLOAT,
    282     path_base   VARCHAR(255),   -- path_base of any supporting files,
    283                                 -- updated WCS for example,
    284                                 -- we may just build a new cmf
    285     flags       UNSIGNED INT,
    286     registered  DATETIME,
    287     time_stamp  DATETIME,
    288     PRIMARY KEY (rel_id),
    289     KEY (rel_id, tess_id, skycell_id),  -- NOTE: no foreign key constraint
    290                                         -- becase we don't reference the
    291                                         -- stack directly. Insert query should
    292                                         -- enforce match.
    293     FOREIGN KEY(rel_id) REFERENCES ippRelease(rel_id),
    294     FOREIGN KEY(skycal_id) REFERENCES skycalRun(skycal_id),
    295     FOREIGN KEY(add_id) REFERENCES addRun(add_id)
    296 ) ENGINE=innodb DEFAULT CHARASET=latin1;
    297 }}}
    298 
    299 Stacks will also be released in two steps:
    300  * When the stack is constructed the relSkycell will be created in state pending. The calibration will not be set
    301  * Once the stack photometry and sky calibration are run the skycal_id will be included.
    302 
     280release, tess_id, and skycell_id.
     281
     282{{
     283mysql> describe relStack;
     284+-------------+------------------+------+-----+---------+----------------+
     285| Field       | Type             | Null | Key | Default | Extra          |
     286+-------------+------------------+------+-----+---------+----------------+
     287| relstack_id | bigint(20)       | NO   | PRI | NULL    | auto_increment |
     288| rel_id      | int(8)           | YES  | MUL | NULL    |                |
     289| skycal_id   | bigint(20)       | YES  | MUL | NULL    |                |
     290| skycell_id  | varchar(64)      | YES  |     | NULL    |                |
     291| tess_id     | varchar(64)      | YES  | MUL | NULL    |                |
     292| state       | varchar(16)      | YES  |     | NULL    |                |
     293| flags       | int(10) unsigned | YES  |     | NULL    |                |
     294| stack_type  | varchar(16)      | YES  |     | NULL    |                |
     295| zpt_obs     | float            | YES  |     | NULL    |                |
     296| zpt_stdev   | float            | YES  |     | NULL    |                |
     297| fwhm_major  | float            | YES  |     | NULL    |                |
     298| path_base   | varchar(255)     | YES  |     | NULL    |                |
     299| registered  | datetime         | YES  |     | NULL    |                |
     300| time_stamp  | datetime         | YES  |     | NULL    |                |
     301+-------------+------------------+------+-----+---------+----------------+
     30214 rows in set (0.00 sec)
     303
     304}}}
     305
     306There are differences between relStack and the single frame table.
     307
     308 * stack_type will have one of three values "deep", "reference", or "nightly"
     309 * fwhm_major will contain the measurement of the seeing as measured by psphotStack. This is currently not stored anywhere in the ipp database so it will be extracted from the cmf header by the process that creates the relStack entry.
    303310
    304311== Data Access ==
    … …  
    307314 * IPP_RELEASE - select data from corresponding ipp release
    308315 * RUN_TYPE - specify stack type (nightly, deep, reference) or difference type (warp-warp, warp-stack, stack-stack)
     316
    309317This will allow users to more efficiently select the data that they want.
    310318
    311 If a particular release name is not supplied an appropriate release will be chosen by the system.
     319If a particular release name is not supplied an appropriate release will be chosen by the system using the priority system described above.
     320
     321TODO:
     322 * Allow selection based on fwhm_max (min??) as requested by several users.