IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Initial Version and Version 1 of DVO_format


Ignore:
Timestamp:
May 29, 2009, 1:38:57 PM (17 years ago)
Author:
beaumont
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • DVO_format

    v1 v1  
     1
     2= DVO Table Structures =
     3
     4The DVO shell is the easiest way to explore data contained in a DVO database. However, involved analysis usually requires custom designed tools. For these tasks, the DVO shell may not be convenient. This document discusses the structure of the DVO database tables, with the intent of outlining how one might work with these tables using external tools. I give particular emphasis to working with DVO tables in IDL.
     5
     6All of the DVO tables are saved as multi-extension FITS files. The first extension of the fits file is usually a dummy extension, and most of the interesting information is in the next extension. To read one of these files into an IDL data structure, use the MRDFITS routine from the IDL Astronomy User's Library.
     7
     8Note that this article is concerned only with reading data from the DVO tables, and not changing their contents. In my opinion, given the number of dependencies and relationships between different quantities in these tables, it is best to use the IPP to update the DVO database.
     9
     10{{{
     11IDL> data = mrdfits( fileName, 1, header)
     12}}}
     13
     14== File Hierarchy ==
     15
     16The top level of a DVO database contains three files and many folders
     17
     18{{{
     19beaumont% ls
     20Images.dat     n0000  n2230  n4500  n6730  s0000  s2230  s4500  s6730
     21Photcodes.dat  n0730  n3000  n5230  n7500  s0730  s3000  s5230  s7500
     22SkyTable.fits  n1500  n3730  n6000  n8230  s1500  s3730  s6000  s8230
     23}}}
     24
     25Each of the folders contains science information for a specific subset of the sky, and has 0, 1, or many sets of the same 4 files:
     26
     27{{{
     28beaumont% ls n0000/
     290148.cpm  0148.cpn  0148.cps  0148.cpt  0149.cpm  0149.cpn  0149.cps  0149.cpt
     30}}}
     31
     32Let us first discuss the 3 files at the top of the hierarchy, and then move onto the 4 types of files in the subdirectories
     33
     34== Top Level Files ==
     35The top level of the DVO database has 3 data files; Images.dat, Photcodes.dat, and SkyTable.fits
     36
     37=== Images.dat ===
     38This file contains information about each 'image' in the DVO database. A single exposure from a telescope is often broken into multiple images. For example, one Megacam exposure is 37 images, 1 for each of the 36 chips, and one dummy image which summarizes the exposure.
     39
     40The header summarizes the data fields in the Images.dat file. Some useful quantities:
     41 * IMAGE_ID : An integer uniquely identifying this image. This is the 1-indexed row number in the Images.dat table
     42 * PHOTCODE: This image's photcode (see Photcodes.dat discussion below)
     43 * NAME:  A string giving the filenam
     44 * EXPTIME: Exposure time, usually in seconds (but check the header)
     45 * NSTAR: Number of stars detected in the image
     46 * NX / NY: Pixel size of image
     47In addition, there are a number of astrometry related keywords like CRVAL, etc. However, I think these are defined in a non-standard way. Luckily, there is more direct information about each object's location in other tables.
     48
     49=== Photcodes.dat ===
     50Each filter / chip combination is associated with a unique photcode. References to photcodes are found in the Images.dat table above, and in the .cpm tables discussed below. The Photcodes.dat file contains information about all of the photcodes recognized by the IPP.
     51
     52The photcodes table format is simple:
     53{{{
     54IDL> p = mrdfits('Photcodes.dat', 1, h)
     55MRDFITS: Binary table.  21 columns by  818 rows.
     56
     57IDL> help, p[150], /structure
     58** Structure <8d1ec2c>, 21 tags, length=108, data length=104, refs=2:
     59   CODE            INT            204
     60   NAME            STRING    'MEGACAM.g.04'
     61   TYPE            STRING    ''
     62   DUMMY           STRING    ''
     63   C_LAM           INT          26460
     64   C_LAM_ERR       INT              0
     65   X_ERR           INT              0
     66   K               FLOAT         -0.150000
     67   C1              LONG                 0
     68   C2              LONG                 0
     69   EQUIV           LONG                 1
     70   NC              LONG                 1
     71   X               FLOAT     Array[4]
     72   ASTROM_ERR_SYS  FLOAT           0.00000
     73   ASTROM_ERR_SCALE
     74                   FLOAT           0.00000
     75   ASTROM_ERR_MAG_SCALE
     76                   FLOAT           1.00000
     77   ASTROM_POOR_MASK
     78                   INT              0
     79   ASTROM_BAD_MASK INT          14472
     80   PHOTOM_ERR_SYS  FLOAT           0.00000
     81   PHOTOM_POOR_MASK
     82                   INT              0
     83   PHOTOM_BAD_MASK INT              0
     84}}}
     85
     86 * CODE: The photcode value which other tables reference via their PHOTCODE data field. Note that this is NOT the same as the row number in the Photcodes.dat table.
     87 * NAME: A more instructive label. In this case, this photcode referes to CCD4 on megacam, using the G filter.
     88 * C_LAM, C_LAM_ERR, X_ERR, K, C1, C2, EQUIV, NC, X: Constants which translate between instrumental magnitudes and AB magnitudes. For examples of their use, see Ohana/src/libdvo/src/dvo_photcode_ops.c in the IPP
     89
     90There are a number of bit masks for each photcode. During reduction, the IPP sets a number of quality flags, which get copied into the PHOT_FLAGS data field in the .cpm files. The photcode fields ASTROM_BAD_MASK, ASTROM_POOR_MASK, PHOTOM_BAD_MASK encode which of those bits indicate failures in an object's astrometry or photometry. See examples at the end for working with these values in IDL.
     91
     92=== SkyTable.fits ===
     93As mentioned above, the objects in the DVO database get grouped into regions based on their position in the sky, and placed in one of the DVO subdirectories like n0000. The SkyTable.fits file gives the sky boundaries of each of these subdirectories, and can be used to locate which directory a particular region of interest is in:
     94
     95{{{
     96IDL> sky = mrdfits('SkyTable.fits', 1, h)
     97MRDFITS: Binary table.  12 columns by  161905 rows.
     98IDL> help, sky[1], /struct
     99** Structure <a04384c>, 12 tags, length=80, data length=80, refs=2:
     100   R_MIN           FLOAT           0.00000
     101   R_MAX           FLOAT           360.000
     102   D_MIN           FLOAT           0.00000
     103   D_MAX           FLOAT           7.50000
     104   CHILD_S         LONG                25
     105   CHILD_E         LONG                50
     106   PARENT          LONG                 1
     107   INDEX           LONG                 1
     108   DEPTH           STRING    ''
     109   CHILD           STRING    ''
     110   TABLE           STRING    ''
     111   NAME            STRING    'n0000'
     112}}}
     113
     114This particular sky region, n0000, is a 7.5 degree wide declination band centered at dec=3.75.
     115
     116== Subdirectory files ==
     117Most of the science content of the DVO database is found in the subdirectories with names like n0000. There are four types of files, with suffixes of .cpm, .cpn, .cps, and .cpt.
     118
     119=== The Averages Table: .cpt ===
     120Each .cpt table has one entry for each object in that region of the sky. It summarizes the average properties of that object ''as long as those properties can be derived independently of the filter used''. This means that the magnitude of the object cannot be found here (it is different for each filter).
     121
     122'''Note: it is important that your DVO database is sorted. Do this by running addstar -resort on your database after initially populating it (usually with addstar -update)'''
     123
     124{{{
     125IDL> t = mrdfits('n0000/0148.cpt', 1, h)
     126MRDFITS: Binary table.  23 columns by  285933 rows.
     127IDL> help, t, /struct
     128** Structure <8f069ac>, 23 tags, length=96, data length=96, refs=1:
     129   RA              DOUBLE           102.62241
     130   DEC             DOUBLE         0.015223970
     131   RA_ERR          FLOAT         0.0480983
     132   DEC_ERR         FLOAT         0.0481240
     133   U_RA            FLOAT           0.00000
     134   U_DEC           FLOAT           0.00000
     135   V_RA_ERR        FLOAT           0.00000
     136   V_DEC_ERR       FLOAT           0.00000
     137   PAR             FLOAT           0.00000
     138   PAR_ERR         FLOAT           0.00000
     139   SIGMA_POS       FLOAT          -374.000
     140   CHISQ_POS       FLOAT           0.00000
     141   NUMBER_POS      INT              0
     142   NMEASURE        INT            136
     143   NMISSING        INT              0
     144   NEXTEND         INT              0
     145   OFF_MEASURE     LONG                 0
     146   OFF_MISSING     LONG                -1
     147   OFF_EXTEND      LONG                -1
     148   FLAGS           LONG                 1
     149   OBJ_ID          LONG                 0
     150   CAT_ID          LONG               578
     151   EXT_ID          LONG      Array[2]
     152}}}
     153
     154 * RA, DEC: Average RA and DEC, in degrees
     155 * RA_ERR, DEC:ERR: Error in position, in arcsec
     156 * U_RA, V_RA, etc: proper motions, in mas/yr (note that these must be explicitly calculated using relastro via the command --update-objects +pm)
     157 * PAR: Parallax in mas (must be populated via relastro --update-objects +par)
     158 * NMEAURE: Number of measurements associated with this object
     159 * NMISSING: Number of times that an object doesn't appear in other exposures. Not currently populated.
     160 * OFF_MEASURE: The zero-indexed row number in the measurement (.cpm) table that contains the first measurement for this object. That row, and the next NMEASURE -1 rows, are the measurements for this object
     161
     162=== The Measurement Table: .cpm ===
     163Each .cpm table contains all of the measurement information for each object in the average (.cpt) table
     164
     165{{{
     166IDL> m = mrdfits('n0000/0148.cpm',1,h, range=[0,5])
     167MRDFITS: Binary table.  42 columns by  6 rows.
     168IDL> help, m, /structure                           
     169** Structure <8ee013c>, 42 tags, length=160, data length=158, refs=1:
     170   D_RA            FLOAT         0.0669184
     171   D_DEC           FLOAT          0.135676
     172   MAG             FLOAT               NaN
     173   M_CAL           FLOAT           0.00000
     174   M_APER          FLOAT               NaN
     175   MAG_ERR         FLOAT               NaN
     176   MAG_CAL_ERR     FLOAT          0.660210
     177   M_TIME          FLOAT           1.21749
     178   AIRMASS         FLOAT           4.57614
     179   AZ              FLOAT          -89.8685
     180   X_CCD           FLOAT          0.393112
     181   Y_CCD           FLOAT           3051.92
     182   SKY_FLUX        FLOAT           17.5981
     183   SKY_FLUX_ERR    FLOAT           3.94064
     184   TIME            LONG        1067008742
     185   AVE_REF         LONG                 0
     186   DET_ID          LONG              1504
     187   IMAGE_ID        LONG                 9
     188   OBJ_ID          LONG                 0
     189   CAT_ID          LONG               578
     190   EXT_ID          LONG      Array[2]
     191   PSF_QF          FLOAT               NaN
     192   PSF_CHISQ       FLOAT               NaN
     193   PSF_NDOF        LONG                 0
     194   PSF_NPIX        LONG                 0
     195   CR_NSIGMA       FLOAT               NaN
     196   EXT_NSIGMA      FLOAT               NaN
     197   FWHM_MAJOR      INT              0
     198   FWHM_MINOR      INT              0
     199   PSF_THETA       INT              0
     200   MXX             INT              0
     201   MXY             INT              0
     202   MYY             INT              0
     203   TIME_MSEC       INT              0
     204   PHOTCODE        INT            307
     205   X_CCD_ERR       INT            358
     206   Y_CCD_ERR       INT             48
     207   PAD             STRING    ''
     208   POSANGLE        INT           8262
     209   PLTSCALE        FLOAT          0.185317
     210   DB_FLAGS        LONG                 0                       
     211   PHOT_FLAGS      LONG         268470272
     212}}}
     213
     214 * D_RA: Offset, in arcseconds, between the average ra (in the .cpt table) and this measurement.
     215 * D_DEC: Offset, in arcseconds, between the average dec (in the .cpt table) and this measurement.
     216 * MAG: Instrumental magnitude for this object
     217 * X_CCD, Y_CCD: Pixel location of this object's centroid
     218 * TIME: Time of exposure. For the data I worked with, this was the Unix time (seconds since UTC Jan 1 1970)
     219 * AVE_REF: The zero indexed row number in the averages (.cpt) table of the object associated with this measurement
     220 * IMAGE_ID: The '''one-indexed''' row number in Images.dat for this measurement's parent image
     221 * X_CCD_ERR, Y_CCD_ERR: The centroid error, in 1/100th of a pixel
     222 * FWHM_MAJOR, FWHM_MINOR: The psf fwhm in 1/100th of a pixel
     223 * Photcode: The photcode corresponding to this measurement's parent image.
     224 * DB_FLAGS: Flags supplied by relastro, relphot, etc
     225 * PHOT_FLAGS: Flags set by the IPP. Bad photcode bit masks are found in the Photcodes.dat table
     226
     227=== The missing table: .cpn ===
     228=== The secfilt table: .cps ===
     229
     230== Examples of Working with DVO tables in IDL ==
     231
     232 * Determine which objects in a .cpm table have bad astrometry
     233
     234{{{
     235m = mrdfits('n0000/0148.cpm',1)
     236p = mrdfits('Photcodes.dat',1)
     237
     238photcode = m[0].photcode
     239index = where(photcode eq p.photcode)
     240mask = p[index].astrom_bad_mask ;- usually the same mask for all photcodes associated with a given camera
     241
     242bad = where((m.phot_flags and mask) ne 0, complement = good)
     243
     244}}}
     245
     246
     247 * Extract all measurements of a given object into a variable ''without reading in the entire .cpm table''
     248{{{
     249t = mrdfits('n0000/0148.cpt', 1)
     250object = 10
     251lo = t[object].off_measure
     252hi = lo + t[object].nmeasure - 1
     253
     254measurements = mrdfits('n0000/0148.cpm',1, range=[lo,hi])
     255}}}