IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changeset 9689


Ignore:
Timestamp:
Oct 20, 2006, 2:58:33 PM (20 years ago)
Author:
Paul Price
Message:

Updated the configuration and concepts sections.

File:
1 edited

Legend:

Unmodified
Added
Removed

  • trunk/doc/modules/ModulesSDRS.tex

    r6762 r9689  
    1 %%% $Id: ModulesSDRS.tex,v 1.77 2006-04-04 18:52:47 eugene Exp $
     1%%% $Id: ModulesSDRS.tex,v 1.78 2006-10-21 00:58:33 price Exp $
    22\documentclass[panstarrs,spec]{panstarrs}
    33
    … …  
    4747PSDC-430-006  &   Pan-STARRS PS-1 IPP Algorithm Design Document \\ \hline
    4848PSDC-430-011  &   Pan-STARRS PS-1 IPP System/Subsystem Design Description \\ \hline
     49PSDC-430-???  &   Pan-STARRS PS-1 IPP Configurations \\ \hline
    4950\DocumentsExternal
    5051Posix Standard & Open Group Based Specifications Issue 6, IEEE Std 1003.1, 2003 \\
    … …  
    6970programs which perform complete data analysis tasks (an ``analysis
    7071stage'').  The modules may be tied together within a C framework or
    71 using a high-level scripting language.  Bindings of the Modules are
    72 made available to the scripting language using the program SWIG.
     72using a high-level scripting language.
    7373
    7474In order to preserve name space, globally-visible structures and
    … …  
    7676Modules''.
    7777
    78 \section{Runtime Configuration Data}
    79 
    80 PSLib defines a \code{psMetadata} structure which can carry labeled
    81 data of arbitrary types.  The associated functions implemented by
    82 PSLib consist of tools to manipulate and extract data from
    83 \code{psMetadata} collections.  A particular application of the
    84 \code{psMetadata} structure within PSLib is to carry the data from a
    85 FITS header.  Other general-purpose information is also carried with
    86 the structure.  Functions are available to fill a \code{psMetadata}
    87 collection from a text-based configuration file using a human-readable
    88 syntax, and to fill a \code{psMetadata} collection from a properly
    89 formatted XML document.
    90 
    91 In the IPP Modules, we use \code{psMetadata} collections to carry
    92 run-time configuration data used by the data analysis modules.  Below,
    93 in the discussion of the various modules, this configuration
    94 information is defined by specifying the name of the data item of
    95 interest, the conceptual meaning of that data item, and the allowed
    96 values for the data item.  In this section, we discuss top-level
    97 concepts related to the configuration information, including the
    98 sources of the run-time configuration data and special
    99 operations used to extract information from the configuration system.
    100 
    101 \subsection{Configuration Data Sources}
    102 
    103 All modules need to load some configuration information defining
    104 parameters which may be configured at run-time.  We break these
    105 parameters down into four levels:
    106 \begin{itemize}
    107 \item Options for the particular site installation of the
    108   pipeline: the {\it site};
    109 \item Options specifying the instrument setup: the {\it camera};
    110 \item Options specifying the format of the FITS file: the {\it
    111   format}; and
    112 \item Options specifying the particular parameter choices that affect
    113   the details of an analysis: the {\it recipe}.
    114 \end{itemize}
    115 Note that these are arranged in an hierarchical order, with the site
    116 configuration being the most general, and the recipe configuration the
    117 most specific.  For example, not all sites will have to deal with all
    118 cameras, and different cameras may require different recipes at
    119 different times according to their particular quirks, analysis
    120 experimentations, or their evolution.
    121 
    122 Each of the levels will have a metadata configuration file.  In the
    123 case of the site configuration, the filename shall be that specified
    124 by the \code{-site} option on the command line if provided, the
    125 environment variable \code{PS_SITE}, if defined, or \code{~/.ipprc}
    126 otherwise.  The camera configuration shall be specified by the
    127 \code{-camera} option on the command line if provided, or shall be
    128 inferred from a FITS header (more detail below).  The recipe
    129 configuration shall be specified by the \code{-recipe} option on the
    130 command line if provided, or from the the camera configuration (more
    131 detail below).
    132 
    133 \subsection{Configuration Files}
    134 
    135 \subsubsection{Site Configuration}
    136 
    137 The site configuration file must contain the following:
    138 \begin{itemize}
    139 \item The database configuration:
    140   \begin{itemize}
    141   \item \code{DBSERVER} of type \code{STR}: The database host name for
    142     \code{psDBInit}.
    143   \item \code{DBUSER} of type \code{STR}: The database user name for
    144     \code{psDBInit}.
    145   \item \code{DBPASSWORD} of type \code{STR}: The corresponding
    146     database password for \code{psDBInit}.
    147   \item \code{DBNAME} of type \code{STR}: The database name for
    148     \code{psDBInit}.
    149   \end{itemize}
    150 \item \code{CAMERAS} of type \code{METADATA}: A list of instruments
    151   that the system understands.  Cameras are specified as separate
    152   metadata entries, with the name of the camera as the key, and the
    153   filename of the camera configuration file (of type \code{STR}) as
    154   the data.
    155 \end{itemize}
    156 and may also contain the following psLib configuration options:
    157 \begin{itemize}
    158 \item \code{TIME} of type \code{STR}: The time configuration file (for
    159   \code{psTimeInitialize}).
    160 \item \code{LOGLEVEL} of type \code{S32}: The log level for
    161   \code{psLogSetLevel}.
    162 \item \code{LOGFORMAT} of type \code{STR}: The log format for
    163   \code{psLogSetFormat}.
    164 \item \code{LOGDEST} of type \code{STR}: The log destination for
    165   \code{psLogSetDestination}.
    166 \item \code{TRACE} of type \code{METADATA}: A list of components with
    167   the desired trace level (of type \code{S32}) for each.
    168 \end{itemize}
    169 
    170 \tbd{No doubt there is a need for better security than storing the
    171 database password directly in the file, but we push this problem onto
    172 the stack for now.}
    173 
    174 \tbd{We will add other data sources in the future, e.g., file paths,
    175 configuration for Nebulous and DVO, etc.}.
    176 
    177 
    178 An example site configuration file:
    179 
    180 \begin{verbatim}
    181 ### Example .ipprc file
    182 
    183 ### Database configuration
    184 DBSERVER        STR     ippdb.ifa.hawaii.edu    # Database host name (for psDBInit)
    185 DBUSER          STR     ipp                     # Database user name (for psDBInit)
    186 DBPASSWORD      STR     password                # Database password (for psDBInit)
    187 
    188 ### Setups for each camera system
    189 CAMERAS         METADATA
    190         MEGACAM_RAW     STR     megacam_raw.config
    191         MEGACAM_SPLICE  STR     megacam_splice.config
    192         GPC1_RAW        STR     gpc1_raw.config
    193         LRIS_BLUE       STR     lris_blue.config
    194         LRIS_RED        STR     lris_red.config
    195 END
    196 
    197 ### psLib setup
    198 TIME            STR     time.config             # Time configuration file
    199 LOGLEVEL        S32     3                       # Logging level; 3=INFO
    200 LOGFORMAT       STR     THLNM                   # Log format
    201 LOGDEST         STR     STDOUT                  # Log destination
    202 TRACE           METADATA                        # Trace levels
    203         psLib.math.psPolynomial         S32     6
    204         psLib.image.psImageConvolve     S32     2
    205 END
    206 \end{verbatim}
    207 
    208 \subsubsection{Camera configuration}
    209 
    210 The camera configuration file is a fairly simple configuration file
    211 containing information particular to a particular camera, regardless
    212 of the file format used to represent that camera.  The camera configuration
    213 consists of the following elements:
    214 \begin{itemize}
    215 \item \code{FORMATS} of type \code{METADATA}: this contains a list of
    216   known FITS file formats with the file names (of type \code{STR}) of
    217   the configuration files;
    218 \item \code{FPA} of type \code{METADATA}: this contains a list of
    219   chips, each with a string list (type \code{STR} of the component
    220   cells; and
    221 \item \code{RECIPES} of type \code{METADATA}: this contains a list of
    222   recipes used for the camera with the file names (of type \code{STR}
    223   of the configuration files.
    224 \end{itemize}
    225 
    226 An example camera configuration file:
    227 
    228 \begin{verbatim}
    229 # Camera configuration file for MegaCam: describes the camera
    230 
    231 # File formats that we know about
    232 FORMATS         METADATA
    233         RAW     STR     megacam_raw.config
    234         SPLICE  STR     megacam_splice.config
    235         SPLIT   STR     megacam_split.config
    236 END
    237 
    238 
    239 # Description of camera --- all the chips and the cells that comprise them
    240 FPA     METADATA
    241         ccd00   STR     left right
    242         ccd01   STR     left right
    243         ccd02   STR     left right
    244         ccd03   STR     left right
    245         ccd04   STR     left right
    246         ccd05   STR     left right
    247         ccd06   STR     left right
    248         ccd07   STR     left right
    249         ccd08   STR     left right
    250         ccd09   STR     left right
    251         ccd10   STR     left right
    252         ccd11   STR     left right
    253         ccd12   STR     left right
    254         ccd13   STR     left right
    255         ccd14   STR     left right
    256         ccd15   STR     left right
    257         ccd16   STR     left right
    258         ccd17   STR     left right
    259         ccd18   STR     left right
    260         ccd19   STR     left right
    261         ccd20   STR     left right
    262         ccd21   STR     left right
    263         ccd22   STR     left right
    264         ccd23   STR     left right
    265         ccd24   STR     left right
    266         ccd25   STR     left right
    267         ccd26   STR     left right
    268         ccd27   STR     left right
    269         ccd28   STR     left right
    270         ccd29   STR     left right
    271         ccd30   STR     left right
    272         ccd31   STR     left right
    273         ccd32   STR     left right
    274         ccd33   STR     left right
    275         ccd34   STR     left right
    276         ccd35   STR     left right
    277 END
    278 
    279 
    280 # Recipe options
    281 RECIPES         METADATA
    282         PHASE2          STR     phase2.config           # Phase 2 recipe details
    283         PSPHOT          STR     psphot.config           # psphot details
    284 END
    285 \end{verbatim}
    286 
    287 
    288 \subsubsection{FITS file format}
    289 
    290 The FITS file format configuration files are somewhat complicated and
    291 involved, since they must not only specify how to translate the pixels
    292 from a FITS file into a focal plane hierarchy
    293 (\S\ref{sec:focalplane}), but must also specify how to derive the
    294 various values the IPP needs (\S\ref{sec:concepts}).  Moreover, they
    295 must be able to do these for the great variety of cameras in use in
    296 the astronomical community.
    297 
    298 Example camera configuration files are included in an appendix, but
    299 below we explain the components.
    300 
    301 \paragraph{FITS File to Focal Plane Hierarchy}
    302 
    303 The Focal Plane hierarchy (\code{pmFPA, pmChip, pmCell, pmReadout}) is
    304 explained in more detail in \S\ref{sec:focalplane}.  The top level, an
    305 FPA contains one or more chips, which correspond to a contiguous piece
    306 of silicon.  A chip contains one or more cells, which correspond to a
    307 single amplifier.  A cell contains one or more readouts, which
    308 correspond to individual reads of the detector.
    309 
    310 The FITS data storage formation is a standard in the astronomical
    311 community for storing astronomical images.  A FITS file consists of an
    312 arbitrary number of coupled human readable \code{ASCII} header
    313 segments and binary data segments.  The headers describe the format
    314 and layout of the data segments.  The first of these groups is
    315 traditionally called the ``primary header unit'' (PHU) and the rest are
    316 referred to as ``extensions''.  The header segments may contain
    317 extensive documentary information related to the interpretation of the
    318 data.  Although the FITS format defines a standard representation of
    319 the data, the header metadata is not so consistently defined within
    320 the astronomical community.  Also, the flexibility of the data format
    321 means that different representations are possible for the same
    322 fundamental collection of data.  The tools presented in this section
    323 provide a method to define and constrain the wide range of possible
    324 FITS representations of astronomical images.
    325 
    326 Within the FITS data representation, there are various choices which
    327 can and have been made for the placement of the pixels in the file.
    328 In the simplest case, the camera consists of a single chip consisting
    329 of a single cell always read with a single readout.  In this case, the
    330 image data could be written as part of the primary header unit.  In a
    331 more complex case with multiple chips and multiple cells, the data may
    332 be organized in several ways.  The data may be distributed into
    333 multiple files or in multiple FITS data extensions.  A single camera
    334 image may be written as a collection of files for individual chips
    335 with separate extensions for each cell (CFH12K.split, GPC).  Another
    336 camera may write a single file with multiple extensions for each cell
    337 (Megacam.raw), or multiple extensions per chip, with each cell
    338 representing portions of the chip image (Megacam.splice, CFHT-IR).
    339 
    340 In all of these representations, there are only two basic distinctions
    341 in how the pixel data is stored: what level in the hierarchy the
    342 entire FITS file corresponds to (FPA, chip, or cell), and what level
    343 the extensions correspond to (chip, cell or no extensions at all).
    344 Knowing these, and having a list of the extensions, we can construct
    345 the focal plane hierarchy.
    346 
    347 Note that a single data extension, consisting of a uniform grid of
    348 pixels, can only naturally represent a cell or a chip.  In order to
    349 represent the entire focal plane array as a single grid, some
    350 artificial choices would be made to fill-in or ignore the gaps between
    351 chips and their relative rotations.  Within our framework, a complete
    352 focal plane mosaic of multiple chips could be represented as a single
    353 extension by treating the collection of pixels as if they were from a
    354 single chip. 
    355 
    356 To define the hierarchy, we specify the following keywords:
    357 \begin{itemize}
    358 \item \code{RULE} of type \code{METADATA}: contains headers with their
    359   respective values that are required to be in the PHU of any FITS
    360   file of this type.
    361 
    362 \item \code{FILE} of type \code{METADATA}: contains information on
    363   the global format of the FITS file with the following entries:
    364   \begin{itemize}
    365   \item \code{PHU} of type \code{STR}: May be one of \code{FPA},
    366     \code{CHIP}, or \code{CHIP}.  This specifies the focal plane level
    367     of the Primary Header Unit, and hence the entire FITS file (the
    368     'class' of the file).
    369 
    370   \item \code{EXTENSIONS} of type \code{STR}: May be one of
    371     \code{CHIP}, \code{CELL} or \code{NONE}, though not of a level
    372     higher than that specified by the \code{PHU}.  This specifies what
    373     each extension represents.
    374 
    375   \item \code{FPA.NAME} of type \code{STR}: Specifies a header keyword
    376     in the primary header for a unique identifier for the FPA name
    377     (e.g., an observation number).
    378 
    379   \item \code{CHIP.NAME} of type \code{STR}: Need only be included if
    380     \code{PHU} is \code{CHIP} or \code{CELL}.  Specifies a header
    381     keyword in the primary header for a unique identifier for the chip
    382     name (e.g., the CCD identification number or name).
    383 
    384   \item \code{CELL.NAME} of type \code{STR}: Need only be included if
    385     \code{PHU} is \code{CELL}.  Specifies a header keyword in the
    386     primary header for a unique identifier for the cell name (e.g.,
    387     the amplifier identification).
    388   \end{itemize}
    389 
    390 \item \code{CONTENTS} of type \code{METADATA}: Specifies what the
    391   contents of the FITS file are.  Each entry is an extension name with
    392   the corresponding value being a string listing the source and the
    393   cell type, separated by a colon (e.g., \code{ccd01:left
    394   ccd01:right}).  If \code{EXTENSIONS=NONE} then the \code{CONTENTS}
    395   is ignored (since there are no extensions to list).
    396 
    397 \item \code{CELLS} of type \code{METADATA}: specifies the cell types.
    398   Entries are the cell types, each of type \code{METADATA}, with the
    399   values being PS concept values appropriate for each cell type (more
    400   detail later) \tbd{link to more detail}.  In the event that
    401   \code{EXTENSIONS=NONE}, the \code{CELLS} is used as a list of all
    402   cells present in the file.
    403 
    404 \item \code{TRANSLATION} of type \code{METADATA}
    405 
    406 \item \code{DEFAULTS} of type \code{METADATA}
    407 
    408 \item \code{DATABASE} of type \code{METADATA}
    409 
    410 \item \code{FORMATS} of type \code{METADATA}
    411 
    412 \end{itemize}
    413 
    414 An example:
    415 
    416 \begin{verbatim}
    417 # The raw MegaCam data comes off the telescope with each of the chips stored in extensions of a MEF file.
    418 
    419 # How to identify this type
    420 RULE    METADATA
    421         TELESCOP        STR     CFHT 3.6m
    422         DETECTOR        STR     MegaCam
    423         EXTEND          BOOL    T
    424         NEXTEND         S32     72
    425 END
    426 
    427 # How to read this data
    428 FORMAT  METADATA
    429         PHU             STR     FPA     # The FITS file represents an entire FPA
    430         EXTENSIONS      STR     CELL    # The extensions represent cells
    431         FPA.NAME        STR     EXPNUM  # A PHU keyword for unique identifier within the hierarchy level
    432 END
    433 
    434 # What's in the FITS file?
    435 CONTENTS        METADATA
    436         # Extension name, chip name:type
    437         amp00           STR     ccd00:left
    438         amp01           STR     ccd00:right
    439         amp02           STR     ccd01:left
    440         amp03           STR     ccd01:right
    441         amp04           STR     ccd02:left
    442         amp05           STR     ccd02:right
    443         amp06           STR     ccd03:left
    444         amp07           STR     ccd03:right
    445         amp08           STR     ccd04:left
    446         amp09           STR     ccd04:right
    447         amp10           STR     ccd05:left
    448         amp11           STR     ccd05:right
    449         amp12           STR     ccd06:left
    450         amp13           STR     ccd06:right
    451         amp14           STR     ccd07:left
    452         amp15           STR     ccd07:right
    453         amp16           STR     ccd08:left
    454         amp17           STR     ccd08:right
    455         amp18           STR     ccd09:left
    456         amp19           STR     ccd09:right
    457         amp20           STR     ccd10:left
    458         amp21           STR     ccd10:right
    459         amp22           STR     ccd11:left
    460         amp23           STR     ccd11:right
    461         amp24           STR     ccd12:left
    462         amp25           STR     ccd12:right
    463         amp26           STR     ccd13:left
    464         amp27           STR     ccd13:right
    465         amp28           STR     ccd14:left
    466         amp29           STR     ccd14:right
    467         amp30           STR     ccd15:left
    468         amp31           STR     ccd15:right
    469         amp32           STR     ccd16:left
    470         amp33           STR     ccd16:right
    471         amp34           STR     ccd17:left
    472         amp35           STR     ccd17:right
    473         amp36           STR     ccd18:left
    474         amp37           STR     ccd18:right
    475         amp38           STR     ccd19:left
    476         amp39           STR     ccd19:right
    477         amp40           STR     ccd20:left
    478         amp41           STR     ccd20:right
    479         amp42           STR     ccd21:left
    480         amp43           STR     ccd21:right
    481         amp44           STR     ccd22:left
    482         amp45           STR     ccd22:right
    483         amp46           STR     ccd23:left
    484         amp47           STR     ccd23:right
    485         amp48           STR     ccd24:left
    486         amp49           STR     ccd24:right
    487         amp50           STR     ccd25:left
    488         amp51           STR     ccd25:right
    489         amp52           STR     ccd26:left
    490         amp53           STR     ccd26:right
    491         amp54           STR     ccd27:left
    492         amp55           STR     ccd27:right
    493         amp56           STR     ccd28:left
    494         amp57           STR     ccd28:right
    495         amp58           STR     ccd29:left
    496         amp59           STR     ccd29:right
    497         amp60           STR     ccd30:left
    498         amp61           STR     ccd30:right
    499         amp62           STR     ccd31:left
    500         amp63           STR     ccd31:right
    501         amp64           STR     ccd32:left
    502         amp65           STR     ccd32:right
    503         amp66           STR     ccd33:left
    504         amp67           STR     ccd33:right
    505         amp68           STR     ccd34:left
    506         amp69           STR     ccd34:right
    507         amp70           STR     ccd35:left
    508         amp71           STR     ccd35:right
    509 END
    510 
    511 # Specify the cell data
    512 CELLS   METADATA
    513         left    METADATA        # Left amplifier
    514                 CELL.NAME               STR     LeftSide
    515                 CELL.BIASSEC.SOURCE     STR     HEADER
    516                 CELL.TRIMSEC.SOURCE     STR     HEADER
    517                 CELL.BIASSEC            STR     BIASSEC
    518                 CELL.TRIMSEC            STR     DATASEC
    519                 CELL.XPARITY            S32     1 # We could have specified this as a DEFAULT, but this works
    520                 CELL.X0                 S32     1
    521                 CELL.Y0                 S32     1
    522         END
    523         right   METADATA        # Right amplifier
    524                 CELL.NAME               STR     RightSide
    525                 CELL.BIASSEC.SOURCE     STR     HEADER
    526                 CELL.TRIMSEC.SOURCE     STR     HEADER
    527                 CELL.BIASSEC            STR     BIASSEC
    528                 CELL.TRIMSEC            STR     DATASEC
    529                 CELL.XPARITY            S32     -1 # This cell is read out in the opposite direction
    530                 CELL.X0                 S32     2048
    531                 CELL.Y0                 S32     1
    532         END
    533 END
    534 
    535 # How to translate PS concepts into FITS headers
    536 TRANSLATION     METADATA
    537         FPA.NAME                STR     EXPNUM
    538         FPA.AIRMASS             STR     AIRMASS
    539         FPA.FILTER              STR     FILTER
    540         FPA.POSANGLE            STR     ROTANGLE
    541         FPA.RA                  STR     RA
    542         FPA.DEC                 STR     DEC
    543         FPA.RADECSYS            STR     RADECSYS
    544         CELL.EXPOSURE           STR     EXPTIME
    545         CELL.DARKTIME           STR     DARKTIME
    546         CELL.GAIN               STR     GAIN
    547         CELL.READNOISE          STR     RDNOISE
    548         CELL.SATURATION         STR     SATURATE
    549         CELL.TIME               STR     MJD-OBS
    550         CELL.XBIN               STR     CCDBIN1
    551         CELL.YBIN               STR     CCDBIN2
    552 END
    553 
    554 # Default PS concepts that may be specified by value
    555 DEFAULTS        METADATA
    556         CELL.READDIR            S32     1               # Cell is read in x direction
    557         CELL.BAD                S32     0
    558         CELL.TIMESYS            STR     UTC
    559         CELL.YPARITY            S32     1
    560 END
    561 
    562 # How to translation PS concepts into database lookups
    563 DATABASE        METADATA
    564         TYPE            dbEntry         TABLE           COLUMN          GIVENDBCOL      GIVENPS
    565 #       FPA.BIAS        METADATA
    566 #               TABLE   STR     Camera
    567 #               COLUMN  STR     gain
    568 #               chipId  STR     {CHIP.NAME}
    569 #               cellId  STR     {CELL.NAME}
    570 #               time    STR     {CELL.TIME}
    571 #       END
    572 #       CELL.GAIN       dbEntry         Camera          gain            chipId,cellId   CHIP.NAME,CELL.NAME
    573 #       CELL.READNOISE  dbEntry         Camera          readNoise       chipId,cellId   CHIP.NAME,CELL.NAME
    574 
    575 # A database entry refers to a particular column (COLUMN) in a
    576 # particular table (TABLE), given certain PS concepts (GIVENPS) that
    577 # match certain database columns (GIVENDBCOL).
    578 END
    579 
    580 
    581 # Where there might be some ambiguity, specify the format
    582 FORMATS         METADATA
    583         FPA.RA          STR     HOURS
    584         FPA.DEC         STR     DEGREES
    585         CELL.TIME       STR     MJD
    586 #       CELL.BINNING    STR     SEPARATE
    587         CELL.X0         STR     FORTRAN
    588         CELL.Y0         STR     FORTRAN
    589 END
    590 \end{verbatim}
    591 
    592 Observe how the \code{CONTENTS} specifies the extension name, which we
    593 know from the \code{EXTENSIONS} is a cell, and that each extension is
    594 associated with a chip, and has a cell type.
    595 
    596 \paragraph{Deriving concept values}
    597 \label{sec:derivingconcepts}
    598 
    599 The PS concepts are described in more detail in \S\ref{sec:concepts}.
    600 Basically, astronomical cameras generally store the important details
    601 (``concepts'') in different ways.  This is generally manifested in the
    602 choice of different FITS header keywords to describe the same concept,
    603 but one can also imagine deriving values from a database or a known
    604 default.
    605 
    606 We therefore specify the following keywords:
    607 \begin{itemize}
    608 \item \code{TRANSLATION} of type \code{METADATA} is a translation
    609   table for understanding PS concepts in terms of FITS headers.  The
    610   PS concept (keyword) is derived from the FITS header given in the
    611   value.
    612 \item \code{DATABASE} of type \code{METADATA} is a formula for
    613   obtaining a PS concept from the database.  Each component is of a
    614   user-specified type containing \code{TABLE}, \code{COLUMN},
    615   \code{GIVENDBCOL} and \code{GIVENPS}.  The idea is that to obtain
    616   the value of a PS concept, one refers to a particular \code{COLUMN}
    617   in a particular \code{TABLE}, where the value of certain PS concepts
    618   (\code{GIVENPS}; multiple values separated by a comma or semicolon)
    619   match certain database columns (\code{GIVENDBCOL}; multiple values
    620   separated by a comma or semicolon).
    621 \item \code{DEFAULTS} of type \code{METADATA} is a set of default
    622   values of PS concepts for the camera.  The PS concept (keyword) is
    623   assigned the value.  There is also limited dependency allowed; see
    624   \S\ref{sec:concepts}.
    625 \end{itemize}
    626 
    627 An example:
    628 \begin{verbatim}
    629 # How to translate PS concepts into FITS headers
    630 TRANSLATION     METADATA
    631         FPA.NAME        STR     EXPNUM
    632         FPA.AIRMASS     STR     AIRMASS
    633         FPA.FILTER      STR     FILTER
    634         FPA.POSANGLE    STR     ROTANGLE
    635         FPA.RA          STR     RA
    636         FPA.DEC         STR     DEC
    637         FPA.RADECSYS    STR     RADECSYS
    638         FPA.MJD         STR     MJD-OBS
    639         CELL.EXPOSURE   STR     EXPTIME
    640         CELL.DARKTIME   STR     DARKTIME
    641         CELL.XBIN       STR     CCDBIN1
    642         CELL.YBIN       STR     CCDBIN2
    643         CELL.SATURATION STR     SATURATE
    644 END
    645 
    646 # Default PS concepts that may be specified by value
    647 DEFAULTS        METADATA
    648         CELL.BAD                S32     0
    649         CELL.PARITY.DEPEND      STR     CHIP.NAME
    650         CELL.PARITY    METADATA
    651                 amp00   S32     1
    652                 amp01   S32     -1
    653                 amp02   S32     1
    654                 amp03   S32     -1
    655         END
    656 END
    657 
    658 # How to translate PS concepts into database lookups
    659 DATABASE        METADATA
    660         TYPE            dbEntry         TABLE           COLUMN          GIVENDBCOL      GIVENPS
    661         CELL.GAIN       dbEntry         Camera          gain            chipId,cellId   CHIP.NAME,CELL.NAME
    662         CELL.READNOISE  dbEntry         Camera          readNoise       chipId,cellId   CHIP.NAME,CELL.NAME
    663 END
    664 \end{verbatim}
    665 
    666 The \code{.DEPEND} entry in the \code{DEFAULTS} will be explained in
    667 \S\ref{sec:concepts}.
    668 
    669 \paragraph{Indentification by rule}
    670 \label{sec:camerarule}
    671 
    672 The function \code{pmConfigCameraFromHeader} requires that the camera
    673 configuration also contains a rule on how to recognise that a FITS
    674 header comes from that camera.
    675 
    676 We therefore specify another keyword: \code{RULE} of type
    677 \code{METADATA}: Contains a list of FITS headers keywords and values
    678 (of the appropriate type) against which actual headers are compared to
    679 determine if it matches the camera type.
    680 
    681 An example is:
    682 \begin{verbatim}
    683 # How to identify this type
    684 RULE    METADATA
    685         TELESCOP        STR     CFHT 3.6m
    686         DETECTOR        STR     MegaCam
    687         EXTEND          BOOL    T
    688         NEXTEND         S32     72
    689 END
    690 \end{verbatim}
    691 
    692 \paragraph{Recipes}
    693 
    694 The camera configuration file must also contain filenames for the
    695 recipe configuration files.  We include \code{RECIPES} of type
    696 \code{METADATA} with component keywords being the various recipe names
    697 and the values (of type \code{STR}) the corresponding recipe
    698 configuration filename.
    699 
    700 An example:
    701 \begin{verbatim}
    702 # Recipes for LRIS
    703 RECIPES METADATA
    704         PHASE1          STR     lris_phase1.config
    705         PHASE2          STR     lris_phase2.config
    706         PHASE4          STR     lris_phase4.config
    707 END
    708 \end{verbatim}
    709 
    710 \subsubsection{Recipe Configuration}
    711 
    712 \tbd{The contents of the recipe configuration file are dependent upon
    713 the particular module, and hence are not specified here at this time.}
    714 
    715 
    716 \subsection{PS Concepts}
    717 
    718 \subsubsection{Ingest}
    719 
    720 For different camera systems, these concepts are not always known by
    721 the same name, nor are they generally obtained in the same manner, and
    722 so their source or value must be specified in the camera configuration
    723 file.  At ingest, the value of a concept shall be found by searching in
    724 the following order:
    725 \begin{itemize}
    726 \item The cell data from the \code{CELLS} metadata in the camera configuration.
    727 \item The FITS header via the \code{TRANSLATION} table.
    728 \item The \code{DATABASE} lookup.
    729 \item The \code{DEFAULTS} value.
    730 \end{itemize}
    731 
    732 \subsubsection{Dependencies for defaults}
    733 
    734 In the \code{DEFAULTS} table in the camera configuration, we allow the
    735 specification of the concept with an additional suffix, \code{DEPEND}.
    736 The value (of type \code{STR}) of the \code{CONCEPT.DEPEND} is the
    737 name of a concept on which the first concept depends.  For example, it
    738 might depend on the chip name.  Then the first concept becomes of type
    739 \code{METADATA}, with the component keywords being the value of the
    740 second concept (on which the first depends).  To avoid infinite
    741 recursion, no further dependency is permitted.  We also allow an entry
    742 \code{CONCEPT.DEFAULT} specifiying the default value of the concept if
    743 a match is not made with the dependcency list.  An example of the
    744 dependency:
    745 
    746 \begin{verbatim}
    747 # Default PS concepts that may be specified by value
    748 DEFAULTS        METADATA
    749         CELL.GAIN.DEPEND     STR     CHIP.NAME
    750         CELL.GAIN.DEFAULT    STR     1.0
    751         CELL.GAIN    METADATA
    752                 ccd00   F32     1.2
    753                 ccd01   F32     3.4
    754                 ccd02   F32     5.6
    755         END
    756 END
    757 \end{verbatim}
    758 
    759 \subsubsection{FORMATS}
    760 
    761 Because of the variety of methods for specifying these concepts
    762 (especially in FITS headers), we must also specify additional
    763 information in the camera configuration that specifies how to
    764 interpret the data provided.  These are provided in an entry
    765 \code{FORMATS} (of type \code{METADATA}) in the camera configuration.
    766 Within the \code{FORMATS} metadata, there is a string for each of the
    767 concepts that requires a format to be specified.
    768 
    769 \paragraph{CELL.TIME}
    770 
    771 The time at which the shutter opens is represented in a variety of
    772 ways in FITS files, so care must be taken to specify what the format
    773 is in the file under consideration.  Permitted values of
    774 \code{CELL.TIME.FORMAT} are:
    775 
    776 \begin{itemize}
    777 \item \code{JD}: The value pointed to by \code{CELL.TIME} is to be
    778   interpreted as a Julian Date.
    779 \item \code{MJD}: The value pointed to by \code{CELL.TIME} is to be
    780   interpreted as a Modified Julian Date.
    781 \item \code{ISO}: The value pointed to by \code{CELL.TIME} is to be
    782   interpreted as an ISO date-time (yyyy-mm-ddThh:mm:ss.ss).
    783 \item \code{SEPARATE}: The date and time are specified separately, and
    784   the \code{CELL.TIME} contains the headers for the date and the time
    785   separated by whitespace or a comma.  Then it is necessary to add
    786   additional qualifiers to specify the formats of these:
    787   \begin{itemize}
    788   \item \code{PRE2000}: The year is in the old style two-digit format
    789     popular before the year 2000, and it should be assumed that the
    790     date is in the twentieth century.
    791   \item \code{BACKWARDS}: The date is in the format dd-mm-yyyy or
    792     dd/mm/yyyy.
    793   \item \code{SOD}: The time is specified as seconds-of-day.
    794   \end{itemize}
    795 \end{itemize}
    796 
    797 Note that the FITS standard is that the time in the header refers to
    798 the {\it start} of the observation. 
    799 
    800 \tbd{the PRE2000 and BACKWARDS qualifiers should be replace with
    801 explicit format definitions in the form YYYY/MM/DD}
    802 
    803 \tbd{In the future, we might add additional qualifiers that calculate
    804 the start time of the observation based on someone foolishly putting
    805 the end- or mid-time in the header.}
    806 
    807 \tbd{Should we move CELL.TIMESYS into the format as well?}
    808 
    809 \paragraph{FPA.RA and FPA.DEC}
    810 
    811 The RA and Declination of the boresight might be specified in a few
    812 ways.  We need to specify both how the value is interpreted and the
    813 units.  \code{FPA.RA.FORMAT} and \code{FPA.DEC.FORMAT} should be one
    814 of the following:
    815 
    816 \begin{itemize}
    817 \item \code{HOURS}: The value pointed to by the concept should be
    818   interpreted as being in hours.
    819 \item \code{DEGREES}: The value pointed to by the concept should be
    820   interpreted as being in degrees.
    821 \item \code{RADIANS}: The value pointed to by the concept should be
    822   interpreted as being in radians.
    823 \end{itemize}
    824 
    825 How the value is interpreted can be determined from the type of the
    826 header: if it is of type \code{STR}, then we can reasonably assume
    827 that it is in sexagesimal format with colons or spaces as separators;
    828 and if it is of type \code{F32} (or \code{F64}), then we can assume
    829 that it is in decimal format.
    830 
    831 \subsubsection{Implicit format information}
    832 
    833 While details like the units of the right ascension in the header must
    834 be specified explicitly, some other details can be determined from
    835 implicit information.
    836 
    837 \begin{itemize}
    838 \item \code{FPA.RA} and \code{FPA.DEC}: if the value on ingest is of
    839 type \code{STRING}, then it may be interpreted as sexagesimal
    840 notation, ``\code{dd:mm:ss.ss}'', or ``\code{dd:mm.mmm}''.  A space
    841 may be used instead of a colon to separate the values.  Otherwise, if
    842 the value is of a numerical type (\code{F32} or \code{F64}), then that
    843 is the appropriate value.
    844 \item \code{CELL.XBIN} and \code{CELL.YBIN}: if the value on ingest is
    845 of type \code{STRING}, then it may be interpreted as ``\code{x,y}'',
    846 where \code{x} is the binning in x, and \code{y} is the binning in y.
    847 A space may be used instead of a comma, and there may even be a space
    848 before or after the comma (or both).  Otherwise, if the value is of a
    849 numerical type (\code{S32}, etc), then that is the appropriate value.
    850 \item \code{CELL.BIASSEC} and \code{CELL.TRIMSEC}: These values on
    851 ingest should always be of type \code{STRING}.  If they contain a
    852 square bracket, then they may be interpreted as a list of standard
    853 region specifications, ``\code{[x0:x1,y0:y1];[x2:x3,y2:y3];...}'',
    854 where the semi-colon may be replaced by spaces.  Otherwise, the string
    855 may be interpreted as a FITS header (or headers, separated by spaces,
    856 commas or semi-colons) that contains the appropriate values.
    857 \end{itemize}
    858 
    859 \tbd{the use of implicit interpretation of formats should be
    860   discouraged: format interpretation guides should be provided}
    861 
    862 \subsection{Configuration APIs}
    863 
    864 \begin{prototype}
    865 bool pmConfigRead(psMetadata **site, psMetadata **camera, psMetadata **recipe,
    866                   int *argc, char **argv, const char *recipeName);
    867 psMetadata *pmConfigCameraFromHeader(const psMetadata *site, const psMetadata *header);
    868 psMetadata *pmConfigRecipeFromCamera(const psMetadata *camera, const char *recipeName);
    869 \end{prototype}
    870 
    871 \code{pmConfigRead} shall load the \code{site} configuration
    872 (according to the above rule for determining the source).  The
    873 \code{camera} configuration shall also be loaded if it is specified on
    874 the command line (\code{argc, argv}); otherwise it shall be set to
    875 \code{NULL}.  The \code{recipe} shall also be loaded from the command
    876 line (if specified) or, if the camera configuration has been loaded,
    877 from the camera configuration and recipe specification therein (see
    878 below).  In dealing with the command line parameters, the functions
    879 shall use the appropriate functions in psLib to retrieve and remove
    880 the relevant options from the argument list; this simplifies
    881 assignment of the mandatory arguments, since all the optional command
    882 line arguments are removed leaving only the mandatory arguments.  The
    883 following psLib setups shall also be performed if they are specified
    884 in the site configuration:
    885 \begin{itemize}
    886 \item the function shall call \code{psTimeInitialize} with the
    887   configuration file specified by \code{TIME}.
    888 \item the function shall call \code{psLogSetLevel} with the logging
    889   level specified by \code{LOGLEVEL}.
    890 \item the function shall call \code{psLogSetFormat} with the log
    891   format specified by \code{LOGFORMAT}.
    892 \item the function shall call \code{psTraceSetLevel} with the component names and
    893   trace levels specified by the \code{TRACE}.
    894 \end{itemize}
    895 Note that additional log/trace command-line options may be specified
    896 and interpretted using the \code{psArgumentVerbosity} function from
    897 psLib.  These options should (in the case of logging) override the
    898 configuration-supplied information or (in the case of tracing)
    899 supplement it.
    900 
    901 \code{pmConfigCameraFromHeader} shall load the \code{camera}
    902 configuration based on the contents of the FITS \code{header}, using
    903 the list of known cameras contained in the \code{site} configuration.
    904 If more than one camera matches the FITS header, a warning shall be
    905 generated and the first matching camera returned.
    906 
    907 \code{pmConfigRecipeFromCamera} shall load the \code{recipe}
    908 configuration based on the \code{recipeName} and the list of known
    909 recipes contained in the \code{camera} configuration (details below).
    910 
    911 \begin{prototype}
    912 bool pmConfigValidateCamera(const psMetadata *camera, const psMetadata *header);
    913 \end{prototype}
    914 
    915 This function, used by \code{pmConfigCameraFromHeader}, shall return
    916 \code{true} if the FITS \code{header} matches the rule contained in
    917 the \code{camera} configuration (see \S\ref{sec:camerarule});
    918 otherwise it shall return \code{false}.
    919 
    920 \begin{prototype}
    921 psDB *pmConfigDB(psMetadata *site);
    922 \end{prototype}
    923 
    924 \code{pmConfigDB} shall use the \code{site} configuration data to open
    925 a database handle.  \tbd{This is fairly straightforward at the moment,
    926 but will change when we beef up security.}
    927 
    928 \subsubsection{Example usage}
    929 
    930 The following is provided as an example of how the above functions
    931 are envisioned in use.
    932 
    933 \begin{verbatim}
    934 int main(int argc, char *argv[])
    935 {
    936     // Parse other command-line arguments here
    937     psMetadata *site = NULL;            // Site configuration
    938     psMetadata *camera = NULL;          // Camera configuration
    939     psMetadata *recipe = NULL;          // Recipe configuration
    940     if (! pmConfigRead(&site, &camera, &recipe, &argc, argv, "moduleName")) {
    941         psLogMsg("moduleName", PS_LOG_ERROR, "Can't find site configuration!\n");
    942         exit(EXIT_FAILURE);
    943     }
    944     // Parse other command-line arguments here
    945 
    946     // The command-line argument list now contains only mandatory arguments
    947     // Assume the first of these is an input image
    948     char *imageName = argv[1];          // Name of FITS file
    949     psFits *imageFH = psFitsOpen(imageName, "r"); // File handle for FITS file
    950     if (! imageFH) {
    951         psLogMsg("moduleName", PS_LOG_ERROR, "Can't open input image %s\n", imageName);
    952         exit(EXIT_FAILURE);
    953     }
    954     psMetadata *header = psFitsReadHeader(NULL, imageFH); // FITS header
    955 
    956     if (!camera && !(camera = pmConfigCameraFromHeader(site, header))) {
    957         psLogMsg("moduleName", PS_LOG_ERROR, "Can't find camera configuration!\n");
    958         exit(EXIT_FAILURE);
    959     }
    960 
    961     if (! recipe && !(recipe = pmConfigRecipeFromCamera(camera, "moduleName"))) {
    962         psLogMsg("moduleName", PS_LOG_ERROR, "Can't find recipe configuration!\n");
    963         exit(EXIT_FAILURE);
    964     }
    965 
    966     // Now go on and do stuff
    967     ....
    968 }
    969 \end{verbatim}
     78%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
     79
     80\section{Configuration}
     81
     82Configuration information in the IPP is provided on four levels: the
     83site, camera, format and recipe configurations.  The use of these, and
     84their contents are described in an additional document, PSDC-430-???,
     85``Pan-STARRS PS1 IPP Configurations''.  Here we define functions that
     86read these configuration files.
     87
     88
     89\subsubsection{Configuration information}
     90
     91This structure stores the configuration information: the site, camera
     92and recipe configuration, the command-line arguments, the pmFPAfiles
     93used, and the database handle.
     94\begin{datatype}
     95typedef struct {
     96    psMetadata *site;                   ///< Site configuration
     97    psMetadata *camera;                 ///< Camera specification
     98    const char *cameraName;             ///< Camera name
     99    psMetadata *recipes;                ///< Recipes for processing
     100    psMetadata *arguments;              ///< Processed command-line arguments
     101    psMetadata *files;                  ///< pmFPAfiles used for analysis
     102    psDB *database;                     ///< Database handle
     103    int *argc;                          ///< Number of command-line arguments
     104    char **argv;                        ///< Command-line arguments (raw version)
     105    // Private members
     106    p_pmRecipeSource recipesRead;       ///< Which recipe sources have been read
     107    psMetadata *recipesSource;          ///< Where each recipe came from
     108} pmConfig;
     109\end{datatype}
     110
     111The site, camera and recipe configurations are represented here.  The
     112camera format configuration is stored elsewhere (within the
     113\code{pmFPA}).
     114
     115Note that the \code{arguments} are different from the \code{argc} and
     116\code{argv}, since it is the parsed version.  This metadata may also
     117be used to carry program options, if desired.
     118
     119The allocator for this structure is:
     120\begin{prototype}
     121pmConfig *pmConfigAlloc(int *argc,      /// Number of command-line arguments
     122                        char **argv     /// Command-line arguments
     123                       );
     124\end{prototype}
     125
     126
     127\subsection{Reading configuration files}
     128
     129Read configuration information from the command line:
     130\begin{prototype}
     131pmConfig *pmConfigRead(int *argc,       ///< Number of command-line arguments
     132                       char **argv      ///< Array of command-line arguments
     133                      );
     134\end{prototype}
     135pmConfigRead loads the site configuration (the file name is specified
     136by "-site SITE_FILE" on the command-line, the PS_SITE environment
     137variable, or it is \code{$HOME/.ipprc}).  The configuration search %$
     138path is set. The camera configuration is loaded if it is specified on
     139the command line ("-camera CAMERA_FILE"). Recipes specified on the
     140command line ("-recipe RECIPE_NAME RECIPE_SOURCE") are also loaded.
     141These command-line arguments are removed from from the command-line,
     142to simplify parsing.  The psLib log, trace and time setups are also
     143performed if specified in the site configuration.
     144
     145Read a configuration file:
     146\begin{prototype}
     147bool pmConfigFileRead(psMetadata **config, ///< Config to output
     148                      const char *name, ///< Name of file
     149                      const char *description ///< Description of file
     150                     );
     151\end{prototype}
     152Read a metadata configuration file into the supplied metadata.
     153Produce an error and return false if there's a problem.
     154
     155Set static configuration information:
     156\begin{prototype}
     157void pmConfigSet(const char *path ///< Search paths for configuration files; colon-delimited directories
     158                );
     159\end{prototype}
     160The search path for the configuration files is a local static
     161variable, set by this function.
     162
     163Free static memory used in the configuration system:
     164\begin{prototype}
     165void pmConfigDone(void);
     166\end{prototype}
     167
     168
     169\subsection{Camera configuration}
     170
     171Validate a header against the camera format:
     172\begin{prototype}
     173bool pmConfigValidateCameraFormat(const psMetadata *cameraFormat, ///< Camera format containing the RULE
     174                                  const psMetadata *header // FITS header for the PHU
     175                                 );
     176\end{prototype}
     177Given a FITS header (the PHU header), check it against the RULE
     178metadata contained within the camera format; return true if it
     179matches.
     180
     181Determine the camera format (and camera if unknown) from examining the
     182header:
     183\begin{prototype}
     184psMetadata *pmConfigCameraFormatFromHeader(pmConfig *config, ///< The configuration
     185        const psMetadata *header ///< The FITS header
     186                                          );
     187\end{prototype}
     188Given a FITS header, check it against all known cameras (unless we
     189already know which camera, from pmConfigRead) and all known formats
     190for those cameras in order to identify which is appropriate.  The
     191first matching format is accepted; further matches produce warnings.
     192The accepted camera is saved in the configuration.  The accepted
     193format is returned.
     194
     195Return the camera configuration specified by name:
     196\begin{prototype}
     197psMetadata *pmConfigCameraByName(pmConfig *config, ///< The configuration
     198                                 const char *cameraName ///< The camera name header
     199                                );
     200\end{prototype}
     201Given a camera name, returns the camera configuration metadata.
     202
     203Make the supplied header conform to the nominated camera format:
     204\begin{prototype}
     205bool pmConfigConformHeader(psMetadata *header, ///< Header to conform
     206                           const psMetadata *format ///< Camera format
     207                          );
     208\end{prototype}
     209Given a FITS header, make it conform to the RULE in the specified
     210camera format.  This is useful for switching between formats, or
     211generating fake data that must be recognised by
     212pmConfigCameraFormatFromHeader.
     213
     214
     215\subsection{Recipes}
     216
     217Recipes are read from a variety of sources: the command line, the site
     218configuration, and the camera configuration.  In addition, recipes
     219specified on the command line may be a symbolic link to recipes
     220defined in the other locations.  Since the various sources of recipes
     221may be read at different times (e.g., the camera configuration may be
     222read from the command line, or it may only be read once a FITS file
     223has been opened for inspection), we specify the following list of
     224recipe sources:
     225\begin{datatype}
     226typedef enum {
     227    P_PM_RECIPE_SOURCE_NONE        = 0x00, ///< None yet
     228    P_PM_RECIPE_SOURCE_SITE        = 0x01, ///< Site configuration
     229    P_PM_RECIPE_SOURCE_CAMERA      = 0x02, ///< Camera configuration
     230    P_PM_RECIPE_SOURCE_CL          = 0x04, ///< Command-line
     231    P_PM_RECIPE_SOURCE_SYMBOLIC    = 0x14, ///< Symbolic link, specified on command-line
     232    P_PM_RECIPE_SOURCE_ALL         = 0xff  ///< All sources
     233} p_pmRecipeSource;
     234\end{datatype}
     235This structure is private to psModules --- there is no need for the
     236user to know about it.
     237
     238See PSDC-430-???, ``IPP Configurations'' for information on symbolic
     239links, and how these are identified and resolved, and also for the
     240precedence of sources.
     241
     242Read recipes:
     243\begin{prototype}
     244bool pmConfigReadRecipes(pmConfig *config ///< Configuration
     245                        );
     246\end{prototype}
     247Attempt to read recipes from the sources that are available but have
     248not already been read.Having read a recipe, attempt to resolve
     249symbolic links that were specified on the command line.
     250
     251
     252\subsection{Database}
     253
     254Setup the database:
     255\begin{prototype}
     256psDB *pmConfigDB(pmConfig *config       ///< Configuration
     257                );
     258\end{prototype}
     259Initialise the database connection using the DBSERVER, DBNAME, DBUSER,
     260DBPASSWORD values provided in the site configuration.  Stores the
     261database handle in the configuration, and also returns it.
     262
     263
     264\subsection{File sets}
     265
     266Read the command-line for files (or a text file containing a list of
     267files):
     268\begin{prototype}
     269psArray *pmConfigFileSets(pmConfig *config, ///< Configuration, containing command-line arguments
     270                          const char *file, ///< CL argument specifying a filename
     271                          const char *list ///< CL argument specifying a text file with a list of filenames
     272                         );
     273\end{prototype}
     274Given the 'file' and 'list' arguments (e.g., "-file" and "-list"),
     275find the arguments associated with these words and interpret them as
     276lists of files.  Return an array of the resulting filenames.
     277
     278Stuff associated files from the command-line into a metadata:
     279\begin{prototype}
     280bool pmConfigFileSetsMD(psMetadata *metadata, ///< Metadata into which to stuff the array
     281                        pmConfig *config, ///< Configuration (which command-line arguments)
     282                        const char *name, ///< Name for array in the metadata
     283                        const char *file, ///< CL argument specifying a filename
     284                        const char *list ///< CL argument specifying a text file with a list of filenames
     285                       );
     286\end{prototype}
     287Calls pmConfigFileSets to parse the command line for filenames (or a
     288list which provides filenames), and stuffs the array of filenames into
     289the metadata under "name".
    970290
    971291%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    … …  
    984304plane hierarchy.
    985305
    986 After ingest (performed in \code{pmFPARead}, the user may safely
    987 assume that all of the above concepts exist at the appropriate level
    988 (meaning the user needn't be hampered by excessive error checking), is
    989 of the specified type (meaning the user doesn't need to worry about
    990 whether the value of interest is stored in, e.g., floating point or
    991 double precision or even a colon-delimited string) and in the
    992 specified format (meaning the user doesn't need to know, e.g., whether
    993 the right ascension is in radians or degrees) --- all the conversions
    994 are handled by the ``concepts'' functions at ingest.
     306After ingest, the user may safely assume that all of the above
     307concepts exist at the appropriate level (meaning the user needn't be
     308hampered by excessive error checking), is of the specified type
     309(meaning the user doesn't need to worry about whether the value of
     310interest is stored in, e.g., floating point or double precision or
     311even a colon-delimited string) and in the specified format (meaning
     312the user doesn't need to know, e.g., whether the right ascension is in
     313radians or degrees) --- all the conversions are handled by the
     314``concepts'' functions at ingest.
    995315
    996316Most of the structures and functions in this section are intended to
    … …  
    1007327
    1008328\begin{datatype}
    1009 typedef psMetadataItem* (*p_pmConceptReadFunc)(pmFPA *fpa, pmChip *chip, pmCell *cell, psDB *db);
    1010 typedef bool (*p_pmConceptWriteFunc)(pmFPA *fpa, pmChip *chip, pmCell *cell, psDB *db);
    1011329typedef struct {
    1012     psMetadataItem *blank;          // Blank value of concept; also contains the name
    1013     p_pmConceptReadFunc read;       // Function to call to read the concept
    1014     p_pmConceptWriteFunc write;     // Function to call to write the concept
    1015 } p_pmConceptSpec;
     330    psMetadataItem *blank;              ///< Blank value of concept; also contains the name and comment
     331    pmConceptParseFunc parse;           ///< Function to call to read the concept
     332    pmConceptFormatFunc format;         ///< Function to call to write the concept
     333} pmConceptSpec;
    1016334\end{datatype}
    1017335
    1018 \code{blank} is a \code{psMetadataItem} that provides the name, type
    1019 and default/blank value for the concept.  \code{read} and \code{write}
    1020 provide the functions to read and write.
    1021 
    1022 A concept specification may be allocated:
    1023 \begin{prototype}
    1024 p_pmConceptSpec *p_pmConceptSpecAlloc(psMetadataItem *blank, pmConceptReadFunc read,
    1025                                       pmConceptWriteFunc write);
    1026 \end{prototype}
     336The allocator is:
     337\begin{prototype}
     338pmConceptSpec *pmConceptSpecAlloc(psMetadataItem *blank, ///< Blank value; contains the name
     339                                  pmConceptParseFunc parse, ///< Function to call to parse the concept
     340                                  pmConceptFormatFunc format ///< Function to call to format the concept
     341                                 );
     342\end{prototype}
     343
     344\code{blank} provides the name, type and default/blank value for the
     345concept.  \code{parse} and \code{format} provide the functions to
     346parse and format the concepts:
     347
     348\begin{datatype}
     349typedef psMetadataItem* (*pmConceptParseFunc)(const psMetadataItem *concept, ///< Concept to parse
     350        const psMetadataItem *pattern, ///< Pattern for parsing
     351        const psMetadata *cameraFormat, ///< Camera format definition
     352        const pmFPA *fpa, ///< FPA for concept, or NULL
     353        const pmChip *chip, ///< Chip for concept, or NULL
     354        const pmCell *cell ///< Cell for concept, or NULL
     355                                             );
     356typedef psMetadataItem* (*pmConceptFormatFunc)(const psMetadataItem *concept, ///< Concept to format
     357        const psMetadata *cameraFormat, ///< Camera format definition
     358        const pmFPA *fpa, ///< FPA for concept, or NULL
     359        const pmChip *chip, ///< Chip for concept, or NULL
     360        const pmCell *cell ///< Cell for concept, or NULL
     361                                              );
     362\end{datatype}
     363
     364A concept that has just been read requires parsing, so that the final
     365product stored in the \code{concepts} metadata is of the correct type
     366and format.  The parsing function requires a \code{pattern} (generally
     367the \code{blank}), which provides the name and comment for the parsed
     368concept.  The camera format (\code{cameraFormat}) allows looking up of
     369the \code{FORMATS} or other information, and the \code{fpa},
     370\code{chip} and \code{cell} allow lookup of other concepts.
     371
     372Before writing, concepts are formatted so that the source's format
     373(likely differing from that of the psModules concepts) may be
     374respected.  Again, providing the camera format (\code{cameraFormat}),
     375\code{fpa}, \code{chip} and \code{cell} allow the formatting function
     376to access information necessary to format.
     377
    1027378
    1028379\subsection{Registering a concept}
    1029380
    1030 The concept specifications that have been registered shall be stored in
    1031 three \code{psMetadata}s, one for each level (FPA, chip, cell).
     381The concept specifications that have been registered are stored in
     382three (static) \code{psMetadata}s, one for each level (FPA, chip,
     383cell).
    1032384
    1033385Registering a concept is achieved by:
    1034386\begin{prototype}
    1035 bool pmConceptRegister(psMetadataItem *blank, pmConceptReadFunc read,
    1036                        pmConceptWriteFunc write, pmConceptLevel level);
    1037 \end{prototype}
    1038 
    1039 \code{pmConceptRegister} shall generate a concept specification from
    1040 the provided \code{blank}, and \code{read} and \code{write} functions,
     387bool pmConceptRegister(psMetadataItem *blank, ///< Blank value; contains the name and default comment
     388                       pmConceptParseFunc parse, ///< Function to call to parse the concept, or NULL
     389                       pmConceptFormatFunc format, ///< Function to call to format the concept, or NULL
     390                       pmFPALevel level ///< Level at which to store concept in the FPA hierarchy
     391                      );
     392\end{prototype}
     393
     394\code{pmConceptRegister} generates a concept specification from the
     395provided \code{blank}, and \code{parse} and \code{format} functions,
    1041396and register it in the metadata specified by the \code{level}.
    1042397
    1043 \code{pmConceptLevel} simply specifies which level in the focal plane
    1044 hiearchy the concept applies to:
    1045 \begin{datatype}
    1046 typedef enum {
    1047     PM_CONCEPT_LEVEL_FPA,               // Store in the FPA
    1048     PM_CONCEPT_LEVEL_CHIP,              // Store in the chip
    1049     PM_CONCEPT_LEVEL_CELL               // Store in the cell
    1050 } pmConceptLevel;
    1051 \end{datatype}
    1052 
    1053 A \code{read} function of \code{NULL} indicates that there is no
     398A \code{parse} function of \code{NULL} indicates that there is no
    1054399special interpretation of the concept required, and that it can be
    1055400used as read.  A \code{write} function of \code{NULL} indicates that
    … …  
    1060405\subsection{Default concepts}
    1061406
    1062 Below is a list of concepts that the IPP will use, with the expected
    1063 type and a short description.
     407Below is a list of concepts that are defined in the psModules, with
     408the expected type and a short description.
    1064409
    1065410\begin{itemize}
    1066 \item \code{FPA.NAME} (\code{psString}): An identifier (e.g., observation number) for the FPA instance
    1067 \item \code{FPA.AIRMASS} (F32): Airmass at which the observation is made (boresight)
    1068 \item \code{FPA.FILTER} (\code{psString}): Filter used in observation
    1069 \item \code{FPA.POSANGLE} (F32): Position angle for camera
    1070 \item \code{FPA.RADECSYS} (\code{psString}): System of RA,Dec (e.g., J2000 or ICRS)
    1071 \item \code{FPA.RA} (F64): Right Ascension of boresight in radians
    1072 \item \code{FPA.DEC} (F64): Declination of boresight in radians
    1073 \item \code{CHIP.NAME} (\code{psString}): The name of the chip (unique within the FPA) --- set at FITS read
    1074 \item \code{CELL.NAME} (\code{psString}): The name of the cell (unique within the parent chip) --- set at FITS read
    1075 \item \code{CELL.GAIN} (F32): CCD gain (e/ADU)
     411\item \code{FPA.CAMERA} (STR): Camera used
     412\item \code{FPA.FOCUS} (F32): Telescope focus
     413\item \code{FPA.AIRMASS} (F32): Airmass at boresight
     414\item \code{FPA.FILTER} (STR): Filter used
     415\item \code{FPA.POSANGLE} (F32): Position angle of instrument
     416\item \code{FPA.RADECSYS} (STR): Celestial coordinate system
     417\item \code{FPA.RA} (F64): Right Ascension of boresight
     418\item \code{FPA.DEC} (F64): Declination of boresight
     419\item \code{FPA.OBSTYPE} (STR): Type of observation
     420\item \code{FPA.OBJECT} (STR): Object of observation
     421\item \code{FPA.ALT} (F64): Altitude of telescope
     422\item \code{FPA.AZ} (F64): Azimuth of telescope
     423\item \code{FPA.TIMESYS} (STR): Time system
     424\item \code{FPA.TIME} (psTime): Time of exposure
     425\item \code{CHIP.XPARITY} (S32): Orientation in x compared to the rest of the FPA, -1 or +1
     426\item \code{CHIP.YPARITY} (S32): Orientation in y compared to the rest of the FPA, -1 or +1
     427\item \code{CHIP.X0} (S32): Position of (0,0) on the FPA
     428\item \code{CHIP.Y0} (S32): Position of (0,0) on the FPA
     429\item \code{CHIP.TEMP} (F32): Temperature of chip
     430\item \code{CELL.GAIN} (F32): CCD gain (e/count)
    1076431\item \code{CELL.READNOISE} (F32): CCD read noise (e)
    1077 \item \code{CELL.SATURATION} (F32): CCD saturation point (ADU)
    1078 \item \code{CELL.BAD} (F32): CCD bad pixel point (ADU)
    1079 \item \code{CELL.XPARITY} (S32): Direction of CCD readout in x relative to the rest of the chip
    1080 \item \code{CELL.YPARITY} (S32): Direction of CCD readout in y relative to the rest of the chip
    1081 \item \code{CELL.READDIR} (S32): Read direction: line (1) or column (2)
    1082 \item \code{CELL.EXPOSURE} (F32): Exposure time of image (sec)
    1083 \item \code{CELL.DARKTIME} (F32): Dark time for image (sec)
    1084 \item \code{CELL.TRIMSEC} (\code{psRegion*}): Trim region
    1085 \item \code{CELL.BIASSEC} (\code{psList*} of \code{psRegion*}): Overscan region(s)
    1086 \item \code{CELL.XBIN} (S32): CCD binning in x
    1087 \item \code{CELL.YBIN} (S32): CCD binning in y
    1088 \item \code{CELL.TIMESYS} (\code{psTimeType}): Time system in use
    1089 \item \code{CELL.TIME} (\code{psTime*}): Time of observation start
    1090 \item \code{CELL.X0} (S32): x position of cell (0,0) on the chip
    1091 \item \code{CELL.Y0} (S32): y position of cell (0,0) on the chip
     432\item \code{CELL.SATURATION} (F32): Saturation level (counts)
     433\item \code{CELL.BAD} (F32): Bad level (counts)
     434\item \code{CELL.XPARITY} (S32): Orientation in x compared to the rest of the chip, -1 or +1
     435\item \code{CELL.YPARITY} (S32): Orientation in y compared to the rest of the chip, -1 or +1
     436\item \code{CELL.READDIR} (S32): Read direction, rows=1, cols=2
     437\item \code{CELL.EXPOSURE} (F32): Exposure time (sec)
     438\item \code{CELL.DARKTIME} (F32): Time since flush (sec)
     439\item \code{CELL.TRIMSEC} (psRegion): Trim section
     440\item \code{CELL.BIASSEC} (psList of psRegions): Bias sections
     441\item \code{CELL.XBIN} (S32): Binning in x
     442\item \code{CELL.YBIN} (S32): Binning in y
     443\item \code{CELL.TIMESYS} (STR): Time system
     444\item \code{CELL.TIME} (psTime): Time of exposure
     445\item \code{CELL.X0} (S32): Position of (0,0) on the chip
     446\item \code{CELL.Y0} (S32): Position of (0,0) on the chip
    1092447\end{itemize}
    1093448
    … …  
    1098453dark time.}
    1099454
    1100 The concept specifications for the above shall be registered by
     455The concept specifications for the above may be registered by
    1101456\code{pmConceptsInit}:
    1102457\begin{prototype}
    … …  
    1106461Since defined concept specifications are required before any concept
    1107462ingest can take place, all functions that work with the concepts must
    1108 call \code{pmConceptsInit} first.
     463call \code{pmConceptsInit} first.  This is usually performed internal
     464to the pmConcepts, so the user need not call this function.
    1109465
    1110466The concept specification metadata containers and the concept
    1111 specifications that have been registered shall all be freed by
    1112 \code{pmConceptsDone}:
     467specifications that have been registered are freed by:
    1113468\begin{prototype}
    1114469void pmConceptsDone(void);
    … …  
    1117472leak, since the metadata containers are defined \code{static}.
    1118473
    1119 \subsection{Reading, Writing and Blanking}
    1120 
    1121 Reading concepts is the act of determining their values and setting
    1122 them in the \code{concepts} metadata in the focal plane hierarchy.
    1123 Writing concepts is the act of taking the \code{concepts} metadata
    1124 which is in the focal plane hierarchy and preparing them for output.
     474
     475\subsection{Concept sources}
     476
     477Concepts may be read from, and written to a variety of sources: the
     478\code{CELLS} information within the camera format, the FITS header
     479(either the primary header, or extension), default values from the
     480camera format, or the database.
     481
     482Since some sources become available at different times from others, we
     483need to provide some specificity to reading and writing concepts (or
     484we're forced to wait until everything's available, which we don't want
     485to do).  Concepts may be read from or written to multiple sources at
     486once by OR-ing them.
     487
     488\begin{datatype}
     489typedef enum {
     490    PM_CONCEPT_SOURCE_NONE     = 0x00,  ///< No concepts
     491    PM_CONCEPT_SOURCE_CELLS    = 0x01,  ///< Concept comes from the camera information
     492    PM_CONCEPT_SOURCE_DEFAULTS = 0x02,  ///< Concept comes from defaults
     493    PM_CONCEPT_SOURCE_PHU      = 0x04,  ///< Concept comes from PHU
     494    PM_CONCEPT_SOURCE_HEADER   = 0x08,  ///< Concept comes from FITS header
     495    PM_CONCEPT_SOURCE_DATABASE = 0x10,  ///< Concept comes from database
     496    PM_CONCEPT_SOURCE_ALL      = 0xff   ///< All concepts
     497} pmConceptSource;
     498\end{datatype}
     499
     500
     501\subsection{Blanking}
     502
    1125503By ``blanking'', we mean setting the concepts to a default or blank
    1126504value (e.g., \code{NaN} for floating point); this takes place before
    1127505reading, and can be used to set up a focal plane hierarchy without
    1128 reading from any particular source.
    1129 
    1130 The following functions shall read, write or blank (as appropriate)
    1131 the concepts at the appropriate level in the focal plane hierarchy:
    1132 \begin{prototype}
    1133 bool p_pmConceptsReadFPA(pmFPA *fpa);
    1134 bool p_pmConceptsReadChip(pmChip *chip);
    1135 bool p_pmConceptsReadCell(pmCell *cell);
    1136 bool p_pmConceptsWriteFPA(pmFPA *fpa);
    1137 bool p_pmConceptsWriteChip(pmChip *chip);
    1138 bool p_pmConceptsWriteCell(pmCell *cell);
    1139 bool p_pmConceptsBlankFPA(pmFPA *fpa);
    1140 bool p_pmConceptsBlankChip(pmChip *chip);
    1141 bool p_pmConceptsBlankCell(pmCell *cell);
    1142 \end{prototype}
    1143 
    1144 Under ordinary circumstances, these functions will be called by
    1145 \code{pmFPARead}, \code{pmFPAWrite} and \code{pmFPAConstruct}.
     506reading from any particular source.  This is not normally done by the
     507user, but it is performed on the focal plane hierarchy at construction
     508(see \code{pmFPAConstruct}).
     509
     510The following functions blank the concepts at the appropriate level:
     511
     512\begin{prototype}
     513bool pmConceptsBlankFPA(pmFPA *fpa);
     514bool pmConceptsBlankChip(pmChip *chip);
     515bool pmConceptsBlankCell(pmCell *cell);
     516\end{prototype}
     517
     518
     519\subsection{Reading}
     520
     521Reading a concept is the act of looking up the value, parsing it, and
     522setting the parsed concept in the \code{concepts} metadata in the
     523focal plane hierarchy at the appropriate level.  It is not normally
     524necessary for the user to call these functions, since they are called
     525when constructing or reading the FPA (see \code{pmFPAConstruct},
     526\code{pmFPARead}).
     527
     528The following functions read the concepts at the appropriate level:
     529
     530\begin{prototype}
     531bool pmConceptsReadFPA(pmFPA *fpa,      ///< FPA for which to read concepts
     532                       pmConceptSource source, ///< Source for concepts
     533                       bool propagateDown, ///< Propagate to lower levels?
     534                       psDB *db         ///< Database handle
     535                      );
     536bool pmConceptsReadChip(pmChip *chip,   ///< Chip for which to read concepts
     537                        pmConceptSource source, ///< Source for concepts
     538                        bool propagateUp, ///< Propagate to higher levels?
     539                        bool propagateDown, ///< Propagate to lower levels?
     540                        psDB *db        ///< Database handle
     541                       );
     542bool pmConceptsReadCell(pmCell *cell,   ///< Cell for which to read concepts
     543                        pmConceptSource source, ///< Source for concepts
     544                        bool propagateUp, ///< Propagate to higher levels?
     545                        psDB *db        ///< Database handle
     546                       );
     547
     548\end{prototype}
     549In each case, we require the hierarchy component for which to read the
     550concepts, an indication of the source from which to read concepts
     551(which source has recently become available), the database handle (for
     552reading concepts from the database), and flags indicating whether we
     553should propagate the read to higher or lower levels in the hierarchy.
     554
     555An additional function reads concepts for as many hierarchy components
     556as are supplied:
     557\begin{prototype}
     558bool pmConceptsRead(pmFPA *fpa,         ///< FPA for which to read concepts
     559                    pmChip *chip,       ///< Chip for which to read concepts, or NULL
     560                    pmCell *cell,       ///< Cell for which to read concepts, or NULL
     561                    pmConceptSource source, ///< The source of the concepts to read
     562                    psDB *db            ///< Database handle
     563                   );
     564\end{prototype}
     565
     566
     567\subsection{Writing}
     568
     569Writing a concept is the act of formatting the concept for the native
     570format, and writing it to the appropriate place (here, ``writing''
     571often means preparing for writing, by updating the header metadata, or
     572checking the concept value against the default).  The user will not
     573normally use these functions directly, since they are called when
     574writing the FPA (see \code{pmFPAWrite}).
     575
     576\begin{prototype}
     577bool pmConceptsWriteFPA(const pmFPA *fpa,     ///< FPA for which to write concepts
     578                        pmConceptSource source, ///< Source for concepts
     579                        bool propagateDown, ///< Propagate to lower levels?
     580                        psDB *db        ///< Database handle
     581                       );
     582bool pmConceptsWriteChip(const pmChip *chip,  ///< Chip for which to write concepts
     583                         pmConceptSource source, ///< Source for concepts
     584                         bool propagateUp,///< Propagate to higher levels?
     585                         bool propagateDown, ///< Propagate to lower levels?
     586                         psDB *db       ///< Database handle
     587                        );
     588bool pmConceptsWriteCell(const pmCell *cell,  ///< FPA for which to write concepts
     589                         pmConceptSource source, ///< Source for concepts
     590                         bool propagateUp, ///< Propagate to higher levels?
     591                         psDB *db       ///< Database handle
     592                        );
     593\end{prototype}
     594In each case, we require the hierarchy component for which to read the
     595concepts, an indication of the source to which to write concepts
     596(which source requires updating), the database handle (for writing
     597concepts to the database), and flags indicating whether we should
     598propagate the write to higher or lower levels in the hierarchy.
    1146599
    1147600
    … …  
    1150603The values of concepts may be copied from one source to another:
    1151604\begin{prototype}
    1152 bool pmFPACopyConcepts(pmFPA *target, pmFPA *source);
    1153 \end{prototype}
    1154 
    1155 \code{pmFPACopyConcepts} shall iterate through the focal plane
    1156 hierarchy, copying the values of the concepts from the \code{source}
    1157 to the \code{target}.
     605bool pmFPACopyConcepts(pmFPA *target, const pmFPA *source);
     606\end{prototype}
     607
     608\code{pmFPACopyConcepts} iterates through the focal plane hierarchy,
     609copying the values of the concepts from the \code{source} to the
     610\code{target}.
    1158611
    1159612
Note: See TracChangeset for help on using the changeset viewer.