IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changeset 11569


Ignore:
Timestamp:
Feb 1, 2007, 6:46:06 PM (19 years ago)
Author:
Paul Price
Message:

Updating recipes, camera configuration. Should be up to date at this point.

File:
1 edited

Legend:

Unmodified
Added
Removed

  • trunk/doc/config/config.tex

    r9815 r11569  
    1 %%% $Id: config.tex,v 1.4 2006-11-01 02:54:16 price Exp $
     1%%% $Id: config.tex,v 1.5 2007-02-02 04:46:06 price Exp $
    22\documentclass[panstarrs,spec]{panstarrs}
    33
    44% Basic document variables
    5 \title{Pan-STARRS PS-1 Image Processing Pipeline Configurations}
    6 \subtitle{Guide}
    7 \shorttitle{IPP Configurations}
     5\title{Pan-STARRS Image Processing Pipeline}
     6\subtitle{Configuration Guide}
     7\shorttitle{IPP CG}
    88\author{Paul Price}
    99\audience{Pan-STARRS PMO}
    … …  
    1111\project{Pan-STARRS Image Processing Pipeline}
    1212\organization{Institute for Astronomy}
    13 \version{11}
     13\version{DR}
    1414\docnumber{PSDC-430-???}
    1515
    … …  
    6363using a human-readable syntax.  We therefore use these ``metadata
    6464configuration'' (MDC) files as the basis for our configuration files.
    65 When referring to entries in an MDC file, we use the convention that
    66 \code{NAME(TYPE)} refers to the item called \code{NAME}, with type
    67 \code{TYPE}.
     65When referring to entries in an MDC file, we use the convention in
     66this document that \code{NAME(TYPE)} refers to the item called
     67\code{NAME}, with type \code{TYPE}.
    6868
    6969The IPP uses configuration parameters on four levels:
    … …  
    9797\item The \code{-site} option\footnote{\code{-site} is used for C
    9898programs.  For Perl programs, we use the \code{Getopt::Long} module
    99 which requires us to use: \code{--site}} on the command line if
     99which requires us to use \code{--site}} on the command line if
    100100provided;
    101101\item The environment variable \code{PS_SITE}, if defined; or
    102 \item \code{$HOME/.ipprc} otherwise. %$
     102\item \code{$HOME/.ipprc} otherwise. %$ --- Balancing the former dollar sign for Emacs
    103103\end{enumerate}
    104104
    … …  
    116116and camera configuration files.
    117117
    118 The \code{WORKDIR(STR)} entry gives a top-level working directory that
    119 is prepended to all filenames used by the \code{pXtools}.  This allows
    120 the database to be moved to a different system (with different
    121 directory structure) without having to search and replace all paths.
     118The \code{DATAPATH(METADATA)} entry contains a series of symbolic
     119links (of tyoe \code{STR}) to data directories.  This allows data to
     120be moved to a different system (with different directory structure)
     121without having to search and replace all paths within the database;
     122and also to juggle multiple projects using the same configuration
     123file.  In the Perl components of the IPP, we use \code{path://DIR} to
     124mean ``look up \code{DIR} in the \code{DATAPATH} to get the
     125directory''.
    122126
    123127\subsubsection{Database setup}
    … …  
    132136\code{DBUSER(STR)} specifies the database user name.
    133137
    134 \code{DBPASSWORD(STR)} specifies the database password.  \tbd{This is
    135 an insecure method of storing what might be sensitive information.
    136 This is to be revised in the future.}
     138\code{DBPASSWORD(STR)} specifies the database password.
     139\tbd{DBPASSWORD is an insecure method of storing what might be
     140sensitive information.  This is to be revised in the future.}
    137141
    138142\subsubsection{Cameras}
    … …  
    156160
    157161\code{TIME(STR)} specifies the location of the time configuration file
    158 for psLib.
     162for psLib.  This is not too important, since the original installation
     163location is known by psLib.
    159164
    160165\code{LOGLEVEL(S32)} specifies the logging level for \code{psLogMsg}.
    … …  
    164169
    165170\code{LOGDEST(STR)} specifies the log destination (see
    166 \code{psMessageDestination} for acceptable formats).
     171\code{psMessageDestination} for acceptable formats).  We recommend
     172setting this to \code{STDERR} to avoid problems associated with
     173higher-level programs attempting to parse unintended input from
     174\code{stdout}.
    167175
    168176\code{TRACEFORMAT(STR)} specifies the trace format (see
    … …  
    170178
    171179\code{TRACEDEST(STR)} specifies the trace destination (see
    172 \code{psMessageDestination} for acceptable formats).
     180\code{psMessageDestination} for acceptable formats).  We recommend
     181setting this to \code{STDERR} to avoid problems associated with
     182higher-level programs attempting to parse unintended input from
     183\code{stdout}.
    173184
    174185\code{TRACE(METADATA)} gives a list of trace facilities and their
    175186accompanying levels (of type \code{S32}); see \code{psTraceSetLevel}.
    176 It is useful to include at least \code{err}, with level \code{10},
    177 since this will print all error messages.
    178 
     187We recommend including at least \code{err}, with level \code{10},
     188since this will print all error messages.  Other useful traces to set
     189are \code{psModules.camera} for camera (and especially
     190\code{pmFPAfile}) operations; and \code{psLib.db} for database
     191lookups.
    179192
    180193\subsection{Example}
    … …  
    184197
    185198PATH            STR     .:/my/home/.ipp # Default search path for configuration files
    186 WORKDIR         STR     /my/data/disk/  # Top-level working directory
     199
     200# Place your data directories here and refer to as path://PATH/remainder
     201DATAPATH        METADATA
     202        HERE    STR     /data/my/host/
     203        THERE   STR     /data/other/host/
     204END
    187205
    188206### Database configuration
    … …  
    194212### Setups for each camera system
    195213CAMERAS         METADATA
    196         MCSHORT         STR     mcshort/camera.config
    197         MCSHORT_CHIP    STR     mcshort_chip/camera.config
    198         MCSHORT_FPA     STR     mcshort_fpa/camera.config
    199         MEGACAM         STR     megacam/camera.config
    200         MEGACAM_CHIP    STR     megacam_chipmosaic/camera.config
    201         MEGACAM_FPA     STR     megacam_fpamosaic/camera.config
    202         MEGACAM_DET     STR     megacam_detrended/camera.config
    203         UCAM            STR     ucam/camera.config
    204         UCAM_MOSAIC     STR     ucam_mosaic/camera.config
    205         GPC1            STR     gpc1/camera.config
    206         LRIS_BLUE       STR     lris_blue/camera.config
    207         LRIS_RED        STR     lris_red/camera.config
    208         ISP             STR     isp/camera.config
    209         SIMPLE          STR     simple/camera.config
     214        CTIO_MOSAIC2    STR     ctio_mosaic2/camera.config      # CTIO MOSAIC2 camera, for ESSENCE
     215        MEGACAM         STR     megacam/camera.config           # Megacam, on CFHT
     216        GPC1            STR     gpc1/camera.config              # Pan-STARRS GigaPixel Camera 1
     217        ISP             STR     isp/camera.config               # Pan-STARRS Imaging Sky Probe
     218        SIMPLE          STR     simple/camera.config            # Simple single-chip camera
    210219END
    211220
    … …  
    218227TRACEFORMAT     STR     THLNM                   # Trace format
    219228TRACE           METADATA                        # Trace levels
    220         err             S32     10
     229        err                 S32     10
     230#       psLib.db            S32     10
     231#       psModules.camera    S32     10
    221232END
    222233
    223234RECIPES         METADATA                # Site-level recipes
    224         PPMERGE         STR             ppMerge_template.config # Recipe for combination
    225         PPSTATS_PHASE0  STR             ppStats_phase0.config   # Recipe for phase 0 processing
     235        PPIMAGE         STR             recipes/ppImage.config  # Image reduction
     236        PPMERGE         STR             recipes/ppMerge.config  # Image combination
     237        PPSTATS         STR             recipes/ppStats.config  # Image statistics
     238        PPSTATS_PHASE0  STR             recipes/ppStats_phase0.config   # Image statistics for Phase 0
     239        PSPHOT          STR             recipes/psphot.config   # Photometry
     240        PSASTRO         STR             recipes/psastro.config  # Astrometry
    226241END
    227242\end{verbatim}
    … …  
    249264the \code{CAMERAS(METADATA)}, which lists cameras by name, with their
    250265corresponding configuration file.  Note that the \code{PATH(STR)} in
    251 the site configuration defines the search paths for these files.
     266the site configuration defines the search path for these files.
    252267
    253268\subsection{Contents}
    … …  
    257272stored in different formats (e.g., one amplifier per extension, vs all
    258273amplifiers spliced together in the PHU).  The camera configuration
    259 contains the formats, the camera description, filter translation table,
    260 recipes, rejection levels and file rules.
     274contains the formats, the camera description, filter translation
     275table, observation type translation table, recipes, rejection levels
     276and file rules.
    261277
    262278\subsubsection{Formats}
    … …  
    280296\subsubsection{Filter translation table}
    281297
    282 \tbd{EAM: This needs a better description of the purpose.}
    283 
    284 \code{FILTER.ID(METADATA)} contains a list of filter identifications
    285 (generally those found within the header), with a general term to
    286 describe the filter, of type \code{STR}.
     298\code{FILTER.ID(METADATA)} contains a list of filter names (generally
     299those found within the FITS header; e.g., \code{r.MP9601}), with an
     300abstract name to describe the filter (e.g., \code{r}), of type
     301\code{STR}.  This allows multiple descriptions of the same filter that
     302may exist in the FITS headers to be resolved as the same thing.
     303
     304\subsubsection{Observation type translation table}
     305
     306\code{OBSTYPE.TABLE(METADATA)} contains a list of observation types
     307(generally those found within the FITS header; e.g., \code{ZERO}),
     308with an abstract name to describe the observation type (e.g.,
     309\code{BIAS}).  This allows multiple descriptions of the same
     310observation type that may exist in the FITS headers to be resolved as
     311the same thing (e.g., \code{BIAS}, \code{ZERO} and \code{PEDESTAL} can
     312all be set to \code{BIAS}).
    287313
    288314\subsubsection{Recipes}
    … …  
    335361  the mean to the mean standard deviation of the mean; in terms of
    336362  standard deviations.  \tbd{Confusing enough?}
     363\item \code{IMFILE.SN}: rejection level for the signal-to-noise at the
     364  imfile level.
     365\item \code{EXP.SN}: rejection level for the signal-to-noise at the
     366  exposure level.
    337367\end{itemize}
    338368Apart from \code{FILTER}, values that are set to zero are ignored.
    … …  
    341371be useful to define a type:
    342372\begin{verbatim}
    343 TYPE    LIMITS  FILTER  EXPECTED        IMFILE.MEAN     IMFILE.STDEV    EXP.MEAN        EXP.STDEV       EXP.MEANSTDEV   ENSEMBLE.MEAN   ENSEMBLE.STDEV  ENSEMBLE.MEANSTDEV
     373TYPE    LIMITS  FILTER  EXPECTED        IMFILE.MEAN     IMFILE.STDEV    EXP.MEAN        EXP.STDEV       EXP.MEANSTDEV   ENSEMBLE.MEAN   ENSEMBLE.STDEV  ENSEMBLE.MEANSTDEV      IMFILE.SN       EXP.SN
    344374\end{verbatim}
    345375
    346376\subsubsection{File rules}
    347377
    348 \tbd{EAM to fill this in}
     378\tbd{EAM to check and supplement this description.}
     379
     380The file rules are one of the most important aspects of the camera
     381configuration, and one of the easiest to get wrong.  When setting up a
     382new camera configuration and getting errors (or worse, segmentation
     383faults), check the file rules first.  Try turning up the
     384\code{psModules.camera} trace level to see what's going on.
     385
     386\code{FILERULES(METADATA)} lists the different types of files used in
     387the image processing, which specify how and when a file is read in and
     388written out.  The files usually are of two or three components,
     389separated by a period (not for any particular reason except that's
     390what's been adopted); the first part specifies the program the file
     391will be used in, the second and third parts identify its role.  For
     392example, \code{PPIMAGE.INPUT} specifies the input file for
     393\code{ppImage}; \code{PPIMAGE.OUTPUT.MASK} specifies the output mask
     394file from \code{ppImage}.
     395
     396\subsubsubsection{Replacements}
     397
     398Throughout the file rules, a syntax for defining strings from
     399variables is used: curly brackets \verb|{}| around an abstract name
     400are replaced by the program to obtain the proper value.  Supported
     401abstract names are:
     402\begin{itemize}
     403\item \verb|{OUTPUT}| --- replaced with the output file root;
     404\item \verb|{CHIP.NAME}| --- replaced with the chip name;
     405\item \verb|{CHIP.N}| --- replaced with the chip number (printed \code{%02d});
     406\item \verb|{CELL.NAME}| --- replaced with the cell name;
     407\item \verb|{CELL.N}| --- replaced with the chip number (printed \code{%02d});
     408\item \verb|{EXTNAME}| --- replaced with the extension name;
     409\item \verb|{FILTER}| --- replaced with the filter name (without
     410  applying the \code{FILTER.ID} translation table);
     411\item \verb|{FILTER.ID}| --- replaced with the filter identifier (after
     412  applying the \code{FILTER.ID} translation table);
     413\item \verb|{CAMERA}| --- replaced with the instrument name (from \code{FPA.INSTRUMENT});
     414\item \verb|{INSTRUMENT}| --- replaced with the instrument name (from \code{FPA.INSTRUMENT});
     415\item \verb|{DETECTOR}| --- replaced with the detector name (from \code{FPA.DETECTOR}); and
     416\item \verb|{TELESCOPE}| --- replaced with the telescope name (from \code{FPA.TELESCOPE}).
     417\end{itemize}
     418
     419(More could potentially be added.  If one you greatly desire is
     420missing, please ask!)
     421
     422
     423\subsubsubsection{Redirections}
     424
     425Entries with type \code{STR} are treated as symbolic links to another
     426line.  For example, specifying:
     427\begin{verbatim}
     428  PPIMAGE.OUTPUT   STR   PPIMAGE.OUTPUT.SPLIT
     429\end{verbatim}
     430means that the program will look up \code{PPIMAGE.OUTPUT.SPLIT} in the
     431place of \code{PPIMAGE.OUTPUT}.  This allows a quick replacement if a
     432different output format is desired (e.g., \code{PPIMAGE.OUTPUT.MEF}
     433instead of \code{PPIMAGE.OUTPUT.SPLIT}).
     434
     435\subsubsubsection{File types}
     436\label{sec:camera-filerules-types}
     437
     438Both the input and output file rules use file types.  The currently
     439supported file types are:
     440\begin{itemize}
     441\item \code{IMAGE} --- image data in FITS image format (treated as \code{F32});
     442\item \code{MASK} --- mask data in FITS image format (treated as \code{U8});
     443\item \code{WEIGHT} --- weight data in FITS image format (treated as \code{F32})
     444\item \code{FRINGE} --- fringe image with fringe tables (one for each
     445  cell) in FITS image format (image treated as \code{F32});
     446\item \code{JPEG} --- image data in JPEG format (output only);
     447\item \code{CMP} --- object data in CMP format;
     448\item \code{CMF} --- object data in CMF format;
     449\item \code{RAW} --- object data in RAW format;
     450\item \code{SX} --- object data in SX format; and
     451\item \code{OBJ} --- object data in OBJ format.
     452\end{itemize}
     453
     454\tbd{EAM to fill in details on the object formats.}
     455
     456
     457\subsubsubsection{Inputs}
     458
     459It is useful to make the following \code{TYPE} declaration, which can
     460be used for all input files:
     461\begin{verbatim}
     462TYPE   INPUT   FILENAME.RULE   FILENAME.XTRA   EXTNAME.RULE   EXTNAME.XTRA   DATA.LEVEL   FILE.TYPE
     463\end{verbatim}
     464The components are:
     465\begin{itemize}
     466\item \code{FILENAME.RULE} --- this specifies the rule for
     467  constructing the filename.  Options for doing so are:
     468  \begin{itemize}
     469  \item A simple filename, perhaps using the replacement syntax
     470    defined above;
     471  \item \code{@DETDB} to look up the appropriate file using the
     472    detrend database (see \S\ref{sec:detrend-database}); or
     473  \item \code{@FILES} to indicate that the input file(s) will be
     474    specified on the command-line of the program.
     475  \end{itemize}
     476\item \code{FILENAME.XTRA} --- \tbd{PAP is not entirely sure what this
     477  is for; it may be unnecessary.}
     478\item \code{EXTNAME.RULE} --- This defines the extension name.
     479  \tbd{Is this true?  PAP thinks the camera format does that; this may
     480    be unnecessary, or it may have to be tied into the camera format.}
     481\item \code{EXTNAME.XTRA} --- \tbd{PAP is not entirely sure what this
     482  is for; it may be unnecessary.}
     483\item \code{DATA.LEVEL} --- the level of the hierarchy at which the
     484  data is to be opened for reading.  This should correspond to the
     485  level of the extension in the FITS file, or higher.  There are some
     486  checks against the camera format that this is sensical, but don't
     487  bet your life on it just yet.  This is an important setting to check
     488  if you're having problems.
     489\item \code{FILE.TYPE} --- the type of file, from the above list
     490  (\S\ref{sec:camera-filerules-types}).
     491\end{itemize}
     492
     493\subsubsubsection{Outputs}
     494
     495It is useful to make the following \code{TYPE} declaration, which
     496can be used for all output files:
     497\begin{verbatim}
     498TYPE   OUTPUT   FILENAME.RULE   FILENAME.XTRA   EXTNAME.RULE   EXTNAME.XTRA   FILE.LEVEL   DATA.LEVEL   FILE.TYPE   FILE.SAVE   FILE.FORMAT
     499\end{verbatim}
     500The components are:
     501\begin{itemize}
     502\item \code{FILENAME.RULE} --- this specifies the rule for
     503  constructing the filename.  You most likely want to include
     504  \verb|{OUTPUT}| somewhere here; Pan-STARRS convention is that
     505  it goes at the front.
     506\item \code{FILENAME.XTRA} --- \tbd{PAP is not entirely sure what this
     507  is for; it may be unnecessary.}
     508\item \code{EXTNAME.RULE} --- This defines the extension name.
     509  \tbd{Is this true?  PAP thinks the camera format does that; this may
     510  be unnecessary, or it may have to be ties into the camera format.}
     511\item \code{EXTNAME.XTRA} --- \tbd{PAP is not entirely sure what this
     512  is for; it may be unnecessary.}
     513\item \code{FILE.LEVEL} --- the level of the hierarchy at which a file
     514  should be opened and the PHU written.  This should correspond to the
     515  level of the PHU.  There are some checks against the camera format
     516  that this is sensical, but don't bet your life on it just yet.  This
     517  is an important setting to check if you're having problems.
     518\item \code{DATA.LEVEL} --- the level of the hierarchy at which an
     519  extension should be written.  This should correspond to the level of
     520  the extensions in the FITS file, or higher.  There are some checks
     521  against the camera format that this is sensical, but don't bet your
     522  life on it just yet.  This is an important setting to check if
     523  you're having problems.
     524\item \code{FILE.TYPE} --- the type of file, from the above list
     525  (\S\ref{sec:camera-filerules-types}).
     526\item \code{FILE.SAVE} --- whether this type of file should be saved
     527  (\code{TRUE}) or not (\code{FALSE}).
     528\item \code{FILE.FORMAT} --- if the file format is to be changed, this
     529  is the name of the file format (from the \code{FORMATS} metadata).
     530  Otherwise, it is \code{NONE}.
     531\end{itemize}
     532
    349533
    350534\subsection{Example}
    … …  
    372556
    373557
    374 # Lookup table to go from filter ID to abstract name
     558# Lookup table to go from FPA.FILTER to abstract name for the filter
    375559FILTER.ID       METADATA
    376560   u.MP9301     STR u
    … …  
    387571END
    388572
     573# Lookup table to go from FPA.OBSTYPE values to abstract name for the exposure type
     574OBSTYPE.TABLE METADATA
     575  bias     STR BIAS
     576  zero     STR BIAS
     577  dark     STR DARK
     578  flat     STR SKYFLAT
     579  skyflat  STR SKYFLAT
     580  domeflat STR DOMEFLAT
     581  object   STR OBJECT
     582  science  STR OBJECT
     583END
    389584
    390585# Recipe options
    391 RECIPES         METADATA
    392         # Recipes for ppImage
    393         PPIMAGE         STR     megacam/ppImage.config          # Default: all (normal) options on
    394         PPIMAGE_O       STR     megacam/ppImage_o.config        # Overscan only
    395         PPIMAGE_OB      STR     megacam/ppImage_ob.config       # Overscan, bias only
    396         PPIMAGE_OBD     STR     megacam/ppImage_obd.config      # Overscan, bias, dark only
    397         PPIMAGE_OBDF    STR     megacam/ppImage_obdf.config     # Overscan, bias, dark, flat only
    398         PPIMAGE_B       STR     megacam/ppImage_b.config        # Bias only
    399         PPIMAGE_D       STR     megacam/ppImage_d.config        # Dark only
    400         PPIMAGE_F       STR     megacam/ppImage_f.config        # Flat only
    401         PPIMAGE_J1      STR     megacam/ppImage_j1.config       # JPEG only; binning 1
    402         PPIMAGE_J2      STR     megacam/ppImage_j2.config       # JPEG only; binning 2
    403         PPIMAGE_N       STR     megacam/ppImage_n.config        # Nothing significant; binning only
    404 
    405         # Recipes for ppMerge
    406         PPMERGE         STR     ppMerge_template.config         # ppMerge recipe
    407         PPMERGE_BIAS    STR     megacam/ppMerge_bias.config
    408         PPMERGE_DARK    STR     megacam/ppMerge_dark.config
    409         PPMERGE_FLAT    STR     megacam/ppMerge_flat.config
    410 
    411         # Other recipes
     586RECIPES         METADATA
     587        # Other recipes
    412588        PSPHOT          STR     megacam/psphot.config           # psphot details
    413589        PSASTRO         STR     megacam/psastro.config          # psastro details
    414         PPSTATS         STR     megacam/ppStats.config          # ppStats recipe
    415 END
    416 
     590        PPSTATS         STR     megacam/ppStats.config          # ppStats recipe
     591        PPIMAGE         STR     megacam/ppImage.config          # ppImage recipe
     592END
    417593
    418594# Rejection levels for detrend creation
    419 REJECTION       METADATA
    420         TYPE    LIMITS  FILTER  EXPECTED        IMFILE.MEAN     IMFILE.STDEV    EXP.MEAN        EXP.STDEV       EXP.MEANSTDEV   ENSEMBLE.MEAN   ENSEMBLE.STDEV  ENSEMBLE.MEANSTDEV
    421         FLAT    MULTI
    422 
    423         BIAS    LIMITS  *       0               0               15              0               15              0               0               0               0
    424         DARK    LIMITS  *       0               0               0               0               0               0               0               0               0
    425         FLAT    LIMITS  *       0               0               0               0               0               0               0               0               0
    426         FLAT    LIMITS  u       0               0               0               0               0               0               0               0               0
    427         FLAT    LIMITS  g       0               0               0               0               0               0               0               0               0
    428         FLAT    LIMITS  r       0               0               0               0               0               0               0               0               0
    429         FLAT    LIMITS  i       0               0               0               0               0               0               0               0               0
    430         FLAT    LIMITS  z       0               0               0               0               0               0               0               0               0
    431 
    432 END
    433                
     595REJECTION       METADATA
     596        TYPE    LIMITS  FILTER  EXPECTED        IMFILE.MEAN     IMFILE.STDEV    EXP.MEAN        EXP.STDEV       EXP.MEANSTDEV   ENSEMBLE.MEAN   ENSEMBLE.STDEV  ENSEMBLE.MEANSTDEV      IMFILE.SN       EXP.SN
     597        FLAT    MULTI
     598
     599        BIAS    LIMITS  *       0               1               5               0.5             3               0.5             3               3               0                       0               0
     600        DARK    LIMITS  *       0               1               5               0.5             3               0.5             3               3               0                       0               0
     601        FLAT    LIMITS  *       0               0               0               0               0               0               0               0               3                       0               0
     602#       FLAT    LIMITS  u       0               0               0               0               0               0               0               0               3                       0               0
     603#       FLAT    LIMITS  g       0               0               0               0               0               0               0               0               3                       0               0
     604#       FLAT    LIMITS  r       0               0               0               0               0               0               0               0               3                       0               0
     605#       FLAT    LIMITS  i       0               0               0               0               0               0               0               0               3                       0               0
     606#       FLAT    LIMITS  z       0               0               0               0               0               0               0               0               3                       0               0
     607        FRINGE  LIMITS  *       0               0               0               0               0               0               0               0               0                       0               0
     608
     609# FILTER is an additional qualifier, and may be "*" (or absent!), in which case it matches everything
     610# EXPECTED is the expected mean value
     611# IMFILE.MEAN is the maximum permitted mean value for an imfile, relative to the standard deviation
     612# IMFILE.STDEV is the maximum permitted standard deviation for an imfile
     613# EXP.MEAN is the maximum permitted mean value for an exposure, relative to the standard deviation
     614# EXP.STDEV is the maximum permitted standard deviation for an exposure
     615# EXP.MEANSTDEV is the maximum permitted mean standard deviation for an exposure relative to the mean
     616# ENSEMBLE.MEAN is the maximum permitted mean for an ensemble of exposures
     617# ENSEMBLE.STDEV is the maximum permitted standard deviation for an ensemble of exposures
     618# ENSEMBLE.MEANSTDEV is the maximum permitted mean standard deviation for an ensemble of exposures
     619# IMFILE.SN is the minimum permitted signal-to-noise for an imfile
     620# EXP.SN is the minimum permitted signal-to-noise for an exposure
     621# These values (all except FILTER) may be zero, in which case no clipping is applied.
     622
     623END
     624
    434625
    435626FILERULES METADATA
    … …  
    496687\section{Camera format configuration}
    497688
    498 The FITS data storage formation is a standard in the astronomical
    499 community for storing astronomical images.  A FITS file consists of an
    500 arbitrary number of coupled human readable \code{ASCII} header
    501 segments and binary data segments.  The headers describe the format
    502 and layout of the data segments.  The first of these groups is
     689The FITS (Flexible Image Transport System) format is a standard in the
     690astronomical community for storing astronomical images.  A FITS file
     691consists of an arbitrary number of coupled human readable \code{ASCII}
     692header segments and binary data segments.  The headers describe the
     693format and layout of the data segments.  The first of these groups is
    503694traditionally called the ``primary header unit'' (PHU) and the rest
    504695are referred to as ``extensions''.  The header segments may contain
    … …  
    507698the data, the header metadata is not so consistently defined within
    508699the astronomical community.  Also, the flexibility of the data format
    509 means that different representations are possible for the same
    510 fundamental collection of data.
     700means that it is possible to construct a variety of different
     701representations for the same fundamental collection of data.
    511702
    512703The purpose of the camera format file is to define how FITS files are
    … …  
    518709The camera formats for a particular camera are listed in the
    519710\code{FORMATS} metadata of the camera configuration file.  Note that
    520 the \code{PATH(STR)} in the site configuration defines the search
     711the \code{PATH} in the site configuration defines the search
    521712paths for these files.
    522713
    … …  
    525716The camera format specifies how a FITS file from a particular camera
    526717is to be read.  Different formats may be defined for a single camera
    527 (e.g., one amplifier per extension, vs all amplifiers spliced together
    528 in the PHU).  The camera format configuration file contains the rules
    529 for recognising the format, how to read the file, the contents of a
    530 FITS file, data appropriate to different types of cells, information
    531 on how to determine the concepts from the headers, default values, or
    532 database, and expected formats for certain concepts.
     718(e.g., one amplifier per extension, or all amplifiers spliced together
     719in the PHU, or anything in between).  The camera format configuration
     720file contains the rules for recognising the format, how to read the
     721file, the contents of a FITS file, data appropriate to different types
     722of cells, information on how to determine the concepts from the
     723headers, default values, or database, and expected formats for certain
     724concepts.
    533725
    534726\subsubsection{Rules for recognising}
    535727
    536 \code{RULE(METADATA)} contains a list of telescope headers with
    537 expected values (of the appropriate type) for this particular
    538 combination of the camera and format.  It is often useful to include
    539 \code{TELESCOP} and \code{DETECTOR}, if possible, along with any other
    540 headers that uniquely identify the camera and format.  Note that all
    541 of the headers must match exactly (modulo leading and trailing spaces
    542 for strings), including the data type and value, for the rule to
    543 match, and that the first format's rule to match is accepted.  If a
    544 rule doesn't match the header, try adjusting the types (especially for
    545 numerical types; try S32 for integers, F32 and F64 for floats).
     728\code{RULE(METADATA)} contains a list of FITS headers with expected
     729values (of the appropriate type) for this particular combination of
     730the camera and format.  It is often useful to include \code{TELESCOP}
     731and \code{DETECTOR}, if possible, along with any other headers that
     732uniquely identify the camera and format.  Note that all of the headers
     733must match exactly (modulo leading and trailing spaces for strings),
     734including the data type and value, for the rule to match, and that the
     735first format's rule to match is accepted.  If a rule doesn't match the
     736header, try adjusting the types (especially for numerical types ---
     737use S32 for integers, F32 and F64 for floats).
    546738
    547739\subsubsection{How to read the file}
    … …  
    551743In the simplest case, the camera consists of a single chip consisting
    552744of a single cell always read with a single readout.  In this case, the
    553 image data could be written as part of the primary header unit.  In a
    554 more complex case with multiple chips and multiple cells, the data may
    555 be organized in several ways.  The data may be distributed into
    556 multiple files or in multiple FITS data extensions.  A single camera
    557 image may be written as a collection of files for individual chips
    558 with separate extensions for each cell (CFH12K.split, GPC).  Another
    559 camera may write a single file with multiple extensions for each cell
    560 (Megacam.raw), or multiple extensions per chip, with each cell
    561 representing portions of the chip image (Megacam.splice, CFHT-IR).
     745image data is generally written as part of the primary header unit.
     746However, in a more complex case with multiple chips and multiple
     747cells, the data may be organized in several ways.  The data may be
     748distributed into multiple files or in multiple FITS data extensions
     749within a single file..  A single camera image may be written as a
     750collection of files for individual chips with separate extensions for
     751each cell (CFH12K.split, GPC).  Another camera may write a single file
     752with multiple extensions for each cell (Megacam.raw), or multiple
     753extensions per chip, with each cell representing portions of the chip
     754image (Megacam.splice, CFHT-IR).
    562755
    563756In all of these representations, there are only two basic distinctions
    … …  
    565758entire FITS file corresponds to (FPA, chip, or cell), and what level
    566759the extensions correspond to (chip, cell or no extensions at all).
    567 Knowing these, and having a list of the extensions, we can construct
    568 the Focal Plane hierarchy.
     760Knowing these, and having a list of the contents of each extension, we
     761can construct the Focal Plane hierarchy.
    569762
    570763\code{FILE(METADATA)} contains information on how to read the FITS
    … …  
    583776  with the same value of \code{FPA.NAME} can be admitted to the same
    584777  FPA structure.
    585 \item \code{CHIP.NAME(STR)} (necessary if \code{PHU} is \code{CHIP} or
     778\item \code{CHIP.NAME(STR)} (only required if \code{PHU} is \code{CHIP} or
    586779  \code{CELL}) specifies a PHU header keyword that identifies the name
    587780  of the chip.  The purpose is to identify to which chip in the
    588781  hierarchy the file belongs.
    589 \item \code{CELL.NAME(STR)} (necessary if \code{PHU} is \code{CELL})
     782\item \code{CELL.NAME(STR)} (only required if \code{PHU} is \code{CELL})
    590783  specifies a PHU header keyword that identifies the name of the cell
    591784  within the chip.  The purpose is to identify to which cell in the
    592785  hierarchy the file belongs.
    593 \item \code{CONTENT(STR)} (necessary if \code{EXTENSIONS} is
     786\item \code{CONTENT(STR)} (only required if \code{EXTENSIONS} is
    594787  \code{NONE} and \code{PHU} is \code{CHIP} or \code{CELL}) specifies
    595788  a key to the \code{CONTENTS} menu (see below).  The purpose is to
    … …  
    599792  \code{CELL.NAME} only; \tbd{future concepts may be permitted in the
    600793  future if there exists sufficient demand}.  This allows such a
    601   construct as \code{\{CHIP.NAME\}_\{CELL.NAME\}} to identify a
     794  construct as \verb|{CHIP.NAME}_{CELL.NAME}| to identify a
    602795  combination of chip and cell.
    603796\end{itemize}
    … …  
    611804contents of an extension: the chip and cell to which a component
    612805belongs, and the type of the cell (see \S\ref{sec:cell_data} for cell
    613 types), with the symbolic names separated by colons.  The triplets may
    614 be listed one after the other, separated by whitespace, where an
    615 extension contains more than one cell.
     806types), with the symbolic names separated by colons.  Where an
     807extension contains more than one cell, the triplets are listed one
     808after the other, separated by whitespace.
    616809
    617810\begin{itemize}
    … …  
    642835here, since these differ according to the cell type.  Since there is
    643836ambiguity in what the values here refer to (if the concept is of type
    644 \code{STR}), we also require an additional entry with \code{.SOURCE}
     837\code{STR}, then the value could be a header name or the actual value
     838to use), we also require an additional entry with \code{.SOURCE}
    645839suffixed to the concept name, with the value (of type \code{STR})
    646840being \code{VALUE} to indicate that the concept is specified by value,
    … …  
    652846name may vary depending on the cell type.  For example, the Megacam
    653847spliced format uses \code{TSECA} and \code{TSECB} to specify the trim
    654 sections for the left and right amplifiers.]
     848sections for the left and right amplifiers, respectively.]
    655849
    656850\subsubsection{Concepts from headers}
    … …  
    661855the concept is ingested.  No distinction is made between the PHU and
    662856extension headers, but inheritance (look at the PHU if it's not in the
    663 extension header) should be the normal behaviour.  Multiple headers
    664 may be given for certain concepts:
     857extension header) should be the normal behaviour.  Multiple header
     858keywords (separated by whitespace) may be given for certain concepts:
    665859\begin{itemize}
    666860\item \code{FPA.TIME} and \code{CELL.TIME} to specify the date and
    667   time in separate headers
     861  time (in that order) are contained in separate header keywords.
    668862\item \code{CELL.BIASSEC} to specify multiple bias regions (e.g., a
    669863  prescan and an overscan).
    … …  
    719913  indicates the date is a julian date.
    720914\item \code{CELL.X0}, \code{CELL.Y0}, \code{CHIP.X0} and
    721   \code{CHIP.Y0}: \code{FORTRAN} indicates that the corner corresponds
    722   to corner (1,1); if missing, assumes that the corner is at (0,0).
     915  \code{CHIP.Y0}: \code{FORTRAN} indicates that the corner lower
     916  left-hand pixel corresponds to coordinates (1,1); if missing,
     917  assumes that the corner is at (0,0).
    723918\end{itemize}
    724919
    … …  
    729924\code{DATABASE}:
    730925\begin{itemize}
    731 \item \code{FPA.CAMERA}: Camera used
     926\item \code{FPA.TELESCOPE}: Telescope used
     927\item \code{FPA.INSTRUMENT}: Instrument used
     928\item \code{FPA.DETECTOR}: Detector used
     929\item \code{FPA.CAMERA}: Camera used; \tbd{To be deprecated?}
    732930\item \code{FPA.FOCUS}: Telescope focus
    733931\item \code{FPA.AIRMASS}: Airmass at boresight
    734932\item \code{FPA.FILTER}: Filter used
     933\item \code{FPA.FILTERID}: Filter identifier (parsed through the
     934  \code{FILTER.ID} translation table in the camera configuration).
    735935\item \code{FPA.POSANGLE}: Position angle of instrument
    736936\item \code{FPA.RADECSYS}: Celestial coordinate system
    … …  
    743943\item \code{FPA.TIMESYS}: Time system
    744944\item \code{FPA.TIME}: Time of exposure
     945\item \code{FPA.TEMP}: Temperature of the focal plane
     946\item \code{FPA.EXPOSURE}: Exposure time for the focal plane
    745947\item \code{CHIP.XPARITY}: Orientation in x compared to the rest of the FPA
    746948\item \code{CHIP.YPARITY}: Orientation in y compared to the rest of the FPA
    … …  
    11741376to.  In addition, they may be specified in the site configuration and
    11751377the camera configuration under the \code{RECIPES} metadata.  Note that
    1176 the \code{PATH(STR)} in the site configuration defines the search
     1378the \code{PATH} in the site configuration defines the search
    11771379paths for these files.
    11781380
    1179 \subsubsection{Symbolic links}
    1180 
    1181 Symbolic links to another recipe may be specified on the command line,
    1182 removing the need for the user to memorise a file name: e.g.,
    1183 \code{-recipe PPIMAGE PPIMAGE_BIAS} might perform a bias subtraction
    1184 only.  A symbolic link is recognised as such if the value is the name
    1185 of a recipe that has already been read (and the link is immediately
    1186 resolved), or if no filename of that name exists (in which case the
    1187 link is to be resolved later, as further sources become available).
    1188 
    1189 \subsubsection{Precedence}
    1190 
    1191 If multiple recipes have the same name, the precedence order is:
    1192 \begin{enumerate}
    1193 \item Command-line
    1194 \item Camera configuration
    1195 \item Site configuration
    1196 \end{enumerate}
    1197 with sources higher in this list having greater precedence.  This
    1198 allows the user to override any recipes using the command-line, and to
    1199 specify bottom-level defaults in the site configuration while also
    1200 having camera-specific recipes in the camera configurations.
     1381\subsubsection{Recipe combination}
     1382
     1383A single recipes are defined at multiple levels (site, camera, and
     1384command-line), so it's important to know how these are loaded.  The
     1385site configuration recipes serve as the default recipes.  Once the
     1386particular camera is known, the values contained within its recipes
     1387(provided either as a filename or as a symbolic link; see below for
     1388symbolic links) override those defined in the site configuration
     1389(unless the value has been declared as \code{MULTI}, in which case it
     1390supplements).  This is useful because recipes often depend on the
     1391camera from which the data being processed originated; for example,
     1392not all cameras require a dark to be subtracted.
     1393
     1394Finally, the command line can be used to provide further refinement.
     1395A recipe can be defined on the command line using \code{-recipe
     1396RECIPE_NAME filename.config} to specify a file containing the recipe,
     1397or \code{-recipe RECIPE_NAME ALTERNATE_RECIPE_NAME} to specify an
     1398symbolic link from which to inget values for the original recipe.
     1399
     1400Symbolic links offer the ability to override the default recipe values
     1401by specifying a name, rather than a filename.  A symbolic link can
     1402refer to a recipe of a different name that has already been defined,
     1403or it can refer to a \code{METADATA} within the recipe of that same
     1404name.
     1405
     1406A few examples are useful here.  Say the site configuration contains:
     1407\begin{verbatim}
     1408RECIPES    METADATA
     1409    RECIPE         STR    recipe_default.config
     1410    RECIPE_EXOTIC  STR    recipe_exotic.config
     1411END
     1412\end{verbatim}
     1413
     1414The camera configuration has:
     1415\begin{verbatim}
     1416RECIPES    METADATA
     1417    RECIPE         STR    recipe_camera.config
     1418END
     1419\end{verbatim}
     1420
     1421\code{recipe_default.config} has:
     1422\begin{verbatim}
     1423VALUE    STR    Default
     1424
     1425RECIPE_DULL    METADATA
     1426    VALUE    STR    Dull
     1427END
     1428\end{verbatim}
     1429
     1430
     1431\code{recipe_exotic.config} has:
     1432\begin{verbatim}
     1433VALUE    STR    Exotic
     1434\end{verbatim}
     1435
     1436And \code{recipe_camera.config} has:
     1437\begin{verbatim}
     1438VALUE    STR    Camera
     1439\end{verbatim}
     1440
     1441Then:
     1442\begin{itemize}
     1443\item If the recipe is examined without knowing the camera,
     1444  \code{VALUE} will be \code{Default}.
     1445\item If the recipe is examined once the camera is known, \code{VALUE}
     1446  will be \code{Camera}.
     1447\item If the command-line contains \code{-recipe RECIPE recipe_exotic.config},
     1448  \code{VALUE} will be \code{Exotic}.
     1449\item If the command-line contains \code{-recipe RECIPE RECIPE_EXOTIC},
     1450  \code{VALUE} will be \code{Exotic}.
     1451\item If the command-line contains \code{-recipe RECIPE RECIPE_DULL},
     1452  \code{VALUE} will be \code{Dull}.
     1453\end{itemize}
     1454
    12011455
    12021456\subsection{Contents}
    … …  
    12431497    good choice).
    12441498  \end{itemize}
     1499  \tbd{Non-linearity correction is implemented but not tested.}
    12451500\item \code{OVERSCAN.SINGLE(BOOL)} indicates if the entire overscan is
    12461501  to be reduced to a single value.
    … …  
    12531508  overscan: \code{MEAN} or \code{MEDIAN}. \tbd{Would like to change
    12541509  this to allow the full range of statistics.}
     1510\item \code{FRINGE.ITER(S32)} specifies the number of rejection iterations for fringe solution.
     1511\item \code{FRINGE.REJ(F32)} specifies the rejection threshold (in standard deviations) for fringe solution.
     1512\item \code{FRINGE.KEEP(F32)} specifies the minimum fraction of points to keep in the fringe solution.
    12551513\item \code{BIN1.XBIN(S32)} gives the level 1 binning in x
    12561514\item \code{BIN2.YBIN(S32)} gives the level 1 binning in y
    … …  
    12581516\item \code{BIN2.YBIN(S32)} gives the level 2 binning in y:
    12591517\item \code{PHOTCODE.RULE(STR)} gives a rule for producing a
    1260   photometry code, with values in curly brackets interpolated.
     1518  photometry code, with values in curly brackets interpolated in the
     1519  same manner as the file rules in the camera configuration.
     1520\item \code{PPIMAGE.JPEG1(METADATA)} and \code{PPIMAGE.JPEG2(METADATA)} give parameters for JPEG scaling, and contains:
     1521  \begin{itemize}
     1522  \item \code{COLORMAP(STR)} specifies the colormap to use:
     1523    \code{greyscale}, \code{-greyscale} (inverse greyscale),
     1524    \code{rainbow}, \code{heat}.
     1525  \item \code{SCALE.MODE(STR)} specifies how the scaling is performed: \code{RANGE} or \code{FRACTION}.
     1526  \item \code{SCALE.MIN(F32)} specifies the minimum scale.
     1527  \item \code{SCALE.MAX(F32)} specifies the maximum scale.
     1528  \end{itemize}
    12611529\end{itemize}
    12621530
    … …  
    13051573OVERSCAN.STAT           STR     MEAN            # MEAN | MEDIAN
    13061574
     1575# Fringe subtraction options
     1576FRINGE.ITER     S32     10              # Number of rejection iterations for fringe solution
     1577FRINGE.REJ      F32     2.0             # Rejection threshold for fringe solution
     1578FRINGE.KEEP     F32     0.5             # Minimum fraction to keep in fringe solution
     1579
    13071580# binned output image options
    13081581BIN1.XBIN               S32     8
    … …  
    13101583BIN2.XBIN               S32     64
    13111584BIN2.YBIN               S32     64
     1585
     1586PPIMAGE.JPEG1  METADATA
     1587  COLORMAP      STR     -greyscale
     1588  SCALE.MODE    STR     RANGE
     1589  SCALE.MIN     F32     -5.0
     1590  SCALE.MAX     F32     20.0
     1591END
     1592
     1593PPIMAGE.JPEG2  METADATA
     1594  COLORMAP      STR     greyscale
     1595  SCALE.MODE    STR     FRACTION
     1596  SCALE.MIN     STR     0.50
     1597  SCALE.MAX     STR     2.00
     1598END
    13121599
    13131600PHOTCODE.RULE           STR     {CAMERA}.{FILTER.ID}.{CHIP.N}
    … …  
    13321619\item \code{MASKVAL(S32)} gives the mask value for input data.
    13331620\item \code{COMBINE(STR)} gives the statistic to use for combination.
    1334 \item \code{BACKGROUND(STR)} gives the statistic to use to measure the background.
     1621\item \code{MEAN(STR)} gives the statistic to use to measure the mean.
     1622\item \code{STDEV(STR)} gives the statistic to use to measure the standard deviation.
     1623\item \code{WEIGHTS(BOOL)} specifies whether image (Poisson) weights should be used in the combination.
     1624\item For combining a fringe:
     1625  \begin{itemize}
     1626  \item \code{FRINGE.NUM(S32)} specifies the number of fringe regions for fringe combination.
     1627  \item \code{FRINGE.SIZE(S32)} specifies the half-size of the fringe regions.
     1628  \item \code{FRINGE.XSMOOTH(S32)} specifies the number of smoothing regions in x.
     1629  \item \code{FRINGE.YSMOOTH(S32)} specifies the number of smoothing regions in y.
     1630  \end{itemize}
     1631\item For generating a shutter correction:
     1632  \begin{itemize}
     1633  \item \code{SHUTTER.SIZE(S32)} specifies the size for shutter measurement regions.
     1634  \item \code{SHUTTER.ITER(S32)} specifies the number of iterations for performing the shutter measurement.
     1635  \item \code{SHUTTER.REJECT(F32)} specifies the rejection limit for shutter measurement.
     1636  \end{itemize}
     1637\item For generating a bad pixel mask:
     1638  \begin{itemize}
     1639  \item \code{MASK.SUSPECT(F32)} specifies the threshold for suspect pixels (in standard deviations).
     1640  \item \code{MASK.BAD(F32)} specifies the threshold for bad pixels
     1641    (in standard deviations); if negative, assume it's something like
     1642    a Poisson distribution.
     1643  \end{itemize}
    13351644\end{itemize}
    13361645
    1337 Statistics specified by a string (for \code{COMBINE} and
    1338 \code{BACKGROUND}) may be one of \code{MEAN}, \code{MEDIAN},
    1339 \code{ROBUST}, \code{FITTED} or \code{CLIPPED}.
     1646Mean statistics specified by a string (for \code{COMBINE},
     1647\code{MEAN}) may be one of \code{MEAN}, \code{MEDIAN}, \code{ROBUST},
     1648\code{FITTED} or \code{CLIPPED}.  The standard deviation statistic
     1649(\code{STDEV}) may be one of \code{STDEV}, \code{ROBUST_STDEV},
     1650\code{FITTED_STDEV}, or \code{CLIPPED_STDEV}.
     1651
    13401652
    13411653\subsubsubsection{Example}
    … …  
    13441656# Recipe configuration for ppMerge
    13451657 
    1346 ROWS            S32     128             # Number of rows to read at once
     1658ROWS            S32     512             # Number of rows to read at once
    13471659ELECTRONS       F32     100.0           # Minimum number of electrons for useful signal
    13481660SAMPLE          S32     100             # Sampling factor for measuring the background
    1349 REJ             F32     3.0             # Rejection threshold (sigma)
    1350 ITER            S32     1               # Number of rejection iterations
    1351 FRACHIGH        F32     0.3             # Fraction of high pixels to reject immediately
    1352 FRACLOW         F32     0.1             # Fraction of low pixels to reject immediately
    1353 NKEEP           S32     5               # Minimum number of pixels in stack to keep
    1354 MASKVAL         S32     0xff            # Mask value for input data
    1355 ### Statistics options: MEAN | MEDIAN | ROBUST | FITTED | CLIPPED
    1356 COMBINE         STR     MEAN            # Statistic to use for combination:
    1357 BACKGROUND      STR     MEDIAN          # Statistic to use to measure the background
     1661REJ             F32     3.0             # Rejection threshold (sigma)
     1662ITER            S32     1               # Number of rejection iterations
     1663FRACHIGH        F32     0.0             # Fraction of high pixels to reject immediately
     1664FRACLOW         F32     0.0             # Fraction of low pixels to reject immediately
     1665NKEEP           S32     5               # Minimum number of pixels in stack to keep
     1666FRINGE.NUM      S32     10000           # Number of fringe regions
     1667FRINGE.SIZE     S32     5               # Half-size of fringe regions
     1668FRINGE.XSMOOTH  S32     5               # Number of smoothing regions in x
     1669FRINGE.YSMOOTH  S32     11              # Number of smoothing regions in y
     1670SHUTTER.SIZE    S32     128             # Size for shutter measurement regions
     1671SHUTTER.ITER    S32     1               # Number of iterations for shutter measurement
     1672SHUTTER.REJECT  F32     2               # Rejection limit for shutter measurement
     1673MASK.SUSPECT    F32     5.0             # Threshold for suspect pixels (sigma)
     1674MASK.BAD        F32     -4.0            # Threshold for bad pixels (sigma)
     1675MASKVAL         S32     0xff            # Mask value for input data
     1676COMBINE         STR     CLIPPED         # Statistic to use for combination
     1677MEAN            STR     ROBUST_MEDIAN   # Statistic to use to measure the mean
     1678STDEV           STR     ROBUST_STDEV    # Statistic to use to measure the stdev
     1679WEIGHTS         BOOL    FALSE           # Use image weights?
    13581680\end{verbatim}
    13591681
Note: See TracChangeset for help on using the changeset viewer.