Changeset 6757 for trunk/doc/modules/ModulesSDRS.tex

Timestamp:

Mar 31, 2006, 5:07:09 PM (20 years ago)

Author:

Paul Price

Message:

In the process of updating the SDRS to match the current implementation.

File:

• trunk/doc/modules/ModulesSDRS.tex (modified) (8 diffs)

trunk/doc/modules/ModulesSDRS.tex

@@ -1 @@
-%%% $Id: ModulesSDRS.tex,v 1.75 2006-02-16 18:57:51 eugene Exp $
+%%% $Id: ModulesSDRS.tex,v 1.76 2006-04-01 03:07:09 price Exp $
 \documentclass[panstarrs,spec]{panstarrs}
 
@@ -103 +103 @@
 All modules need to load some configuration information defining
 parameters which may be configured at run-time.  We break these
-parameters down into three levels:
+parameters down into four levels:
 \begin{itemize}
 \item Options for the particular site installation of the
   pipeline: the {\it site};
-\item Options specifying the instrument setup, and in particular the
-  format of the FITS file: the {\it camera}; and
+\item Options specifying the instrument setup: the {\it camera};
+\item Options specifying the format of the FITS file: the {\it
+  format}; and
 \item Options specifying the particular parameter choices that affect
   the details of an analysis: the {\it recipe}.
@@ -205 +206 @@
 \end{verbatim}
 
-\subsubsection{Camera Configuration}
-
-The camera configuration is somewhat complicated and involved, since
-it must not only specify how to translate the pixels from a FITS file
-into a focal plane hierarchy (\S\ref{sec:focalplane}), but it must
-also specify how to derive the various values the IPP needs
-(\S\ref{sec:concepts}).  Moreover, it must be able to do these for the
-great variety of cameras in use in the astronomical community.
+\subsubsection{Camera configuration}
+
+The camera configuration file is a fairly simple configuration file
+containing information particular to a particular camera, regardless
+of the file format used to represent that camera.  The camera configuration
+consists of the following elements:
+\begin{itemize}
+\item \code{FORMATS} of type \code{METADATA}: this contains a list of
+  known FITS file formats with the file names (of type \code{STR}) of
+  the configuration files;
+\item \code{FPA} of type \code{METADATA}: this contains a list of
+  chips, each with a string list (type \code{STR} of the component
+  cells; and
+\item \code{RECIPES} of type \code{METADATA}: this contains a list of
+  recipes used for the camera with the file names (of type \code{STR}
+  of the configuration files.
+\end{itemize}
+
+An example camera configuration file:
+
+\begin{verbatim}
+# Camera configuration file for MegaCam: describes the camera
+
+# File formats that we know about
+FORMATS         METADATA
+        RAW     STR     megacam_raw.config
+        SPLICE  STR     megacam_splice.config
+        SPLIT   STR     megacam_split.config
+END
+
+
+# Description of camera --- all the chips and the cells that comprise them
+FPA     METADATA
+        ccd00   STR     left right
+        ccd01   STR     left right
+        ccd02   STR     left right
+        ccd03   STR     left right
+        ccd04   STR     left right
+        ccd05   STR     left right
+        ccd06   STR     left right
+        ccd07   STR     left right
+        ccd08   STR     left right
+        ccd09   STR     left right
+        ccd10   STR     left right
+        ccd11   STR     left right
+        ccd12   STR     left right
+        ccd13   STR     left right
+        ccd14   STR     left right
+        ccd15   STR     left right
+        ccd16   STR     left right
+        ccd17   STR     left right
+        ccd18   STR     left right
+        ccd19   STR     left right
+        ccd20   STR     left right
+        ccd21   STR     left right
+        ccd22   STR     left right
+        ccd23   STR     left right
+        ccd24   STR     left right
+        ccd25   STR     left right
+        ccd26   STR     left right
+        ccd27   STR     left right
+        ccd28   STR     left right
+        ccd29   STR     left right
+        ccd30   STR     left right
+        ccd31   STR     left right
+        ccd32   STR     left right
+        ccd33   STR     left right
+        ccd34   STR     left right
+        ccd35   STR     left right
+END
+
+
+# Recipe options
+RECIPES         METADATA
+        PHASE2          STR     phase2.config           # Phase 2 recipe details
+        PSPHOT          STR     psphot.config           # psphot details
+END
+\end{verbatim}
+
+
+\subsubsection{FITS file format}
+
+The FITS file format configuration files are somewhat complicated and
+involved, since they must not only specify how to translate the pixels
+from a FITS file into a focal plane hierarchy
+(\S\ref{sec:focalplane}), but must also specify how to derive the
+various values the IPP needs (\S\ref{sec:concepts}).  Moreover, they
+must be able to do these for the great variety of cameras in use in
+the astronomical community.
 
 Example camera configuration files are included in an appendix, but
@@ -231 +313 @@
 segments and binary data segments.  The headers describe the format
 and layout of the data segments.  The first of these groups is
-traditionally called the 'primary header unit' (PHU) and the rest are
-referred to as 'extensions'.  The header segments may contain
+traditionally called the ``primary header unit'' (PHU) and the rest are
+referred to as ``extensions''.  The header segments may contain
 extensive documentary information related to the interpretation of the
 data.  Although the FITS format defines a standard representation of
@@ -246 +328 @@
 In the simplest case, the camera consists of a single chip consisting
 of a single cell always read with a single readout.  In this case, the
-image data could be written as part of the primary data unit.  In a
+image data could be written as part of the primary header unit.  In a
 more complex case with multiple chips and multiple cells, the data may
 be organized in several ways.  The data may be distributed into
@@ -260 +342 @@
 entire FITS file corresponds to (FPA, chip, or cell), and what level
 the extensions correspond to (chip, cell or no extensions at all).
-Knowing these, and having a list of the components, we can construct
+Knowing these, and having a list of the extensions, we can construct
 the focal plane hierarchy.
 
@@ -274 +356 @@
 To define the hierarchy, we specify the following keywords:
 \begin{itemize}
-
-\item \code{PHU} of type \code{STR}: May be one of \code{FPA},
-  \code{CHIP}, or \code{CHIP}.  This specifies the focal plane level
-  of the Primary Header Unit, and hence the entire FITS file (the
-  'class' of the file) .
-
-\item \code{EXTENSIONS} of type \code{STR}: May be one of \code{CHIP},
-  \code{CELL} or \code{NONE}, though not of a level higher than that
-  specified by the \code{PHU}.  This specifies what each extension
-  represents.
-
-\item \code{CONTENTS} which may be of type \code{METADATA} or
-  \code{STR}, depending upon the \code{PHU} and \code{EXTENSIONS},
-  specifies what the contents of the FITS file are:
+\item \code{RULE} of type \code{METADATA}: contains headers with their
+  respective values that are required to be in the PHU of any FITS
+  file of this type.
+
+\item \code{FILE} of type \code{METADATA}: contains information on
+  the global format of the FITS file with the following entries:
   \begin{itemize}
-    \item \code{PHU=FPA, EXTENSIONS=CHIP}: Type \code{METADATA} with
-      the component keywords being the extension names and the values
-      the names of the cells, separated by commas or whitespace.
-    \item \code{PHU=FPA, EXTENSIONS=CELL}: Type \code{METADATA} with
-      the component keywords being the extension names and the values
-      the chip name and the cell type, separated by a colon.
-    \item \code{PHU=FPA, EXTENSIONS=NONE}: Type \code{METADATA} with
-      the component keywords being the chip names and the values the
-      names of the cells, separated by commas or whitespace.
-    \item \code{PHU=CHIP, EXTENSIONS=CELL}: Type \code{METADATA} with
-      the component keywords being the extension names and the values
-      the corresponding cell type.
-    \item \code{PHU=CHIP, EXTENSIONS=NONE}: Type \code{STR} with the
-      value being the cell types separated by commas or whitespace.
+  \item \code{PHU} of type \code{STR}: May be one of \code{FPA},
+    \code{CHIP}, or \code{CHIP}.  This specifies the focal plane level
+    of the Primary Header Unit, and hence the entire FITS file (the
+    'class' of the file).
+
+  \item \code{EXTENSIONS} of type \code{STR}: May be one of
+    \code{CHIP}, \code{CELL} or \code{NONE}, though not of a level
+    higher than that specified by the \code{PHU}.  This specifies what
+    each extension represents.
+
+  \item \code{FPA.NAME} of type \code{STR}: Specifies a header keyword
+    in the primary header for a unique identifier for the FPA name
+    (e.g., an observation number).
+
+  \item \code{CHIP.NAME} of type \code{STR}: Need only be included if
+    \code{PHU} is \code{CHIP} or \code{CELL}.  Specifies a header
+    keyword in the primary header for a unique identifier for the chip
+    name (e.g., the CCD identification number or name).
+
+  \item \code{CELL.NAME} of type \code{STR}: Need only be included if
+    \code{PHU} is \code{CELL}.  Specifies a header keyword in the
+    primary header for a unique identifier for the cell name (e.g.,
+    the amplifier identification).
   \end{itemize}
-\item \code{CELLS} of type \code{METADATA} with the component keywords
-  being the cell names or types, each of type \code{METADATA}.  Within
-  each cell should be specified various PS concept values appropriate
-  for each cell.
+
+\item \code{CONTENTS} of type \code{METADATA}: Specifies what the
+  contents of the FITS file are.  Each entry is an extension name with
+  the corresponding value being a string listing the source and the
+  cell type, separated by a colon (e.g., \code{ccd01:left
+  ccd01:right}).  If \code{EXTENSIONS=NONE} then the \code{CONTENTS}
+  is ignored (since there are no extensions to list).
+
+\item \code{CELLS} of type \code{METADATA}: specifies the cell types.
+  Entries are the cell types, each of type \code{METADATA}, with the
+  values being PS concept values appropriate for each cell type (more
+  detail later) \tbd{link to more detail}.  In the event that
+  \code{EXTENSIONS=NONE}, the \code{CELLS} is used as a list of all
+  cells present in the file.
+
+\item \code{TRANSLATION} of type \code{METADATA}
+
+\item \code{DEFAULTS} of type \code{METADATA}
+
+\item \code{DATABASE} of type \code{METADATA}
+
+\item \code{FORMATS} of type \code{METADATA}
+
 \end{itemize}
 
@@ -313 +415 @@
 
 \begin{verbatim}
+# The raw MegaCam data comes off the telescope with each of the chips stored in extensions of a MEF file.
+
+# How to identify this type
+RULE    METADATA
+        TELESCOP        STR     CFHT 3.6m
+        DETECTOR        STR     MegaCam
+        EXTEND          BOOL    T
+        NEXTEND         S32     72
+END
+
 # How to read this data
-PHU             STR     FPA     # The FITS file represents an entire FPA
-EXTENSIONS      STR     CELL    # The extensions represent cells
- 
+FORMAT  METADATA
+        PHU             STR     FPA     # The FITS file represents an entire FPA
+        EXTENSIONS      STR     CELL    # The extensions represent cells
+        FPA.NAME        STR     EXPNUM  # A PHU keyword for unique identifier within the hierarchy level
+END
+
 # What's in the FITS file?
-CONTENTS        METADATA
-        # Extension name, chip name:type
-        amp00   STR     ccd00:left
-        amp01   STR     ccd00:right
-        amp02   STR     ccd01:left
-        amp03   STR     ccd01:right
-        amp04   STR     ccd02:left
+CONTENTS        METADATA
+        # Extension name, chip name:type
+        amp00           STR     ccd00:left
+        amp01           STR     ccd00:right
+        amp02           STR     ccd01:left
+        amp03           STR     ccd01:right
+        amp04           STR     ccd02:left
+        amp05           STR     ccd02:right
+        amp06           STR     ccd03:left
+        amp07           STR     ccd03:right
+        amp08           STR     ccd04:left
+        amp09           STR     ccd04:right
+        amp10           STR     ccd05:left
+        amp11           STR     ccd05:right
+        amp12           STR     ccd06:left
+        amp13           STR     ccd06:right
+        amp14           STR     ccd07:left
+        amp15           STR     ccd07:right
+        amp16           STR     ccd08:left
+        amp17           STR     ccd08:right
+        amp18           STR     ccd09:left
+        amp19           STR     ccd09:right
+        amp20           STR     ccd10:left
+        amp21           STR     ccd10:right
+        amp22           STR     ccd11:left
+        amp23           STR     ccd11:right
+        amp24           STR     ccd12:left
+        amp25           STR     ccd12:right
+        amp26           STR     ccd13:left
+        amp27           STR     ccd13:right
+        amp28           STR     ccd14:left
+        amp29           STR     ccd14:right
+        amp30           STR     ccd15:left
+        amp31           STR     ccd15:right
+        amp32           STR     ccd16:left
+        amp33           STR     ccd16:right
+        amp34           STR     ccd17:left
+        amp35           STR     ccd17:right
+        amp36           STR     ccd18:left
+        amp37           STR     ccd18:right
+        amp38           STR     ccd19:left
+        amp39           STR     ccd19:right
+        amp40           STR     ccd20:left
+        amp41           STR     ccd20:right
+        amp42           STR     ccd21:left
+        amp43           STR     ccd21:right
+        amp44           STR     ccd22:left
+        amp45           STR     ccd22:right
+        amp46           STR     ccd23:left
+        amp47           STR     ccd23:right
+        amp48           STR     ccd24:left
+        amp49           STR     ccd24:right
+        amp50           STR     ccd25:left
+        amp51           STR     ccd25:right
+        amp52           STR     ccd26:left
+        amp53           STR     ccd26:right
+        amp54           STR     ccd27:left
+        amp55           STR     ccd27:right
+        amp56           STR     ccd28:left
+        amp57           STR     ccd28:right
+        amp58           STR     ccd29:left
+        amp59           STR     ccd29:right
+        amp60           STR     ccd30:left
+        amp61           STR     ccd30:right
+        amp62           STR     ccd31:left
+        amp63           STR     ccd31:right
+        amp64           STR     ccd32:left
+        amp65           STR     ccd32:right
+        amp66           STR     ccd33:left
+        amp67           STR     ccd33:right
+        amp68           STR     ccd34:left
+        amp69           STR     ccd34:right
+        amp70           STR     ccd35:left
+        amp71           STR     ccd35:right
 END
 
 # Specify the cell data
-CELLS   METADATA
-        left    METADATA        # Left amplifier
-                CELL.BIASSEC            STR     BIASSEC
-                CELL.TRIMSEC            STR     DATASEC
-                CELL.PARITY             S32     1
-        END
-        right   METADATA        # Right amplifier
-                # This cell is read out in the opposite direction
-                CELL.BIASSEC            STR     BIASSEC
-                CELL.TRIMSEC            STR     [1025:2048,1:2048]
-                CELL.PARITY             S32     -1
-        END
+CELLS   METADATA
+        left    METADATA        # Left amplifier
+                CELL.NAME               STR     LeftSide
+                CELL.BIASSEC.SOURCE     STR     HEADER
+                CELL.TRIMSEC.SOURCE     STR     HEADER
+                CELL.BIASSEC            STR     BIASSEC
+                CELL.TRIMSEC            STR     DATASEC
+                CELL.XPARITY            S32     1 # We could have specified this as a DEFAULT, but this works
+                CELL.X0                 S32     1
+                CELL.Y0                 S32     1
+        END
+        right   METADATA        # Right amplifier
+                CELL.NAME               STR     RightSide
+                CELL.BIASSEC.SOURCE     STR     HEADER
+                CELL.TRIMSEC.SOURCE     STR     HEADER
+                CELL.BIASSEC            STR     BIASSEC
+                CELL.TRIMSEC            STR     DATASEC
+                CELL.XPARITY            S32     -1 # This cell is read out in the opposite direction
+                CELL.X0                 S32     2048
+                CELL.Y0                 S32     1
+        END
+END
+
+# How to translate PS concepts into FITS headers
+TRANSLATION     METADATA
+        FPA.NAME                STR     EXPNUM
+        FPA.AIRMASS             STR     AIRMASS
+        FPA.FILTER              STR     FILTER
+        FPA.POSANGLE            STR     ROTANGLE
+        FPA.RA                  STR     RA
+        FPA.DEC                 STR     DEC
+        FPA.RADECSYS            STR     RADECSYS
+        CELL.EXPOSURE           STR     EXPTIME
+        CELL.DARKTIME           STR     DARKTIME
+        CELL.GAIN               STR     GAIN
+        CELL.READNOISE          STR     RDNOISE
+        CELL.SATURATION         STR     SATURATE
+        CELL.TIME               STR     MJD-OBS
+        CELL.XBIN               STR     CCDBIN1
+        CELL.YBIN               STR     CCDBIN2
+END
+
+# Default PS concepts that may be specified by value
+DEFAULTS        METADATA
+        CELL.READDIR            S32     1               # Cell is read in x direction
+        CELL.BAD                S32     0
+        CELL.TIMESYS            STR     UTC
+        CELL.YPARITY            S32     1
+END
+
+# How to translation PS concepts into database lookups
+DATABASE        METADATA
+        TYPE            dbEntry         TABLE           COLUMN          GIVENDBCOL      GIVENPS
+#       FPA.BIAS        METADATA
+#               TABLE   STR     Camera
+#               COLUMN  STR     gain
+#               chipId  STR     {CHIP.NAME}
+#               cellId  STR     {CELL.NAME}
+#               time    STR     {CELL.TIME}
+#       END
+#       CELL.GAIN       dbEntry         Camera          gain            chipId,cellId   CHIP.NAME,CELL.NAME
+#       CELL.READNOISE  dbEntry         Camera          readNoise       chipId,cellId   CHIP.NAME,CELL.NAME
+
+# A database entry refers to a particular column (COLUMN) in a
+# particular table (TABLE), given certain PS concepts (GIVENPS) that
+# match certain database columns (GIVENDBCOL).
+END
+
+
+# Where there might be some ambiguity, specify the format
+FORMATS         METADATA
+        FPA.RA          STR     HOURS
+        FPA.DEC         STR     DEGREES
+        CELL.TIME       STR     MJD
+#       CELL.BINNING    STR     SEPARATE
+        CELL.X0         STR     FORTRAN
+        CELL.Y0         STR     FORTRAN
 END
 \end{verbatim}