Changeset 13881

Timestamp:

Jun 19, 2007, 11:38:53 AM (19 years ago)

Author:

eugene

Message:

updated text

File:

• trunk/doc/manual/manual.tex (modified) (9 diffs)

trunk/doc/manual/manual.tex

@@ -1 @@
- %%% $Id: manual.tex,v 1.8 2007-06-19 03:34:03 eugene Exp $
-\documentclass[panstarrs,spec]{panstarrs}
+ %%% $Id: manual.tex,v 1.9 2007-06-19 21:38:53 eugene Exp $
+\documentclass[panstarrs,psreport,spec]{panstarrs}
 
 % basic document variables
@@ -24 +24 @@
 % version     Date         Description
 DR      & 2007-01-31 & First draft \\ \hline
-\RevisionsEnd
+\hline
+\end{tabular}
+\end{center}
 
 \inserttbd
@@ -409 +411 @@
 \note{specifying the dvo output database}
 
-\subsubsection{Pipeline Reduction : CFHT Queue Runs}
-
-\subsubsection{Pipeline Reduction : PS1 Survey Operations}
+\subsubsection{Pipeline Reduction : PI-Oriented Observatory}
+
+An PI-oriented observatory has the opportunity to operate the analysis
+system on a somewhat larger scale than the individual user.  At a
+PI-oriented observatory, there will normally be a rotating suite of
+instruments, each of which is mounted on the telescope for a
+substantial number of nights.  Over the course of months and years,
+the observatory will collect enough data from the instruments to
+perform a much more detailed level of analysis of the instrument than
+is possible in a single observing run by a single PI.  In such a
+situation, the observatory may use the IPP to generate and track the
+high-quality master detrend images, to apply the detrend images to the
+data as it is being collected, and to monitor the instrumental zero
+points and other calibration information.  The basic concepts of
+running the IPP at a PI observatory are identical to that of a single
+user.  There are, however, a few modifications which may be
+interesting.
+
+First, it is natural to automatically inject and register the data as
+it is obtained at the telescope.  The telescope readout software could
+be modified to call \code{ipp_serial_inject.pl} after each image is
+obtained.  Alternatively, this step could take place downstream
+wherever the data are archived.  In an observatory environment, more
+so than in a single-user situation, it may be perfereable to use the
+Nebulous infrastructure to manage the location and copies of the image
+data.  \tdb{define an inject mechanism that uses Nebulous}.
+
+Second, it is interesting to consider on what timescale to test and
+re-build the master detrend images.  In the single user case, the
+possible input detrend images are fairly limited.  As illustrated
+above, the single user may use the IPP to define master detrend images
+and test the consistency of the inputs against these masters.  In an
+observatory environment, it may be more appropriate to build the
+masters only on regular intervals when there is a suspicion the
+instrument may have changed.  The periods in which the instrument is
+mounted define an obvious possible starting point for this period.
+Alternatively, the detrend creation steps in the IPP can be defined to
+run in verify mode on a nightly basis to test the current state of the
+instrument.  For the PI-oriented observatory, the number of variables
+may make it difficult to rely on these results in an automatic
+fashion; user examination of the results with \code{ippMonitor} may be
+needed to decide if a new master should be built.
+
+\subsubsection{Pipeline Reduction : Survey Observatory}
+
+A telescope used as a survey instrument has a somewhat different, more
+limited range of circumstances, than a PI-oriented observatory.  There
+is generally a pre-defined schedule, perhaps with only a single
+instrument.  The instrument configuration is more controlled, and the
+types of observations are more consistently defined.  In this case, an
+automatic validation process can be more reliably used.  The IPP can
+be set up to run the detrend validation stage at the beginning of
+every night, to automatically generate a new master if any of the
+input detrend images are out of spec, and to start automatic
+processing of the night's data with the new master images when they
+are ready. 
 
 \section{IPP Components}
@@ -419 +474 @@
 
 \subsection{Analysis Programs}
+
 \subsubsection{psphot}
+
+The detection and characterization of astronomical objects within an
+image is performed by the IPP component called \code{psphot}.  This
+software may be used as a stand-alone program, or it may be called by
+other IPP programs to operate on the images they generate.  A more
+detailed guide is available at REF.  Here we summarize the basic
+operation and command-line options.
+
+The basic call to the stand-alone version of \code{psphot} is:
+%
+\code{psphot -file input.fits output [options]}
+%
+where \code{[options]}
+represents options to the IPP configuration system as well as some
+specific to \code{psphot}.  In the case of a ``split'' camera format,
+where separate chips or amps are stored in separate files, the
+``input.fits'' may be a UNIX file glob which will expand to the set of
+file; note that the glob must be protected with double quotes for
+psphot. Alternatively, the input file list may be written to a text
+file and the \code{-file input.fits} entry replace with \code{-list
+  file.txt}.  The output entry is used as the root of a number of
+possible output files selected by the user.
+
+Standard IPP-wide command-line options (document elsewhere):
+\begin{itemize}
+\item C{-site site.mdc}
+\item C{-camera camera-name}
+\item C{-recipe NAME VALUE}
+\item C{-D KEY VALUE}
+\item C{-Df KEY VALUE}
+\item C{-Db KEY VALUE}
+\item C{-Di KEY VALUE}
+\end{itemize}
+
+Here are other possible command-line options specific to psphot:
+\begin{itemize}
+\item -version : report version info and exit
+\item -mask mask.fits 
+\item -weight weight.fits
+\item -modeltest X Y : run the object model on the object at the given coordinates
+\begin{itemize}
+\item -model
+\item -fitmode
+\item -fitset
+\end{itemize}
+\item -photcode : manually specify the photcode for this image
+\item -region : limit the analysis to a specific image area
+\item -chip : limit the analysis to the given list of chips
+\item -psf : use the specified psf model (do not build a psf model)
+\item -src : perform photometry on the positions given in the list
+\end{itemize}
+
+There are many configuration options in the \code{psphot.config}
+recipe file.  These are defined in the \code{psphot} manual.
+
 \subsubsection{psastro}
+
 \subsubsection{ppStats}
+
 \subsubsection{ppImage}
+
 \subsubsection{ppMerge}
+
 \subsubsection{ppNorm}
 
 \subsection{Pipeline Infrastructure}
-\subsubsection{ippdb}
+
+\subsubsection{ippdb / dbconfig}
+
+The IPP uses a mysql database to track the data to be processed,
+the current state of the processing steps, and summary information
+from each of the analysis steps.  The database schema is defined by
+the set of MDC files in \code{dbconfig}.  These files are used to
+generate a set of C-level APIs to access the database tables.  
+
 \subsubsection{ippTools}
+
 \subsubsection{ippScripts}
+
 \subsubsection{ippTasks and panTasks}
+
 \subsubsection{ippMonitor}
 
-\subsubsection{Nebulous}
+\subsection{Nebulous}
 
 \subsection{DVO}
+
 \subsubsection{DVO Shell}
+
 \subsubsection{Adding and Removing Data}
+
 \subsubsubsection{addstar}
+
 \subsubsubsection{delstar}
+
 \subsubsubsection{getstar}
+
 \subsubsection{Database Level Calibrations}
+
 \subsubsubsection{relphot}
+
 \subsubsubsection{uniphot}
+
 \subsubsubsection{relastro}
+
 \subsubsection{Other Tools}
+
 \subsubsubsection{sky cell tools}
 
 \subsection{Software Architecture}
-\subsection{psLib}
-\subsection{psModules}
-\subsection{Perl Modules}
+
+\subsubsection{psLib}
+
+\subsubsection{psModules}
+
+\subsubsection{Perl Modules}
 
 \section{Configuration}
 
-Correct use of the IPP depends on several configuration files.  We
-distinguish below between configuration files required for the image
-processing and those for running the process scheduler, PanTasks.
-Note, however, that the Perl scripts called by PanTasks to run the
-processing do use the site and camera configuration files principally
-used for the image processing.
-
-\subsection{Image Processing}
+Correct use of the IPP depends on a collection of configuration files.
+These files define information about the processing environment, such
+as the location of reference data or the name of computers available
+for analysis or storage.  They also define the operating parameters
+for the different programs, and choices about which analysis steps to
+perform.
 
 Configuration information for the image processing is provided on four
-levels: the site, camera, format and recipe configurations.  Each uses
-the ``MetaData Configuration'' (MDC) file format, which is briefly
-described below; for a more detailed description, see the psLib SDRS
-(PSDC-430-007).
+levels: the site-specific information, camera-specific details,
+information related to different formats of data from the same camera,
+and recipe configurations for the different analysis programs or
+steps.  The configuration information is stored in files using the
+``psMetadataConfig'' (MDC) file format, which is briefly described
+below (\S\ref{sec:MDC}); for a more detailed description, see the
+psLib SDRS (PSDC-430-007).
 
 The configuration levels for the image processing components of the
 IPP are:
 \begin{itemize}
-\item Options for the particular site installation of the
-  pipeline: the {\it site};
-\item Options specifying the instrument setup: the {\it camera};
+\item Options for the particular installation of the
+  IPP: the {\it site};
+\item Options specifying the details about the instrument: the {\it
+camera};
 \item Options specifying the format of the FITS file: the {\it
   format}; and
@@ -493 +635 @@
 (PSDC-430-???) has all the detailed information.
 
+\subsubsection{Setting up configuration files}
+
+When the IPP software is installed, it generates a copy of the
+top-level {\it site} configuration file:
+\code{PREFIX/share/ippconfig/ipprc.config}. Programs look for this
+information in the user's home directory at \code{~/.ipprc}.  A new
+user may want to link \code{~/.ipprc} to the installed copy
+\code{PREFIX/share/ippconfig/ipprc.config}, in which case new
+installations will be immediately available.  Alternatively, the user
+may want to copy the file in order to make their own modifications, if
+for example the user needs to modify the recipes for a specific
+camera.
+
+After the {\it site} configuration file is read by IPP programs, they
+then attempt to read the {\it camera} configuration files defined in
+the {\it site} file by the list of cameras.  These entries refer to
+files for each camera.  The IPP configuration system searches for these
+files (and others referred there in) in the PATH defined at the top of
+the {\it site} file.  This value mimics the UNIX PATH variable, and
+consists of a colon-separated list of locations.  The first entry
+found from this list is used by the configuration system, allowing a
+user to override, for example, the globally installed camera
+configuration for some camera.  
+
 \subsubsection{Overview of MDC format}
+\label{sec:MDC}
 
 psLib defines a \code{psMetadata} structure which can carry labeled
@@ -577 +744 @@
 current level.
 
-
-\subsubsection{Setting up configuration files}
-
-You will generally want to link \code{~/.ipprc} to the site
-configuration file (\code{ipprc.config} which gets installed in
-\code{PREFIX/share/ippconfig/} where \code{PREFIX} is your
-installation prefix).  Then link \code{~/.ipp} to the \code{ippconfig}
-directory to save hacking the \code{PATH} in the site configuration.
-
-\subsection{PanTasks}
-
-\subsubsection{Paths}
-
-Throughout PanTasks, a file may be referred to as:
+\subsection{Filenames : UNIX Paths, Abstract Paths, Nebulous}
+
+The IPP programs recognize three types of file names: a standard UNIX
+path, an abstract path with a top-level location defined by the IPP
+configuration system, and a Nebulous path, in which the file location
+is completely abstracted by the Nebulous file management system:
 \begin{itemize}
-\item \code{/path/to/file.ext} --- not a URI, but it should work.
-\item \code{file:///path/to/file.ext} --- the URI version of the above.
-\item \code{path://PATH/file.ext} --- Uses the \code{DATAPATH} in the site configuration to set the path.
+\item \code{file:///path/to/file.ext} : a UNIX path, always relative
+  to the root of the file system.  It is valid to drop the
+  \code{file:/} component of the name.  The IPP accepts an arbitrary
+  number of slashes after \code{file:}.
+\item \code{path://PATH/file.ext} : An abstract path. The value of
+  PATH is iterpolated from the corresponding entry in the
+  \code{DATAPATH} table given in the site configuration file.
+\item \code{neb://full/nebulous/name} : A nebulous filename.  The IPP
+  programs query Nebulous for the true path of (one copy of) the file.
 \end{itemize}
+If you wish to expand a file name in any of these forms to a standard
+UNIX path, use the program \code{ipp_datapath.pl}.  
 
 \section{Installation}
@@ -653 +820 @@
 When building software 
 
-\subsubsection{jhbuild}
+\subsection{jhbuild}
 
 JH uses \code{jhbuild} even though the 'jh' in \code{jhbuild} doesn't
@@ -935 +1102 @@
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\subsubsection{Aliases}
+\subsection{Aliases}
 
 PAP puts the following in his \code{~/.tcshrc}:
@@ -966 +1133 @@
 \end{itemize}
 
-\subsection{Perl}
+\subsection{Manual Perl Module Installation}
 
 Here we describe setting up the Perl dependencies followed by the