Changeset 13869

Timestamp:

Jun 18, 2007, 5:34:03 PM (19 years ago)

Author:

eugene

Message:

more updates

File:

• trunk/doc/manual/manual.tex (modified) (7 diffs)

trunk/doc/manual/manual.tex

@@ -1 @@
- %%% $Id: manual.tex,v 1.7 2007-06-15 20:52:48 eugene Exp $
+ %%% $Id: manual.tex,v 1.8 2007-06-19 03:34:03 eugene Exp $
 \documentclass[panstarrs,spec]{panstarrs}
 
@@ -275 +275 @@
 processing. \note{halt does not current stop pcontrol}.  
 
-\subsubsection{ippTools}
+\subsubsection{ippTools \& ippScripts}
+
+The IPP uses a \code{mysql} database to keep track of the data to be
+analysed and the results of analysis steps.  The IPP user interacts
+with this database via a set of programs called \code{ippTools} and
+\code{ippScripts}.  The difference between these is that the
+\code{ippTools} are somewhat lower-level programs to interact directly
+with the database, while the ippScripts perform a more complex
+operation, sometimes wrapping multiple calls to commands within
+\code{ippTools}.   
+
+\begin{itemize}
+\item \code{ipp_serial_inject.pl} : inject data into the IPP database.
+\item \code{dettool -definebyquery} : define a new detrend run.  With
+  this command, the user defines a set of criteria which define a
+  collection of input images to be used to generate a detrend image.
+  The parameters which define the output detrend image also
+  specified.
+\item \code{dettool -updatedetrun} : manually set the state of a
+  detrend run.  
+\item \note{extend this list...}
+\end{itemize}
+
+The ippTools accept a common set of command-line arguments which
+modify their behavior:
+\begin{itemize}
+\item \code{-pretend} : show the expected result, but do not apply the
+  results to the database
+\item \code{-simple} return results with one line for each output
+  component.  Without this flag, the results are returned with the
+  versbose, but self-documenting, psMetadataConfig format.
+\item \code{-dbname} specify the database to use
+\item \note{extend this list...}
+\end{itemize}
 
 \subsection{Operational Scenarios}
 
 \subsubsection{Reducing an Observing Run}
+
+For a single observing run, the user will typically have data from
+several nights, both science and raw detrend images.  The following
+steps illustrate the analysis with data from a sample observing run.
+We assume the user has already installed the IPP software and the
+external software (mysql, apache, php, etc), set up the configuration
+for the camera, defined a project for this analysis, started pantasks,
+and loaded the analysis tasks.  This example shows the steps to build
+the detrend data and process the science data.
+
+The first step is to inject all of the data into the IPP database with
+the command \code{ipp_serial_inject.pl}.  It is possible to define
+abstract data storage paths in the user's \code{.ipprc} file in the
+\code{DATAPATH} section. Each line gives the relationship between an
+abstract data location and the real-world UNIX path to that data.  The
+script \code{ipp_serial_inject.pl} will interpret the location of the
+data in terms of the DATAPATH hierarchy when identifying the data for
+the database. When injecting the data, it is convenient to define a
+working directory with the \code{--workdir} option to
+\code{ipp_serial_inject.pl}.  The science processing steps will use
+this location to store the output data products.
+
+% complete command here
+
+As the images are injected, they will be placed in the table of
+\code{new} exposures.  If \code{pantasks} is running with the modules
+loaded, the images will be processed for registration.  In this step,
+a minor analysis of the image statistics and an examination of the
+header information is performed, and the results used to provide
+further information about the images in the database.  As images are
+registered, they migrate from the \code{new} to the \code{raw}
+tables.  These tables can be view in the \code{ippMonitor} under the
+\code{Load & Setup} pages.
+
+Once the images have been injected, the user will want to create, in
+order, bias, dark, shutter, flat, and fringe frames.  As these are
+created, they may be used to generate the next level of image.  It is
+important not to define, eg, a flat run before the lower-precedence
+elements are defined.  At the moment, it is not possible to define a
+detrend analysis which blocks until a previous detrend analysis is
+complete.  To start, define a bias based on a query to the database:
+
+% example command hrere
+
+\code{dettool -dbname name -definebyquery -det_type bias etc}
+
+For a small observing run, it is probably possible to include all bias
+images in the initial pass.  IPP will iteratively reject complete
+input images which are outliers from the ensemble, as well as
+individual pixels.  In the \code{ippMonitor}, rejected images will be
+greyed out in the residual pages.  Examine the resulting residual
+images.  If there are patterns to the residuals, it may be necessary
+to define subgroups, perhaps by time or ccd temperature.  For example:
+
+% example command here.
+
+After a bias or set of biases are defined, follow the same process for
+a dark frame.  It may be necessary to supply a non-linear dark current
+correction (perhaps one for each amplifier or chip).  The script
+\code{ipp_darkstats.pl} can be used to examine this trend.  
+
+If a shutter correction is necessary, make sure to obtain data which
+can be used to generate the correction: a series of flat-field images
+with a range of exposure times, particularly near the short end of the
+range.  If the shutter correction will be skipped, set the SHUTTER
+entries in the camera's ppImage.config file to FALSE.
+
+A flat-field image is next.  Initially, we must define a raw
+flat-field image, from dome or twilight flat-field images.  Below, we
+will discuss the generation of a flat-field correction image from
+dithered science images.  Do not forget to specify the filter for each
+flat-field image
+
+% example command.
+
+Up to now, we have been ignoring the bad pixel mask.  At this point,
+the user may generate a bad pixel mask from the collection of
+processed flat-field images.  Use the script \code{UNKNOWN} to build a
+mask image.  Turn on the use of the mask by setting \code{MASK} to
+true in the \code{ppImage.config} file.  It is probably a good idea to
+now re-run the previous stages with the mask applied: this will result
+in better statistics for the residual images.
+
+If certain filters require a fringe image, the user may inform the IPP
+of this fact by setting the FRINGE.FILTERS entry in the
+\code{ppImage.config} file.  Use a set of night-time sky images to
+generate a fringe frame. \note{how to specify more than one fringe
+  mode?}
+
+Activate the chip-level science analysis tasks in pantasks: chip.on.
+With this off, no data will be processed before the detrend images are
+ready.  Once this is turned on, the science images will start to be
+processed \note{discuss blocks; re-queue; versions}.
+
+\note{specifying the dvo output database}
 
 \subsubsection{Pipeline Reduction : CFHT Queue Runs}
@@ -498 +626 @@
 \subsection{psconfig}
 
+\note{use the write-up in psconfig for this section}
+
+\begin{verbatim}
 * pscheckperl : search for and install, if needed, external Perl modules
 * pschecklibs : search for and install, if needed, external C libraries
@@ -504 +635 @@
 * psdist : build IPP distributions (requires CVS access)
 * tagsets : tables defining the C and Perl components to be built
+\end{verbatim}
 
 The IPP is a large and complex software system.  A major goal of the
@@ -1339 +1471 @@
 \subsection{missing libX11.so}
 
-RedHat RHEL 3 & 4 (likely other RedHat variants as well) for \emph{amd64} seem
+RedHat RHEL 3 \& 4 (likely other RedHat variants as well) for \emph{amd64} seem
 to often be installed without a \code{libX11.so} under
 \code{/usr/X11R6/lib64/}.  This will cause 64bit builds trying to link against
@@ -1347 +1479 @@
 yourself.  Eg..
 
-\begin{verbatium}
+\begin{verbatim}
 su -
 cd /usr/X116/lib64
 ln -s libX11.so.6.2 libX11.so
-\end{verbatium}
+\end{verbatim}
 
 or with \code{sudo}:
 
-\begin{verbatium}
+\begin{verbatim}
 cd /usr/X116/lib64
 sudo ln -s libX11.so.6.2 libX11.so
-\end{verbatium}
+\end{verbatim}
 
 
@@ -1365 +1497 @@
 installing IPP software.  Note that \code{\$prefix/lib} will also need to be manually added the enviroment variable \code{LD_LIBRARY_PATH} if your not using jhbuild. E.g for ksh/bash users:
 
-\begin{verbatium}
+\begin{verbatim}
 export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/[foo]/[yourinstallprefix]/lib
-\end{verbatium}
+\end{verbatim}