Changeset 11555

Timestamp:

Feb 1, 2007, 12:30:42 PM (19 years ago)

Author:

Paul Price

Message:

Adding description on MDC format. Cleaning up.

File:

• trunk/doc/manual/manual.tex (modified) (16 diffs)

trunk/doc/manual/manual.tex

@@ -1 @@
- %%% $Id: manual.tex,v 1.1 2007-02-01 04:32:04 price Exp $
+ %%% $Id: manual.tex,v 1.2 2007-02-01 22:30:42 price Exp $
 \documentclass[panstarrs,spec]{panstarrs}
 
@@ -125 +125 @@
 
 \begin{figure}
-\psfig{file=ipp.ps,height=9in,angle=0}
+\psfig{file=ipp.ps,height=8in,angle=0}
 \caption{Dependency chart for the IPP binaries}
 \label{fig:ipp-dependencies}
@@ -132 +132 @@
 The following external libraries are required to build \code{psLib}:
 \begin{itemize}
-\item \code{gsl}
-\item \code{fftw}
-\item \code{mysql}
-\item \code{cfitsio}
-\item \code{libjpeg}
-\item \code{openssl}
-\item \code{doxygen} (optional --- for generating documentation)
+\item \code{gsl} --- \code{http://www.gnu.org/software/gsl/}
+\item \code{fftw} --- \code{http://www.fftw.org}
+\item \code{mysql} --- \code{http://www.mysql.org}
+\item \code{cfitsio} ---
+  \code{http://heasarc.gsfc.nasa.gov/docs/software/fitsio/fitsio.html}
+\item \code{libjpeg} --- \code{http://www.ijg.org}
+\item \code{openssl} --- \code{http://www.openssl.org}
+\item \code{doxygen} (optional: for generating documentation) ---
+  \code{http://www.stack.nl/~dimitri/doxygen/}
 \end{itemize}
 
-\code{libpng} is required to build the \code{Ohana} package.
+In addition to the above, \code{libpng} (\code{http://www.libpng.org})
+is required to build the \code{Ohana} package.
 
 
@@ -147 +150 @@
 
 \begin{figure}
-\psfig{file=perl.ps,height=9in,angle=0}
+\psfig{file=perl.ps,height=8in,angle=0}
 \caption{Dependency chart for the IPP Perl components}
 \label{fig:perl-dependencies}
 \end{figure}
 
-The following external modules are required to build the Perl scripts:
+The following external modules, available from CPAN (see
+\S\ref{sec:installing-perl-dependencies}), are required to build the
+Perl scripts:
 \begin{itemize}
 \item \code{Apache2::SOAP}
@@ -188 +193 @@
 \section{Installation}
 
-After the prerequisites have been satisfied (\S\ref{sec:dependencies}),
-the IPP packages should be installed in the following order to satisfy
-the dependencies:
+After the dependencies (\S\ref{sec:dependencies}) have been satisfied,
+the IPP packages should be installed in the following order:
 \begin{itemize}
 \item \code{Ohana}
@@ -216 +220 @@
 \subsection{Binaries}
 
-Installation of the binaries is complicated by the fact that it may be
-used on multiple architectures.  The three developers based at
+Installation of the binaries is complicated by the fact that they may
+be used on multiple architectures.  The three developers based at
 Pan-STARRS HQ each use a different method for configuring the
-environment and installing the binaries to deal with this problem.
-We describe each of these below.  Choose one that works for you!
-
-\subsubsection{psconfig}
-
-\tbd{EAM to document psconfig.}
+environment and installing the binaries to deal with this problem.  We
+describe each of these below.  Choose one that works for you!
 
 \subsubsection{Aliases}
@@ -258 +258 @@
 
 
+\subsubsection{psconfig}
+
+\tbd{EAM to document psconfig.}
+
+
 \subsubsection{jhbuild}
 
+JH uses \code{jhbuild} even though the 'jh' in \code{jhbuild} doesn't
+really refer to him.
+
 \subsubsubsection{What is it?}
 
-According to the introduction on the jhbuild website:
+According to the introduction on the \code{jhbuild} website:
 
 \begin{quote}
-Jhbuild is a program that can be used to pull a number of modules from
-CVS and build them in the correct order. Unlike some build scripts,
-jhbuild lets you specify what modules you want built and it will then
-go and build those modules plus dependencies.
-
-Although jhbuild was originally developed to build [WWW]Gnome, it is
-now able to build a number of the modules in freedesktop.org
-CVS. Extending it to handle new modules is usually trivial (assuming
-the build infrastructure matches the other modules it handles).
+\code{jhbuild} is a program that can be used to pull a number of
+modules from CVS and build them in the correct order. Unlike some
+build scripts, \code{jhbuild} lets you specify what modules you want
+built and it will then go and build those modules plus dependencies.
+
+Although \code{jhbuild} was originally developed to build
+\code{[WWW]Gnome}, it is now able to build a number of the modules in
+\code{freedesktop.org} CVS. Extending it to handle new modules is
+usually trivial (assuming the build infrastructure matches the other
+modules it handles).
 \end{quote}
 
@@ -280 +289 @@
 FTP.
 
-\code{jhbuild} has been adopted as an official freedesktop.org
+\code{jhbuild} has been adopted as an official \code{freedesktop.org}
 project. You can find more information on the project's homepage
 (\code{http://www.freedesktop.org/Software/jhbuild}). Bugs can be
@@ -473 +482 @@
 \subsubsubsection{Dependancies}
 
-\code{jhbuild} has a fairly minimal set of dependencies. Far less then
-what may be required to actually compile and install any
-packages. However; if your system can meet the base requirements,
+\code{jhbuild} has a fairly minimal set of dependencies --- far less
+than what may be required to actually compile and install any
+packages. However, if your system can meet the base requirements,
 \code{jhbuild} should be able to bootstrap your build environment.
 \begin{itemize}
-\item A working C compiler (eg. gcc)
-\item A working libc (eg. glibc)
-\item Perl 5 with the XML::Parser module (needed by libtool)
+\item A working C compiler (eg. \code{gcc})
+\item A working \code{libc} (eg. \code{glibc})
+\item Perl 5 with the \code{XML::Parser} module (needed by
+  \code{libtool})
 \item Python 2.?
-\item Either wget or curl
-\item GNU M4 1.4
-\item tar
-\item gzip
-\item bzip2
+\item Either \code{wget} or \code{curl}
+\item GNU \code{M4} 1.4
+\item \code{tar}
+\item \code{gzip}
+\item \code{bzip2}
 \end{itemize}
 
@@ -492 +502 @@
 
 \code{jhbuild} has a limited ability to install some of the necessary
-tools for maintaining software that configure it's build environment
+tools for maintaining software that configure its build environment
 with the GNU autotools.
 
 This step is probably required on OSX and Solaris. Your mileage will
 vary per Linux distribution but you can probably skip this step if
-your distribution is less then two years old (ie. RedHat 9 or newer).
+your distribution is around RedHat 9 vintage or newer.
 
 \begin{verbatim}
@@ -537 +547 @@
 \subsection{Perl}
 
+Here we describe setting up the Perl dependencies followed by the
+IPP components.
+
 \subsubsection{Dependencies}
+\label{sec:installing-perl-dependencies}
 
 If you have access to the \code{root} account, installation as
 \code{root} is much easier.  If not, you will have to go through
-the more flaky installation as a privileged user.
+the more flaky installation as an unprivileged user.
 
 \subsubsubsection{Installation as root}
@@ -560 +574 @@
 \end{verbatim}
 
-If you get into trouble, try \code{force install MODULE_NAME}.
+Follow the prompts.  It's usually safe to accept the default (simply
+hit enter) in response to most questions.
+
+If you get into trouble, try: \code{force install MODULE_NAME}.
 
 You can also try to use the \code{Bundle::PS} as described below if
@@ -567 +584 @@
 \subsubsubsection{Installation as unprivileged user}
 
-To install modules from CPAN with CPAN.pm interface you need to setup
-a CPAN configuration file in your home directory.  The CPAN.pm will
-walk you through setting up the most important configuration values.
-Unfortunately, there is some variation in the behavior of the various
-versions of CPAN.pm that have shipped with Perl.  Some (most) of these
-variants will not correctly create a configuration files that allows a
-non-root user to install modules outside of "system" paths.  In order
-to make sure that you get a "correct" CPAN configuration file you need
-to "prime" it with a few values.
+To install modules from CPAN with the \code{CPAN.pm} interface, you
+need to setup a CPAN configuration file in your home directory.  Then
+\code{CPAN.pm} can walk you through setting up the most important
+configuration values.  Unfortunately, there is some variation in the
+behavior of the various versions of \code{CPAN.pm} that have shipped
+with Perl.  Some (most) of these variants will not correctly create a
+configuration files that allows a non-\code{root} user to install
+modules outside of "system" paths.  In order to make sure that you get
+a "correct" CPAN configuration file you need to ``prime'' it with a
+few values.
 
 First you need to create the directory in which the CPAN configuration file will live.
@@ -597 +615 @@
 %$ --- Emacs needs this to balance the previous dollar sign
 
-Now you need to invoke CPAN.pm so it can walk you through configuring
-the rest of the required values.  This is an example of one possible
-configuration with CPAN.com version 1.8802.  \textbf{Your version of
-CPAN.pm may present you with different prompts.}  Use your common
-sense.  If in doubt, it is generally safe to simply hit enter (and
-accept the default).
+Now you need to invoke \code{CPAN.pm} so it can walk you through
+configuring the rest of the required values.  This is an example of
+one possible configuration with \code{CPAN.pm} version 1.8802.
+\textbf{Your version of CPAN.pm may present you with different
+prompts.}  Use your common sense.  If in doubt, it is generally safe
+to simply hit enter (and accept the default).
 
 \begin{verbatim}
@@ -838 +856 @@
 
 Now we should install the basic compliment of helper modules that
-CPAN.pm needs to function fully.  Go back into cpan (\code{perl -MCPAN
--e shell}) and:
+\code{CPAN.pm} needs to function fully.  Go back into CPAN (\code{perl
+-MCPAN -e shell}) and:
 
 \begin{verbatim}
@@ -899 +917 @@
 \section{Configuration}
 
-\subsection{Site}
-\subsection{Camera}
-\subsection{Recipes}
+Correct use of the IPP depends on several configuration files.  We
+distinguish between configuration files required for the image
+processing and those for running the process scheduler, PanTasks.
+
+\subsection{Image Processing}
+
+Configuration information for the image processing is provided on four
+levels: the site, camera, format and recipe configurations.  Each uses
+the ``MetaData Configuration'' (MDC) file format, which is briefly
+described below; for a more detailed description, see the psLib SDRS
+(PSDC-430-007).
+
+The configuration levels for the image processing components of the
+IPP are:
+\begin{itemize}
+\item Options for the particular site installation of the
+  pipeline: the {\it site};
+\item Options specifying the instrument setup: the {\it camera};
+\item Options specifying the format of the FITS file: the {\it
+  format}; and
+\item Options specifying the particular parameter choices that affect
+  the details of an analysis: the {\it recipe}.
+\end{itemize}
+Note that these are arranged in an hierarchical order, with the site
+configuration being the most general, and the recipe configurations
+the most specific.  For example, not all sites will have to deal with
+all cameras, and different cameras may require different recipes at
+different times according to their particular quirks, analysis
+experimentations, or their evolution.
+
+We have provided examples of each of these configurations in the
+\code{config} component of the IPP, which should be a useful guide for
+setting up your own.  The Pan-STARRS IPP Configuration Guide
+(PSDC-430-???) has all the detailed information.
+
+\subsubsection{Overview of MDC format}
+
+psLib defines a \code{psMetadata} structure which can carry labeled
+data of arbitrary types.  Originally designed for carrying the data in
+FITS headers, the \code{psMetadata} have proved so generally useful
+that we use them for our configurations (and a multitude of other
+uses!).  We have designed a human-readable text-based format --- the
+``MetaData Configuration'' (MDC) format --- which we use for this end.
+
+Each simple entry in an MDC file must contain the name, type and
+value.  Each of these is on a single line, separated by whitespace,
+and in that order.  Comments may be placed at the end of the line (or
+on a blank line), after a hash mark (\code{#}).  Whitespace at the
+beginning and end of strings (either the name, value or comment) are
+stripped.
+
+The simple types follow the psLib types.  Integers are specified by a
+letter indicating if the integer is signed (\code{S}) or unsigned
+(\code{U}) and a number indicating the dynamic range in bits (8, 16,
+32 or 64); e.g., \code{U8} is commonly used for bit mask values,
+\code{S32} is commonly used for ordinary integer values.  Floating
+point values are specified by the letter \code{F} and a number
+indicating the precision in bits (32 or 64): \code{F32} (single
+precision) or \code{F64} (double precision).  Strings are specified by
+\code{STR}.  Times may be specified with the following types:
+\code{UTC,UT1,TAI,TT}; values for the time are expected to be in
+ISO8601 format (\code{YYYY-MM-DDTHH:MM:SS.sZ}).
+
+Names are traditionally all-caps, though there is no reason why they
+must be; the names are case-sensitive.  A name may not be repeated
+unless it has previously been declared to be of type \code{MULTI} (no
+value should be provided with this declaration):
+\begin{verbatim}
+COMMENT    MULTI
+COMMENT    STR    Having more than one COMMENT like this
+COMMENT    STR    is permitted because of the MULTI.
+\end{verbatim}
+
+A hierarchy can be made using the \code{METADATA} type, which signals
+a new level:
+\begin{verbatim}
+JANITOR    METADATA
+    NAME          STR    John Doe
+    PAY           F32    1234.56
+    ECCENTRICITY  STR    9.87
+END
+\end{verbatim}
+Note that a \code{METADATA} block is closed by an \code{END}.  No
+identing need be done within a \code{METADATA} block, but it is useful
+to be able to see the levels at a glance (just like in a C program).
+\code{METADATA} blocks may be nested within \code{METADATA} blocks,
+probably down as far as you have the patience to try.  Note that
+\code{MULTI} declarations only apply to the current level --- there is
+no inheritance.
+
+The above format can be long if there are many \code{METADATA}s with
+similar contents.  For this reason, we provide the \code{TYPE}
+declaration, which generates a \code{METADATA} with the contents each
+of type \code{STR}:
+\begin{verbatim}
+TYPE          EMPLOYEE    NAME       PAY        ECCENTRICITY
+\end{verbatim}
+Now, the type \code{EMPLOYEE} may be used, with string values (NB: no
+spaces allowed!) to specify multiple entries:
+\begin{verbatim}
+JANITOR       EMPLOYEE    JohnDoe    1234.56    9.87
+PROGRAMMER    EMPLOYEE    FooBar     2345.67    1.00
+\end{verbatim}
+This is the same as the much longer block:
+\begin{verbatim}
+JANITOR       METADATA
+    NAME            STR     JohnDoe
+    PAY             STR     1234.56
+    ECCENTRICITY    STR     9.87
+END
+PROGRAMMER    METADATA
+    NAME            STR     FooBar
+    PAY             STR     2345.67
+    ECCENTRICITY    STR     1.00
+END
+\end{verbatim}
+Like the \code{MULTI}, \code{TYPE} declarations only apply to the
+current level.
+
+
+
 \subsection{PanTasks}