\documentclass[panstarrs,spec]{panstarrs}

\title{IPP Postage Stamp Server} % put in your title
\subtitle{ICD}
\author{Bill Sweeney}
\audience{IPP}
\shorttitle{Postage Stamp ICD}
\group{Pan-STARRS IPP}
\project{Pan-STARRS IPP}
\organization{Institute for Astronomy}
\version{1.0}
\docnumber{PSDC-940-011-00}

\newcommand\ugriz{$u^\prime g^\prime r^\prime i^\prime z^\prime$}
\newcommand\grizy{$g r i z y$}


\begin{document}

\maketitle
\tableofcontents

\pagebreak
\pagenumbering{arabic}

\section{Overview}

The basic functionality of the postage stamp server is to accept lists
of postage stamp requests, determine the corresponding images, extract
the desired pixels from the images, and return the postage stamps to
the user.  The nominal mode of interaction as an interface to MOPS and
the other Science Clients is for the postage stamp requests to be FITS tables
which are placed as incoming data on the IPP Data Store (or the data
store of the requesting system) and for the results to be placed back
on the IPP Data Store, along with a FITS table description of the
resulting output image files.  For the generic user, the requests and
the result may be accessed directly via a web server.  

The basic information needed to by the postage stamp server (PSS) to honor a
request consists of: 1) the image of interest and 2) the region on the
image desired (ROI).  This information may be represented in a number of ways
which are discussed in section 2.1. % How do I include a reference by tag?

\section{Postage Stamp Requests}


Postage stamp requests are packaged for submission to the PSS in a file set containing
one or more FITS binary tables called \emph{request tables}.
Each row in a request table contains a postage stamp request specification.
Each request specification causes zero or more postage stamp jobs to be processed.
Images are added to the output file set for each successful job.

The request specification give the parameters that the PSS uses to determine the images of interest
and the Region of Interest. 

\subsection{Request Table Contents}

The required header keywords for a request table
are given in Table~\ref{RHeaderDef}.  
The definition for the table's columns is in Table~\ref{RTableDef}.

\begin{table}[h]
\begin{center}
\caption{Postage Stamp Request File Header Keywords}\label{RHeaderDef}
\begin{tabular}{lll}
\hline
\hline
{\bf FITS Keyword} & {\bf Datatype} &  {\bf Description } \\
\hline
\code{EXTNAME}  & string     & \code{PS1_PS_REQUEST} \\
\code{EXTVER}   & string     & 1  \\
\code{REQ_NAME} & string     & unique name of the request (see text) \\
\hline
\end{tabular}
\end{center}
\end{table}

The IPP will be configured to periodically query a set of Data Stores for new request file sets.
We will refer to each Data Store as a \emph{requester}.
Each request table submitted must have a unique value for the keyword \code{REQ_NAME}.
This value is used for the name of the request's output file set.

The postage stamp server supports three types of "postage stamp" job specified by the value
the column \code{JOB_TYPE}. If the value is \emph{stamp} postage stamp images will be extracted
from the images of interest.  If the value of \code{JOB_TYPE} is \emph{get\_image} 
the data from the selected images' "runs"  with (satellite streaks removed) are bundled
in and placed on the output file set.

\begin{table}[h]
\begin{center}
\caption{Postage Stamp Request Table Columns}\label{RTableDef}
\begin{tabular}{llll}
\hline
\hline
{\bf Column Name} & {\bf Datatype} & {\bf Values}       &        {\bf Notes } \\
\hline
\code{ROWNUM}   & U32       &                           &        must be unique in file\\
\code{PROJECT}  & string    & gpc1                      &        \\
\code{JOB_TYPE} & string    & stamp or get\_image       &        \\
\\
\code{OPTION_MASK} & U32    & see text                  &      \\ 
\\
\code{REQ_TYPE} & string    & bycoord, byexp, byid, bydiff, or byskycell   &        \\
\code{IMG_TYPE} & string    & raw, chip, warp, diff, or stack  & \\
\code{ID} & string          & depends on REQ\_TYPE& \\
\code{TESS_ID}  & string    & see text                  &        \\
\code{COMPONENT} & string   & see text                  &        \\
\\
\code{COORD_MASK} &  U32    & \code{[0-3]} see text     &        not null if \code{JOB_TYPE} = stamp \\
\code{CENTER_X} &  F64      & see text                  &        not null if \code{JOB_TYPE} = stamp \\
\code{CENTER_Y} &  F64      & see text                  &        not null if \code{JOB_TYPE} = stamp \\
\code{WIDTH}    &  F64      & see text                  &        not null if \code{JOB_TYPE} = stamp \\
\code{HEIGHT}   &  F64      & see text                  &        not null if \code{JOB_TYPE} = stamp \\
\\
\code{DATA_GROUP} &   string  & any                      & \\
\code{REQFILT}  &   string  & any                       & \\
\code{MJD_MIN}  &   F64     & Modified Julian Day       & \\
\code{MJD_MAX}  &   F64     &                           & \\
\\
\code{COMMENT}  &   string  & copied to the results flie & \\
\hline
\end{tabular}
\end{center}
\end{table}

\subsubsection{Image Types}
All requests need IMG\_TYPE. This parameter selects the stage of the processing pipeline
for the images of interest.
The allowed values for IMG\_TYPE are raw, chip, warp, diff, and stack.

\subsubsection{Request Types}
The method used to select the images of interest is determined by the value of the REQ\_TYPE column.
There are three allowed values. 

\begin{itemize}

\item{\textbf{bycoord}} The center of the region of interest is specified in celestial coordinates.
Images are selected by querying the project's DVO database for images with the provided
IMG\_TYPE that contain the center.
The set of images of interest may be limited by specifying certain criteria as described
in Section \ref{ImageCuts}.

\item{\textbf{byexp}}  The image of interest is the image resulting from a particular exposure 
name given by the value of the ID column.

\item{\textbf{byid}}  The image with given IMG\_TYPE and ID is the image  of interest.
The ID parameter refers to IPP image database key that corresponds to the IMG\_TYPE. 
e.g: either exp\_id, chip\_id, warp\_id, diff\_id, or stack\_id.

\item{\textbf{bydiff}}  The image with given IMG\_TYPE corresponding to the difference image
whose diff\_skyfile\_id is equal to ID. This type is useful for requesting stamps based on difference detections.

\item{\textbf{byskycell}}  Images are selected from the skycell with skycell\_id equal to the
value for COMPONENT from the tessellation identified by TESS\_ID. This request type may be
used with IMG\_TYPE warp, stack, and diff.
The set of images may be limited by filter and date as described in Section \ref{ImageCuts}.


\end{itemize}

Note that at this time the REQ\_TYPE byexp is not implemented for IMG\_TYPE stack.

COMPONENT must also be supplied for request specifications where all of the following are true

\begin{itemize}

\item the ROI center is specified in pixel coordinates

\item JOB\_TYPE=stamp

\item REQ\_TYPE=byexp or REQ\_TYPE=byid 

\end{itemize}

The special value 'all' may be used to select all component images but this is not recommended.

\subsubsection{Image Selection Cuts}\label{ImageCuts}

A request specification may refer to many images. The scope of the search can be
limited to certain dates with the parameters MJD\_MIN and MJD\_MAX. The specified range is inclusive.
Note: these cuts have no effect for stack stage requests.

DATA\_GROUP restricts the search to runs with the given value. Note that the SQL database lookups
are done with a "LIKE" comparison so SQL wild cards may be used.

The REQFILT parameter may be used to restrict the search for images taken with a given filter.
If the filter is a single character (g, r, i, z, y) the server appends a '\%' character to the
sql query.


\subsection{The Region of Interest}

Postage stamps are extracted from a rectangular region on the input image. The region is defined by
a center,  width, and height.

The center is defined by the columns CENTER\_X and CENTER\_Y and the dimensions of the rectangle are
the values of WIDTH and HEIGHT. The two least signficant bits in value for the column COORD\_MASK
determine the type for the coordinates

\begin{table}[h]
\begin{center}
\caption{}\label{Rcoordmask}
\begin{tabular}{ll}
\hline
\hline
{\bf COORD\_MASK value} &  {\bf Description } \\
\hline
0                       &   center in RA/DEC; width \& height in arc seconds\\
1                       &   center in pixel coordinates; width and height in arc seconds\\
2                       &   center in RA/DEC; width and height in pixels\\
3                       &   center x/y, width, and height in pixel coordinates\\
\hline
\end{tabular}
\end{center}
\end{table}

Another way to describe this is that COORD\_MASK is two bit mask that bit mask that
determines the format of the ROI values. In the C language:

\begin{itemize}
\item\code{#define PSTAMP_CENTER_IN_PIXELS 1}
\item\code{#define PSTAMP_RANGE_IN_PIXELS  2}
\end{itemize}

If \code{COORD_MASK & PSTAMP_CENTER_IN_PIXELS} is zero, CENTER\_X is the right ascension of
the ROI center in degrees and CENTER\_Y  is its declination in degrees.

If \code{COORD_MASK & PSTAMP_RANGE_IN_PIXELS} is zero, WIDTH and HEIGHT are measured in seconds of arc.

When pixel coordinates are used the coordinate system is that of the image. (For chip level images, before
the stamps are created the cells are mosaicked into a single image.) 

\subsection{Request Options}

For \code{JOB_TYPE} stamp the value of \code{OPTION_MASK} is used to select various other options.
It is a bitwise OR of the following values.

\begin{table}[h]
\begin{center}
\caption{}\label{optionmask}
\begin{tabular}{lll}
\hline
\hline
{\bf C Macro name}      & {\bf Value} &  {\bf Description } \\
\hline
PSTAMP\_SELECT\_IMAGE     &       1  &   create postage stamp of the image pixels \\
PSTAMP\_SELECT\_MASK      &       2  &   create postage stamp of the mask pixels \\
PSTAMP\_SELECT\_VARIANCE  &       4  &   create postage stamp of the variance pixels \\
PSTAMP\_SELECT\_CMF       &       8  &   return the sources file (.cmf) for the images' run \\
PSTAMP\_SELECT\_PSF       &      16  &   return the psf file for the images' run \\
PSTAMP\_SELECT\_BACKMDL   &      32  &   return the background model file for the images chipRun (applies only for chip) \\
PSTAMP\_SELECT\_INVERSE   &    1024  &   make stamps from inverse difference images (applies only for diff) \\
PSTAMP\_SELECT\_UNCONV    &    2048  &   make stamps from unconvolved stack images (applies only for stack) \\
PSTAMP\_RESTORE\_BACKGROUND &  4096  &   (TODO) restore background correction to chip stage images before making stamp \\
PSTAMP\_USE\_IMFILE\_ID   &   16384  &   for byid requests select images using image id instead of stage id \\
PSTAMP\_NO\_WAIT\_FOR\_UPDATE &  32768  &   if images are not available because they have been cleaned don't wait \\
\hline
\end{tabular}
\end{center}
\end{table}

If a bit is set in the OPTION\_MASK that doesn't apply to a particular IMG\_TYPE it is silently ignored.

Note that bit zero is also currently ignored. A stamp of the image pixels is always included in the
results.

For requests with IMG\_TYPE diff or if \code{REQ_TYPE} is bydiff and 
the corresponding difference run is a 'bothways' diff the images are selected based on the inverse difference
images if \code{(OPTION_MASK & PSTAMP_SELECT_INVERSE)} is non-zero.
Otherwise from the images are selected based on the 'positive' images.

For \code{REQ_TYPE} byid normally the \code{ID} value is the "Stage ID" or "Run ID" of the requested image.
If the bit \code{PSTAMP_USE_IMFILE_ID} is included in the \code{OPTION_MASK} for a byid request,
the \code{ID} for the request is the "Image ID" instead of the "Stage ID". 
Table ~\ref{byid_ids} gives the IPP database column that corresponds to these identefiers for the various
stages. The "Image ID" is useful because it may be used to select component of a run. For example
the chip\_imfile\_id values are recorded in the DVO image table so it is easy to create a postage
stamp request from a DVO measurement. XXX: point to an example.

\begin{table}[h]
\begin{center}
\caption{}\label{byid_ids}
\begin{tabular}{lll}
\hline
\hline
{\bf IMG\_TYPE/stage}      & {\bf stage ID} &  {\bf Image ID } \\
\hline
raw & exp\_id & raw\_imfile\_id \\
chip & chip\_id & chip\_imfile\_id \\
warp & warp\_id & warp\_skyfile\_id \\
diff & diff\_id & diff\_skyfile\_id \\
stack & stack\_id & stack\_id \\
\hline
\end{tabular}
\end{center}
\end{table}
Due to limited space on the IPP cluster, images files are deleted some time after they are created. Data in
this state is referred to as "cleaned". By default the postage stamp server will automatically regenerate
the pixels for requested images, if possible. This is referred to as the update process.
If the user does not wish to wait for the updates to complete he should include 
\code{PSTAMP_NO_WAIT_FOR_UPDATE} in the \code{OPTION_MASK}.


\section{Results File}

Each request specification (each row in a request file) results in zero or more postage stamp jobs
to be queued and executed. Each job results in a row in the Postage stamp server results table.


\subsection{Results Table Contents}

The required header keywords for a results table
are given in Table~\ref{ResHeaderDef}.  
The definition for the table's columns is in Table~\ref{ResTableDef}.

\begin{table}[h]
\begin{center}
\caption{Postage Stamp Results File Header Keywords}\label{ResHeaderDef}
\begin{tabular}{lll}
\hline
\hline
{\bf FITS Keyword} & {\bf Datatype} &  {\bf Description } \\
\hline
\code{EXTNAME}  & string     & \code{PS1_PS_RESULTS} \\
\code{EXTVER}   & string     & 1  \\
\code{REQ_NAME} & string     & name of the request the results correspond to ) \\
\code{REQ_ID} & S64     & internal postage stamp id for the request \\
\hline
\end{tabular}
\end{center}
\end{table}

\begin{table}[h]
\begin{center}
\caption{Postage Stamp Results Table Columns}\label{ResTableDef}
\begin{tabular}{lll}
\hline
\hline
{\bf Column Name} & {\bf Datatype} & {\bf Values}       \\
\code{ROWNUM}       & U32       & identifier for this row in table\\
\code{ERROR_CODE}   & U32       & Error code (see text) \\
\code{ERROR_STR}    & U32       & Name corresponding to \code{ERROR_CODE}\\
\code{IMG_NAME}     & string    & Name of image in Data Store file set \\
\code{JOB_ID}       & S64       & Internal postage stamp server job id that produced this image \\
\\
\code{RA_DEG}       & F64       &  RA of stamp center in degrees \\
\code{DEC_DEG}      & F64       &  Declination of stamp center in degrees \\
\\
\code{MJD_OBS}      & F64       &  actual start time of exposure \\
\code{RA_OBS}       & F64       &  J2000 field center RA at start of exposure MJD \\
\code{DEC_OBS}      & F64       &  J2000 field center Dec at start of exposure MJD \\
\code{FILTER}       & string    &  Actual filter retrieved (may differ from REQFILT\\
\code{EXPTIME}      & F64       &  exposure time of parent image \\
\code{FPA_ID}       & string    &  orginal FPA\_ID used at injection \\
\\
\code{PROJECT}  & string    & Values from initial request specification\\
\code{JOB_TYPE} & string    & \\
\\
\code{REQ_TYPE} & string    & \\
\code{IMG_TYPE} & string    & \\
\code{TESS_ID} & string    & \\
\code{COMPONENT} & string    & \\
\\
\code{COORD_MASK} &  U32    & \\
\code{CENTER_X} &  F64      & \\
\code{CENTER_Y} &  F64      & \\
\code{WIDTH}    &  F64      & \\
\code{HEIGHT}   &  F64      & \\
\\
\code{REQFILT}  &   string  & \\
\code{MJD_MIN}  &   F64     & \\
\code{MJD_MAX}  &   F64     & \\
\code{DATA_GROUP}  &   string     & \\
\code{COMMENT}  &   string     & \\
\\
\hline
\end{tabular}
\end{center}
\end{table}

Each line in the results file corresponds to a job to create a postage stamp image. ROWNUM gives
the corresponding row in the request file.
Note that if more than one bit is set in the OPTION\_MASK, a single job may generate multiple images
and so there may be multiple rows in the results file with the same value of ROWNUM.

IMG\_NAME give the name of the image in the Data Store file set.

The image file names will be in the form \code{ROWNUM_I.fits}  where I is an integer that ranges from 1
to the number of jobs generated for the row's request
specification.

Mask images will be named \code{ROWNUM_I.mk.fits} and variance images will
have the name \code{ROWNUM_I.mk.fits}

JOB\_ID is the value of the postage stamp server's internal job id. (This is provided primarily as a debugging
aid.)

The column named PROJECT and following columns are copies from the input request specification.

If ERROR\_CODE is non-zero an error occurred when processing the request specification or the job.
The values for ERROR\_CODE are listed in table\ref{Rerrorcodes}.



\begin{table}[h]
\begin{center}
\caption{Postage Stamp Error Codes}\label{Rerrorcodes}
\begin{tabular}{lll}
\hline
\hline
{\bf Name} & {\bf ERROR\_CODE value} &  {\bf Description } \\
\hline
PSTAMP\_SUCCESS           & 0  &    No errors \\
PSTAMP\_SYSTEM\_ERROR     & 10 &    some unspecified system error occurred during processing\\
PSTAMP\_NOT\_IMPLEMENTED  & 11 &    Feature not yet implemented \\
PSTAMP\_UNKNOWN\_ERROR    & 12 &    Unknown error \\
PSTAMP\_DUP\_REQUEST      & 20 &    Request name is a duplicate \\
PSTAMP\_INVALID\_REQUEST  & 21 &    Error in a request specification. See parse\_error.txt \\
PSTAMP\_UNKNOWN\_PRODUCT  & 22 &    Unknown product in request specification \\
PSTAMP\_NO\_IMAGE\_MATCH  & 23 &    No images matched the request \\
PSTAMP\_NOT\_DESTREAKED   & 24 &    Image matched, but not yet de-streaked \\
PSTAMP\_NOT\_AVAILABLE    & 25 &    Image not available (temporary) \\
PSTAMP\_GONE              & 26 &    Image is no longer available \\
PSTAMP\_NO\_JOBS\_QUEUED  & 27 &    Request specification yielded no jobs. See parse\_error.txt \\
PSTAMP\_NO\_OVERLAP       & 28 &    Center coordinates are not contained in any image of interest \\

\hline
\end{tabular}
\end{center}
\end{table}

If parsing of the request specification file results in error output, the contents will be placed
in the file 'parse\_error.txt' in the resulting file set.

\end{document}

