%%% $Id: PSDC-940-005.tex,v 1.7 2006/04/04 18:05:09 denneau Exp $
\documentclass[panstarrs,spec]{panstarrs}
\RequirePackage{aasbib}

\newcommand\arcdeg{\mbox{$^\circ$}}
\newcommand\degr{\arcdeg}
\newcommand\ps{Pan-STARRS\ }
\newcommand\SDD{PSDC-530-003}
\newcommand\psmopsdc{\texttt{PS::MOPS::DC}}

% basic document variables
\title{The Future of MOPS}
\subtitle{Finding Moving Objects with PS1 and PS2}
%\shorttitle{MOPS LSD Design}
\author{Larry Denneau, Jr.}
\audience{Pan-STARRS PMO}
\group{Pan-STARRS IPP \& MOPS}
\project{Pan-STARRS PS-1}
\organization{Institute for Astronomy, University of Hawaii}
\version{01}
\docnumber{PSDC-5XX-YYYY}

\begin{document}
\maketitle
\sloppy

\RevisionsStart
% version  Date            Description
00 & 2008 Feb 02 & Initial \\ \hline
\RevisionsEnd

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\DocumentsInternal
PSDC-230-001  &   PS-1 Design Reference Mission \\ \hline
PSDC-430-011  &   Pan-STARRS PS-1 IPP System/Subsystem Design Description \\ \hline
PSDC-510-001  &   Pan-STARRS PS-1 MOPS System Concept Definition \\ \hline
PSDC-530-003  &   Pan-STARRS PS-1 MOPS Software Design Description \\ \hline
PSDC-XXX-YYY  &   MOPS CDR Document \\ \hline
\DocumentsExternal
\DocumentsEnd

\tableofcontents
\pagebreak 
\pagenumbering{arabic}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Scope}

\subsection{Identification}

This document describes a design for attribution and precovery of low-significance
detections in the MOPS dataset.

\subsection{System Overview}

The \ps Moving Object Processing System (MOPS) is the
Pan-STARRS subsystem dedicated to discovering asteroids in Pan-STARRS
detection catalogs.

\begin{tabular}{ll}
Project sponsor:&	AFRL, United States Air Force \\
Acquirer:       &	University of Hawaii Institute for Astronomy \\
User: 		&	Astronomical community \\
Developer:      &	University of Hawaii Institute for Astronomy, participating \\
                &       institutions, and associated subcontractors	
\end{tabular}

\subsection{Document Overview}

Attribution and precovery are processes by which \emph{tracklets} consisting of single-occurrence transient (SOT) detections separated by a transient-time interval (TTI) are associated with known \emph{derived objects} by MOPS.

Within MOPS, there is a processing distinction between high-significance (HS) and low-significance (LS) transients due to the substantial relative volume of LS transients compared to HS transients (typically 1000X). Since the LS transient data collection (DC) by definition consist primarily of false detections, MOPS stores and searches the LS DC using different storage mechanisms and algorithms.  Low-significance attribution and precovery comprise the extensions to standard MOPS attribution and precovery to search the LS data collection during the precovery and attribution process.

The actual definition of high-significance and low-significance will be defined by operational parameters from the live PS1 IPP data stream, but for simulations and design this distinction has been drawn at 5$sigma$, with a lower bound for LS SOTs at 3$sigma$. At these cutoffs, we expect on the order of $10^3$ HS SOTs per PS1 field, and $10^6$ LSSOTs per 
PS1 field, most of which will be false detections.

Within this document, the phrase ``precovery'' will be used when referring the general attribution and precovery processes. These processes are identical except for the directions in time for the search (attribution searches forward, precovery backward), and attribution only operates one night at a time, whereas precovery searches from the current night backward through all processed data.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\DocumentsInternalSection
PSDC-230-001  &   PS-1 Design Reference Mission \\
PSDC-230-002  &   PS-1 System Concept Definition \\
PSDC-530-004  &   Data Exchange Standard for Solar System Object Detections and Orbits, Version 2.02 \\ \hline
\DocumentsExternalSection
POSIX Standard & Open Group Based Specifications Issue 6, IEEE Std 1003.1, 2003 \\
FITS Standard & Definition of the Flexible Image Transport System (FITS),
March 29, 1999, NOST 100-2.0 \\
\DocumentsEnd

\section{MOPS Processing Overview}

MOPS locates asteroids by performing intra-night linking of transient
detections into \emph{tracklets}. Then MOPS forms inter-night
linkages called \emph{tracks} across multiple nights within a lunation.
Orbits are then computed for all tracks found by MOPS, and tracks meeting
various quality criteria become MOPS \emph{derived objects}.

Prior to nightly linking of tracklets into tracks, MOPS attempts to associate
the night's tracklets with existing derived objects, a procedure called
\emph{attribution}.

Following the creation of all derived objects for a night, the MOPS 
pipeline then starts a process called \emph{end-of-night} (EON), where
a queue of all modified derived objects (either from attribution or derived
object creation) are then processed against the entire pool of existing derived
objects and unattributed tracklets. The first stage of EON processing is called \emph{orbit identification},
in which EON queue objects are compared in orbital element space against
the current population of derived objects. If any close matches are
found a particular orbit, an orbit is fit to the combination of
detetctions for both derived objects, and if the orbit meets necessary criteria,
the objects are merged.

Next in EON processing is \emph{precovery}, a procedure in which all EON queue derived objects
have their positions predicted for all fields in the past so that
the fields can be searched for unattributed tracklets that belong to these derived objects. The nominal precovery design searches only
detections already in the MOPS database, which are always high-significance (HS) detections.
The low-significance (LS) precovery process, discussed in this document, incorporates an
additional search into offline detection data archive in flat files instead of a database.
If tracklets are found in the LS data that are compatible with existing
derived objects, then the LS precovery process provides an additional
mechanism, called \emph{tracklet promotion}, that places the LS detections in
the MOPS database.

Due to the volume of LS SOT data (around 1000 times the HS SOT data), LS
detections are not stored in the MOPS database. Instead they are archived
into indexed binary files so that they may be searched rapidly.

Clearly the precovery search scales linearly by number of nights into the PS1 survey, and the number of derived
objects found by MOPS also grows linearly, so a naive precovery implementation
scales with $n^2$. This quadratic scaling, combined with the 1000X overhead 
of LS SOT data, make the LS precovery implementation challenging.

\section{MOPS Terminology}

A MOPS \emph{observing cycle} number (OC) is a lunation-like interval beginning on a full
moon. The term observing cycle was chosen to be distinct from a lunation, which is normally numbered starting with a new moon.
OC 1 begins at 0UT with the first full moon on 21 January 2000.

A \emph{night number} is the highest integer less than the MJD (or floor() of the local MJD)
at the local noon. By this definition, all detections for a single night, or between successive
noons, have the same night number.  Note that in Hawaii, the night number is usually
one less than the MJD observations occur on, since the MJD rolls over at 1400 HST.

A \emph{target field} is the last field in time used to create a tracklet or track. In other
words, it is the field in which the tracklet or track comes into existence.  All tracklets
and tracks have a single target field.

A \emph{free tracklet} or \emph{unattributed tracklet} is a tracklet in the MOPS database that has not been assigned to a derived object.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{MOPS Synthetic Objects}

MOPS incorporates an 11-million population of orbits called the MOPS
solar system model (SSM) that are injected into the processing stream as though
they were observed by the PS1 telescope.  The consituents of this 
population are various sub-populations: near-Earth objects (NEOs), main-belt objects, 
Kuiper belt objects, and so on.  There is also a small population
of "extreme objects", e.g. hyperbolic objects, long-period comets,
that are present to test MOPS's capability of finding and producing
orbits for these objects. The SSM is desribed in greater detail
in document XXX.

In the low-significance processing regime, not all detections (in fact most of them) do not actually reside in the MOPS database and thus do not have detection IDs.  To provide accurate accounting of all combinations of synthetic, nonsynthetic, low-significance and high-significance detections, detections are identified uniquely using a combination of a MOPS database detection ID and an enumerated detection number within the detection's field. Table \ref{t.StorageMatrix} shows the storage and labeling matrix for detections within MOPS under the low-significance processing regime.

\begin{deluxetable}{lll}
\tabletypesize{\scriptsize}
\tablecaption{Identification and storage of detections with MOPS\label{t.StorageMatrix}}
\tablewidth{0pt}
\tablehead{
  \colhead{Detection Type} & 
  \colhead{Identifier} & 
  \colhead{Physical Location}
}
\startdata
 Synthetic HS & Database detection ID & LS archive and MOPS database \\
 Synthetic LS & Database detection ID & LS archive and MOPS database \\
 Nonsynthetic HS & Enumerated detection number and database detection ID & LS archive and MOPS database \\
 Nonsynthetic LS & Enumerated detection number only & LS archive only \\
 Upgraded nonsynthetic LS & Database detection ID and enumerated detection number & LS archive and MOPS database \\
\enddata
\end{deluxetable}



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Precovery Overview}

Precovery and attribution are processes in which positions for MOPS derived objects are predicted
for a night, and tracklets near these predictions are tested for compatibility via an
orbit determination.  For a given derived object, if a tracklet near the prediction is deemed compatible, then the tracklet
is permanently associated with the derived object and made unavailable for any other associations.
Figure \ref{f.hsprec1} shows a block diagram of precovery processing on the MOPS master node. Figure \ref{f.hsprec2} details processing on a client node.

In ``standard'' precovery, the MOPS pipeline obtains from the EON queue a list of derived objects to be precovered.  For each night to be precovered, positions of these derived objects are then computed at standard time, e.g. the local 6 p.m., midnight, and 6 a.m. Then the FieldProximity program produces field-object associations to jump-start the search for candidate tracklets within target fields. The groups of derived objects and their candidate fields (as determined by FieldProximity) is then split by derived object into equal-sized chunks to be processed across the MOPS cluster. Each cluster node then searches all candidate fields for each of the node's derived objects and performs an orbit determination for all plausible tracklets. The set of tracklets deemed plausible for a derived object is called the derived object's \emph{candidate tracklets}. Orbits that satisfy a quality control are then returned back to the master node, where results for all derived objects are combined, conflicts resolved, and updates performed on the MOPS database.

Details regarding low-level precovery operations, such as selection of a search radius for a given derived object, or arbitration rules when two nodes produce candidate precoveries for the same tracklet, are beyond the scope of this document and can be found in document XXX precovery document.

\section{LS Precovery Overview}

In the low-significance regime, MOPS needs to search not only its database of free tracklets but also a relatively large offline
dataset of LS nonsynthetic detections and perform various accounting to preserve
consistency of both the nonsynthetic LS archive and the MOPS DB.  The key difficulty is that only detections, not tracklets, are stored in the LS archive. So the LS precovery module must first perform localized detection searches near derived object predictions, then perform FindTracklets on the set of available detections, then evaluate these tracklets against available precoverable objects. The tracklets during this stage are provisional, in that they exist only within the precovery processing code on the cluster node and not in the database. Thus a temporary mapping must exist that allows tracklets to be returned to the master node with all the necessary information to ``upgrade'' the tracklet if necessary. Furthermore, the LS precovery code must know which low-significance detections have been used previously so that they can be omitted in future LS precovery processing.

\subsection{LS Precovery Detail}

The LS precovery code can be considered an ``overlay'' onto the standard high-significance precovery code, with additional functionality at a minimal number of code points to inject additional precovery data from the LSD archive.  This additional functionality is activated by a single configuration item for a MOPS instance. Thus, in environments where there is no distinction between high- and low-significance processing, MOPS can operate with no additional overhead.

Each precovery job performs the following operations on its client node:

\begin{itemize}
\item Search the LS archive for detections for all of the job's derived objects.

\item Strip LS detections reported by the DB as attributed from the result detection lists so that
they are not re-used in LS precovery tracklet search. LS detections that were upgraded
from a previous successful precovery operation will reside in the DB as well, and
these detections need to be reported back to the LS precovery system.

\item Integrate the night's set of synthetic HS and LS detections into detection
lists.

\item Identify tracklets created by LS precovery that may already exist in the MOPS DB. Re-use
these tracklets instead of creating new ones.

\item Upgrade compatible LS detections from compatible tracklets into the MOPS DB
and create DB tracklet objects for them.

\end{itemize}

Figure \ref{f.hclcflowchart} shows a modified precovery flowchart indicating additional
processing steps (in grey) required to search the LS data. Figure \ref{f.lsddetail} shows a high-level of MOPS processing, indicating the data paths of LS detections in the pipeline.

\begin{figure}
\includegraphics{flowchart.pdf}
\caption{\label{f.hclcflowchart} MOPS low-significance (LS) processing overview.}
\end{figure}



\subsection{LSD Ingest and Merge}
During field and detection ingest, MOPS must identify nonsynthetic detections (NSDs) in the data stream and archive these NSDs in its LSD archive for further precovery processing.  Additionally, MOPS must integrate synthetic detections into the LSD archive so that all detections are available for LSD processing (MOPS generates synthetic low-significance detections as well, and these are precovered using the general LSD processing mechanism).

After the ingest of a single field, MOPS does not know if it has ingested all fields, so the detections in the field are archived temporarily in a single file per field. After all fields for a night have been ingested, the fields are \emph{merged} into a single, large binary file, and indexes are created so that detections can be located in the file by field ID and right ascension.


\subsection{LSD Archive File Formats}

The LS archive is a read-only set of packed binary data files and free-formatted text index files.
The PS1 IPP will provide MOPS FITS files containing transient detections.  MOPS will
strip the HS detections from these FITS files and insert them into the MOPS DB.  The
original FITS files will then be reduced to a minimal binary format and archived for LS precovery.  At a LS threshold of 3$sigma$, there
will be approximately $10^6$ detections per PS1 field; a source FITS file containing this
number of LS detections is about 100MB. Thus some form of indexing is
required so that entire fields do not need to be scanned repeatedly when searching for detections.



Fields will be identified in the archive by a tag concatenating their observing cycle (OC) number,
MOPS night number, and UT time of observation to five decimal places.  Inside a directory
identified using this method will be files for each RA strip.

\begin{deluxetable}{lcl}
\tabletypesize{\scriptsize}
\tablecaption{\label{t.lcdetectiondata} Detection data stored in the LS archive.}
\tablewidth{0pt}
\tablehead{
  \colhead{Input} & 
  \colhead{Format} & 
  \colhead{Description}
}
\startdata
 DETECTION\_ID & char(20) & IPP or MOPS-assigned detection ID \\
 FILTER & char(4) & grizy filter \\
 EPOCH\_MJD & float(8) & Epoch of observation \\
 RA\_DEG & float(8) & Right ascension, degrees \\
 DEC\_DEG & float(8) & Declination, degrees \\
 RA\_SIGMA\_DEG & float(8) & RA error, degrees \\
 DEC\_SIGMA\_DEG & float(8) & DEC error, degrees \\
 MAG & float(8) & Magnitude \\
 MAG\_SIGMA & float(8) & Magnitude error \\
 S2N & float(8) & Signal-to-noise \\
\enddata
\end{deluxetable}


\subsection{LSD Indexes}


\subsection{LSD Fetch Detections}

The LS precovery system will need to process tens of thousands of precoveries
for each possible night precoveries may be available.  This process requires
the searching of the LS archive for detections near the predicted positions
of precoverable objects.  Each precovery worker pipeline instance is responsible
for completely processing a subset of field-object associations and reporting the
successful precoveries back to the precovery master. The inputs passed to the worker pipeline
are shown in \ref{t.lcfetchinputs}.

\begin{deluxetable}{ll}
\tabletypesize{\scriptsize}
\tablecaption{\label{t.lcfetchinputs} Per-object inputs to LS detection search.}
\tablewidth{0pt}
\tablehead{
  \colhead{Input} & 
  \colhead{Description}
}
\startdata
 Object identifier & MOPS derived object name \\
 Ephemeris & Predicted position and velocity with uncertainties \\
 Field IDs & IDs of fields where object may be found, reported by FieldProximity \\
\enddata
\end{deluxetable}

\begin{itemize}
\item Compute the precise predicted position for the object within the near fields reported by FieldProximity.
\item Search a region bounded by the ephemeris uncertainty for detections and store the detections.
\end{itemize}


\subsection{HS/LS Filter and Merge}

After detections are retrieved from the LS archive for an object, the detections need
to be compared against the used-detection list provided as an input.  After building
detection lists for precovery candidates, but before finding tracklets, the precovery
worker must fetch from the MOPS DB all detections in the searched fields and remove
those detections that are already associated with some derived object in MOPS.
Additionally, the filter and merge operation must integrate the synthetic detections
into the detection lists since synthetics are never stored in the LS archive.

\subsection{Low-Significance FindTracklets}

The resulting set of detections, which consists of unattributed synthetic and nonsynthetic
HS and LS detections, can then be searched for tracklets. Each individual object to be
precovered by the worker pipeline will have all detections from its field stack assembled 
into input to FindTracklets.  The allowed sky-plane velocty will be restricted to reduce
the tracklet search. Afterward, the output tracklet set will be further reduced to match
the predicted direction of the object. These reductions allow for the smallest set of 
candidate orbits that need to be computed.

After all tracklets have been found for all objects and precoveries found, the resultant
orbits and tracklets are packaged and sent back to the precovery master, where some
validation is performed against detection re-use and surviving LS detections are
upgraded into the MOPS DB.

\subsection{LS Upgrade}

The LS upgrade procedure examines successfully precovered tacklets from the worker
pool and creates detections and tracklets as necessary before marking tracklets
as associate to derived objects.  If a tracklet is found to consist entirely of detections 
already in the MOPS DB, then the existing tracklet ID is fetched and marked. In
all other cases, each detection is created, then a tracklet consisting of the
detections is created, and the new tracklet marked.

\section{Summary}

Figure \ref{f.lsworkerdetail} shows a detailed flowchart of processing
within a precovery worker pipeline.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%% END MATTER
\pagebreak

\section{Appendix}
\subsection{Acronyms}

\begin{table}[h]
\begin{center}
\small
\caption{Acronyms used in this document\label{table-acronyms}}
\begin{tabular}{ll}
\hline
\hline
{\bf Acronym}            & {\bf Definition} \\
\hline
 ADD & Algorithm Design Description \\
 AFRL & Air Force Research Laboratory \\
 DC  & Data Collection \\
 DTCTL & Detections and Tracklets Controller \\
 GUI & Graphical User Interface \\
 HS & High-significance \\
 HSSOTDC & High-Significance Single-Occurrence Transient DC \\
 HTML & Hypertext Markup Language \\
 HTTP & Hypertext Transfer Protocol \\
 IfA & Institute for Astronomy \\
 IOD & Initial Orbit Determination \\
 IPP & Image Processing Pipeline \\
 LS & Low-significance \\
 LSSOTDC & Low-Significance Single-Occurrence Transient DC \\
 LODCTL & Linking and Orbit Determination Controller \\
 MOPS & Moving Object Processing System \\
 MOID & Minimum Orbit Intersection Distance \\
 OD  & (Differential) Orbit Determination \\
 PS1 & Pan-STARRS 1 proto-type telescope \\
 PS4 & Pan-STARRS 4 full system \\
 $S/N$ & Signal-to-Noise \\
 SCD & System Concept Definition \\
 SDD & Software Design Description \\
 SOT & Single Occurence Transient \\
 SRS & Software Requirements Specification \\
 SS  & Solar System \\
 SSM  & Solar System Model \\
 TBD & To Be Determined \\
 TBR & To Be Reviewed \\
 TTI & Transient Time Interval \\
 UH & University of Hawaii \\
\hline
\end{tabular}
\end{center}
\end{table}

\end{document}

