%\usepackage{geometry}

\documentclass[panstarrs]{panstarrs}%
\usepackage[onehalfspacing]{setspace}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{graphicx}%
\setcounter{MaxMatrixCols}{30}
%TCIDATA{OutputFilter=latex2.dll}
%TCIDATA{Version=5.50.0.2890}
%TCIDATA{CSTFile=40 LaTeX article.cst}
%TCIDATA{Created=Sunday, May 14, 2006 18:51:53}
%TCIDATA{LastRevised=Friday, October 19, 2007 02:29:46}
%TCIDATA{<META NAME="GraphicsSave" CONTENT="32">}
%TCIDATA{<META NAME="SaveForMode" CONTENT="1">}
%TCIDATA{BibliographyScheme=Manual}
%TCIDATA{<META NAME="DocumentShell" CONTENT="Standard LaTeX\Blank - Standard LaTeX Article">}
%TCIDATA{Language=American English}
%BeginMSIPreambleData
\providecommand{\U}[1]{\protect\rule{.1in}{.1in}}
%EndMSIPreambleData
\newtheorem{theorem}{Theorem}
\newtheorem{acknowledgement}[theorem]{Acknowledgement}
\newtheorem{algorithm}[theorem]{Algorithm}
\newtheorem{axiom}[theorem]{Axiom}
\newtheorem{case}[theorem]{Case}
\newtheorem{claim}[theorem]{Claim}
\newtheorem{conclusion}[theorem]{Conclusion}
\newtheorem{condition}[theorem]{Condition}
\newtheorem{conjecture}[theorem]{Conjecture}
\newtheorem{corollary}[theorem]{Corollary}
\newtheorem{criterion}[theorem]{Criterion}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{example}[theorem]{Example}
\newtheorem{exercise}[theorem]{Exercise}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{notation}[theorem]{Notation}
\newtheorem{problem}[theorem]{Problem}
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{remark}[theorem]{Remark}
\newtheorem{solution}[theorem]{Solution}
\newtheorem{summary}[theorem]{Summary}
\newenvironment{proof}[1][Proof]{\noindent\textbf{#1.} }{\ \rule{0.5em}{0.5em}}
\shorttitle{MPC Export}
\begin{document}

\title{PS1-MOPS To MPC Data Export}
\author{Daniel Chang}
\maketitle

\pagenumbering{arabic}

\section{Overview}

PS1-MOPS Derived Objects (DOs) and associated data will be exported to the MPC
according to the criteria specified in the Pan-STARRS Data Release Policy
(PSDC-530-005). MOPS software objects that are referenced during export
include DerivedObjects, Orbits, Tracklets, and Detections. Data will be
reported in the PS1-MOPS DES (Data Exchange Standard) format (PSDC-530-004).
PS1-MOPS to MPC export will consist of two sequential processes. In the first
process, exportable data is determined. Following export determination, data
is exported to the MPC. Both of these processes will run for the lifetime of
PS1-MOPS on a periodic schedule.

The first process determines exportable DOs. The DO is the primary reference
object for the export process. On a daily basis, the MOPS database will be
scanned to determine DOs meeting release criteria. Data associated with DOs
include detections, tracklets and orbits. For each DO satisfying release
parameters, a revision snapshot will be recorded in the database table
\textbf{mpcexport.} This record will include the DO's ID. The detections for
that DO will be recorded in table \textbf{mpcexport\_dets.} The parent
revision record will be given an export status of pending.

The second process exports the pending export records to DES format for
retrieval by the MPC. For each pending revision record in mpcexport, the
related data including an orbit along with its reportable detections, will be
extracted and saved in DES format. These files will then be placed in a
PS1-MOPS DataStore where they may be retrieved by the MPC via an HTTP-protocol
interface. The revision record export status will then be marked as reported.

These processes are tentatively designed to be invoked on a daily basis. They
will synchronize the export records with all of the exportable data in the
MOPS database and export the data for retrieval by the MPC. They serve to
maintain the current set of exportable data in the database along with
providing data necessary for the MPC\ to maintain synchrony with the MOPS data set.

\section{Export determination and maintenance}

The determination of the exportability of a DO is based on whether the data
for that object satisfies export criteria. This is determined by a software
implementation of the PS Data\ Release Policy. The set of export data requires
maintenance as detections will be added and removed from DOs. By creating
revision records for every set of detections to be exported, the set of
exportable detections may be synchronized with the detections in the MOPS
database. New detections that are added to an object are added to the export
set. Detections that are removed from the database will be flagged for removal.

\section{Data export}

Data will be exported to the MPC\ in DES\ format via a DataStore interface.
Export files will consist of text files formatted according to the guidelines
specified in the Data Exchange Standard. These files will be placed on a
server implementing a DataStore interface. The individual files will be
bundled using standard archive utilities. Data verification will be provided
by the particular archive tool that is used. The MPC will be able to retrieve
the archived files through the DataStore. The DataStore connection is able to
be encrypted. The server hosting the DataStore may require an SSH tunnel to be
accessed. It is intended that all data transfer will occur via an encrypted connection.

\section{Implementation}

\subsection{Determining exportable data}

The criteria for exportable data are defined in the data release policy and
include orbit quality, number of observations, number of lunations, number of
oppositions and object class. The release schedule for objects varies
according to the object's class. Each DO will be analyzed by calling library
routines that implement the data release criteria. For DOs that satisfy
release criteria, their associated detections are considered exportable. These
exportable detections are compared with the corresponding set of export
detections, if it exists. This allows for the maintenance of a current set of
exportable detections.

\subsection{Maintaining export data}

In general, the database records for MPC export serve as a revision tracking
system. Their purpose is to provide a means of incrementally synchronizing the
set of exported data with the MPC as changes occur over time. This is
accomplished by maintaining a history of revisions necessary to produce a set
of exportable data for any particular export time. In the initial state of MPC
export, no exportable data will be recorded. Every existing DO\ may then be
analyzed for release. For suitable DOs, their detections will be added to
table mpcexport\_dets. During subsequent analyses of DOs, detections may or
may not change for a given DO.

The algorithm for MPC export data maintenance, illustrated in Figure
\ref{flowchart}, is for each DO satisfying release criteria, if the DO\ is not
in table mpcexport or has updated data, then it is recorded in the table in a
revision record. The set of exportable detections is retrieved for the DO. The
detections are compared with the existing set of detections in
mpcexport\_dets. If no detections are present, then all of the exportable
detections are added to mpcexport\_dets. If detections exist for the DO\ in
mpcexport\_dets, then the exportable detections will be compared with the
pre-existing export detections. If a detection is new, it will be added to the
export set. If a detection exists in the export set, but is not present in the
current set of exportable detections, it will be marked for removal.%

%TCIMACRO{\FRAME{ftbpFU}{7.6287in}{7.9088in}{0pt}{\Qcb{Flowchart for
%maintaining export data. The paths shown are for affirmative evaluations in
%the decision diamonds. From table derivedobjects, each DO is analyzed. If the
%DO satisfies release criteria, then the set of MOPS detections and the set of
%export detections are retrieved. These sets are compared. If differences
%exist, a revision record is added to table mpcexport. If the detection is not
%in mpcexport\_dets, it is considered new and added to the table. If there is a
%mpcexport\_det detection that is not in the MOPS detection set, then that
%detection will be marked for removal.}}{\Qlb{flowchart}}{Figure}%
%{\special{ language "Scientific Word";  type "GRAPHIC";
%maintain-aspect-ratio TRUE;  display "USEDEF";  valid_file "T";
%width 7.6287in;  height 7.9088in;  depth 0pt;  original-width 7.5112in;
%original-height 7.7879in;  cropleft "0";  croptop "1";  cropright "1";
%cropbottom "0";  tempfilename 'JOT9G902.wmf';tempfile-properties "XPR";}}}%
%BeginExpansion
\begin{figure}
[ptb]
\begin{center}
\includegraphics[
natheight=7.787900in,
natwidth=7.511200in,
height=7.9088in,
width=7.6287in
]%
{JOT9G902.wmf}%
\caption{Flowchart for maintaining export data. The paths shown are for
affirmative evaluations in the decision diamonds. From table derivedobjects,
each DO is analyzed. If the DO satisfies release criteria, then the set of
MOPS detections and the set of export detections are retrieved. These sets are
compared. If differences exist, a revision record is added to table mpcexport.
If the detection is not in mpcexport\_dets, it is considered new and added to
the table. If there is a mpcexport\_det detection that is not in the MOPS
detection set, then that detection will be marked for removal.}%
\label{flowchart}%
\end{center}
\end{figure}
%EndExpansion


During the analysis of each DO, it is necessary to retrieve the full export
set for comparison. The revision recording algorithm as described provides for
data storage savings at the expense of computation time during the rebuilding
of export sets. Each export set, during set comparison, requires its
re-creation from the revision records stored in the database. This amounts to
performing the recorded additions and removals from the export data set. At
the end of an MPC\ export invocation, the export detection set and the
exportable MOPS detection set should by synchronized.

\subsection{Handling updates to DOs}

As new tracklets become available for existing DOs, their associated orbits
will be recalculated. These updates are managed through the end-of-night (EON)
processing. DOs that have updated data are placed in the EON queue. At this
point, the MPC export process will be invoked to update the export records.
For each DO in the EON queue, the existing detections will be compared with
the export detections. Any changes to the detections will be handled by the
maintenance protocol previously described.

\subsection{Data verification and redundancy}

By design, synchronization between the MOPS detections and the export
detections is maintained after every MPC export process invocation. The export
data may be verified by comparing it to the MOPS data. For the purpose of
redundancy, a separate log file will be maintained that will record every
action performed on the export data tables. This will allow for the
verification of the data should the database become corrupted. It should also
allow for the rebuilding of the export data set.

\subsection{Exporting data}

The data associated with DOs to be exported will be formatted in the PS1
DES\ format. The current calculated orbit for a particular DO, at the time of
the preparation of this data, will be exported along with the set of
detections previously recorded in mpcexport\_dets. The export data will also
include corrections to previously submitted data in the form of detections
flagged for removal by the MPC. It is the intended that all data transfers
will occur over secure connections.

\subsubsection{DataStore access}

During development of the export process, data will be copied to a DataStore
that may be accessed via HTTP. Data export will likely occur at 24-hour
intervals. After a certain period of time, the files will expire and be
removed from the DataStore. The MPC may configure their retrieval schedule
according to the available data window.

\subsubsection{File format}

Data will be exported in ASCII\ text files with space-delimited data columns.
Each file will contain one type of data with one data item per line. Column
format templates are given below. A brief description of the column and its
units, if applicable, is listed after the format template. The formats being
used are intermediate formats that roughly implement the DES. They are subject
to change with further development.

\paragraph{Detection column template}%

\[
\text{\texttt{do\_id det\_id epoch ra dec mag filter obscode raSigma decSigma
magSigma s2n removal}}%
\]


\begin{description}
\item[do\_id] MOPS unique Derived Object ID. Not part of the DES.

\item[det\_id] MOPS unique Detection ID. Not part of the DES.

\item[epoch] Epoch (MJD).

\item[ra] Right ascension (deg).

\item[dec] Declination (deg).

\item[mag] Apparent magnitude.

\item[filter] Filter type.

\item[obscode] MPC observatory code.

\item[raSigma] Error in right ascension.

\item[decSigma] Error in declination.

\item[magSigma] Error in magnitude.

\item[s2n] Signal-to-noise ratio.

\item[removal] Detection no longer associated with the DO\ in the MOPS\ database.
\end{description}

\paragraph{Orbit column template}%

\[
\text{\texttt{do\_id orbit\_id orbit\_type q e i node argPeri timePeri hV
epoch}}%
\]


\begin{description}
\item[do\_id] MOPS unique Derived Object ID. Not part of the DES.

\item[orbit\_id] MOPS unique Orbit ID. Not part of the DES.

\item[orbit\_type] Value of COM. Orbit data will be reported in cometary
format as described in the DES.

\item[q] Perihelion (AU).

\item[e] Eccentricity.

\item[i] Inclination (deg).

\item[node] Longitude of node (deg).

\item[argPeri] Argument of perihelion (AU).

\item[timePeri] Time of perihelion (MJD).

\item[hV] Absolute magnitude.

\item[epoch] Epoch (MJD).
\end{description}

\subsubsection{File naming scheme}

Exported files will be named according to the following template,%
\[
\text{\texttt{yyyymmdd\_hhmmss-\{data\ type\}-nnn.txt}}%
\]
where yyyy is the four-digit year, mm is the month number, dd is the day
number, hh is the hour, mm is the minute, ss is the second, data type is
either `detections' or `orbits' and nnn is the file number for this particular
sequence. All numerical values will be zero-padded. File numbers begin with
001 and advance sequentially. Multiple file numbers exist so that large files
may be split into smaller files circumventing file system limitations. The
time indicated in the filename corresponds to the time of the last pending
revision record in the data set being exported. Current split limits are
250,000 lines per detection file and 250,000 lines per orbit file.

\subsection{Program files}

MPC export is implemented as a Perl module named PS::MOPS::DC::MPCExport and a
Perl script named mpc-export.pl that uses the library module.

\subsection{Database schema}

The MPC export process is accomplished by the addition of two tables to the
PS1-MOPS database. These tables are named mpcexport and mpcexport\_dets. They
respectively contain the DO IDs and detection IDs for objects that will be
exported to the MPC. These tables and their relationships are diagrammed in
Figure \ref{schema}. The table definitions are documented in Tables
\ref{mpcexport_def} and \ref{mpcexport_dets_def}. These tables have foreign
key relationships with tables derivedobjects and detections. Once records are
added they are not intended to be modified or deleted except to update export
status and export time.%

%TCIMACRO{\FRAME{ftbpFU}{7.3672in}{1.2866in}{0pt}{\Qcb{Database schema diagram
%for MPC\ export. The two tables that are added to implement MPC export are
%mpcexport and mpcexport\_dets. The table mpcexport contains derived object IDs
%along with a revision ID to be used for grouping detection sets. Detection IDs
%are found in table mpcexport\_dets. This table includes a column to indicate
%that a detection should be removed by the MPC. References to tables
%derivedobjects and detections are included to note their relationship with the
%export tables.}}{\Qlb{schema}}{Figure}{\special{ language "Scientific Word";
%type "GRAPHIC";  maintain-aspect-ratio TRUE;  display "USEDEF";
%valid_file "T";  width 7.3672in;  height 1.2866in;  depth 0pt;
%original-width 7.2531in;  original-height 1.2432in;  cropleft "0";
%croptop "1";  cropright "1";  cropbottom "0";
%tempfilename 'JOCVX202.wmf';tempfile-properties "XPR";}}}%
%BeginExpansion
\begin{figure}
[ptb]
\begin{center}
\includegraphics[
natheight=1.243200in,
natwidth=7.253100in,
height=1.2866in,
width=7.3672in
]%
{JOCVX202.wmf}%
\caption{Database schema diagram for MPC\ export. The two tables that are
added to implement MPC export are mpcexport and mpcexport\_dets. The table
mpcexport contains derived object IDs along with a revision ID to be used for
grouping detection sets. Detection IDs are found in table mpcexport\_dets.
This table includes a column to indicate that a detection should be removed by
the MPC. References to tables derivedobjects and detections are included to
note their relationship with the export tables.}%
\label{schema}%
\end{center}
\end{figure}
%EndExpansion


\begin{table}[p]
\caption{Table definition for mpcexport. Records in this table serve to group
detection set snapshots or revisions. Each DO has a set of detections
associated with it. The exportable set of detections are stored in
mpcexport\_dets. }%
\label{mpcexport_def}
\begin{center}%
\begin{tabular}
[c]{llll}\hline\hline
Column & Type & Size (bytes) & Description\\\hline
revision\_id & bigint & 8 & Primary key. Used to group detection sets.\\
rev\_time & datetime & 8 & Time of addition of revision record.\\
do\_export\_id & bigint & 8 & Derived object ID. Foreign key from
derivedobjects.\\
export\_time & datetime & 8 & Time of export.\\
export\_status & char & 8 & Indicates state of export. \{P$\implies$pending,
E$\implies$exported\}\\\hline\hline
\end{tabular}
\end{center}
\end{table}

\begin{table}[p]
\caption{Table definition for mpcexport\_dets. This table contains the sets of
detections that are associated with the DO references in mpcexport. The column
named removal indicates that a particular detection is flagged for removal by
the MPC.}%
\label{mpcexport_dets_def}
\begin{center}%
\begin{tabular}
[c]{llll}\hline\hline
Column & Type & Size (bytes) & Description\\\hline
det\_export\_id & bigint & 8 & Detection ID. Foreign key from detections.\\
rev\_id & bigint & 8 & Foreign key from mpcexport.\\
det\_id & bigint & 8 & Foreign key from detections.\\
removal & char & 8 & Indicates that detection should be removed by the
MPC.\\\hline\hline
\end{tabular}
\end{center}
\end{table}

\section{Summary}

PS1-MOPS to MPC export will consist of two sequential processes. Derived
objects will be scanned to determine if they are exportable. For those that
satisfy export requirements, as detailed in the PS1-MOPS Data Release Policy,
a revision record will be written to table mpcexport and they will be marked
as pending for export. Pointers to the detections associated with the DO\ will
be added to mpcexport\_dets. If there are changes in the detections for an
existing exportable DO, then the changes in the detections for that DO\ will
be recorded. Those detections requiring removal by the MPC will be
appropriately annotated. Subsequently, the associated data of the DOs---to
include orbits, tracks, and detections---will be exported to the MPC in the
PS1-MOPS DES format. This data will be made available through a DataStore
interface allowing retrieval by the MPC over an encrypted HTTP connection. The
MPC may develop their own schedule for the processing of the data. The MPC
export process facilitates the timely export of high-quality MOPS data to the
MPC and provides for its synchronization with the MPC over the lifetime of PS1.


\end{document}