\documentclass[panstarrs,spec]{panstarrs}

\title{PanTasks}
\subtitle{the IPP Scheduler and Controller system}
\author{Eugene Magnier}
\audience{IPP}
\shorttitle{PanTasks}
\group{Pan-STARRS IPP}
\project{Pan-STARRS IPP}
\organization{Institute for Astronomy}
\version{DR}
\docnumber{PSDC-xxx-xxx}

\begin{document}
\maketitle

\tableofcontents
\pagebreak 
\pagenumbering{arabic}

\section{Overview}

This document discusses the design of PanTasks.  PanTasks is the IPP
tool which manages the sequencing of data analysis steps and, with the
related tool `PControl', distributes the data processing across a
cluster of computers.

The purpose of PanTasks is to manage the automatic construction and
execution of inter-related (often repetative) operations.  PanTasks
uses a set of rules to define UNIX commands, and their corresponding
command-line arguments, to be performed on some regular, repeated
basis.  The utility of PanTasks is that it can easily define an
analysis system which is completely state-based, as opposed to an
event-driven system.  

The two basic units of PanTasks operation are the 'task' and the
'job'.  A 'job' is simply a command the user would execute on a
command line; it consists of a command along with optional command
line arguments.  A 'task' is a generic description of a type of job in
which the details of any specific piece of data are omitted.  The task
defines the UNIX command which corresponds to the job, and it provides
rules for determining the identity of data for the job.  The task also
defines tests which are used to decide if the job may be executed.
Finally, it defines a polling frequency in which PanTasks should
attempt to construct a new job for the task.

For example, we may want to regularly copy files from one location to
another.  Perhaps the name of the next available file is available in
a database table, and can be retrieved with the command 'nextFile'.
Perhaps we wish to check for new files every 60 seconds.  Thus, the
task is to copy files, while a specific job of this task might be of
the form \code{cp newfile newpath}.  With PanTasks, we would define a
copy task somewhat like the following:

\begin{verbatim}
  task copyfile
    periods -exec 60.0

    task.exec
      $file = `nextFile`
      if ($file == "none")
        break
      end
      command cp $file $newpath
    end

    task.exit 0
      queuepush copied $file
    end

    task.exit 1
      queuepush failure $file
    end
  end
\end{verbatim}

In this simple example, the task is attempted every 60 seconds.  If
there is no new file (output of 'nextFile' is 'none'), then no job
results from this task.  If there is a new file, the copy command is
performed.  This example is deceptively simple because one could
easily imagine writing a stand-along program which performs the same
thing.  The important advantages of using PanTasks for this type of
operation are: 
\begin{itemize}
\item the output from one job can be used to spawn a number of other tasks.
\item the success or failure state of each job can be used to spawn
  other tasks.
\item the details of the job and data test are kept separated from the
  rules which connect tasks together.  
\item the relationships between tasks are kept together in a single
  location.
\end{itemize}

In addition to these organizational advantages, PanTasks also provides
a direct connection to the tool for monitoring parallel jobs,
pcontrol.  Thus the jobs spawned by PanTasks can be defined to run in
the background locally or on any of the computers in the parallel
processing cluster.


\section{PanTasks : the Scheduler}

The purpose of PanTasks is to manage the automatic construction and
execution of inter-related (often repetative) operations. PanTasks uses
a set of rules to define UNIX commands, and their corresponding
command-line arguments, to be performed on some regular, repeated
basis. The utility of PanTasks is that it can easily define an analysis
system which is completely state-based, as opposed to an event-driven
system.

Consider, for example, a telescope which obtains a collection of
images over the course of a night. Every minute or two, it takes an
image and writes the image to some disk. An event-driven analysis
system would involve having the telescope initiate a process at the
end of the exposure. This process would perform an analysis, write
some output, then send trigger another process. This type of operation
works very well for a simple set up with reliable hardware. Such a
system becomes more difficult to maintain when hardware failures occur
or when multiple systems need to interact with each other. When
failures occur, the triggering information (the events) is easily
lost, thus some mechanisms are needed to detect these failures and
either re-send the trigger or send an alternative failure-mode
trigger. Or, if two systems need to interact, one or the other system
must block for results from the first. Stopping and restarting such an
analysis system is very delicate since the appropriate triggers must
be set up some how, eg by noticing which images have not succeeded and
restarting them at the appropriate stage. All of these types of
methods of handling complexity and failures are essentially
state-based rules. PanTasks allows the easy definition of a totally
state-based analysis system.

In a state-based system, some mechanism examines the state of the
system and decides which actions to perform based on the current
state. In the illustration above, the mechanism could examine the
images available (either by examining the disk or by examining the
state of a data table) and decide to perform an operation based on
what images are available. This makes it very easy to handle
complexity and errors. If an analysis fails, the state either is not
successfully updated or the error state is recorded, both situations
being easy to detect and easy to handle. Restarting the system simply
involves starting the state-monitoring mechanism. Combining results
from multiple input sources simply involves watching for the multiple
inputs to be available. PanTasks provides a mechanism to define state
monitors, and to define the actions which are performed when those
states occur. PanTasks action consist of initiating UNIX commands, where
the arguments of those commands may depend on the results of the state
tests.  

\subsection{Tasks vs Jobs}

The primary function of PanTasks is to repeatedly perform tasks, and
execute jobs on the basis of those tasks. A task consists of a set of
rules which describe system state tests to perform on a regular time
scale. Based on the results of those state tests, the task will then
choose whether or not to construct a job. The task also defines
actions to perform upon the completion of a job, based upon the output
and exit status of the job. A task thus defines the repeat period. It
may optionally define valid or invalid time ranges (eg, Mon-Fri or
10:00-17:00, etc). The task may also specify that the job be run
locally (ie, in the background on the same computer as PanTasks) or
remotely by the parallel process controller (pcontrol). A job may even
be restricted to a specific computer managed by pcontrol. An example
of a simple tasks is given below.

\begin{verbatim}
   task datalist
     command ls /data/foo
     periods -exec 5.0
     periods -timeout 50.0
     periods -poll 1.0
 
     task.exit 0
       queueprint stdout
       queuedelete stdout
     end
  
     task.exit 1
       queuepush failure "task failed"
     end
   end
 \end{verbatim}


This task does not perform any system state tests; it is simply
constructs a new job every 5.0 seconds. The job in this case is always
the same: ls /data/foo . When the job finished, if the job exit status
is 0 (normal UNIX success status), the resulting output is printed to
the screen. If the job returns an exit status of 1 (a failure), the
failure queue receives a single entry. Although they are not defined
in this case, it is also possible to specify the action to be taken if
the job crashes (does not exit normally) or if it times out (runs
beyond the specified timeout period). A slightly more complex task
which performs a state test and constructs a command based on that
test is shown below

\begin{verbatim}
   task datalist
     periods -exec 5.0
     periods -timeout 50.0
     periods -poll 1.0
 
     task.exec 
       $file = `next.file`
       if ($file == "none")
         break
       end
       command cp /data/foo/$file /data/bar
     end
 
     task.exit 0
       queueprint stdout
       queuedelete stdout
       queuepush copied $file
     end
  
     task.exit 1
       queuepush failure $file
     end
   end
\end{verbatim}

 The task.exec macro is executed by PanTasks every 5.0 seconds. This
 macro executes a (hypothetical user-defined) UNIX command (next.file)
 which examines the system state, return either a filename or the word
 "none". If the result of this test is "none", the task does nothing:
 no job is constructed. Otherwise, a job is constructed using the name
 of the file returned by the state test. Successful jobs have the
 filename added to the 'copied' queue, while failed jobs add the
 filename to the 'failure' queue.

\subsection{Parallel vs Local Job Processing}

Jobs which are generated by PanTasks tasks may either be run locally
(forked in the background on the same machine as PanTasks) or run on the
IPP parallel process controller, pcontrol. The default is for the job
to be run locally. If a job should be run on the parallel controller,
this can be specified by including the command host (hostname) in the
definition of a task. If the value of (hostname) is 'anyhost', then
pcontrol may select any of its host computers to run the job according
to its own rules. If the value of (hostname) is one of the computers
managed by pcontrol, then that machine will be selected for the job,
if it is available. This amounts to a preference to use that machine,
but pcontrol is allowed to substitute a different machine if it
chooses. If the host command is given the option -required, then
pcontrol is forced to use the named host, even if the machine is down,
unknown, or otherwise unavailable. If the machine is not available,
pcontrol will simply hold onto the job until the machine is available
or the job is deleted. Note that PanTasks may delete jobs from pcontrol
if they remain pending for too long (see period -timeout).

It is possible to interact directly with the parallel processor to
examine the current status, halt the parallel processor, etc. Commands
to the parallel processor are defined under the controller
command. The following controller commands are available:

\begin{itemize}

\item controller host (command) (hostname): Manage the parallel
  controller collection of hosts. This command can be used to add a
  new host, the delete one of the existing hosts, to turn a host on or
  off, and to check the status of a host

  \begin{itemize}
  \item controller host add (hostname): add a new host.

  \item controller host delete (hostname): delete a host.

  \item controller host on (hostname): tell pcontrol that the host is on.

  \item controller host off (hostname): tell pcontrol that the host is off.

  \item controller host retry (hostname): tell pcontrol to retry the host connection.

  \item controller host check (hostname): check the current status of a host. 
  \end{itemize}

  \item controller exit: stop controller execution.

  \item controller status: report controller current status.

  \item controller check: check job or host status.

  \item controller output: print accumulated messages from the controller. 
\end{itemize}

It is also possible to specify a host for a task which has not been
identified to the controller. If such a host is required, the
controller will simply keep the associated jobs in the pending state
until such a machine exists. See the pcontrol documentation for
further discussion of the controller manipuation of jobs and hosts.

\subsection{Task Restrictions}

Tasks may have restrictions on when they create jobs and how
frequently they create jobs. The task command trange is used to
specify a valid or invalid time range for a task. A valid time range
limits the task evaluation to that time period. An invalid time range
excludes task evaluation from the time period. Any number of time
range restrictions may be defined, and the union of all restrictions
will define if a job may be created. By default, the time range is an
inclusive time range: the task is evaluated only if the current time
falls within the specified time range. Alternatively, if the -exclude
flag is given, the time range is exclusive, in which case the task is
not evaluated if the current time falls within this range.

The time range may be given as a range of absolute dates as follows:

\begin{verbatim}
 trange YYYY/MM/DD,HH:MM:SS YYYY/MM/DD,HH:MM:SS 
\end{verbatim}

where the two dates specify the start and end of the time range. In
either of these date representations, the least-significant elements
of the date and time may be dropped, defaulting to 00 (in the case of
hours, minutes, and seconds) or 01 (in the case of day and
months). Rather than specifying an end date, it is also valid to
specify a time interval from the starting date. The time interval is
specified as a number followed by a unit indicated by a single letter:
d (days), h (hours), m (minutes), s (seconds).

The time range may also be specified as a repeated period of time,
either as a time of day or a day and time of week. In the first case,
the time range is specified as follows:

 
\begin{verbatim}
 trange HH:MM:SS HH:MM:SS
 \end{verbatim}


where again the least-significant elements may be dropped and default
to 00. This type of restriction defines a time range which is valid
every day. The alternative is to specify a time range within the week,
in the following form:

\begin{verbatim}
 trange DAY@HH:MM:SS DAY@HH:MM:SS
\end{verbatim}

where the value of DAY may take on any of the three letter day-of-week
names (Sun, Mon, Tue, etc). This restriction specifies a start and end
time within a week which is evaluated for each week.

Below are several examples of valid time range restrictions

\begin{verbatim}
 trange 2005/01/01 2005/12/31   (only run during 2005!)
 trange 18:00 00:00             (only run from 6pm until midnight)
 trange 00:00 06:00             (only run from midnight until 6am)
 trange Mon@08:00 Fri@17:00     (only run between Mon morning and Fri afternoon)
 trange -exclude 12:00 13:00    (skip 1 hour from noon)
\end{verbatim}

Note that the current definition of trange does not include time zone
information. This means that all times are relative to UT. This should
be addressed by adding a timezone environment variable to PanTasks and
by allowing the trange to define a timezone offset.

It is also possible to restrict the total number of jobs which are
spawned for a given task. This is done with the nmax command, which is
given as part of the task definition. Once a task has constructed nmax
jobs, it stops task evaluation. It is possible to redefine the value
of nmax at any time by redefining the task. Any time the task is
redefined, the new values for any task concept will override the
existing values for the task concept.

\subsection{Inter-Task and Inter-Job Communications}

There are several ways in which the results of jobs may be used to
influence other jobs. These include:

\begin{itemize}
\item external communications
\item job exit status
\item job stdout parsing 
\end{itemize}

It is always possible for the interprocess communication to be
performed externally: all jobs may simply write results to an external
data source which is queried as part of the task evaluation. PanTasks
may interact with UNIX programs using Opihi system interaction
functions. These interaction methods include: the backticks for
setting Opihi variables:

\begin{verbatim}
 $variable = `UNIX Command`
\end{verbatim}

The exec command (which executes a UNIX command) and the backticks
both receive the UNIX command exit status, setting the variable
\code{$STATUS}. It is also possible to set a variable list to the
output of a UNIX command:

\begin{verbatim}
 list var -x "UNIX Command"
\end{verbatim}

In this last case, the values \code{$var:0 - $var:N-1} are set to the
value of the stdout lines from the UNIX command, and the value
\code{$var:n} is set to the number of output lines.

Fine-grained control over the job exit status is available with the
task.exit macro command. This allows a task to define an exit macro
which is performed for different exit status conditions. The argument
to the task.exit command is the exit status value which triggers the
macro. This may consist of any valid numeric exit status value
(0-255). It may also have the value crash, in which case the macro is
executed if the program exited as a result of a signal (ie,
segmentation fault, etc). Finally, if may have the value default, in
which case, the macro is run if no other macro describes the exit
status.

Jobs may transmit their results back to PanTasks for further evaluation
through the standard output and standard error streams. Whenever a job
exits, the complete stdout and stderr streams from the job are pushed
onto the PanTasks queues stdout and stderr. The job exit macros may then
parse these queues, moving the results into other PanTasks / Opihi data
containers (queues, variables, vectors, whatever is appropriate). Note
that currently, the output data is simply pushed onto these output
queues. It is currently the responsibility of the PanTasks programmer to
use or dispose of the data in these queues. This may change in the
future: the queues may be flushed for each job completion.

\subsection{Running the scheduler}

Once a set of tasks has been defined, the scheduler can be
started. The scheduler will run in the background, at regular
intervals examining the collection of tasks and jobs. In these
periods, the scheduler attempts to construct new jobs and checks on
the status of jobs which may have finished, either locally or on the
controller. To start the scheduler, give the command run. To stop the
scheduler, given the command stop. The current status of the
scheduler, controller, and any jobs which have been spawned are listed
with the status command.

It is also possible to kill or delete individual jobs by hand with the
commands kill (jobID) or delete (jobID).  Other features

\subsection{PanTasks Command Summary}

\begin{verbatim}
 controller                  -- controller commands
 task                        -- define a schedulable task
 host                        -- define host machine for a task
 nmax                        -- define maximum number of jobs for a task
 trange                      -- define valid/invalid time periods for a task
 task.exit                   -- define exit macros for a task
 task.exec                   -- define pre-exec macro for a task
 command                     -- define executed command for a task
 periods                     -- define time scales for a task
 run                         -- run the scheduler
 stop                        -- stop the scheduler
 pulse                       -- set the scheduler update period
 status                      -- get system status
 kill                        -- kill job
 delete                      -- delete job
 verbose                     -- set/toggle verbose mode
\end{verbatim}

\begin{figure}
\begin{center}
\includegraphics[scale=0.85]{pics/pantasks.01.ps}
\caption{\label{queues} PanTasks queues and MDDB tables}
\end{center}
\end{figure}

\begin{figure}
\begin{center}
\includegraphics[scale=0.85]{pics/pantasks.02.ps}
\caption{\label{queues} PanTasks queues and MDDB tables}
\end{center}
\end{figure}

\begin{figure}
\begin{center}
\includegraphics[scale=0.85]{pics/pantasks.03.ps}
\caption{\label{queues} PanTasks queues and MDDB tables}
\end{center}
\end{figure}

\newpage
\section{pcontrol : the PanTasks parallel controller}

The IPP uses a group of computers to store and process images and to
manipulate collections of detections. These computers perform any of a
large number of analysis stages or other processing tasks without
significant interprocess communication. It is necessary to have a
mechanism which initiates computing tasks on the different computers,
which monitors the tasks as they are executed, which handles the
output and the errors from these tasks, and which reacts to the
failure of any of the computing nodes. The system responsible for the
tasks in the IPP is pcontrol.

\subsection{Host States}

pcontrol maintains a table of available processing computers (hosts)
and tracks their status. Hosts managed by pcontrol are allowed to be
in one of several states: off, down, idle, busy, and done. These
states have the following meanings:

If the host is off, it is known to pcontrol, but pcontrol does not
have an active connection to the machine. Hosts which are off are not
available for jobs, and pcontrol does not attempt to initiate a
connection to them.

When pcontrol is told to consider a machine on, the machine is moved
from the off state to the down state. Pcontrol attempts to initiate a
connection to the host. Connections are made by running a remote
client on the host, using the specified connection method. The
connection method may be ssh, rsh, or an equivalent remote shell
connection. The choice is specified by the COMMAND Opihi variable. The
remote connection starts a dedicated remote client which must accept
the pcontrol client commands and respond appropriately. The provided
remote client is called pclient, though in principal other equivalent
programs could be used by setting the Opihi variable SHELL (this
feature more generally allows a user to specify a path to the remote
client, if it is not in the user's path). A pcontrol user may force a
host to transition to the off state with the command host off
(hostname). ( Note that this command will set only one of the
connections to the named host to off. If multiple connections to a
machine have been defined, multiple off commands must be sent).

If the remote connection is successful, the connected host is moved by
pcontrol from the down state to the idle state. If the connection is
unsuccessful, pcontrol will try again after a certain period of
time. If the connection continues to be unsuccessful, the retry period
is doubled for each successiver connection attempt. If the user wants
to force pcontrol to retry the connection to a machine (if, for
example, the timeout is now very long, but the user knows the
machine's ethernet cable has been re-inserted...), this can be
achieved with the command host retry (hostname). A host which is down
is in the limbo state between off and idle.

Once pcontrol has made a successful connection to the host, the host
is in the idle state. At this point, it is ready to accept jobs from
pcontrol for execution. Pcontrol repeatedly queries the hosts to check
that they are still alive. If a host is discovered to be unresponsive,
and particularly if the remote pipe connection has closed, then the
machine is moved back to the down state.

Hosts which are idle may accept a job from pcontrol. A job simply
consists of a bare UNIX command, without redirection of standard input
or standard output. The host will initiate the job, and pcontrol will
place the host into the busy state. The remote client, pclient, runs
the job in the background and will continue to accept input from
pcontrol. pcontrol will continue to check the status of the host, and
now also the status of the specific job. As before, if the connection
breaks, pcontrol will migrate the host to the down state. Any job
already initiated on a host which goes down will be returned for later
processing, so the job will not be lost.

When the job exits, pclient tells pcontrol that the job is completed,
and specifies the exit status. At this point, pcontrol will move the
host from busy to done state. It will stay in this state until
pcontrol can determine the ending conditions and reset the remote
client. pcontrol requests the standard error and standard output from
the job from pclient. pcontrol stores this data with its information
about the completed job, and send a reset command to the remote
client. Once these cleanup tasks are successfully completed, pcontrol
will move the host to the idle state, ready for further jobs.

Each physical computer may have multiple processors. pcontrol treats
each processor independently. It is up to the system configuration if
each computer needs to reserve one of its CPUs to manage background
tasks or if pcontrol should attempt to send one task per CPU and let
the operating system handle the I/O load. some of this behavior will
probably be eventually more intelligent. For example, the commands
which turn a host on or off should be able to do the same operation to
all host connections for the same machine name.

A machine may be completely removed from pcontrol's host tables with
the command host delete (hostname).

\subsection{Jobs}

The pcontrol accepts new jobs with the command job ..., in which the
ellipsis represents the command and arguments of a valid UNIX
command. The commands are run under sh, and are executed in the user's
home directory. (If it is desired, we can easily add a command to tell
pclient to perform cd). Users should be wary of the conditions under
which the remote jobs are run. If the nodes in question all
cross-mount the same home directories, multiple jobs which interact
with the same named file may produce unexpected results. The
controller cannot enforce good behavior on the part of the remote
jobs; it is the responsibility of the user to ensure that conflicts do
not arise by, eg, always using unique output file names.

Other issues may arise from the fact that pcontrol may be choosing any
of the hosts to run the job. Typical failures arise if the user does
not realize that specific jobs do not behave the same on all machines,
or if a necessary resource (eg, some input data file) is only
available or accessible from some of the hosts. It is the
responsibility of the task to wait for network lags (ie, NFS delays).

pcontrol gives each task a unique internal identifier (Job ID)
equivalent to the process ID used in UNIX. When a job is submitted to
pcontrol, the command echoes back the Job ID. This ID may be used by
other pcontrol commands to obtain information about or interact with
the job.

A job may specify a specific host for the task execution. The host
specified for a job may be required, or desired. In the first case,
pcontrol, will only run the job on the specified host, waiting until
it is available before attempting the job. In the second case,
pcontrol will attempt to send the job to the specified host, but if
the host is unavailable (how long? what conditions?), pcontrol will
allow the job to be sent to an alternative host. pcontrol attempts to
honor the requests for required and desired hosts, giving priority
first to required-host jobs, then to the desired-host jobs, and
finally to all other jobs. To specify a host for a job, the following
commands are used:

\begin{verbatim}
 job -host (command and arguments...)
 job +host (command and arguments...)
\end{verbatim}


The first case specifies a desired host, while the second specifies a
required host. It is also possible to specify the special host name
anyhost, which is equivalent to not specifying a host at all.

Job priority / urgency levels are not implemented at this time.

I/O vs CPU tasks are not currently distinguished by pcontrol

pcontrol stores the stdout and stderr for each completed job. To
retrieve these data from these streams, the user issues the commands
stdout (JobID) and stderr (JobID). The result is a single line
specifying the number of bytes to expect, followed by a dump of the
buffers, followed by the prompt. It is the user's responsibility to
relieve pcontrol of this data load by deleting jobs once they are no
longer needed. Job deletion is performed with the command delete
(JobID).

Jobs are moved between the following states by pcontrol:

\begin{itemize}
\item pending: the job has not yet been executed.
\item busy: the job is currently being executed.
\item done: the job has completed, but the stdout/stderr has not been processed by pcontrol.
\item exit: the job has completed with a valid exit status
\item crash: the job has completed with a crash status (exit on signal). 
\end{itemize}

\subsection{Miscellaneous Commands}

It is possible to check the status of a single host or job with the
user command check.

pcontrol continuously examines the stack of jobs, adjusting their
state as needed and extracting their output when it is ready. These
checks are performed in the background, with pcontrol ready to accept
further commands from the user in the foreground. These checks are
performed after every keystroke, and also after an inactivity
timeout. The interrupt interval defaults to 1 second, but may be
adjusted with the pulse command, which takes as an argument, the
number of microseconds for the timeout.

the pcontrol system status may be examined with the command
status. This provides a dump of the job stacks and the host stacks.

It is possible to list the jobs currently in a specific stack,
corresponding to the list of jobs with a given state. This is done
with the command jobstack (stackname). The valid stack names are
pending, busy, exit, crash, and done. The result is a list of all jobs
on the specified stack. This is useful to determine quickly which jobs
have exited or crashed.

A specific job may be killed with the command kill (JobID). This
command is only valid for a job in the busy state. Any job in the
pending, exit, or crash state may be deleted with the delete (JobID)
command. This is necessary to free the memory associated with the job
and its output streams.

The command verbose (mode) turns the verbosity of the pcontrol
operations on or off.

The pcontrol and the IPP Image Server have related needs for
information from the combined storage-and-processing nodes regarding
which nodes are available. It is not yet clear if this information is
best stored in a single location (either pcontrol or IPP Image
Server), which provides the information to other systems on demand, or
if both systems should maintain the information. Also, it may be
necessary to distinguish nodes which are available for processing from
those that are available to serve data as part of the IPP Image
Server.

\subsection{Command Summary}

\begin{verbatim}
 check                -- get job or host status
 delete               -- delete job
 host                 -- add / delete / modify host
 job                  -- add job
 jobstack             -- list jobs for a single stack
 kill                 -- kill job
 pulse                -- set system pulse
 status               -- get system status
 stderr               -- get stderr buffer for job
 stdout               -- get stdout buffer for job
 verbose              -- set the verbose mode for job
\end{verbatim}

\begin{figure}
\begin{center}
\includegraphics[scale=0.85]{pics/pantasks.04.ps}
\caption{\label{queues} PanTasks queues and MDDB tables}
\end{center}
\end{figure}

\begin{figure}
\begin{center}
\includegraphics[scale=0.85]{pics/pantasks.05.ps}
\caption{\label{queues} PanTasks queues and MDDB tables}
\end{center}
\end{figure}

\begin{figure}
\begin{center}
\includegraphics[scale=0.85]{pics/pantasks.06.ps}
\caption{\label{queues} PanTasks queues and MDDB tables}
\end{center}
\end{figure}

\newpage
\section{pclient}

pclient is the remote process monitor for pcontrol, the parallel
process controller.

The program pclient is used to support the remote jobs which are run
on the remote hosts by pcontrol. The concept of pclient is to act as a
buffer between the job running on the remote host and pcontrol. The
pcontrol design uses (by default) ssh connections initiated by
pcontrol to the remote hosts. These connections execute the remote
program of pclient. The use of a remote login process lets the UNIX
system take care of the user authentication issues. In this case, the
recommended practice is to set up ssh to allow the connection to the
remote host without additional authentication using the appropriate
authorized keys (see this article on ssh issues).

It is convenient to keep a continuous connection to the remote
hosts. This avoids incurring the overhead of authentication for each
command which is executed, while keeping a high-quality user
authentication process in place.

pclient acts as a buffer between pcontrol and the remote background
process, allowing the continuous connection to remain viable without
samping pcontrol with output from the jobs.  

\section{Command Summary}

pclient has a very limited command set, as follows:

\begin{verbatim}
 job           : start the job (UNIX command) in the background.
 check         : return the current job status
 status        : return the current job status (?)
 stdout        : dump the stdout stream accumulated from the job
                 back to the calling program.
 stderr        : dump the stderr stream accumulated from the job
                 back to the calling program.
 reset         : kill (if needed) the job and reset to accept
                 another job.
\end{verbatim}

\begin{figure}
\begin{center}
\includegraphics[scale=0.85]{pics/pantasks.07.ps}
\caption{\label{queues} PanTasks queues and MDDB tables}
\end{center}
\end{figure}

\appendix
\pagebreak

\section{Example : Pcopy.pro}

Below is an example script for psched which demonstrates the
scheduling system. This parallel-copying script implements the
Pan-STARRS image copying system, which requests images from the summit
and copies them to the appropriate computer. The first task in the
script queries an external system for new image names with the
function new.images. In the case of Pan-STARRS, this would be a
request from OTIS, the observatory controlling system. The second task
initiates the individual image copies, with separate CCDs being copied
to separate computers. This script uses the concept of having specific
machines assigned to specific CCDs (as Pan-STARRS intends to
operate). The association is determined by calling the external
function chip.host, providing the identifier of the chip in
question. This returns an appropriate host. The copy.image function
copies the file and also sends a message to the summit system to
inform it that the image has been successfully copied.

\begin{verbatim}
verbose on
 queueinit newImages
 exec echo 0 > new.last
 exec cp -f raw.list new.list
 
 controller host add po01
 controller host add po02
 controller host add po03
 controller host add po04
 
 # identify the images ready for copy 
 # new entries are added to queue newImages
 # need to compare the new list with the ones already being processed
 task	       new.images
   command      new.images
   host         local
 
   periods      -poll 1
   periods      -exec 5
   periods      -timeout 5
 
   # success
   task.exit    0
     local i j Nstdout Nimages
     # compare output with new.image queue
     # keep only new entries
     queuesize stdout -var Nstdout
     for i 0 $Nstdout
       queuepop stdout -var line
       queuepush newImages -uniq -key 0 "$line"
     end
   end
 
   # locked list
   task.exit    1
     echo       "new.images: exec failure"
     $new.image.failure ++
   end
 
   # default exit status
   task.exit    default
     echo       "new.images: unknown exit status: $EXIT"
     $new.image.failure ++
   end
 
   # operation times out?
   task.exit    timeout
     echo       "new.images: timeout"
     $new.image.failure ++
   end
 end
 
 # copy new images, sending job to desired host
 task	       copy.images
   periods      -poll 0.2
   periods      -exec 1
   periods      -timeout 5
 
   task.exec
     queuesize  newImages -var N
     if ($N == 0) break
     # if ($network == 0) break
     # if ($filesystem == 1) break
     
     queuepop newImages -var line
     list tmp -split $line
     $filename   = $tmp:0
     $chip       = $tmp:1
     $state      = $tmp:2
     if ($state == new) 
       # copy this image
       queuepush newImages -replace -key 0 "$filename $chip run"
     else
       # ignore this image
       queuepush newImages -replace -key 0 "$filename $chip $state"
       break
     end
     # echo $chip
     $host = `chip.host $chip`
     # echo $host
     host $host
     # echo "starting copy for $filename on $host..."
     command copy.image $filename $chip
   end
 
   # can I have access to argc,argv?
 
   # success
   task.exit    0
     echo "done copy..."
     queuepop stdout -var line
     list tmp -split $line
     $filename   = $tmp:0
     $chip       = $tmp:1
     exec mark.image $filename
     queuepush newImages -replace -key 0 "$filename $chip copy"
   end
 
   # default exit status
   task.exit    default
     echo       "new.images: unknown exit status: $EXIT"
     $new.image.failure ++
   end
 
   # operation times out?
   task.exit    timeout
     echo       "new.images: timeout"
     $new.image.failure ++
   end
 end
 
\end{verbatim}

\end{document}
