Pantasks Controller Interactions

(Up to the IPP Description Page)

Pantasks interacts with a parallel job manager. The current version is 'pcontrol', but we would like to allow interactions with condor as well. This file documents the pantasks / controller interactions and the requirements for implementing a condor (or other) interface.

Some terminology

• controller : the abstract concept of the software or system which manages parallel jobs for pantasks.

• pcontrol : the default (Ohana-native) controller implementation

Pantasks Controller Commands

The following commands are available within the pantasks shell to send commands to the controller. Within the pantasks shell, these are invoked with "controller (command) [options]". When running the pcontrol shell on its own, these commands are called directly.

• check job (jobID)
• check host (hostID)

check on the status of a single job or host. the jobID or hostID is an integer value [1]. The return is a block of information giving the status and some other infomation.

For 'host', the return is of the form:

host (state) STATUS 1

where (state) is one of IDLE, BUSY, RESP, DONE, DOWN, OFF.

For 'job', the return is of the form:

STATUS (status) EXITST (Nexit) STDOUT (Nbytes) STDERR (Nbytes) DTIME (elapsed time)

where:

(status) is one of PENDING, BUSY, DONE, EXIT, CRASH (Nexit) is the exit status of the command (ie, as if it were run on

the UNIX command line).

(Nbytes) is the size of the standard out and standard error buffers

from the job

(elapsed time) is the number of seconds it took to run the command;

this is only set on exit.

• exit

Tell the controller to exit. In pantasks, it is necessary to give this command in the form 'controller exit TRUE'.

• host [options] : manipulate hosts managed by the controller. The

following commands may be given:

• host add (hostname) [-threads N] : add a new connection to the host (hostname). The optional -threads N argument specifies the default value for this machine for the @MAX_THREADS@ directive. A job sent to this machine with @MAX_THREADS@ in the command line will have that value replaced by "-threads N" for this machine, were N is the value specified in this host add command.

• host on (hostname) : tell the controller to activate the specified host

• host off (hostname) : tell the contoller to de-activate the specified host

• host check (hostname) : check the status of the given host by hostname (returns 'host (hostname) is (status)' where (status) is one of the list given above for check host

• host retry (hostname) : tell the controller to re-attempt a connection to the specified machine NOW (if the connection failed, pcontrol attempts to connect with an increasingly long timeout. If the timeout is long, but the user knows the machine is now alive, they may desire to force a connection attempt sooner rather than waiting for the timeout to complete).

• host delete (hostname) : remove the named host from the list of managed hosts.

Note that the controller manages connections to host machines by name. Multiple connections are not generally tied together -- as far as pantasks normally is aware, they are not related. Thus, if a given session has N connections to a given machine (host add was called N times), then manipulation of the state of the machine may require N calls of the same function. (Note that commands which change a machine state, such as 'on', 'off', 'delete' only affect machines in the appropriate state. eg, 'host on ipp050' is invalid if ipp050 is not currently off.

• hoststack (stack) : list the hosts in the given 'stack'. A collection of hosts in a given state is called a 'stack' of hosts. This command lists all of the hosts in one of the stacks. Stack names are case-insensitive and may be one of the following:

• IDLE : machines which are currently unoccupied with processing
• BUSY : machines which are currently active
• RESP : machines which are currently active and responding to

another command

• DONE : machines which have completed a job, but are not yet ready

to accept a new job.

• DOWN : machines which are currently unresponsive (pcontrol will

try to reconnect after an interval)

• OFF : machines which are currently off (pcontrol will not try to reconnect)

The response to this command is a list of the machines (a series of lines each with ID NAME on a single line).

• jobstack (stack) : list the jobs in the given 'stack'. Like the hosts, a collection of jobs in a given state is called a 'stack' of jobs. This command lists all of the jobs in one of the stacks. Stack names are case-insensitive and may be one of the following

• PENDING : job is waiting for a host
• BUSY : job is running on a host
• RESP : job is running on a host, and responding to another command
• DONE : job has finished, but its completion state has not yet

been assessed

• EXIT : job finished with a valid exit status (ie, no abort or segfault)
• CRASH : job aborted or segfaulted
• KILL : kill has been requested for the job

• machines : list the status of the hosts by unique machine name. This command is one of the few which work with the multiple connections to a single named machine as the same connection. The command lists the number of connections to the give machine, the number of jobs currently running on that host, and the number of jobs running on another host which requested the given host.

• parameters : set several pcontrol internal parameters. this function lets the user interact with the pcontrol shell and set some internal state variables. The 3 currently allowed options are:

• parameters connect_time (time) : set the maximum time a pclient is kept alive before pcontrol attempts to reset the connection (time in seconds?)
• parameters wanthost_wait (time) : set the amount of time pcontrol will wait before sending a job to a host other than the desired host (time in seconds)
• parameters unwanted_host_jobs (Njobs) : set the number of jobs allowed to run which desire a specific host. this parameters prevents pcontrol from overloading some specific machine due to I/O operations when the process operation is elsewhere.

• output : print the bufers which carry the controller output. when pcontrol starts up, the output can be redirected to a file. if it is not redirected, it is stored by pantasks. the buffer is not normally dumped in a regular fashion, and can fill the pantasks memory usage. this command dumps the output to pantasks, and can also flush the buffer (if the "flush" option is given).

• run : set the run level for the controller. the command is of the form "run (level)". pcontrol may be in one of 4 run levels:
  • all : all normal pcontrol ops (this is aslo set with no optional

argument to the run command).

• reap : keep the machines running (maintain comms and turn on/off

as needed) and harvest results from jobs, but do not spawn

new jobs.

• hosts : manage the machines, but do not manage jobs (spawn or

harvest).

• none : stop all pcontrol processing

• status : report the current status of the controller: list all known jobs and all known hosts, giving their status.

• stop : stop all pcontrol processing (equivalent to "run none")

• verbose : turn on verbose mode for pcontrol (output to pcontrol.log or use "controller output").

• version : print version info for pcontrol

• pulse : in non-threaded pcontrol mode (deprecated), set the readline timeout.

[1] : note that the command 'controller status' includes the job and host ID values in the form X.X.X.X where X is a hexidecimal number. Unfortunately, pcontrol does not understand this format for the IDs for the 'check' command.