Changes between Version 4 and Version 5 of pantasks_controller_interactions
- Timestamp:
- Oct 4, 2012, 1:20:06 PM (14 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
pantasks_controller_interactions
v4 v5 22 22 pcontrol shell on its own, these commands are called directly. 23 23 24 * check job (jobID) 25 * check host (hostID) 24 ==== check job (jobID) ==== 25 26 check on the status of a single job. the jobID is an integer value^1^. The return is a block of information giving the 27 status and some other infomation. 28 29 ==== check host (hostID) ==== 26 30 27 31 check on the status of a single job or host. the jobID or hostID is … … 48 52 }}} 49 53 50 *where54 where 51 55 * (status) is one of PENDING, BUSY, DONE, EXIT, CRASH 52 56 * (Nexit) is the exit status of the command (ie, as if it were run on the UNIX command line). … … 54 58 * (elapsed time) is the number of seconds it took to run the command; this is only set on exit. 55 59 56 * exit 60 ==== exit ==== 57 61 58 62 Tell the controller to exit. In pantasks, it is necessary to give this command in 59 63 the form 'controller exit TRUE'. 60 64 61 * host [options] : manipulate hosts managed by the controller. The 62 following commands may be given: 65 ==== host [options] ==== 66 67 manipulate hosts managed by the controller. The following commands may be given: 63 68 * host add (hostname) [-threads N] : add a new connection to the host 64 69 (hostname). The optional -threads N argument specifies the default … … 93 98 currently off. 94 99 95 * hoststack (stack) : list the hosts in the given 'stack'. A 96 collection of hosts in a given state is called a 'stack' of hosts. 97 This command lists all of the hosts in one of the stacks. Stack 98 names are case-insensitive and may be one of the following: 100 ==== hoststack (stack) ==== 101 102 list the hosts in the given 'stack'. A collection of hosts in a given state is called a 'stack' of hosts. 103 This command lists all of the hosts in one of the stacks. Stack 104 names are case-insensitive and may be one of the following: 105 99 106 * IDLE : machines which are currently unoccupied with processing 100 107 * BUSY : machines which are currently active … … 107 114 lines each with ID NAME on a single line). 108 115 109 * jobstack (stack) : list the jobs in the given 'stack'. Like the 110 hosts, a collection of jobs in a given state is called a 'stack' of 111 jobs. This command lists all of the jobs in one of the stacks. 112 Stack names are case-insensitive and may be one of the following 116 ==== jobstack (stack) ==== 117 118 list the jobs in the given 'stack'. Like the hosts, a collection of jobs in a given state is called a 'stack' of 119 jobs. This command lists all of the jobs in one of the stacks. 120 Stack names are case-insensitive and may be one of the following 121 113 122 * PENDING : job is waiting for a host 114 123 * BUSY : job is running on a host … … 119 128 * KILL : kill has been requested for the job 120 129 121 * machines : list the status of the hosts by unique machine name. 130 ==== machines ==== 131 list the status of the hosts by unique machine name. 122 132 This command is one of the few which work with the multiple 123 133 connections to a single named machine as the same connection. The … … 126 136 jobs running on another host which requested the given host. 127 137 128 * parameters : set several pcontrol internal parameters. this 129 function lets the user interact with the pcontrol shell and set some 130 internal state variables. The 3 currently allowed options are: 138 ==== parameters ==== 139 set several pcontrol internal parameters. this 140 function lets the user interact with the pcontrol shell and set some 141 internal state variables. The 3 currently allowed options are: 142 131 143 * parameters connect_time (time) : set the maximum time a pclient is 132 144 kept alive before pcontrol attempts to reset the connection (time … … 140 152 I/O operations when the process operation is elsewhere. 141 153 142 * output : print the bufers which carry the controller output. when 143 pcontrol starts up, the output can be redirected to a file. if it 154 ==== output ==== 155 print the bufers which carry the controller output. when 156 pcontrol starts up, the output can be redirected to a file. if it 144 157 is not redirected, it is stored by pantasks. the buffer is not 145 158 normally dumped in a regular fashion, and can fill the pantasks … … 147 160 also flush the buffer (if the "flush" option is given). 148 161 149 * run : set the run level for the controller. the command is of the 150 form "run (level)". pcontrol may be in one of 4 run levels: 162 ==== run ==== 163 set the run level for the controller. the command is of the 164 form "run (level)". pcontrol may be in one of 4 run levels: 165 151 166 * all : all normal pcontrol ops (this is aslo set with no optional argument to the run command). 152 167 * reap : keep the machines running (maintain comms and turn on/off as needed) and harvest results from jobs, but do not spawn new jobs. … … 154 169 * none : stop all pcontrol processing 155 170 156 * status : report the current status of the controller: list all known jobs and all known hosts, giving their status. 157 158 * stop : stop all pcontrol processing (equivalent to "run none") 159 160 * verbose : turn on verbose mode for pcontrol (output to pcontrol.log or use "controller output"). 161 162 * version : print version info for pcontrol 163 164 * pulse : in non-threaded pcontrol mode (deprecated), set the readline timeout. 171 ==== status ==== 172 173 report the current status of the controller: list all known jobs and all known hosts, giving their status. 174 175 ==== stop ==== 176 177 stop all pcontrol processing (equivalent to "run none") 178 179 ==== verbose ==== 180 181 turn on verbose mode for pcontrol (output to pcontrol.log or use "controller output"). 182 183 ==== version ==== 184 185 print version info for pcontrol 186 187 ==== pulse ==== 188 189 In non-threaded pcontrol mode (deprecated), set the readline timeout. 165 190 191 === Pantasks / Controller Interactions === 192 193 ==== Pantasks Threads & Controller Interactions ==== 194 195 Pantasks has a server/client mode and a stand-alone mode. These share 196 the bulk of code, but there are some minor difference. In stand-alone 197 mode, there are 3 active threads. The main thread interprets the 198 commands (accepted by readline); a second thread manages the tasks and 199 jobs known to pantasks; the third thread manages interactions with the 200 parallel controller. In server/client mode, a fourth thread manages 201 the communcations with the remote clients. 202 203 The task/job thread is responsible for monitoring the task rules and 204 constructing the commands which corresponds to jobs when appropriate. 205 Those jobs which are defined to be local are executed on the local 206 machine, while remote jobs are sent to the thread which interacts with 207 the parallel controller. 208 209 The primary job of the controller thread is to monitor the status of 210 jobs submitted to the controller and to harvest jobs which have 211 finished. A secondary job is to flush the stdout and stderr buffers 212 of the pantasks / pcontrol connection. 213 214 Jobs are submitted to the controller directly by the Task/Job thread. 215 Other operations, check as manual checks of the job status are 216 performed by the main thread. 217 218 ==== Commands sent to pcontrol ==== 219 220 In addition to the user-level commands discussed above, the following 221 messages are sent to the controller: 222 223 jobstack exit : the controller thread checks for the set of completed 224 jobs (which did not crash) by sending this command. The response is 225 a list of jobs ready for harvest. 226 227 * jobstack crash : the controller thread checks for the set of 228 complete jobs which crashed by sending this command. The response 229 is a list of jobs ready for harvest. 230 231 * delete : jobs which have completed and for which the stderr/stdout 232 have been received can be deleted from the controller. For 233 'pcontrol', this is necessary to free up resources managing the 234 specific job. 235 236 * check job : once the controller gets a list of jobs which have 237 exited or crashed, pantasks loops over those jobs, harvesting their 238 results. this command is used to get the needed stats (size of 239 stderr buffer, size of stdout buffer, exit status, etc). 240 241 * job : Submit a job to the controller. this command is used by the 242 function SubmitControllerJob to send a new job to the controller. 243 the function appends options such as +host, -nice as needed. the 244 function expects to receive a job ID from the controller for future 245 interactions. 246 247 * quit : shutdown the controller 248 249 --- 250 251 pantasks & condor 252 253 There are three classes of pcontrol operations which pantasks 254 currently perform: 255 256 * submit jobs to pcontrol 257 * detect completion and harvest job output 258 * manage the hosts used by pcontrol 259 260 The communication between pantasks and pcontrol uses a pipe; pcontrol 261 is run as a forked child of pantasks. The communication is done via 262 text blocks with fairly minimal hand-shaking. 263 264 If pantasks is integrated with condor, it is critical that it be able 265 to perform the first two of these functions. It is not required that 266 pantasks manage the condor hosts. 267 268 Condor client / server interaction could be performed via forked 269 commands or we could use the SOAP interface. I suspect that SOAP will 270 be more effective for the high-rate job harvesting process. 271 272 However jobs are submitted to condor, it is critical that condor be 273 able to respect the limit on the number of mistargeted jobs currently 274 active. 166 275 167 276 ^1^ : note that the command 'controller status' includes the job and^
