Changes between Version 6 and Version 7 of shepherding_the_pstamp_server

Timestamp:

Jun 23, 2011, 1:21:13 PM (15 years ago)

Author:

bills

Comment:

--

shepherding_the_pstamp_server

@@ -12 +12 @@
 
 Each file is represented in the database by a row in the table pstampRequest. The request starts
-life in state 'new'. Once it is done parsing the state is set to 'run'. Request files are parsed into jobs by the task pstamp.job.run. The working directory for a request is /data/ippdb02.0/pstamp/work/$datestr/$req_id.
+life in state 'new'. Once it is done parsing the state is set to 'run'. Request files are parsed into jobs by the task pstamp.job.run. The working directory for a request is /data/ippc17.0/pstamp/work/$datestr/$req_id.
 
 For example here is the entry for a request while it is being parsed 
@@ -32 +32 @@
 Each row in the request table contains a request specification. Each row causes one or more pstampJobs to
 be entered into the database. Jobs are often entered with state = 'stop' and fault > 10. These are due
-to requests wanting images that haven't been destreaked or have been purged, bycoord requests that
-have no images or errors in the request. The set of errors is listed at the bottom of this page.
+to requests for images that can not be satisfied. The set of errors is listed at the bottom of this page.
 
-Once the file has completed parsing successfully it's state is set to 'run'. Once the request is
+Once the request table has completed parsing successfully it's state is set to 'run'. Once the request is
 in state run its jobs are eligible to run and will be output by pstamptool -pendingjob
 
@@ -49 +48 @@
 the outdir of the first request that needs the component. 
 
-Dependents can start being processed before parsing completes. The -pendingdependent query is
-
-
-{{{
-SELECT DISTINCT
-    pstampDependent.*,
-    IFNULL(Label.priority, 10000) AS priority
-FROM pstampDependent
-JOIN pstampJob USING(dep_id)
-JOIN pstampRequest USING(req_id)
-LEFT JOIN Label ON pstampRequest.label = Label.label
-WHERE pstampDependent.state = 'new'
-    AND pstampDependent.fault = 0
-    AND pstampJob.state = 'run'
-    AND (pstampRequest.state = 'run' OR pstampRequest.state = 'new')
-    AND (Label.active OR Label.active IS NULL)
- ORDER BY priority DESC, req_id  LIMIT 1
-
-}}}
-
 
 
@@ -81 +60 @@
 
 
-Since the postage stamp / data store database is on ippdb02 and not ippdb01
+Since the postage stamp / data store database is on ippc17 and not ippdb01
 some care must be taken when running commands.
 
-I define the environment variable PSDBSERVER=ippdb02 and the alias
+I define the environment variable PSDBSERVER=ippc17 and the alias
 
  
@@ -119 +98 @@
 
 Requests can fault for two reasons either at the parse stage or when finishing the request. Request
-faults rarely happen and are usually due to software bug. pst -revertreq doesn't work for requests that
-haven't finished parsing. (The scripts set the state to run when it shouldn't. I don't want to fix this
-problem right now. I want to be sure I don't break anything).
+faults rarely happen and are usually due to software bug. 
 
 Jobs can fault due to typical nfs errors. There is a revert task for jobs. It only resets jobs with
-fault > 0 and < 10. Faults >= 10 are faults for the users. Jobs which are faulted in this way should
+fault > 0 and < 10. Faults >= 10 are faults that are to be reported to the users. Jobs which are faulted in this way should
 have had their state set to stop thus they are finished.
 
@@ -167 +144 @@
 to avoid faulted dependents from filling up the pantasks queue thus preventing other dependents from running.
 
-Once the chip fault has been cleared
+Later the dependent fault will be reverted and the dependency checking will run again. Hopefully the chip fault
+will have reverted and finished. After some number of dependency faults (3 currently) the server gives up
+and faults the job.
+
+So if we have long periods of cluster trouble, postage stamp users will get lots of failed jobs.
 
 
@@ -199 +180 @@
 == Monitoring the Apache Server ==
 
-It is a good idea to check the Apache error log (/var/log/apace2/error_log) from time to time to look for unusual activity. On June 4 I found a user furtively looking for a results file for a request that was submitted May 24.
+It is a good idea to check the Apache error log (/var/log/apace2/error_log) from time to time to look for unusual activity. On June 4, 2010 I found a user furtively looking for a results file for a request that was submitted May 24, 2010.