pstampRequest 5977 request was submitted through the web upload interaface on May 27. The request file has 2624 rows all of which are bycoord requests for chip stage data. bycoord requests are the slowest to parse. It takes on the order of 2 seconds per request row to process these.

The request started parsing at 13:38 HST May 27. We can see this because that is the time stamp on the request file. At 15:55 parsing is about half done. 4994 jobs have been queued so far and all but 200 are requests for cleaned up data. So the parsing will probably complete around 18:00

The following is a diary of me watching this request process. If some faults occur this may provide a debugging and repair tutorial.

# The program pstampdump can be used to list the contents of a fits table

(ippdb02:work/20100527/5977) bills% pstampdump -simple web_1695.fits > dumpreq.txt

(ippdb02:work/20100527/5977) bills% wc dumpreq.txt
  2624  49856 256353 dumpreq.txt

# Each row has a ROWNUM which must be unique in the file.


(ippdb02:work/20100527/5977) bills% grep Collected psparse.5977.log | tail -n 1
Collected 1 rows beginning with row 100595. bycoord chip null null 

# it's working on the 1486th row now

(ippdb02:work/20100527/5977) bills% grep -n 100595 dumpreq.txt 
1486:100595 205.550874 28.351276 100 100 2 stamp 1 gpc1 bycoord chip null null null null y 0 0 search 

# We're up to 5123 jobs now.

mysql> select count(*) from pstampJob where req_id >= 5977 and fault = 0 ;
+----------+
| count(*) |
+----------+
|     5123 | 
+----------+
1 row in set (0.02 sec)


# All but 200 of these are for data that has been cleaned.

mysql> select count(*) from pstampJob where req_id >= 5977 and fault = 0 and dep_id = 0;
+----------+
| count(*) |
+----------+
|      200 | 
+----------+
1 row in set (0.02 sec)

# where are we now?

(ippdb02:work/20100527/5977) bills% date ; grep -n `grep Collected psparse.5977.log | tail -n 1 | awk '{print $7}'` dumpreq.txt
Thu May 27 16:06:56 HST 2010
1665:100774 205.543583 28.359453 100 100 2 stamp 1 gpc1 bycoord chip null null null null y 0 0 search 

1665 of 2564 64%

# There are lots of dependents pending. The poll limit in pstamp  pantasks is 256 so 1/4 of these have run and have queued
their updates.

(ippdb02:work/20100527/5977) bills% pst -pendingdependent -simple | wc
   1027   11297   94484

# No faults in the pstamp or update pantasks as yet.  The update pantasks queue is full with chip processing and destreaking.

# looks like my 18:00 finish time estimate was pessimistic. It's 88% done at 16:42

(ippdb02:work/20100527/5977) bills% date ; grep -n `grep Collected psparse.5977.log | tail -n 1 | awk '{print $7}'` dumpreq.txt | awk '{print $1}'
Thu May 27 16:41:54 HST 2010
2264:140277

6892 jobs, 6778 in run state, 114 faulted

mysql> select fault, count(job_id) from pstampJob where req_id = 5977 and fault > 0 group by fault;
+-------+---------------+
| fault | count(job_id) |
+-------+---------------+
|    23 |            93 |  PSTAMP_NO_IMAGE_MATCH  center coordinate not contained in any "magicked" images (or perhaps wrong error code)
|    26 |            23 |  PSTAMP_GONE  these are probably from purged or cleaned up data
+-------+---------------+
2 rows in set (0.04 sec)

# 97% done at 17:00

# 100% done at 17:06
# Final tally
7691 jobs
7570 to run
4793 dependents (chip images to regenerate (update))
121 faulted
+-------+---------------+
| fault | count(job_id) |
+-------+---------------+
|    23 |            98 | 
|    26 |            23 | 
+-------+---------------+

11797 seconds to parse 3.27 hours