dist.process.run jobs can run twice

[bills]

disttool -toprocess can take several seconds to run

We've had a couple of cases where by the time it returns with
a new list of jobs a job in the list may have already run
to completion.

The second job then overwrites the files but the database
insert fails due to a duplicate entry.

We've had a couple of cases where the receive client has failed
to get a file because the size and md5sum do not match the
values listed in the data store (database)

[bills]

Example:
dist_id 8367mysql> select * from rcDSFileset where name = 'o5018g0142o.warp.9411.8367.12';
+-------+---------+---------+-------------------------------+-------+---------------------+-------+
| fs_id | dist_id | prod_id | name | state | registered | fault |
+-------+---------+---------+-------------------------------+-------+---------------------+-------+
| 17219 | 8367 | 12 | o5018g0142o.warp.9411.8367.12 | full | 2009-08-05 13:57:20 | 0 |
+-------+---------+---------+-------------------------------+-------+---------------------+-------+
1 row in set (0.01 sec)

mysql> select * from distComponent where dist_id = 8367 and component = 'skycell.1306.140';
+---------+------------------+----------+----------------------------------+-------+------------------------------------------------------------------+-------+
| dist_id | component | bytes | md5sum | state | name | fault |
+---------+------------------+----------+----------------------------------+-------+------------------------------------------------------------------+-------+
| 8367 | skycell.1306.140 | 47349643 | bf537e5b386caa1e0696b916bfead64a | full | o5018g0142o.85258.wrp.9411.skycell.1306.140.skycell.1306.140.tgz | 0 |
+---------+------------------+----------+----------------------------------+-------+------------------------------------------------------------------+-------+
1 row in set (0.00 sec)

mysql> select * from distRun where dist_id = 8367;
+---------+-----------+-------+----------+-------------------------------+------------------------------------------------------------+-------+----------+-------+---------------------+-------+
| dist_id | target_id | stage | stage_id | label | outroot | clean | no_magic | state | time_stamp | fault |
+---------+-----------+-------+----------+-------------------------------+------------------------------------------------------------+-------+----------+-------+---------------------+-------+
| 8367 | 370 | warp | 9411 | ThreePi_SouthernRegion.090724 | /data/ipp049.0/gpc1_distribution/20090805/o5018g0142o/warp | 0 | 0 | full | 2009-08-05 23:57:11 | 0 |
+---------+-----------+-------+----------+-------------------------------+------------------------------------------------------------+-------+----------+-------+---------------------+-------+
1 row in set (0.00 sec)

Size on disk is 47349516

mysql> select 47349643 - 47349516;
+---------------------+
| 47349643 - 47349516 |
+---------------------+
| 127 |
+---------------------+
1 row in set (0.00 sec)

Note that the files for this component are dated a couple of hours after
the others. The UTC date changed in the meantime.

# cd /data/ipp049.0/gpc1_distribution/20090805/o5018g0142o/warp.8367
# ls -lrt | tail

-rw-rw-r-- 1 ipp users 13378009 Aug 5 13:56 o5018g0142o.85258.wrp.9411.skycell.1396.057.skycell.1396.057.tgz
-rw-rw-r-- 1 ipp users 1593 Aug 5 13:56 dist.8367.skycell.1396.057.log
-rw-rw-r-- 1 ipp users 1593 Aug 5 13:56 dist.8367.skycell.1396.043.log
-rw-rw-r-- 1 ipp users 205913 Aug 5 13:57 dbinfo.warp.9411.mdc
-rw-rw-r-- 1 ipp users 3935 Aug 5 13:57 dirinfo.warp.9411.mdc
-rw-rw-r-- 1 ipp users 136023 Aug 5 13:57 dist.advance.8367.log
-rw-rw-r-- 1 ipp users 64522 Aug 5 13:57 makefs.8367.12.log
-rw-rw-r-- 1 ipp users 47349516 Aug 5 15:56 o5018g0142o.85258.wrp.9411.skycell.1306.140.skycell.1306.140.tgz
-rw-rw-r-- 1 ipp users 4944 Aug 5 15:56 dist.8367.skycell.1306.140.log

[bills]

the queries have been rewritten. This problem is no longer observed