Changes between Version 224 and Version 225 of ippToPsps

Timestamp:

Apr 2, 2012, 2:37:47 PM (14 years ago)

Author:

rhenders

Comment:

--

ippToPsps

@@ -133 +133 @@
 == loader.py ==
 
-{{{loader.py}}} is the program that actually processes and publishes data to PSPS. You can run one or more instances of {{{loader.py}}} as the code has been designed so taht they will run concurrently without causing any problems. Note, however, that clients on machines where the DVO database is not on a local disk will run more slowly due to the increased burden of accessing DVO over the network. Also, {{{loader.py}}} will only run on a host with a scratch database setup. Instructions for this are [here].
-
-{{{loader.py}}} programs actually process the data and upload it to the datastore for PSPS. They look to the queue tables in the {{{ippToPsps}}} database for a list of items to process. They first attempt to secure a  a 'stripe' of RA (from the [wiki:ippToPsps_Database#Thestripetable stripe table]) that is not being processed by any other {{{loader.py}}} clients. If all available stripes have already been reserved, then it chooses the stripe with the largest number of pending items.
+{{{loader.py}}} is the program that actually processes and publishes data to PSPS. You can run one or more instances of {{{loader.py}}} as the code has been designed so taht they will run concurrently without causing any problems. Note, however, that clients on machines where the DVO database is not on a local disk will run more slowly due to the increased burden of accessing DVO over the network. Also, {{{loader.py}}} will only run on a host with a scratch database setup. Instructions for this are [wiki:ippToPsps#Settingupaloadingmachine here].
+
+The first thing a {{{loader.py}}} instance does is look to the queue tables in the {{{ippToPsps}}} database for a list of items to process. It first attempt to secure a 'stripe' of RA (from the [wiki:ippToPsps_Database#Thestripetable stripe table]) that is not being processed by any other {{{loader.py}}} clients. If all available stripes have already been reserved, then it chooses the stripe with the largest number of pending items.
 
 With a stripe chosen, {{{loader.py}}} gathers a list of box_ids (each stripe is sub-divided into boxes). It works its way through each box, processing all pending items.
 
 Because it is possible, and usual, to run multiple versions of {{{loader.py}}} (in order to speed up loading time), the method to begin processing a new batch is part of a critical section. This simply means that the 'batch' table in the {{{ippToPsps}}} database is locked by a client looking for a new item to process, then released afterwards, ensuring that two separate {{{loader.py}}} instances do not attempt to process the same item. 
-
 == pollodm.py ==