Changes between Version 194 and Version 195 of ippToPsps

Timestamp:

Mar 9, 2012, 2:32:53 PM (14 years ago)

Author:

rhenders

Comment:

--

ippToPsps

@@ -112 +112 @@
 == loader.py ==
 
-{{{loader.py}}} programs actually process the data and upload it to the datastore for PSPS. They look to the queue tables in the {{{ippToPsps}}} database for a list of items to process. They first attempt to process a stripe of RA (from the [wiki:ippToPsps_Database#Thestripetable stripe table]) that is not being processed by any other {{{loader.py}}} client. If all available stripes have already been reserved, then it chooses the stripe with the largest number of pending items.
-
-With a stripe chosen, {{{loader.py}}} obtains a list of box_ids (each stripe is sub-divided into boxes). It works its way through each box, processing all pending items.
+{{{loader.py}}} programs actually process the data and upload it to the datastore for PSPS. They look to the queue tables in the {{{ippToPsps}}} database for a list of items to process. They first attempt to secure a  a 'stripe' of RA (from the [wiki:ippToPsps_Database#Thestripetable stripe table]) that is not being processed by any other {{{loader.py}}} clients. If all available stripes have already been reserved, then it chooses the stripe with the largest number of pending items.
+
+With a stripe chosen, {{{loader.py}}} gathers a list of box_ids (each stripe is sub-divided into boxes). It works its way through each box, processing all pending items.
+
+Because it is possible, and usual, to run multiple versions of {{{loader.py}}} (in order to speed up loading time), the method to begin processing a new batch is part of a critical section. This simply means that the 'batch' table in the {{{ippToPsps}}} database is locked by a client looking for a new item to process, then released afterwards, ensuring that two separate {{{loader.py}}} instances do not attempt to process the same item. 
 == metrics.py ==