Changes between Version 222 and Version 223 of ippToPsps
- Timestamp:
- Apr 2, 2012, 2:35:03 PM (14 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
ippToPsps
v222 v223 130 130 131 131 Each config requires an entry in the '''{{{epoch}}}''' column. An ''epoch'' in the context of {{{ippToPsps}}} is the date that we count as ''the beginning''. If we loaded IPP data to PSPS once and only once, this would not be necessary: we would queue up all available exposures and stacks and simply load them. However, the early stages of the project have required multiple re-loads of data while the IPP perfects the science, necessitating that the same exposures and stacks are loaded more than once. Because {{{queue.py}}} queues available items by taking into account those that have already been published to PSPS, we have to give it an epoch date from which to accurately determine those exposure that have been loaded or not. Generally speaking, the epoch is reset every time we delete the main PSPS database. 132 132 133 == loader.py == 134 135 {{{loader.py}}} is the program that actually processes and publishes data to PSPS. You can run one or more instances of {{{loader.py}}} as the code has been designed so taht they will run concurrently without causing any problems. Note, however, that clients on machines where the DVO database is not on a local disk will run more slowly due to the increased burden of accessing DVO over the network. Also, {{{loader.py}}} will only run on a host with a scratch database setup. Instructions for this are [here]. 133 136 134 137 {{{loader.py}}} programs actually process the data and upload it to the datastore for PSPS. They look to the queue tables in the {{{ippToPsps}}} database for a list of items to process. They first attempt to secure a a 'stripe' of RA (from the [wiki:ippToPsps_Database#Thestripetable stripe table]) that is not being processed by any other {{{loader.py}}} clients. If all available stripes have already been reserved, then it chooses the stripe with the largest number of pending items. … … 209 212 This program simply sets up a scratch database ready for processing. Details of it's use can be found [wiki:ippToPsps#Settingupaloadingmachine here]. 210 213 211 = Using the software = 212 213 214 == Loading == 215 216 Loading data from IPP to PSPS means running an instance of the {{{load.py}}} program. It is possible, and usually necessary, to run multiple versions of {{{load.py}}}. The code has been designed so that the user can start as many clients as he or she likes, and they will run concurrently without causing any problems. Note, however, that clients on machines where the DVO database is not on a local disk will run more slowly due to the increased burden of accessing DVO over the network. 217 218 Any number of clients can run concurrently on any number of hosts, so long as they have a local 'scratch' database set up properly, as detailed below. 219 220 === Setting up a loading machine === 221 222 Although this can be changed in the config, {{{ippToPsps}}} uses a 'scratch' MySQL database on {{{localhost}}} while processing. So, to to any loading, a local MySQL server should be up-and-running, with a database created as the {{{ipp}}} user for use by {{{ippToPsps}}}. The default name for this is {{{ipptopsps_scratch}}}. 214 215 == Setting up a loading machine == 216 217 Each instance of {{{loader.py}}} uses a 'scratch' MySQL database on {{{localhost}}} while processing. So, to do any loading, a local MySQL server should be up-and-running with a database created as the {{{ipp}}} user for use by {{{ippToPsps}}}. The default name for this is {{{ipptopsps_scratch}}}. 223 218 224 219 Because {{{ippToPsps}}} uses this local database as a scratch database, there are large number of inserts and deletions, meaning large log files. For this reason, it is important to configure MySQL to store its data on a local partition, and not {{{/var}}} as is the default. For example, on {{{ipp005}}}:
