Changes between Version 8 and Version 9 of AlternateDistributionClient

Timestamp:

Aug 12, 2009, 11:38:23 AM (17 years ago)

Author:

welling

Comment:

--

AlternateDistributionClient

@@ -1 +1 @@
 = An Alternate Data Distribution Client =
 
-update_mirror.py is a simple client for the IPP distribution system.  It requires only Python 2.6 to run; the IPP software need not be installed.  The script understands the IPP distribution system communication protocol and is intended to be run from ''cron'' to implement daily updates of a mirror site.
+''update_mirror.py'' is a simple client for the IPP distribution system.  It requires only Python 2.6 to run; the IPP software need not be installed.  The script understands the IPP distribution system communication protocol and is intended to be run from ''cron'' to implement daily updates of a mirror site.
 
 The advantages of the script are simplicity, brevity, ease of installation and ease of operation.  Potential disadvantages are:
@@ -10 +10 @@
 == Installation Instructions ==
 
- 1. Create an empty directory on the filesystem where you want to mirror the data distribution.  On Odyssey the corresponding directory is /n/panstarrs/data/MIRROR .
+ 1. Create an empty directory on the filesystem where you want to mirror the data distribution.  On Odyssey the corresponding directory is /n/panstarrs/data/MIRROR .  The source files are attached below.
  1. Copy update_mirror.py and parse_config.py into that directory.
  1. Customize update_mirror.py for your site, by editing the variables at the top of the file and making the following changes:
@@ -32 +32 @@
 }}}
 
+== Mode of Operation ==
+
+The distribution system provides a list of products; new data is appended to those products over time.  The client must remember the most recent update time for a product and only download new material posted since that time.  It must also keep track of previously failed downloads so that they can be re-tried later.
+
+''update_mirror.py'' stores its status information in two comma-separated-value files, the default names for which are ''product_status.csv'' and ''download_failures.csv''.  It maintains a log called ''update_mirror.log'' by default.  If it is interrupted it will also save the state of its internal queues in a file called ''update_mirror_leftover_work.pkl'' so that it can pick up where it left off when it is restarted.  When it starts up, it:
+
+ 1. Re-queues leftover work from a previous crash, if any.
+ 1. Re-queues any entries from the download failures file.
+ 1. Checks for updates to desired products which are more recent than the times recorded in the product_status.csv file, and adds those filesets to the download queue.
+ 1. Commences to work through the download queues using multiple instances of wget and tar, as specified by the IPP distribution protocol.
+
+When it finishes working through the queue it updates its status files and exits.  '''Note: if you try to kill it with SIGTERM (the normal kill signal) it will finish processing the currently in-progress filesets before quitting.  This can take a while, but it's worth it in terms of cleanup.''' If you need to stop it immediately kill it with SIGKILL (''kill -9'').  You are then responsible for cleanup.
+