Changes between Version 1 and Version 2 of DistribFileSets

Timestamp:

Apr 9, 2009, 10:37:55 AM (17 years ago)

Author:

bills

Comment:

--

DistribFileSets

@@ -1 +1 @@
 == Contents of IPP Distribution File Sets ==
+Distribution bundles are built and placed in file sets on the outgoing data store by the
+server side of the ipp distribution system.
+
+Each destination site runs the client sideis programmed to query a particular data store
+product for new filesets.
+
+Fileset Names
+-------------
+Runs that are the result of a particular exposure
+
+
+    $exp_name.$stage.$stage_id.$dist_id.$prod_id
+
+$exp_name is rawExp.exp_name
+$stage is raw, chip, camera, fake, warp, or diff
+$dist_id is distRun.dist_id       (This table is internal to the server side of the distribution system)
+$prod_id is rcDSProduct.prod_id   (This table is internal to the server side of the distribution system)
+
+
+Examples
+
+
+
+
+    o4741g0239o.raw.36625.27.1
+    o4741g0239o.chip.11955.2.1
+
+
+For stackRun distributions the fileset name is
+
+ 
+   $stack.$stack_id.$dist_id.$prod_id
+
+
+for example
+
+
+    stack.9554.42.2
+
+
+The name of detrend filesets is TBD.
+
+
+Contents of the fileset
+-----------------------
+1) An mdc file containing the database information for the run with the name
+
+    dbinfo.$stage.$stage_id.mdc
+
+for example
+
+    dbinfo.chip.11955.mdc
+
+This file is the output from the -exportrun mode of the stage's ippTool (regtool, chiptool, etc).
+If the run is exported with the -clean option when the run is in full state 
+the state and data state are changed from 'full' to 'cleaned'
+
+XXX: Should we be doing something with the nebulous paths in this file here? Here I assume this
+is the client's job.
+
+2) A set of gzipped tar files containing the files for a given component of the run.
+
+For stages raw, chip, camera, and fake the name of the file for each component is
+
+    basename($component_path_base).$class_id.tgz
+
+and for stages warp, diff, and stack the filenames are
+
+    basename($component_path_base).tgz
+
+For example the files for chip XY01 for chipRun 11955 the path_base is
+
+    neb://anyhost/md01_20081002/o4741g0239o.36625/o4741g0239o.36625.ch.11955
+
+so the name of the tarfile is
+
+    o4741g0239o.36625.ch.11955.XY05.tgz
+
+The filenames in the tarfile are the basenames of the nebulous paths. For example
+
+        % tar tf  o4741g0239o.36625.ch.11955.XY05.tgz
+        ./
+        ./o4741g0239o.36625.ch.11955.XY05.ppImage.mdc
+        ./o4741g0239o.36625.ch.11955.XY05.b2.fits
+        ./o4741g0239o.36625.ch.11955.XY05.stats
+        ./o4741g0239o.36625.ch.11955.XY05.cmf
+        ./o4741g0239o.36625.ch.11955.XY05.trace
+        ./o4741g0239o.36625.ch.11955.XY05.log
+        ./o4741g0239o.36625.ch.11955.XY05.mdl.fits
+        ./o4741g0239o.36625.ch.11955.XY05.b1.fits
+
+
+Notes on the process of importing the data into the mirror system
+------------------------------------------------------------------
+The client system runs a task that queries the ipp distribution data store product for new
+filesets
+
+        dsproductls --uri $uri --last_fileset $last_fileset_processed
+
+For each fileset to be downloaded
+        dsget each file in the fileset to local temporary storage
+
+        parse the dbinfo file to find the workdirs and path_base
+        foreach component
+            edit paths as necessary for the mirror installation
+            foreach file insert into nebulous
+        import the dbinfo into the database using xxxtool -importrun
+
+ISSUES/Question?
+---------------
+