Contents of IPP Distribution File Sets

Distribution bundles are built and placed in file sets on the outgoing data store by the server side of the ipp distribution system.

Each destination site runs the client sideis programmed to query a particular data store product for new filesets.

Product List Format The data store product list is the list of file sets in the given product. The product listing contains information about the contents of the fileset. Example

# filesetID                |time registered     |type     |target_id|stage|stage_id|exp_name|
o4741g0234o.fake.2799.11.1 |2009-04-15T01:22:12Z|notset   |5|fake|2799 |o4741g0234o|
o4741g0236o.chip.14519.33.1|2009-04-15T03:02:11Z|notset   |1|chip|14519|o4741g0236o|
o4741g0235o.chip.14518.32.1|2009-04-15T03:02:12Z|notset   |1|chip|14518|o4741g0235o|
o4741g0234o.chip.14517.31.1|2009-04-15T03:02:12Z|notset   |1|chip|14517|o4741g0234o|
o4741g0237o.chip.14520.34.1|2009-04-15T03:02:13Z|notset   |1|chip|14520|o4741g0237o|
o4741g0240o.chip.14522.36.1|2009-04-15T03:02:14Z|notset   |1|chip|14522|o4741g0240o|
o4741g0239o.chip.14524.38.1|2009-04-15T03:02:14Z|notset   |1|chip|14524|o4741g0239o|
o4741g0241o.chip.14523.37.1|2009-04-15T03:02:15Z|notset   |1|chip|14523|o4741g0241o|
o4741g0238o.chip.14521.35.1|2009-04-15T04:03:19Z|notset   |1|chip|14521|o4741g0238o|

Note: the fileset type for IPP distribution bundles has not yet been defined. (notset in this example)

Fileset Names

Runs that are the result of a particular exposure

$exp_name.$stage.$stage_id.$dist_id.$prod_id

$exp_name is rawExp.exp_name $stage is raw, chip, camera, fake, warp, or diff $dist_id is distRun.dist_id (This table is internal to the server side of the distribution system) $prod_id is rcDSProduct.prod_id (This table is internal to the server side of the distribution system)

Examples

o4741g0239o.raw.36625.27.1 o4741g0239o.chip.11955.2.1

For stackRun distributions the fileset name is

$stack.$stack_id.$dist_id.$prod_id

for example

stack.9554.42.2

The name of detrend filesets is TBD.

Contents of the fileset

1) An mdc file containing the database information for the run with the name

dbinfo.$stage.$stage_id.mdc

for example

dbinfo.chip.11955.mdc

This file is the output from the -exportrun mode of the stage's ippTool (regtool, chiptool, etc). If the run is exported with the -clean option when the run is in full state the state and data state are changed from 'full' to 'cleaned'

XXX: Should we be doing something with the nebulous paths in this file here? Here I assume this is the client's job.

2) A set of gzipped tar files containing the files for a given component of the run.

For stages raw, chip, camera, and fake the name of the file for each component is

basename($component_path_base).$class_id.tgz

and for stages warp, diff, and stack the filenames are

basename($component_path_base).tgz

For example the files for chip XY01 for chipRun 11955 the path_base is

neb://anyhost/md01_20081002/o4741g0239o.36625/o4741g0239o.36625.ch.11955

so the name of the tarfile is

o4741g0239o.36625.ch.11955.XY05.tgz

The filenames in the tarfile are the basenames of the nebulous paths. For example

% tar tf o4741g0239o.36625.ch.11955.XY05.tgz ./ ./o4741g0239o.36625.ch.11955.XY05.ppImage.mdc ./o4741g0239o.36625.ch.11955.XY05.b2.fits ./o4741g0239o.36625.ch.11955.XY05.stats ./o4741g0239o.36625.ch.11955.XY05.cmf ./o4741g0239o.36625.ch.11955.XY05.trace ./o4741g0239o.36625.ch.11955.XY05.log ./o4741g0239o.36625.ch.11955.XY05.mdl.fits ./o4741g0239o.36625.ch.11955.XY05.b1.fits

Notes on the process of importing the data into the mirror system The client system runs a task that queries the ipp distribution data store product for new filesets

dsproductls --uri $uri --last_fileset $last_fileset_processed

For each fileset to be downloaded

dsget each file in the fileset to local temporary storage

parse the dbinfo file to find the workdirs and path_base foreach component

edit paths as necessary for the mirror installation foreach file insert into nebulous

import the dbinfo into the database using xxxtool -importrun

ISSUES/Question?

---