Requirements

Note that publishing requires installation of the DataStoreServer.

Components

The detections publishing mechanism consists of the following components:

• ippdb database tables: publishClient, publishRun, publishDone
• ippTools: pubtool
• ippScripts: publish_file.pl
• ippTasks: publish.pro (publish.trigger, publish.load, publish.run)
• ppMops (translation for IPP-MOPS product)

Database

• publishClient: IPP clients (they may be "science servers" to the rest of the PS1SC, but they are clients to the IPP) that want to receive detections via the DataStore. They must nominate a product (used for the DataStore as the product name and type), the stage of interest (currently only "diff" and "camera" are accepted) and a working directory (meaningless for the clients themselves, but may be important for us if we're translating formats, since we need a place to store the translated data; note this does not have anything to do with the DataStore). A comment field is also available, unused by the system (though it can be searched on using LIKE for some of the pubtool-generated queries).
• publishRun: A run consists of pushing a single exposure, specified by a stage_id (which corresponds to either a diff_id or cam_id, depending on the stage in publishClient) to a client. A run has state 'new' when created, and 'full' when completed. A label is provided for grouping.
• publishDone: On completion of a run, an entry is made here with the path_base and fault code.

Workflow

An entry in the publishClient table is set up manually:

pubtool -defineclient -stage diff -comment "This is an aid to memory" -product IPP-MOPS -workdir neb://any/where/

With the client set up, pantasks (task publish.trigger) regularly looks for data that is ready to be published. It does so with:

pubtool -definerun -label LABEL

This looks for a completed diffRun or camRun (as specified by the clients) with the provided label, and generates a corresponding publishRun with the same label.

Pending publishRuns are recognised by pantasks (task publish.load) using:

pubtool -pending -label LABEL

These are then executed (task publish.run):

publish_file.pl --pub_id 123 --camera GPC1 --workdir neb://any/where/ --product IPP-MOPS --stage diff --stage_id 456 --redirect-output

This script retrieves the appropriate input files (using either difftool or camtool), applies any translation (ppMops is used for the IPP-MOPS product) and the results bundled off to the DataStore. Finally, it adds the result:

pubtool -add -pub_id 123 -path_base neb://any/where/IPP-MOPS.123

In the event of an error, the fault code is added. The run can be retried following reversion:

pubtool -revert -pub_id 123

Use

Add a client using "pubtool -defineclient", install the DataStoreServer package, ensure the DataStore configurations are in your site configuration (DS_DBSERVER, DS_DBNAME, DS_DBUSER, DS_DBPASSWORD, DATA_STORE_ROOT) and you should be good to go:

pantasks: module pantasks.pro
pantasks: module publish.pro
pantasks: add.label LABEL
pantasks: add.database DATABASE
pantasks: controller host add HOST_1
pantasks: controller host add HOST_2
pantasks: controller host add HOST_3
pantasks: run

A directory is set up for the ipp user on the Production (Maui) cluster to run this on a specified machine.

Debugging

The publish_file.pl script recognises the usual debugging flags:

• --verbose
• --no-update
• --save-temps