Changes between Initial Version and Version 1 of PublishingDetections

Timestamp:

Jun 19, 2009, 1:48:52 PM (17 years ago)

Author:

Paul Price

Comment:

Adding information about publishing detections.

PublishingDetections

@@ +1 @@
+== Requirements ==
+
+Note that publishing requires installation of the DataStoreServer.
+
+== Components ==
+
+The detections publishing mechanism consists of the following components:
+
+ * ippdb database tables: publishClient, publishRun, publishDone
+ * ippTools: pubtool
+ * ippScripts: publish_file.pl
+ * ippTasks: publish.pro (publish.trigger, publish.load, publish.run)
+ * ppMops (translation for IPP-MOPS product)
+
+== Database ==
+
+ * publishClient: IPP clients (they may be "science servers" to the rest of the PS1SC, but they are clients to the IPP) that want to receive detections via the DataStore.  They must nominate a product (used for the DataStore as the product name and type), the stage of interest (currently only "diff" and "camera" are accepted) and a working directory (meaningless for the clients themselves, but may be important for us if we're translating formats, since we need a place to store the translated data; note this does not have anything to do with the DataStore).  A comment field is also available, unused by the system (though it can be searched on using LIKE for some of the pubtool-generated queries).
+ * publishRun: A run consists of pushing a single exposure, specified by a stage_id (which corresponds to either a diff_id or cam_id, depending on the stage in publishClient) to a client.  A run has state 'new' when created, and 'full' when completed.  A label is provided for grouping.
+ * publishDone: On completion of a run, an entry is made here with the path_base and fault code.
+
+== Workflow ==
+
+An entry in the publishClient table is set up manually:
+{{{
+pubtool -defineclient -stage diff -comment "This is an aid to memory" -product IPP-MOPS -workdir neb://any/where/
+}}}
+
+With the client set up, pantasks (task publish.trigger) regularly looks for data that is ready to be published.  It does so with:
+
+{{{
+pubtool -definerun -label LABEL
+}}}
+
+This looks for a completed diffRun or camRun (as specified by the clients) with the provided label, and generates a corresponding publishRun with the same label.
+
+Pending publishRuns are recognised by pantasks (task publish.load) using:
+
+{{{
+pubtool -pending -label LABEL
+}}}
+
+These are then executed (task publish.run):
+
+{{{
+publish_file.pl --pub_id 123 --camera GPC1 --workdir neb://any/where/ --product IPP-MOPS --stage diff --stage_id 456 --redirect-output
+}}}
+
+This script retrieves the appropriate input files (using either difftool or camtool), applies any translation (ppMops is used for the IPP-MOPS product) and the results bundled off to the DataStore.  Finally, it adds the result:
+
+{{{
+pubtool -add -pub_id 123 -path_base neb://any/where/IPP-MOPS.123
+}}}
+
+In the event of an error, the fault code is added.  The run can be retried following reversion:
+
+{{{
+pubtool -revert -pub_id 123
+}}}
+
+
+== Use ==
+
+Add a client using "pubtool -defineclient", install the DataStoreServer package, ensure the DataStore configurations are in your site configuration (DS_DBSERVER, DS_DBNAME, DS_DBUSER, DS_DBPASSWORD, DATA_STORE_ROOT) and you should be good to go:
+
+{{{
+pantasks: module pantasks.pro
+pantasks: module publish.pro
+pantasks: add.label LABEL
+pantasks: add.database DATABASE
+pantasks: controller host add HOST_1
+pantasks: controller host add HOST_2
+pantasks: controller host add HOST_3
+pantasks: run
+}}}
+
+A directory is set up for the ipp user on the Production (Maui) cluster to run this on a specified machine.
+
+== Debugging ==
+
+The publish_file.pl script recognises the usual debugging flags:
+ * --verbose
+ * --no-update
+ * --save-temps