Changeset 20257

Timestamp:

Oct 17, 2008, 4:00:52 PM (18 years ago)

Author:

bills

Message:

various postage stamp server updates and bug fixes

Location:

trunk/pstamp

Files:

• doc/psnotes.txt (modified) (8 diffs)
• scripts/pstamp_finish.pl (modified) (2 diffs)
• scripts/pstamp_results_file.pl (modified) (1 diff)
• test/pstamp_req_create (modified) (1 diff)
• test/sample_pstamp.txt (modified) (1 diff)
• test/sample_simtest.txt (added)

trunk/pstamp/doc/psnotes.txt

@@ -6 +6 @@
 This note describes the installation and operation of the IPP Postage Stamp Server.
 We assume that IPP source for examination and that the user has the standard
-ipp configuration environment set up.
-
-This component of the IPP system is not intended for general distribution to ipp users
+IPP configuration environment set up.
+
+This component of the IPP system is not intended for general distribution to IPP users
 and does not have streamlined installation procedures at this time.
 
@@ -91 +91 @@
 
 
-The ipp includes a number of tools for accessing a data store. An introduction to these tools may
+The IPP includes a number of tools for accessing a data store. An introduction to these tools may
 be seen by using the program perldoc.
 
@@ -109 +109 @@
 
 The files for the Data Store Server located in the IPP source tree in the directory
-DataStoreServer (this directory may not be available in the distributed ipp
+DataStoreServer (this directory may not be available in the distributed IPP
 tarballs).
 
@@ -115 +115 @@
 -----------------------------
 
-To set up the database go to the DataStoreServer/scripts and start up mysql.
-
-mysql> create database mydatabase;
-mysql> use mydatabase;
-mysql> source scripts/tabledefs.sql;
-
-Note: if you already have a current IPP database that you would like to use
-for the Postage Stamp Request tables, when skip you can skip the create step
-and use that database.
-
+The tables for the PSS are part of the regular IPP database tables.
+
+The Data Store implementation has been designed to not require the IPP for its
+operation so the Data Store's tables are not created as part of the IPP database setup.
+
+Currently however, the PostageStamp Server implementation assumes that the data
+store and postage stamp tables are in the same mysql database (this will be
+fixed soon).
+
+To set up the database go to the directory DataStoreServer/scripts and start up mysql.
+
+If you do not already have a current IPP database that you would like to use
+for the Postage Stamp Request tables issue the following commands
+
+    mysql> create database mydatabase;
+    mysql> use mydatabase;
+    mysql> source scripts/tabledefs.sql;
+
+Now exit mysql and run the command
+    pxadmin -create -dbname mydatabase
+
+If you already have a database set up, then just issue the commands
+    mysql> use mydatabase;
+    mysql> source scripts/tabledefs.sql;
 
 Web Server Configuration
@@ -170 +184 @@
         ScriptAlias /ds-cgi /DATASTOREDIR/ds-cgi/
 
+(another sample of this configuration file may be found in 
+    DataStoreServer/web/conf/httpd-datastore.conf
+
 Here we have assumed that the data store cgi scripts will be installed in
 /DATASTOREDIR/ds-cgi and the data store data files will be stored in
@@ -282 +299 @@
     state       enabled
     dbname      ps_simtest
-    dvodb       currently not used
+    dvodb       NULL      (currently not used)
     camera      SIMTEST
     telescope   SimScope
     need_magic  0
 
-The column dbname containes the name of the database that is used to look up images. To
-create a project use pstamptool. In this case the ps_simtest is a database
+The column dbname contains the name of the database that is used to look up images.
+To create a project use pstamptool. In this example ps_simtest is a database
 refererencing images made by a simtest run.
 
@@ -298 +315 @@
 
 Postage Stamp requests are submitted in the form of a FITS binary table. The
-format of this table is described in a document which can be found by going
-to the page 
-
-    PostageStampServer
-
-on the IPP wiki.
-
-
-The tables for the PSS are part of the regular IPP database
-tables. The Data Store implementation has been designed to not require the IPP for it's
-operation. 
-So the Data Store's tables are not created as part of the ipp database setup.
-
-However, the current PostageStamp Server implementation assumes that the data
-store and postage stamp tables are in the same mysql database (this will be
-fixes soon).
-
-As will be explained below, the request database does not necessarily have to be the
-same as the image database. 
+format of this table is described in a document which may be found on the IPP wiki.
+http://kiawe.ifa.hawaii.edu/IPPwiki/index.php/PostageStampServer
+
+Only the first table extension found in the request file is processed. 
+
+Each postage stamp request file submitted to the PSS generates a "Request".
+The name of the request is given by the value for the keyword REQ_NAME in
+the extension's fits header. Each postage stamp request must have a unique REQ_NAME must be unique.
+
+To process the request the PSS parses the request file.
+Each row in a request table contains a request specification. Each request specification
+causes zero or more "Jobs" to be queued for processing. Each Job operates on
+one input image.
+
+When all of the Jobs for a given Request complete, a Postage stamp results
+file is created and it along with the images produced by the jobs are
+registered in a fileset on the output Data Store.
+
+There are two job types specified by the value for the JOB_TYPE keyword in the
+request specification.
+
+    JOB_TYPE = "stamp" creates a "postage stamp" image.
+    JOB_TYPE = "get_image" builds a fileset containing copies of images
+
+There are two programs that may be used to create a postage stamp request.
+pstamprequest creates a request file with a single request specification.
+
+For example
+
+    pstamprequest -req_name REQ_00001 -project simtest req.fits \
+                  -byexp chip simtest.004.000 null \
+                  -pixcenter 500 500 -pixrange 100 100
+
+creates a request file req.fits containing a request specification for a 100 x 100 pixel 
+postage stamp image centered at (500,500) in the source.
+The source image is chip processed image for one of the object exposures from a simtest run.
+
+Another program that may be used to create a request file is located in
+pstamp/test/pstamp_req_create.
+
+This program a text file and creates a multi-row postage stamp request.
+
+
 
 XXXXXXXXXXXXXXXXXXXXXX The following sections are to be completed
@@ -322 +363 @@
 --------------------
 pantasks tasks
-Database
+
 DataStore for input and output
 Web interface (limited functionality prototype only)
 
 
-Request files
-Link to Spec
-
-
 
 Examples

trunk/pstamp/scripts/pstamp_finish.pl

@@ -110 +110 @@
     my ($rlf, $reglist_name) = tempfile ("$out_dir/reglist.XXXX", UNLINK => !$save_temps);
     print $rlf "results.fits|||table|\n";
+
+    # XXX: this file is in the request's workdir not the out_dir
     my $err_file = "$out_dir/parse_error.txt";
     if (-e $err_file ) {
@@ -138 +140 @@
             # including those that produced no jobs.
             # for now add an entry for rownum 1 and a phony error code.
-            # we've included parse_results.txt to the fileset if it exists
+            # we've included parse_error.txt to the fileset if it exists
             #
             my $rownum = 0;

trunk/pstamp/scripts/pstamp_results_file.pl

@@ -91 +91 @@
 
         # output parameters
-#        { name => 'STAMP_NAME', type => '16A', writetype => TSTRING },  
         { name => 'OPTION_MASK',type => 'J', writetype   => TULONG },     
 

trunk/pstamp/test/pstamp_req_create

@@ -49 +49 @@
         { name => 'JOB_TYPE',   type => '16A', writetype => TSTRING },
 
-        { name => 'STAMP_NAME', type => '16A', writetype => TSTRING },
         { name => 'OPTION_MASK',type => 'J',   writetype => TULONG },
 

trunk/pstamp/test/sample_pstamp.txt

@@ -15 +15 @@
 # subsequent lines define the rows in the table
 
-# ROWNUM PROJECT       JOB_TYPE STAMP_NAME OPTION_MASK REQ_TYPE IMG_TYPE ID     CLASS_ID COORD_MASK CENTER_X CENTER_Y WIDTH HEIGHT REQFILT MJD_MIN MJD_MAX
-1        megacam-mops   stamp    null       1           byid     chip      419       null        3   1500    1500     100    100     null   0    0
-2        megacam-mops   stamp    null       1           byid     chip      419       null        3   1600    1500     100    100     null   0    0
-3        megacam-mops   stamp    null       1           byid     chip      419       null        3   1700    1500     100    100     null   0    0
+# ROWNUM PROJECT       JOB_TYPE OPTION_MASK REQ_TYPE IMG_TYPE    ID  CLASS_ID COORD_MASK CENTER_X CENTER_Y WIDTH HEIGHT REQFILT MJD_MIN MJD_MAX
+1        megacam-mops   stamp     1           byid     chip      419       null        3   1500    1500     100    100     null   0    0
+2        megacam-mops   stamp     1           byid     chip      419       null        3   1600    1500     100    100     null   0    0
+3        megacam-mops   stamp     1           byid     chip      419       null        3   1700    1500     100    100     null   0    0
 
 # the following row will create several jobs