Changes between Version 138 and Version 139 of ippToPsps

Timestamp:

Aug 3, 2011, 8:51:36 AM (15 years ago)

Author:

rhenders

Comment:

--

ippToPsps

@@ -29 +29 @@
 Essentially, the job of ippToPsps is to take a collection of IPP tables and convert them to tables suitable for ingestion to PSPS. Some mappings between IPP and PSPS are direct (eg IPP exposure ID = PSPS frame ID), while many require a format conversion, or can only be derived from multiple IPP fields; some require mining info from other IPP sources, such as DVO databases or the IPP MySQL database, {{{gpc1}}}.
 
-The tools chosen for {{{ippToPsps}}} are those considered to be the most effective to tackle the task at hand. {{{ippToPsps}}} is not, therefore, consistant with the rest of the IPP code-base, but since its role is essentially ''outside'' the IPP, this, to me, is not an issue.
 
 == Languages and tools used ==
 
+The tools chosen for {{{ippToPsps}}} are those considered to be the most effective to tackle the task at hand. As a result, {{{ippToPsps}}} is not consistant with the rest of the IPP code-base (which is, predominately, Perl and C), but since its role is essentially ''outside'' the IPP, this, to me, is not an issue.
+
 === MySQL ===
 
-Because we are dealing with table data that requires some intensive manipulation, it follows that a relational database be used. Relational databases are highly optimized to provide the fastest possible query times, especially when indexing in incorporated. Thus we gain speed over the more obvious route of read-a-table-from-FITS-into-an-array-then-loop-through-each-value etc.
+Because we are dealing with table data that requires some intensive manipulation, it follows that a relational database be used. Relational databases are highly optimized to provide extremely fast query times, especially when indexing in incorporated. Thus we gain speed over the more obvious route of read-a-table-from-FITS-into-an-array-then-loop-through-each-value etc.
 
-An added bonus is that, by using keys that enforce uniqueness in a given column, we protect ourselves against the risk of duplicates making into PSPS.
+An added bonus is that, by using keys that enforce uniqueness in a given column (or columns), we protect ourselves against the risk of duplicates making into PSPS.
 
-{{{ippToPsps}}} uses two MySQL databases, a 'scratch' database, used to import tables and manipulate then, and the 'ippToPsps' database, which keeps track of which batches have been processed, published to PSPS etc.
+{{{ippToPsps}}} uses two MySQL databases, a ''scratch'' database, used to import tables and manipulate then before discarding them, and the ''ippToPsps'' database, which keeps track of which batches have been processed, published to PSPS etc.
 
 === Jython and STILTS ===
 
-{{{ippToPsps}}} is written in [http://www.jython.org/ Jython], this is in part to take full advantage of the [http://www.star.bris.ac.uk/~mbt/stilts/ STILTS] package, which enables very fast and efficient processing of astronomical catalog data tables. Since it supports FITS, VOTable, and SQL, it is a perfect fit for this project. It is also software maintained elsewhere, reducing this burden for us.
+{{{ippToPsps}}} is written in [http://www.jython.org/ Jython], this is in part to take full advantage of the [http://www.star.bris.ac.uk/~mbt/stilts/ STILTS] package, which enables very fast and efficient processing of astronomical catalog data tables. Since it supports FITS, VOTable, and SQL, it is a perfect fit for this project. It is also software maintained elsewhere, reducing this burden on us.
 
-Jython is simply a Java implementation of Python, a modern, high-level and object-oriented language enabling {{{ippToPsps}}} to be written in a minimal number of lines of code, thus helping it to be both more readable and maintainable.
+Jython is simply a Java implementation of Python, a modern, high-level and object-oriented language enabling {{{ippToPsps}}} to be written in a minimal number of lines of code, helping it be both more readable and maintainable.
 
 == High-level design ==
@@ -53 +54 @@
 {{{ippToPsps}}} works like this:
 
-- reads all relevant FITS tables from a given {{{smf}}} or {{{cmf}}} into temporary tables in a 'scratch' MySQL database
-- creates empty MySQL tables for PSPS output (also in the 'scratch' database). These tables match the shape of the final PSPS database tables exactly.
-- copies all relevant columns from the temporary IPP tables into the PSPS tables, discarding duplicates where necessary
-- accesses the DVO database and creates temporary MySQL tables containing all the detections for this {{{smf}}} or {{{cmf}}}
-- updates the PSPS tables with the IDs from the DVO MySQL tables
-- replaces any NULL values with -999, as required by the PSPS loader
-- exports the PSPS tables to a FITS file
-- publishes FITS file, complete with a batch 'manifest' file, to the datastore
+ * reads all relevant FITS tables from a given {{{smf}}} or {{{cmf}}} into temporary tables in a 'scratch' MySQL database
+ * creates empty MySQL tables for PSPS output (also in the 'scratch' database). These tables match the shape of the final PSPS database tables exactly.
+ * copies all relevant columns from the temporary IPP tables into the PSPS tables, discarding duplicates where necessary
+ * accesses the DVO database and creates temporary MySQL tables containing all the detections for this {{{smf}}} or {{{cmf}}}
+ * updates the PSPS tables with the IDs from the DVO MySQL tables
+ * replaces any NULL values with -999, as required by the PSPS loader
+ * exports the PSPS tables to a FITS file
+ * publishes FITS file, complete with a batch 'manifest' file, to the datastore
 
 The reading and export of FITS tables is done using STILTS. For import, we can specify which columns we wish to import from the IPP smf and cmf files (we don't need everything).
@@ -73 +74 @@
 = Using the software =
 
-Notes on how to run ippToPsps can be found on the individual batch-type pages, linked to at the top of this page.
+{{{ippToPsps}}} is configured for use entirely by a config file (see above). So, running the software is simply a case of editing a config and passing it as the sole argument to the program, like this:
+
+{{{
+cd trunk/ippToPsps/jython
+./run.sh ipptopsps.py someConfig.xml
+}}}
+
+The {{{./run.sh}}} prefix above is necessary to invoke the correct Java virtual machine, while including the relevant jar files in the CLASSPATH (all included in the {{{trunk/ippToPsps/Jars}}} subdir) 
 
 == Deletion policy ==