| 22 | | = Configuration = |
| | 22 | |
| | 23 | = Architecture and design = |
| | 24 | |
| | 25 | == Outline == |
| | 26 | |
| | 27 | Essentially, the job of ippToPsps is to take a collection of IPP tables and convert them to tables suitable for ingestion to PSPS. Some mappings between IPP and PSPS are direct (eg IPP exposure ID = PSPS frame ID), while many require a format conversion, or can only be derived from multiple IPP fields; some require mining info from other IPP sources, such as DVO databases or the IPP MySQL database, {{{gpc1}}}. |
| | 28 | |
| | 29 | == Languages and tools used == |
| | 30 | |
| | 31 | === MySQL === |
| | 32 | |
| | 33 | Because we are dealing with table data that requires some intensive manipulation, it follows that a relational database be used. Relational databases are highly optimized to provide the fastest possible query times, especially when indexing in incorporated. Thus we gain speed over the more obvious route of read-a-table-from-FITS-into-an-array-then-loop-through-each-value etc. |
| | 34 | |
| | 35 | An added bonus is that, by using keys that enforce uniqueness in a given column, we protect ourselves against the risk of duplicates making into PSPS. |
| | 36 | |
| | 37 | {{{ippToPsps}}} uses two MySQL databases, a 'scratch' database, used to import tables and manipulate then, and the 'ippToPsps' database, which keeps track of which batches have been processed, published to PSPS etc. |
| | 38 | |
| | 39 | === Jython and STILTS === |
| | 40 | |
| | 41 | {{{ippToPsps}}} is written in [http://www.jython.org/ Jython], this is in part to take full advantage of the [http://www.star.bris.ac.uk/~mbt/stilts/ STILTS] package, which enables very fast and efficient processing of astronomical catalog data tables. Since it supports FITS, VOTable, and SQL, it is a perfect fit for this project. It is also software maintained elsewhere, reducing this burden for us. |
| | 42 | |
| | 43 | Jython is simply a Java implementation of Python, a modern, high-level and object-oriented language enabling {{{ippToPsps}}} to be written in a minimal number of lines of code, thus helping it to be both more readable and maintainable. |
| | 44 | |
| | 45 | == High-level design == |
| | 46 | |
| | 47 | Each batch type produced by {{{ippToPsps}}} (detection, stack, init etc) has its own class, all of which inherit from the Batch base-class, which handles features common to all batches, such as creating and opening output FITS files, connecting to the GPC1 database, connecting with the DVO database etc. This keeps duplicated code at a minimum. The Batch class is an abstract class, i.e. it should not, and cannot, be instantiated. |
| | 48 | |
| | 49 | {{{ippToPsps}}} works like this: |
| | 50 | |
| | 51 | - reads all relevant FITS tables from a given {{{smf}}} or {{{cmf}}} into temporary tables in a 'scratch' MySQL database |
| | 52 | - creates empty MySQL tables for PSPS output (also in the 'scratch' database). These tables match the shape of the final PSPS database tables exactly. |
| | 53 | - copies all relevant columns from the temporary IPP tables into the PSPS tables, discarding duplicates where necessary |
| | 54 | - accesses the DVO database and creates temporary MySQL tables containing all the detections for this {{{smf}}} or {{{cmf}}} |
| | 55 | - updates the PSPS tables with the IDs from the DVO MySQL tables |
| | 56 | - replaces any NULL values with -999, as required by the PSPS loader |
| | 57 | - exports the PSPS tables to a FITS file |
| | 58 | - publishes FITS file, complete with a batch 'manifest' file, to the datastore |
| | 59 | |
| | 60 | The reading and export of FITS tables is done using STILTS. For import, we can specify which columns we wish to import from the IPP smf and cmf files (we don't need everything). |
| | 61 | |
| | 62 | |
| | 63 | == Configuration == |
| 28 | | = Architecture = |
| 29 | | |
| 30 | | {{{ippToPsps}}} is written in [http://www.jython.org/ Jython], this is in part to take full advantage of the [http://www.star.bris.ac.uk/~mbt/stilts/ STILTS] package, which enables very fast and efficient processing of astronomical catalog data tables. Since it supports FITS, VOTable, and SQL, it is a perfect fit for this project. It is also software maintained elsewhere, reducing this burden for us. |
| 31 | | |
| 32 | | == Design == |
| 33 | | |
| 34 | | Each batch type (detection, stack, init etc) is its own class, all of which inherit from the Batch base-class, which handles features common to all batches, such as creating and opening output FITS files, connecting to the GPC1 database, connecting with the DVO database etc. This keeps duplicated code at a minimum. The Batch class is an abstract class, i.e. it should not, and cannot, be instantiated. |
| 35 | | |
| 36 | | |
| 37 | | = Running = |
| | 69 | = Using the software = |
