| 12 | | {{{ippToPsps}}} is the interface between IPP and PSPS. In short, {{{ippToPsps}}} creates FITS files from data found in IPP {{{smf}}} and {{{cmf}}} files (as well as relevant DVO databases), then publishes them to a datastore in the form of ''jobs''. On the PSPS side, the DXLayer polls the datastore, collects jobs when they become available, then converts the contents to {{{csv}}} files before sending them on to special MSSQL loader software, which ''merges'' them into the PSPS database. Ultimately there will be feedback from PSPS regarding errors in the received data, to which {{{ippToPsps}}} will need to respond. |
| 13 | | |
| 14 | | It is intended that the binary tables in the FITS files generated by {{{ippToPsps}}} will match the PSPS database schemas perfectly, the consequence being that any alterations to the PSPS database schema will only affect {{{ippToPsps}}} code, and not the DXLayer. A certain amount of data validation will be performed by {{{ippToPsps}}} before publication. |
| | 12 | {{{ippToPsps}}} is the interface between IPP and PSPS. In short, {{{ippToPsps}}} creates FITS files from IPP {{{smf}}} and {{{cmf}}} files (as well as relevant DVO databases), then publishes them to a datastore in the form of ''jobs''. On the PSPS side, the DXLayer polls the datastore, collects jobs when they become available, then converts the contents to {{{csv}}} files before sending them on to special MSSQL loader software, which ''merges'' them into the PSPS database. Ultimately there will be feedback from PSPS regarding errors in the received data, to which {{{ippToPsps}}} will need to respond. |
| | 13 | |
| | 14 | It is intended that the binary tables in the FITS files generated by {{{ippToPsps}}} match the PSPS database schemas perfectly, the consequence being that any alterations to the PSPS database schema will only affect {{{ippToPsps}}} code, and not the DXLayer. A certain amount of data validation will be performed by {{{ippToPsps}}} before publication, more validation occurring at the loading and merge stages on the PSPS side. |
| | 15 | |
| | 16 | The 'products', or 'batch types' generated by {{{ippToPsps}}} are as follows: |
| | 17 | |
| | 18 | * ''''Initialisation' batch (IN)'''[[BR]] |
| | 19 | * ''''Detection' batch (P2)'''[[BR]] |
| | 20 | * ''''Difference' batch (OB)'''[[BR]] |
| | 21 | * ''''Stack' batch (ST)'''[[BR]] |
| | 22 | |
| | 23 | The initialisation batch contains numerous metadata tables that describe the data in the other batches, for example, filter IDs, survey IDs etc. |
| 20 | | {{{ippToPsps}}} is a C program within the IPP build. When given the correct arguments it will generate a single FITS for the specified product (above). The program is run from a perl script, which itself generates a list of exposure IDs based on arguments provided by the user (label etc). An instance of {{{ippToPsps}}} is run per exposure ID. Upon completion, the calling script publishes bundles the resultant FITS files up as a ''job'', then publishes it to the datastore, ready for the DXLayer to pick it up. |
| 21 | | |
| 22 | | == Configuration == |
| 23 | | |
| 24 | | {{{ippToPsps}}} uses XML configuration files, as this affords the most flexibility. |
| 25 | | |
| 26 | | === FITS table shapes === |
| 27 | | |
| 28 | | XML files detailing PSPS database tables are used to automatically generate equivalent FITS tables. Because the PSPS schema can change, and we want to avoid code changes as much as possible, these XML files are created using a script ({{{pspsSchema2xml.pl}}}) so that whenever the schema changes, new configs can easily be generated. The table shape configs are: |
| 29 | | |
| 30 | | {{{tables_init.xml}}}[[BR]] |
| 31 | | {{{tables_detection.xml}}}[[BR]] |
| 32 | | {{{tables_diff.xml}}}[[BR]] |
| 33 | | {{{tables_stack.xml}}}[[BR]] |
| 34 | | |
| 35 | | === FITS table contents === |
| 36 | | |
| 37 | | Most data to be loaded into the FITS tables comes from {{{smf}}} files. For many columns, there is a direct mapping between the the {{{smf}}} and the PSPS database column. These mappings are held, in XML format, in ... |
| 38 | | |
| 39 | | The initialisation batch contains metadata that describes valid contents for the other batch types. Because these values will be in a state of flux for awhile, these are held in an editable XML file as... |
| 40 | | |
| 41 | | == DXLayer == |
| | 29 | {{{ippToPsps}}} is a C program within the IPP build. When given the correct arguments it will generate a single FITS for the specified product (above). The program is run from a perl script, which itself generates a list of exposure IDs based on arguments provided by the user (label etc). An instance of {{{ippToPsps}}} is run per exposure ID. Upon completion, the calling script bundles the resultant FITS files up as a ''job'', then publishes it to the datastore, ready for collection by the DXLayer. |
| | 30 | |
| | 31 | = Configuration = |
| | 32 | |
| | 33 | Due to the potential for changes in both input and output for {{{ippToPsps}}}, the code is heavily configurable. Config files are in an XML format as this affords the most flexibility. {{{ippToPsps}}} is pointed to a config directory, under which subdirectories for each batch type hold the various XML config files. |
| | 34 | |
| | 35 | == Table shapes == |
| | 36 | |
| | 37 | All FITS tables mirror PSPS database tables. Since the schema may change, {{{ippToPsps}}} reads table shapes from an XML config. This config can be regenerated from the master PSPS schema using a script ({{{pspsSchema2xml.pl}}}) in the scripts directory. |
| | 38 | |
| | 39 | == Initialisation data == |
| | 40 | |
| | 41 | The table shapes of the initialisation batch are handled as above. The actual initialisation data (lists of filters etc), which is liable to change, is held in an XML config and used by {{{ippToPsps}}} to populate the tables in the FITS file. This data is also used when generating other batch types, detections for example, as look-up tables for setting survey ID etc. |
| | 42 | |
| | 43 | == IPP to PSPS mappings == |
| | 44 | |
| | 45 | Most data to be loaded into the FITS tables comes from IPP {{{smf}}} or {{{cmf}}} files. For many columns, there is a direct mapping between these files and the PSPS database column. These mappings are detailed in a config. |
| | 46 | |
| | 47 | = DXLayer = |
