Changes between Version 37 and Version 38 of ippToPsps
- Timestamp:
- Mar 18, 2010, 12:45:37 PM (16 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
ippToPsps
v37 v38 14 14 It is intended that the binary tables in the FITS files generated by {{{ippToPsps}}} match the PSPS database schemas perfectly, the consequence being that any alterations to the PSPS database schema will only affect {{{ippToPsps}}} code, and not the DXLayer. A certain amount of data validation will be performed by {{{ippToPsps}}} before publication, more validation occurring at the loading and merge stages on the PSPS side. 15 15 16 The outputs of {{{ippToPsps}}} are referred to as 'batches', and 16 The outputs of {{{ippToPsps}}} are referred to as 'batches', and are detailed below. 17 17 18 18 ||'''Batch name''' || '''PSPS name''' || '''Description''' || '''IPP Source''' || … … 22 22 || Stack || ST || stack image detections || generated from... || 23 23 24 [[BR]] 24 25 25 26 = Configuration = 26 27 27 Due to the potential for changes in both input and output for {{{ippToPsps}}}, the code is heavily configurable. Config files are in an XML format as this affords the most flexibility . {{{ippToPsps}}} is pointed to a config directory, under which subdirectories for each batch type hold the various XML config files.28 Due to the potential for changes in both input and output for {{{ippToPsps}}}, the code is heavily configurable. Config files are in an XML format as this affords the most flexibility (XML is both human and machine readable, expandable, self-description etc). {{{ippToPsps}}} is pointed to a config directory, under which subdirectories for each batch type hold the various XML config files. 28 29 29 30 == Table shapes == 30 31 31 All FITS tables mirror PSPS database tables. Since the schema may change, {{{ippToPsps}}} reads table shapes from an XML config. This config can be regenerated from the master PSPS schema using ascript ({{{pspsSchema2xml.pl}}}) in the scripts directory.32 All FITS tables mirror PSPS database tables. Since the PSPS schema will probably remain in a state of flux for some time, rather than hard-coding table descriptions, instead {{{ippToPsps}}} reads table shapes from an XML config. This config can be regenerated from the master PSPS schema using a Perl script ({{{pspsSchema2xml.pl}}}) in the scripts directory. 32 33 33 34 == Initialisation data == 34 35 35 The table shapes of the initialisation batch are handled as above. The actual initialisation data (lists of filters etc), which is liable to change, is held in a n XMLconfig and used by {{{ippToPsps}}} to populate the tables in the FITS file. This data is also used when generating other batch types, detections for example, as look-up tables for setting survey ID etc.36 The table shapes of the initialisation batch are handled as above. The actual initialisation data (lists of filters etc), which is liable to change, is held in a config and used by {{{ippToPsps}}} to populate the tables in the FITS file. This data is also used when generating other batch types, detections for example, as look-up tables for setting survey ID etc. 36 37 37 38 == IPP to PSPS mappings == 38 39 39 40 Most data to be loaded into the FITS tables comes from IPP {{{smf}}} or {{{cmf}}} files. For many columns, there is a direct mapping between these files and the PSPS database column. These mappings are detailed in a config. 40 41 41 42 42 = Architecture = … … 52 52 == ODM Loader == 53 53 54 Performs validation on incoming data based on metadata previously loaded as an initialisation batch (see above). If validation is successful ly, new batches are merged into the PSPS database.54 Performs validation on incoming data based on metadata previously loaded as an initialisation batch (see above). If validation is successful, new batches are merged into the PSPS database. One basic requirement of the ODM is that all detections in a detection batch have unique object IDS. Object IDs are assigned by the IPP DVO to each detection on a chip. The number is formulated from the Ra and Dec of the detection. 55 55 56 = Batch types =56 = Notes about the different batch types = 57 57 58 58 == Detections == 59 59 60 '''Input''' 61 62 Input is one IPP camera-stage {{{smf}}} file for a given exposure and an associated DVO database from which to retrieve object, and other, IDs. 63 64 '''Output''' 60 The input for the detection batch is one IPP camera-stage {{{smf}}} file for a given exposure, as well as an associated DVO database from which to retrieve object, and other, IDs. One FITS file is generated for each exposure. The extensions are: 65 61 66 62 A FITS output file is generated with the following extensions: 67 63 68 64 1 '''{{{Primary}}}''' extension[[BR]] 65 1 '''{{{FrameMeta}}}''' extension[[BR]] 66 1 '''{{{ImageMeta}}}''' extension per chip[[BR]] 67 1 '''{{{Detection}}}''' extension per chip[[BR]] 68 1 '''{{{SkinnyObject}}}''' extension per chip[[BR]] 69 1 '''{{{ObjectCalColor}}}''' extension per chip[[BR]] 69 70 70 1 '''{{{FrameMeta}}}''' extension[[BR]] 71 72 1 '''{{{ImageMeta}}}''' extension per chip[[BR]] 73 74 1 '''{{{Detection}}}''' extension per chip[[BR]] 75 76 So, 122 extensions in all, included the obligatory primary header. 77 78 '''Validation''' 79 80 Some validation is performed to aid the data's insertion into the PSPS database. 71 So, 242 extensions in all, including the obligatory primary header. 81 72 82 73 == Diffs == 83 74 84 '''Input''' 75 The input for difference batches is a set of {{{cmf}}} files, one for each skycell covered by a particular exposure. A FITS output file is generated with the following extensions: 85 76 86 A set of {{{cmf}}} files, one for each skycell covered by a particular exposure.87 88 '''Output'''89 90 A FITS output file is generated with the following extensions:91 92 ...93 94 '''Validation'''95 77 96 78 = Unresolved fields =
