IPP to PSPS interface: ippToPsps

ippToPsps is the interface between IPP and PSPS. In short, ippToPsps creates FITS files from IPP data, then publishes them to a datastore in the form of batches. On the PSPS side, the DXLayer polls the datastore, collects batches when they become available, then converts the contents to csv files before sending them on to SQL Server loader software, which merges them into the PSPS database. Ultimately there will be feedback from PSPS regarding errors in the received data, to which ippToPsps will need to respond.

It is intended that the binary tables in the FITS files generated by ippToPsps match the PSPS database schemas perfectly, the consequence being that any alterations to the PSPS database schema will only affect ippToPsps code, and not the DXLayer. A certain amount of data validation will be performed by ippToPsps before publication, more validation occurring at the loading and merge stages on the PSPS side.

The outputs of ippToPsps are referred to as 'batches', and are detailed below.

Batch name	PSPS name	Description	IPP Source
Initialisation	IN	Metadata for the other batches eg filter ID, survey ID etc	generated from XML config
Detection	P2	Single exposure detections	generated from one smf file per exposure plus associated DVO database
Difference	?	difference image detections	generated from one cmf file per skycell per exposure
Stack	ST	stack image detections	generated from...

Configuration

Due to the potential for changes in both input and output for ippToPsps, the code is heavily configurable. Configuration files are in an XML format as this affords the most flexibility (human and machine readable, expandable, self-describing etc). ippToPsps is pointed to a config directory, under which subdirectories for each batch type hold the various XML config files.

Table shapes

All FITS tables mirror PSPS database tables. Since the PSPS schema will probably remain in a state of flux for some time, rather than hard-coding table descriptions, instead ippToPsps reads table shapes from an XML config. This config can be regenerated from the master PSPS schema using a Perl script (pspsSchema2xml.pl) in the scripts directory. The same script also generates C header files for each batch-type. These headers contains enums for each PSPS table and are used by the code at runtime. This helps minimise code changes.

Initialisation data

The table shapes of the initialisation batch are handled as above. The actual initialisation data (lists of filters etc), which is liable to change, is held in a config and used by ippToPsps to populate the tables in the FITS file. This data is also used when generating other batch types, detections for example, as look-up tables for setting survey ID etc.

IPP to PSPS mappings

Most data to be loaded into the FITS tables comes from IPP smf or cmf files. For many columns, there is a direct mapping between these files and the PSPS database column. These mappings are detailed in a config.

Architecture

ippToPsps

ippToPsps is a C program within the IPP build. When given the correct arguments it will generate a single FITS for the specified product (above). The program is run from a Perl script, which itself generates a list of exposure IDs based on arguments provided by the user (label etc). An instance of ippToPsps is run per exposure ID. Upon completion, the calling script bundles the resultant FITS files up as a batch, then publishes it to the datastore, ready for collection by the DXLayer.

DXLayer

The DXLayer polls the datastore waiting for new batches. Upon receipt of a new batch, the FITS files are converted to a csv format, suitable for ingest by the ODM.

ODM Loader

Performs validation on incoming data based on metadata previously loaded as an initialisation batch (see above). If validation is successful, new batches are merged into the PSPS database. One basic requirement of the ODM is that all detections in a detection batch have unique object IDS. Object IDs are assigned by the IPP DVO to each detection on a chip. The number is formulated from the Ra and Dec of the detection.

Notes about the different batch types

Detections

The input for the detection batch is one IPP camera-stage smf file for a given exposure, as well as an associated DVO database from which to retrieve object, and other, IDs. One FITS file is generated for each exposure. The extensions are:

1 Primary extension
1 FrameMeta extension
1 ImageMeta extension per chip
1 Detection extension per chip
1 SkinnyObject extension per chip
1 ObjectCalColor extension per chip

So, 242 extensions in all, including the obligatory primary header. The 'object ID' is featured in the last three tables, and must remain unique across the exposure (it is generated within DVO). In the merged PSPS database, the primary key on the detections table is both the object ID and detection ID, meaning the the same object can appear in multiple, overlapping exposures as they will have different detection IDs.

Diffs

The input for difference batches is a set of cmf files, one for each skycell covered by a particular exposure. A FITS output file is generated with the following extensions:

Unresolved fields

Below are tables detailing which fields in the PSPS FITS files are still not populated by ippToPsps.

Unresolved fields for camera stage detections

FrameMeta
PSPS field	PSPS type	PSPS Description	IPP source	Comments
analysisVer	SMALLINT	version number of analysis		pending
p1Recip	SMALLINT	IPP phase 1 recipe ID		pending
p2Recip	SMALLINT	IPP phase 2 recipe ID		pending
p3Recip	SMALLINT	IPP phase 3 recipe ID		pending
numPhotoRef	INT	number of photometric reference sources		have values per chip, but global?
ImageMeta
bias	REAL	detector bias level		pending
biasScat	REAL	scatter in bias level		pending
numPhotoRef	INT	number of photometric reference sources	-999 for now
psfModelID	INT	PSF model identifier		pending
qaFlags	INT	Q/A flags		Get from DVO...
Detection
psfLikelihood	REAL	PSF likelihood	???
extendedLikelihood	REAL	Likelihood the source is extended
infoFlag	LONG	flag indicating provenance information
activeFlag	LONG	indicates whether this detection/orphan is still a detection/orphan
SkinnyObject
projectionCellID	LONG	projection cell identifier at discovery time
ObjectCalColor
calColor	FLOAT	color adopted for magnitude calculation (unit = mag)
calColorErr	FLOAT	error in calibrating color (unit = mag)

Unresolved fields for diff stage

StackMeta
PSPS field	PSPS type	PSPS Description	IPP source	Comments
stackMetaID	BIGINT	stack identifier
skycellID	INT	skycell region identifier		missing from cmf files
surveyID	TINYINT	survey flag identifier		e.g. 3PI, M31, MD, STS - need common enum
photoCalID	INT	photometry code numerical id
filterID	TINYINT	filter identifier		IPP uses string (g.00000). Need enum.
stackTypeID	TINYINT	stack type identifier
refImageID	BIGINT	identifier of image used as reference for analysis
subtrImageID	BIGINT	identifier of image subtracted to generate difference image
analVer	SMALLINT	analysis version index
nP2Images	SMALLINT	number of P2 images contributing to this cell
astroScat	FLOAT	0.astrometric scatter for chip
photoScat	FLOAT	0.photometric scatter for chip
nAstroRef	INT	number of astrometric reference sources
nPhoRef	INT	number of photometric reference sources
psfFwhm	FLOAT	PSF full width at half maximum
psfmodelID	SMALLINT	PSF model identifier
psfExtra1	FLOAT	PSF parameters
psfExtra2	FLOAT	PSF parameters
photoZero	FLOAT	local derived photometric zero point
photoColor	FLOAT	local derived photometric color term
polyOrder	TINYINT	polynomial order of astrometry fit
pca1x3y0	FLOAT	polynomial coefficients for the astrometric fit		not available
pca1x2y1	FLOAT	polynomial coefficients for the astrometric fit		not available
pca1x1y2	FLOAT	polynomial coefficients for the astrometric fit		not available
pca1x0y3	FLOAT	polynomial coefficients for the astrometric fit		not available
pca1x2y0	FLOAT	polynomial coefficients for the astrometric fit		not available
pca1x1y1	FLOAT	polynomial coefficients for the astrometric fit		not available
pca1x0y2	FLOAT	polynomial coefficients for the astrometric fit		not available
pca2x3y0	FLOAT	polynomial coefficients for the astrometric fit		not available
pca2x2y1	FLOAT	polynomial coefficients for the astrometric fit		not available
pca2x1y2	FLOAT	polynomial coefficients for the astrometric fit		not available
pca2x0y3	FLOAT	polynomial coefficients for the astrometric fit		not available
pca2x2y0	FLOAT	polynomial coefficients for the astrometric fit		not available
pca2x1y1	FLOAT	polynomial coefficients for the astrometric fit		not available
pca2x0y2	FLOAT	polynomial coefficients for the astrometric fit		not available
calibModNum	SMALLINT	calibration modification number
dataRelease	TINYINT	Data release
StackToImage
stackMetaID	BIGINT	stack identifier
imageID	BIGINT	hashed exposure-ccdID identifier
StackLowSigDelta/StackHighSigDelta
partitionKey	BIGINT	partition key
stackDiffID	BIGINT	ODM detection identifier
ippDetectID	BIGINT	detection id generated by IPP
filterID	TINYINT	filter ID: g=1, r=2, i=3, z=4, y=5, w=6,..
psfLikelihood	FLOAT	PSF likelihood
infoFlag	INT	indicator of strange propeties
nFrames	INT	number of frames contributing to source
htmID	BIGINT	HTM index for the detection
zoneID	INT	Zone index
raErr	FLOAT	estimated error in ra
decErr	FLOAT	estimated error in dec
cx	FLOAT	cartesian x
cy	FLOAT	cartesian y
cz	FLOAT	cartesian z
calMag	FLOAT	calibrated magnitude
calMagErr	FLOAT	estimated error in calMag
dataRelease	TINYINT	Data release when this detection was taken

Unresolved fields for stack stage

Not yet...

Links

​Datastore test area for PSPS