IPP Data Release Management

The IPP does not have a good way of keeping track of what data is intended for or has been released to the PS1SC community. Up until now this has not been a big problem, but several issues have recently led us to believe that some new functionality is necessary to make it easy for users to get access to IPP data products that meet their expectations.

The specific goals are

• To record which versions of the data products have been (and should be) "released" to the community through the various distribution channels.
• To allow users to get the "right" versions of the data in their requests to the postage stamp server
• To record within the IPP domain the results of various re-calibrations and re-processings that may occur

These goals will be met with some enhancements to the IPP data base schema and through some lightweight changes to the procedures for processing and releasing data.

IPP Release Channels

Data is released through several channels.

• The IPP Distribution data store
• The Published Science Products Subsystem (PSPS)
• The IPP Postage Stamp Server
• DVO catalogs

The data is the results of runs through the various stages of the IPP pipeline. For the most part the IPP does not have a very good record of which data went into the various releases. For example given a single frame detection in the IPP database, there is no direct method to determine which version of the processed pixels the detections were measured on. Making the answer to this and other similar questions easy to obtain is the primary purpose of this project.

IPP Data Products

There are three categories of data products to be covered by the IPP release mechanism

• Single Frame Data Products
• Stacked Image Data Products
• Difference Image Data Products

Single Frame Results

Single exposures move through the IPP pipeline in a number of stages. The data products of interest to end users are the results of the chip and camera stage (focal plane coordinates) and the warp stage. The end results are two sets of images and catalogs of detected sources. The catalogs are then imported into the DVO system and eventually into a PSPS data base. Exposures are also queued for difference processing which will be discussed separately.

Nightly Science

On the night that the exposures are acquired they are run through the IPP "Nightly Science" processing. The resulting catalogs are posted to file sets on the IPP Data Store web server. For certain exposures warped images are placed on the data store as well. This is the first "release" for a given exposure.

The catalog files (smf and cmf) currently remain on the data store indefinitely. For various reasons this needs to be changed. A new system for on demand access to these files is under development. The image files posted to the data store are removed after some time to conserve space.

DVO

Occasionally the DVO catalogs are made available to the community. These consist of calibrated measurements and object associations of individual exposures. We currently have no method of directly tracking in the IPP database whether or not the measurements for a particular exposure were "released" in a given DVO catalog or for remembering the results of the calibrations performed by the DVO programs. This information can be figured out from the DVO tables, but since they are subject to change this information can become "stale".

"Grand" Reprocessing

As time goes on and the algorithms of the IPP improve and the exposures are reprocessed from scratch. This is commonly known as the grand reprocessing. Internally the IPP team refers to this process for the 3Pi survey as "Large Area Processing". For the medium deep fields, the terminology has changed over time but the concepts are somewhat similar.

The results of this processing are added to a new DVO database and from there into PSPS. We currently do not have an efficient method of tracking which sets of processed pixels went into a given PSPS database.

Various Test runs

We currently have no way of tracking which of the various processing runs are intended for end user use and those that are the results of testing. The postage stamp server currently attempts to work around this problem but the results have been mixed. By specifically identifying the data that should be released this problem will be solved.

Stacked Images

The warped images are combined to form stacked images. While for single frames the logical unit is the exposure, stacks are performed on areas of the sky we call skycells. A skycell has a tessellation ID and a skycell number. For example (RINGS.V3, skycell.1406.016) or (MD09.V3, skycell.042)

Three types of stacks have been identified

• Nightly Stacks - constructed from exposures on a single night
• Reference Stacks - constructed from exposures with good image quality that provide good coverage of a region of the sky
• Deep Stacks - constructed from virtually all "good" exposures for a given area of the sky

For the medium deep surveys all three stack types are used. Deep Stacks provide the greatest sensitivity, while reference stacks are used as templates for difference imaging and science that requires the highest resolution. Nightly stacks are used for transient detection.

For the 3PI survey the stacks are produced in the LAP processes. For the purposes of this discussion they are considered "Deep Stacks" even though they are made from significantly fewer input exposures.

Right now the various "releases" of stacks are managed in an ad hoc manner. It is very difficult for a user without access to the IPP database and detailed knowlege of the schema to track what data corresponds to a particular release.

Difference Images

Management of difference images is a subject for further research and will be discussed later.