Update Processing Versus Magic - Problems and potential solutions

There will not be enough disk space in the IPP cluster to retain all data products for the PS1 survey.

The static sky images and catalogs will be retained but most of the processed images will be deleted sometime after they are processed and perhaps released. The deletion process is known as cleanup. During cleanup enough information is retained so that the pixels can be regenerated. The process of recreating the images is known as update.

When dealing with destreaked images performing 'updates' becomes somewhat complicated. This note discusses these issues and describes the approach that we propose to implement.

Overview

When a Run (chipRun, warpRun, or diffRun) is cleaned it's state in the database is set to 'cleaned'. The run is queued for update by changing it's state to 'update'. After the run has been updated it's state is set to 'full'.

When a run has dependencies for example a warpRun needs the dependent chipRun's to be in full state. The various pending queries of the ippTools enforce this.

We assume that the update is being performed because some user wants to see the pixels. For example the postage stamp server needs to queue update processing when stamps from cleaned images are requested. This means that the resulting images must have been de-streaked, i.e. detected satellite streaks must be masked.

There are two ways that the de-streaked requirement can be met.

• A. By performing the update processing on input images that have been de-streaked
• B. By re-running the destreak program on images containing the streak pixels

Plan A is much easier to implement in the context of normal IPP operations.