Update Processing Versus Magic - Problems and potential solutions

There will not be enough disk space in the IPP cluster to retain all data products for the PS1 survey.

The static sky images and catalogs will be retained but most of the processed images will be deleted sometime after they are processed and perhaps released. The deletion process is known as cleanup. During cleanup enough information is retained so that the pixels can be regenerated. The process of recreating the images is known as update.

When dealing with destreaked images performing 'updates' becomes somewhat complicated. This note discusses these issues and describes the approach that we propose to implement.

Overview

When a Run (chipRun, warpRun, or diffRun) is cleaned it's state in the database is set to 'cleaned'. The run is queued for update by changing it's state to 'update'. After the run has been updated it's state is set to 'full'.

When a run has dependencies for example a warpRun needs the dependent chipRun's to be in full state. The 'pending' queries for the various ippTools enforce this.

We assume that the update is being performed because some user wants to see the pixels. For example the postage stamp server will need to queue update processing when stamps are requested from cleaned images. This means that the resulting images must have been destreaked, i.e. detected satellite streaks must be masked.

There are two ways that the