
Releasing GPC1 images

This note discusses the process of preparing GPC1 images for release and distributing the
data to the PS1 Science Consortium.

There are a number of somewhat unique features of the PS1 project that must be considered

   1. The requirement that satellite streaks be masked in the images
   2. The massive size of the data
   3. The fact that the data is stored inside secure facility at MHPCC

The note is rather free form. Lines marked with '*' indicate tasks that need to get
done to complete this project.

Magic Masking
-------------

All images to be released must be run through the magic destreaking process. In this process
difference images are examined for potential satellite streaks and pixels along the line
of the streaks and are removed from the images. For the raw images the excised pixels are
saved so that the complete original images may be recovered if necessary.

Additionally those pixels that were not included in the science and magic analyses
must be masked because they may contain undeteced satellite streaks.
This masking is not performed as part of the magic process so that they may be included in
subsequent analysis inside the IPP system.  This masking is applied to images as part of the
release process.


De-streaking
--------------------

A pantasks task is needed that runs periodicaly to queue destreaking runs with the command
magicdstool -definebyquery  (this mode does not exist yet and thus needs to be coded)

* Create the pantasks 'destreak' task
* Create the magicdstool -definebyquery mode

A pantasks task performs the destreaking operations replacing the images in the nebulous database
with the destreaked versions. The original images are deleted and a flag is set in the
images' row in the corresponding database table that indicates that magic has been performed.

Once the destreaking runs are complete the images are ready for release processing.

Releasing images
----------------

The release processing consists of the following steps

   1. identify those data that are to be prepared for release
   2. the 'excise the masked pixel' step for science images
   3. build DataStore file sets containing the images and any other data products (.cmf files, psf etc)

This process is very similar to the processing done by the Postage Stamp server and so it makes sense
to build upon that request processing system.

* We need to be able to specify criteria for what science images are to be released.

For example, do we only release raw and processed detrend images?
Do we release chip processed or warp images from certain surveys? (The Pan planets team has
shown an interest in analyzing the chip level images).

* We need a task that applies the criteria and creates 'get_image' type postage stamp request.
pstamptool -definebyquery ?
(* this mode does not currently exist. Postage stamp jobs are specified
by a request file which is externally prepared and then referenced by the db. The request file is
parsed into a set of zero or more jobs. It probably makes sense to have the capability to skip the
request file step and create the jobs directly.)

* We need to define how we organize the data into Data Store products and file sets.

We have a whole lot of data. The DataStore only has 3 levels of hierarchy
   root
       product
           filesetA1
               file
               file
           filesetA2
               file
               file
       productB
           filesetB2
           etc.

We probably want to have filesets correspond to exposures. To keep the products from getting too
long, we may want to divide up the products by date.

Perhaps something like:
product    20081106_4776
fileset       o4776g0144o_raw
file             o4776g0144o.ota01.fits


Since these data will quickly get large, (we don't have storage space budgeted for a complete second
copy of the data):

* We need to be able to purge release file sets based on some criteria.

* For images that are being released in order to create a backup copy, it would make sense to get
notification from MPG that the images have been successfully transferred. Need a mechanism for that.

* The data store needs to be able to recognize and report to users when requested file sets
are no longer available because they have been purged.


Presentation and data delivery
------------------------------
The primary interface to the data is through the Data Store. The data store is implementated
as cgi scripts exectuted by an apache web server.

Since we do not have space in Manoa to store the images here we need to leave them at MHPCC.
Since the network there is restricted we will need to provide a reverse proxy server in
Manoa which will forward the requests to the apache server(s) on Maui.

* We need to restrict access to the data store by user. Perhaps by product?

* We need to log bandwith used by these users had have the ability to control throttle usage to
insure that the primary users are serviced.

XXXX The following isn't very thought out. There are two issues I'm trying to solve at once.
   1. Maximizing bandwidth for data transfer.
   2. Simplifying the client interface.

The standard data store use pattern is for users to query the product list for new file sets
and then copy the files with HTTP GET requests. This method is used by the IPP for the summit
copy process. This method has the drawback that the client must handle any exceptional situations
and manage errors and retries when necessary.

* It might be more efficient to supply an rsync server interface to the data repository. This program
already implements error correction features, detection of differnces between the source and
destination, etc.

We would need to provide a reverse proxy server in manoa to handle forward the rsync
procotol requests to maui.

On the other hand for efficiency we probably want to distribute the data store's files among
a set of nodes.  (This would make the rsync solution problematic.)
This way we could use a set of apache servers and redirect the requests to the various nodes.
Perhaps we would need a set of reverse proxy servers in manoa as well.


"Mirror of data at MPG"
-----------------------

The destreaked raw images will be copied to the MPG cluster in Garching to serve as a backup and
a source for analysis there.

It will be very confusing to have different database ids for the images located there.
We need a way to copy the database tables for the images so that they can be preserved.

It has been suggested that what might be done is to copy the database and
set all of the processed image entries to 'cleaned'. Then they could be regenerated using 'update'.

* We would need to provide the metadata config dump of the configuration that was used to process
the images.

* Will it be a problem that the reprocessing is done with the streak and masked pixels removed?

* Perhaps we could provide the database dump and the config files as part of the raw image transfer
