Notes on a PS1 postage stamp server web interface

The postage stamp server (PSS) is primarily designed to handle requests for many postage stamps by science clients.

It is also desired to have a user friendly web interface that allows a user to input parameters for a request (or requests), submit them to the server for processing, and retrieve the results.

This note discusses features that such a web interface might have.

First we provide some technical context by describing the operation of the postage stamp server and the prototype web interface. Those interested only in potential web interface features may skip ahead to the section "[Features for the IPP PSS Web Interface]"

A section has been added about the Pittsburgh prototype interactive stampserver interface. Some of the user interface experience from this experiment addresses issues raised in this discussion.

Overview of the operation of the IPP Postage Stamp Server

The postage stamp server consists of a set of IPP programs that run in under pantasks. The sequence of requests and jobs is managed with the use of a mysql database.

The nominal mode of operation is to query a set of predefined web sites for new request files. These web sites follow the data store interface.

When a new fileset on a datastore is discovered the postage stamp server creates a new request in the database for each file in the fileset. All files are assumed to be fits binary tables with EXTNAME=PS1_PS_REQUEST. The definition of this files may be found in the postage stamp server interface control document on the wiki.

​http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/PostageStampServer

Each row in a request file is a "request specification". The columns in the request specification server three purposes

• to select the images of interest
• to define the region of interest (the pixels in the input image that will be included in the stamp)
• to select which data products that are included in the results (e.g. image stamp, mask stamp, variance stamp, psf, background model)

Sometime after the requests have been queued, they are processed. The request file is copied to the PSS network and parsed. The parser reads each row in the request file and creates zero or more postage stamp jobs. Currently each job creates one stamp but there is an expectation that in the future jobs request specifications that reference the same images of interest will be processed by a single job.

When all jobs have finished, the postage stamp server builds a fits binary table that describes the results of each job. Then that file and the results of the jobs are packaged into a fileset on a PSS data store product specific to the requestor.

Web Interface

Now all of the above is very good for science clients but not at all convienent for random user requests.

What we need is some sort of form based web pages that users may use to create requests, submit them for processing, and "get" the results.

We have a "prototype" web interface which is very simple, feature incomplete, and still not particularly friendly. It's a set of php forms. I do not expect that any of it will be used in the "real web interface".

The way the prototype interacts with the PSS is worth reviewing.

The site is ​http://pstamp.ipp.ifa.hawaii.edu/request.php

It is password protected with the "standard" PS1 science consortium username and password.

The form allows users to input some of the supported parameters. At the top we have data for the images of interest For example:

    Select Images by:   Exposure Name
    Image Type:         warp
    ID:                 o5013g0086o
    Sky     RA:         293.3888     DEC:                16.04312
    Pixels  width       200          height: 200

If the user hits submit, the apache web server hosting the page executes a cgi script that runs a program that creates a request table file containing a single row. Then the cgi script runs a command that enters a postage stamp request into the database.

On the form the radio button "Get Status" gets selected. If the submit button is pushed again it shows the status of the request (and any previous requests entered in the given http session.

Once all of the jobs from the request have finished processing part of the the status line turns into a link to the data store fileset where the results have been posted. For example: ​http://datastore.ipp.ifa.hawaii.edu/pstampresults/web_373/

If we go to this location we find two files: results.fits and parse_error.txt Unfortunately this example generated postage stamp images.

results.fits is the results table. If we download it and examine it with fv we find that it contains 4 rows all of which have an error code set.

• The first two rows have ERROR_CODE=24 PSTAMP_NOT_DESTREAKED.
• The other two rows have ERROR_CODE=25 PSTAMP_NOT_AVAILABLE.

The first error is somewhat self explanatory. The images haven't been run through magic and thus can't be released. PSTAMP_NOT_AVAILABLE is the error that happens when the input images have been cleaned up (deleted to recover disk space).

If these images hadn't been cleaned up postage stamp images would have been placed in the fileset as well.

Soon the postage stamp server will (optionally by user request) queue update runs which will ask the IPP to regenerate the images. This will of course increase the latency for processing requests significantly. parse_error.txt gives any errors printed to stderr by the request file parser. In this example it contains the lines

    skipping non-magicked image neb://ipp035.0/gpc1/ThreePi_NorthernRegion.090729//o5013g0086o.83651/o5013g0086o.83651.wrp.9992.skycell.1662.134.fits
    skipping non-magicked image neb://ipp023.0/gpc1/ThreePi_NorthernRegion.090729//o5013g0086o.836515013g0086o.83651.wrp.9992.skycell.1748.006.fits

These lines correspond to the jobs in the results file with ERROR_CODE=24.

---

Desired Features for the IPP PSS Web Interface

Form elements for all parameters

The elements presented to the user should be context sensitive. For example the form elements where the ID and COMPONENT are to be entered should be labeled depending on the REQ_TYPE: if the REQ_TYPE is "byexp" then the ID element should be labeled "Exposure ID". Similar considerations apply to the COMPONENT parameter.

This suggests using some code that operates on the client side in the implementation (e.g. Javascript or Java).

Form elements need not follow the exact names in the request specification.

Parameter validation

Where efficient, parameters should be validated as they are entered and the user should be notified of errors. (The prototype delays what checking it does until submission). Some parameters that are "invalid" on the server side but legal in the request interface should be caught and either corrected or raised as an error. For example o5013g0086o is a valid GPC1 exposure name but o50130086o is not.

That said perhaps the user should be allowed to enter values that map to the exp_name.

    MJD: 55013  SEQ: 86

(Note this would require either the ability to read the ipp database or some other table that maps from date and sequence number to exposure name).

User Accounts

We want to define a set of authorized users each with it's own password. The PSS server will be enhanced to include the identity of the user in the submitted requests. The PSS server may use this to ration usage or prioritize the processing of jobs.

Since requests can have significant latency it would be useful to allow users to "save" the state of a session so that it can be resumed later.

User experience for Request submission and status

One of the biggest limitations of the prototype interface is the interaction with the server. The whole push a button to get status is rather primitive.

Presentation of Results

When a request completes, the results file should be parsed and important data presented to the user in an easy to understand way.

It might be useful to optionally display jpeg images of the stamps. (An option to create these should be added to the PSS request specification).

A way to convenient way to download the fits files is desired.

Availablity of Data

This may be feature creep but it would be useful if a user could determine through the web interface whether data is available for a given request, short of actually queuing the request and getting success or failure.

Filtering images of interest

Right now if you ask the PSS for say warps for a given exposure, we try to get all warps ever made for the exposure. This includes the various reprocessings that we're doing as part of finishing the software. Perhaps we need to be able to specify: "give me only the latest".

The Pittsburgh Prototype Web Interface

Joel Welling and Michael Wood-Vasey have put together a stamp server. It is currently on line at ​http://serenity.phyast.pitt.edu/PS1/stampmaker.php . The interesting part of the form is that which allows you to find observations, select from the generated list, and then generate collections of stamps based on the selection.

Selection is by sky coordinates, and optionally date range and filter. This example shows decimal coordinates but sexigesimal values work as well. Records are fetched from a MySQL database and shown as a table; observations which are not available are grayed out. The user can check-mark some or all of the available records, and then request data products for those records. The data products are produced immediately from the stored images and delivered in a .tar.gz file.

Patterns of Use

Presumably the most common usage pattern will be to collect observations of a transient. We've included a feature to generate a table of grayscale images within an html index. This turns out to be a nice way to select 'useful' images. These can be selected and more detailed information (image, weight and mask) can be downloaded. The trick is to avoid images for which the source of interest is hidden by a mask.

Comparison to IPP Stampserver Functionality

For this system the source images are precomputed and available, whereas they must be computed on demand in the IPP framework. On the other hand, the source data for *all* observations is available to the IPP framework. Would it be possible to display observations for which computation was necessary in gray? The user could then select them and click a button to cause them to be processed; the display would update to show them no longer gray when the processing was complete.

It would be hugely helpful to have the masks immediately available. The user could then select images for further processing based on the masks, avoiding the computation of unnecessary images.

User accounts can easily be added via the usual Apache mechanism.

Underlying Software

The page is written in python, running under the mod_python plug-in in an apache web server. SQLAlchemy is used to interface to a MySQL database of images. The database table is derived from the IPP database in an offline operation; the IPP database tables are organized to describe data processing rather than the simple existence of images. The table knows which observations exist and which images are available. Only the most recent processing run for a given observation is included.

Most of the form value checking is done on the server side; the form includes a small amount of javascript code. It would probably be better to do more of the checking with javascript. Table sorting could be added easily via ​sorttable.