Changes between Initial Version and Version 1 of pstamp_web_interface_features_2012_09

Timestamp:

Sep 4, 2012, 7:36:52 AM (14 years ago)

Author:

bills

Comment:

--

pstamp_web_interface_features_2012_09

@@ +1 @@
+{{{
+The PS1 software team is finally applying resources to produce a good interactive
+inteface to the IPP postage stamp server. This will be included as part of the PS1 SCI (term)
+interface along side PSPS. 
+
+The original web interface prototype has many significant limitations.  The most important is that
+it is quite IPP centric.  The second is that the user experience is not friendly. Finally, it 
+supports neither the the complete feature set of the existing postage stamp server interface or various other
+features that users require.
+
+The purpose of this note is to outline a more user oriented interface and certain enhancements
+to the IPP system that are required to implement them.
+
+There are three major functions of the current postage stamp request interface
+
+    1. Select the images of interest
+    2. Identify the region of interest - the set of pixels to be extracted
+    3. Select the output data products to be returned.
+
+Some of these functions are really "database operations" yet the postage stamp request
+specification does not provide very complete set of selectors.
+
+In this document we propose splitting many of those functions off.
+
+In the end it changes from a "fire and hope" system, to a more controlled user managed system.
+
+Review of the Postage Stamp Interface
+-------------------------------------
+The postage stamp server interface is described in the interface control document (LINK).
+
+Requests are submitted to the server as fits tables. Each row in the table specifies one set
+of coordinates and parameters to select the input images and output data products. When processed
+by the server each row generates zero or more postage stamp jobs.  When all of the jobs 
+queued for a given request the results are bundled into a fileset on the IPP data store.
+A fits file containing a table is also created. Each row in this results file lists the files
+included in the output and various parameters of the source images.
+
+Images of Interest - Stage
+--------------------------
+The postage stamp server interface is based on the various stages of the IPP. Images are
+selected from one of the stages of the IPP pipeline
+
+    raw     
+    chip   
+    warp  
+    stack
+    diff
+
+Raw stage
+---------
+The images acquired by the GPC1 camera. 60 fits files per exposure on for each OTA (chip), 
+64 or more extensions per file 1 for each cell. Raw images are of limited use to end users.
+
+Chip stage
+----------
+Outputs from the IPP chip and camera stage.  At the chip stage images have been detrended. These
+operations include dark, flatfield, burntool, and various other chip (and cell) specific corrections.  
+These images have had a fit to the sky background subtracted. Photometry is performed at this
+stage. At the camera stage the chip stage outputs for the 60 OTAs are analyzed together to arrive
+at the astrometric solution and the initial zero point is measured.
+
+The postage stamp server uses the camera stage astrometric solution to select the pixels
+and includes the solution as the WCS in the headers of the output files. The zero point measured
+at the camera stage is not currently included. That must be fixed.
+
+Warp stage
+----------
+After the camera stage the images are transformed (warped) to a given skycell tessellation.
+The resulting images have right ascension and declination as the X and Y axes. Each skycell gets
+contributions from chip stage images from one or more of the OTAs. The photometry outputs from
+the various input chips are combined.  Photometry is *not* repeated at the warp stage. The sources
+files contain the sources detected at the chip stage with coordinates transformed to skycell
+coordinates.
+
+Stack Stage
+-----------
+At the stack stage multiple warps are combined together. More details on this process will be
+discussed below in the section on data products.
+
+Diff Stage
+----------
+At the Diff stage various combinations of images are subtracted from each other to find differences.
+These are used to find transient events and moving objects. The major difference modes are
+    Warp-Warp   Two individual frames are compared
+    Warp-Stack  A single frame is compared to a template stack image
+    Stack-Stack Stack created from a single night's exposures are compared to a template stack
+
+There are some other stages of the IPP pipline which are currently ignored by the current postage stamp
+interface. Many of these are critical to a complete understanding of the data
+    
+    DVO - Associates source detections from single frames and stacks into objects. Zero point and
+          astrometric measurements are refined. The results are exported to PSPS but  are not fed
+          back into the images returned by the postage stamp server.
+
+    Static Sky and Sky Calibration - Performs photometry and astrometric correction to stacked images
+        including extended source measurments. These results are imported into PSPS but for more rapid
+        turn around and easy association with images these results should be avialable through the pstamp server.
+
+    Chip Background Stage - Certain images undero further processing to remove the background correction 
+    that was applied at the chip stage. (Currently only used for M31.) It has been recommended to add a
+    feature to the postage stamp server to optionally perform this operation on an arbitrary image.
+    Work has been done to implement this but haa not yet been deployed.
+
+    Warp Background Stage  chip background outputs warped to sky coordinates.
+
+    Background restored stacks - For analysis of large extended objects the background correction is
+    problematic. It is a requirement to the IPP to make avaliable through the postage stamp server
+    stack images with the background restored. A project has been started to implement the 
+    background restoration.
+
+Suggested New user interface
+----------------------------
+The selection by stage makes perfect sense for the implementation but requires some knowlege of the
+workings of the IPP. The IPP terminolgy is somewhat different than that used by PSPS and many users
+are not familar with it. We assert here that they shouldn't need to be.
+
+The inteface should be presented as a series of choices which more reflect the nature of the 
+underlying data than the architecture of the IPP
+
+The image type choices should be presented as follows
+
+    1. Single Frame Images
+        chip coordinate pixels
+        warped coordinate pixels
+
+    2. Stacked Images
+
+    3. Difference Images
+
+    4. Raw images
+
+The UI should present options 1 and 2 prominently and make 3 and especially 4 advanced options.
+
+    
+Selection of source images
+--------------------------
+Once the stage is identified, the actual input images need to be selected. The server supports several
+image selection methods. Note that currently postage stamps are extracted from a *single* IPP
+image product. No combination between chips or skycells is performed.
+
+    bycoord     Images are selected based on supplied RA and DEC 
+    byexp       Single frame images selected by name (IPP rawExp.exp_name PSPS FrameMeta.name)
+    byid        Specific images are selected by IPP database ID (exp_id, chip_id, warp_id, stack_id, diff_id)
+    byskycell   warp, stack, and diff images selected based on tessellation id (tess_id) and skycell_id
+    bydiff      specialized method used by MOPS to select images from a particular difference detection
+
+Image of Interest filtering
+---------------------------
+A number of parameters can be supplied with the request to narrow down the set of target images. Two
+obvious ones are filter and observation date. The IPP data_group can be used to narrow down the
+data set. In practice this is only useful with advice from the IPP team or access to a copy of the
+IPP's gpc1 mysql data base.
+
+Problems with this interface for end users
+------------------------------------------
+Other than bycoord these methods are not friendly to end users. Unless the user has
+access to a copy of the gpc1 database or software to find out which skycell contains
+a given point int the sky it is difficult to narrow down images to specific
+data sets. 
+
+Now that PSPS is being loaded with more PSPS data this problem is relieved somewhat. However
+given inevitable the time lag between the exposures and frames information that PSPS does not currently
+have is needed to select recent data efficeintly.
+
+Given the vast data set of pan starrs selection by coordinates has it's problem. Asking for all chip
+images for a spot in one of the medium deep fields is easy to do and probably not what a user
+really wants.
+
+Also frames have been procssed multiple times.  The IPP has no notion of what the right set of
+runs should be "released" to end users. A solution to this problem has been proposed. In this
+document we will assume that those features will be available.
+
+Suggested Features for the New UI
+---------------------------------
+It is desirable to be able to preview what the results of a given set of requests will include.
+In the current interface this is problematic because this information is obtained by parsing
+the request file which is an extremely inefficent operation.
+
+Since the new UI is inside a "database environment"  we have an opportunity to improve this
+significantly.
+
+While finding all of the images for a given stage that correspond to a set of coordinates is
+somewhat expensive finding the individual frames is more straightforward. 
+
+We suggest moving the selection of frames for processing from the postage stamp server to
+the web interface. The Web interface will thus only do byexp or byid requests.
+
+
+Choices in the UI - Single Frame Images
+---------------------------------------
+
+Coordinates
+    Sky coordinates RA DEC decimal degrees and sexagesimal standard formats
+    pixel coordinates (advanced)
+
+Note that it is desirable to accept a list of coordinates and process them together. Users
+should be able to upload lists with certain parameters to the UI for example
+RA DEC FILTER dateobs_min dateobs_max.
+
+The user should also be able to provide a list of object or detection ids obtained from a query.
+
+Choices in the UI - All image types
+-----------------------------------
+
+Stamp Size
+    pixels
+    arcseconds
+    entire image
+
+Image Selection
+    by coordinates
+    by frame id
+    by skycell id (advanced)
+
+Data Release
+    Best    (default)
+    PSPS data / release version
+    Original Nightly Processing (not sure if we need this but it's easy)
+
+This is a new concept which requires some work on the IPP side. It should probably eventually
+default to the version in PSPS database for a given survey but at the current time this is too 
+restrictive as it hides much of the data.
+    
+If we are not selecting by frame id a few more selectors should be made available
+
+Survey
+    3PI
+    MD (includes M31, CNP, and STS)
+    SAS
+    SS
+
+To use this information requires a new interface with the postage stamp server. Note the individual
+frames can be identified by rawExp.obs_mode.... but this is buggy for old data. Perhaps something
+should be added to the new ippRelease table for the given frame.
+
+Choices in the UI - Single Frame Images
+---------------------------------------
+Filter
+    hints should perhaps be provided. For example w shouldn't be a choice in the 3pi or MD surveys.
+    STS is only i.
+
+Observation Date
+    min Default to the beginning of the mission to avoid buggy data. User can change to accept
+        earlier data if desired
+    max default no limit
+
+
+Data Group  This exists in the current interface and should be included but not displayed prominently.
+
+Preview Function
+----------------
+Once the user has selected the parameters he can command the system to perform the image lookup for
+preview and selection.
+
+Ideally this would be a PSPS operation but since we want to make all PS1 data available we propose
+using the gpc1 database (or a replicated copy or subset).
+
+The query would use the selection parameters described above to query the new table ipp release table.
+This table will have entries for each exposure that indicate which images in the IPP have been 
+"released" by the IPP. Paremeters available
+    exp_id
+    chip_id
+    warp_id
+    diff_id 
+    survey_id
+    release (or data version)
+
+Other parameters (ra, dec, filter, dateobs) will be included by joining to the rawExp table.
+
+When the query returns the relevant data will be presented to the users in a format that allows
+them to select the target frames for the postage stamp requests. 
+
+    frameID survey filter "released version" comment
+
+By released version I mean something that tells whether the data is from one of the grand reprocessed data
+sets or is the original nightly processing.
+
+If a list of coordinates was provided the presentation of the results to the user in friendly
+way may be tricky.
+
+
+Image Selection from results of PSPS queries
+--------------------------------------------
+One obvious use case is to request postage stamps from the objects or detections in
+results of a specific PSPS query.
+
+Object coordinates could be used to create a bycoord request to the server. If individual detections are
+available then the frameID may be obtained and byexp requests could be used. A user might also
+be interested in looking at stamps from frames where the object was not detected so that should
+be supported as well.
+
+Stacked images
+--------------
+The IPP will need a table to manage releases for Stacked images as well. The fundamental unit in this
+case is the Skycell.
+    stack_id
+    skycell_id
+    survey_id
+    release
+
+Other parameters, ra, dec, filter, tess_id will be available through join to the tables stackRun 
+and skycell
+
+QUESTION: One feature that might be interesting is to find the stacks that a particular frame is included
+in. Currently this information is not availalble from the PSPS or postage stamp server. It can be
+derived from the IPP database. Is this useful?
+
+
+Data Products Available Current Interface
+-----------------------------------------
+    Image
+    Mask
+    Variance
+    Sources (CMF)
+
+Chip stage
+    psf
+    model
+
+Stack
+    Unconvolved images
+
+Processing
+----------
+
+
+
+
+}}}