IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links
wiki:Basic_GPC1_Processing_for_End_Users

Version 18 (modified by Michael Wood-Vasey, 17 years ago) ( diff )

--

This is a quick kind to the basic processing of the IPP steps that an end user might want to redo at his or her own institution. This was written with the Jan 2009 GPC1 commissioning data release in mind. There's probably some good integration that could be done with previous such pages by Jester et al.

Executive Summary

Detailed Step-by-Step guide

I have written these with the Odyssey cluster installation at Harvard as the example. Please contact your local DRAVG representative to learn where images are kept at your institution and how to set up the IPP environment.

At Harvard we use the 'module load' system to configure things on the cluster 

 module load hpc/ipp
 psconfig default

and here's where we keep the data 

 cd /n/data1/panstarrs/GPC1/MD01_20081002_1

In this directory are each of the different observations. The directories are named by the observation number. These span the course of several weeks. 

 [wwoodvas@heroint2 MD01_20081002_1]$ ls
 36475/	36476/	36477/	36478/	36479/	36480/	36481/	36625/	stack/

I created the 'stack' directory as a place where we can put our coadd stacked images. The coadded images were not distributed with this data release. It wouldn't make sense to put the coadds under any particular directory because they don't belong to any particular observation so I've just created a directory for them here at the base level.

Each of the observation number directories contains directories for the raw, chip, warp, and diff images 

 [wwoodvas@heroint2 MD01_20081002_1]$ ls 36475
 chip/  diff/  raw/  warp/

There are .log files in these directories detailing the steps and output that were used to create them. You may find referring to them a useful exercise if you want to understand all of the details of what was done to derive the images.

Since we are not interested in recreating the basic detector calibration at this point we will start with the 'chip' images.

Photometry

  • Basic photometry with 'psphot' 
 cd MD01_20081002_1/36475
 psphot -file chip/o4741g0234o.36475.ch.11948.XY55.ch.fits \
   -mask chip/o4741g0234o.36475.ch.11948.XY55.ch.mk.fits \
   -weight chip/o4741g0234o.36475.ch.11948.XY55.ch.wt.fits \
   -F PSPHOT.OUTPUT PSPHOT.OUT.CMF.MEF \
   testout

will create testout.cmf (the "-F PSPHOT.OUTPUT PSPHOT.OUT.CMF.MEF" is needed to avoid '{CHIP.NAME}' being put in the name of your output file since in normal processing psphot thinks that if it's using the GPC1 camera it should use 'recipes/filerules-split.mdc'). If you want to see what psphot is doing as it runs, you may be interested in the '-visual' option 

 psphot -file chip/o4741g0234o.36475.ch.11948.XY55.ch.fits \
        -mask chip//o4741g0234o.36475.ch.11948.XY55.ch.mk.fits \
        -weight chip/o4741g0234o.36475.ch.11948.XY55.ch.wt.fits \
        -visual \
        -F PSPHOT.OUTPUT PSPHOT.OUT.CMF.MEF \
        testout

The main output from the psphot command will be the .cmf file, which is a binary FITS table that contains information about the objects as gleaned from the image. One of the fields that generates the most questions is the FLAGS value for objects. This is a bitmask. See IPP Detection Bitmasks for more details.

  • psphot options and argument.

There are a variety of options and settings for psphot. These are controlled by command line optional arguments as well as by the "recipes" file located in the "share/ippconfig/" directory in your IPP installation. On my machine this is "/Volumes/data/PS1/code/ipp/default.darwin/share/ippconfig". The default recipes are in "share/ippconfig/recipes", with camera-specific overriding configuration files in the camera subdirectories in "share/ippconfig", e.g. "share/ippconfig/gpc1" for the PS1 GPC1 camera.

As extracted from the psphot options and meanings from psphot/src/psphotArguments.c from an early January 2009 CVS build: 

-file        # Input FITS file
-mask        # Mask FITS file
-weight      # Weight FITS file (variance)
<outbasename>  # Output base name; 
             # The output CMF file will be <outbasename>.cmf
             # The output model background file will be <outbasenamne>.mdl.fits
             # (Although I'm getting {CHIP.NAME} in my output files)
  
-version     # Version string
-threads     # Number of processing threads to run simultaneously
-modeltest   # run the test model (requires X,Y coordinate)
-model       # specify the modeltest model
-fitmode     # specify the test fit mode
-fitset      #  ???
-photcode    # photcode : used in output to supplement header data (argument or recipe?)
-visual      #   visual : interactive display mode
-break       #    break : used from recipe throughout psphotReadout-fitmode     
-fitmode     #  fitmode : used from recipe throughout psphotReadout
             #         [MWV: Seems inconsistent with previous fitmode]
-region      # analysis region : overrides recipe value, used in psphotReadout/psphotEnsemblePSF
-chip        # chip selection is used to limit chips to be processed

The bulk of psphot behavior is specified by the recipes. Here's an example of a recipes file for psphot: 

SAVE.OUTPUT                         BOOL  TRUE
SAVE.BACKMDL                        BOOL  FALSE
SAVE.BACKMDL.STDEV                  BOOL  FALSE
SAVE.BACKGND                        BOOL  FALSE
SAVE.BACKSUB                        BOOL  FALSE
SAVE.RESID                          BOOL  FALSE
SAVE.PSF                            BOOL  FALSE
LOAD.PSF                            BOOL  FALSE
SAVE.PLOTS                          BOOL  FALSE

# the zero point is used to set a basic scale for DVO
# XXX it may not currently be read : double check this (EAM)
ZERO_POINT                          F32   25.000          # zero point used by DVO
ZERO_PT                             F32   25.000          # zero point used by DVO

OUTPUT.FORMAT                       STR   PS1_DEV_1

# these parameter govern how the background is measured
BACKGROUND.XBIN                     S32   128             # size of background superpixels
BACKGROUND.YBIN                     S32   128             # size of background superpixels
IMSTATS_NPIX                        S32   10000           # number of pixels to use for sky estimate boxes:

SKY_BIAS                            F32   0.0             # offset applied to measured sky (FOR TESTING)
SKY_FIT_ORDER                       S32   0
SKY_FIT_LINEAR                      BOOL  FALSE
SKY_STAT                            STR   FITTED_MEAN_V4  # statistic used to measure background
SKY_CLIP_SIGMA                      F32   2.0             # statistic used to measure background
SKY_SIG                             F32   1.0             # optional sky error for 

# allowed values for SKY_STAT: 
# SAMPLE_MEAN, SAMPLE_MEDIAN, CLIPPED_MEAN, ROBUST_MEAN, ROBUST_QUARTILE, FITTED_MEAN

# masking parameters (XXX EAM : rework this to use psRegion like ANALYSIS_REGION)
XMIN                                F32   0               # minimum valid x-coord
XMAX                                F32   0               # maximum valid x-coord
YMIN                                F32   0               # minimum valid y-coord
YMAX                                F32   0               # maximum valid y-coord

# peak finding 
PEAKS_SMOOTH_SIGMA                  F32   1.0             # smoothing kernel sigma in pixels
PEAKS_SMOOTH_NSIGMA                 F32   2.0             # smoothing kernel width in sigmas
PEAKS_NSIGMA_LIMIT                  F32   25.0            # peak significance threshold
PEAKS_NSIGMA_LIMIT_2                F32   5.0             # peak significance threshold
PEAKS_NMAX                          S32   0               # on first pass, only keep NMAX peaks (0 == all)
PEAKS_MIN_GAUSS                     F32   0.5             # Minimum valid fraction of kernel

# parameters to control the selection of the peak in the Sx,Sy plane
MOMENTS_SCALE                       F32   0.25       
MOMENTS_SN_MIN                      F32   100.0           # min S/N to measure moments
MOMENTS_SX_MAX                      F32   50.0
MOMENTS_SY_MAX                      F32   50.0
MOMENTS_AR_MAX                      F32   1.5             # maximum axial ratio: 1 / AR < (sx / sy) < AR

# basic object statistics
SKY_INNER_RADIUS                    F32   15              # square annulus for local sky measurement
SKY_OUTER_RADIUS                    F32   25              # square annulus for local sky measurement
PSF_MOMENTS_RADIUS                  F32   3               # calculate initial source moments with this radius
PSF_SN_LIM                          F32   50              # minimum S/N for stars used for PSF model
PSF_MAX_NSTARS                      S32   200             # limit number of stars used for PSF model
PSF_CLUMP_NSIGMA                    F32   1.5             # region of Sx,Sy plane to use for selecting PSF stars
PSF_CLUMP_GRID_SCALE                F32   0.5             # size of Sx,Sy image pixel

PSF_CLUMP_NX                        S32   1               # subdivide image in to NX x NY regions for 
PSF_CLUMP_NY                        S32   1               # selecting PSF stars

Currently 'psphot' does not output its options or have an very useful help message. This is something that is being worked on and in fact might even be a result of our efforts to understand psphot better.

DVO

  • Let's say we have a set of cmf files and we want to use DVO to combine the detections from these files. A series of commands such as 
 cd MD01_20081002_1
 addstar -D CAMERA gpc1 -D CATDIR mycatdir -accept-astrom 36475/chip/o4741g0234o.36475.ch.11948.XY55.cmf
 addstar -D CAMERA gpc1 -D CATDIR mycatdir -accept-astrom 36476/chip/o4741g0235o.36476.ch.11949.XY55.cmf
 addstar -D CAMERA gpc1 -D CATDIR mycatdir -accept-astrom 36477/chip/o4741g0236o.36477.ch.11950.XY55.cmf
 addstar -D CAMERA gpc1 -D CATDIR mycatdir -accept-astrom 36478/chip/o4741g0237o.36478.ch.11951.XY55.cmf

will create a DVO database and fill it with the detections in the given CMF files. Since these are the same pointings we should get multiple detections (matches) when we put together the catalog. Currently -accept-astrom is still necessary until the WCS solutions finalizes and we know our true tolerances for good WCS. Need other page reference or more notes to explore DVO further.

Warping, Stacking, and Subtraction

Warping

  • To create warped images that conform to a given position on the sky, we need to define how we want to break up th sky. This tessellation is what IPP will use to map the chip images to what we call skycells. 
 skycells -D CAMERA gpc1 8 -scale 0.2 -D CATDIR default
 skycells -D CAMERA gpc1 -level 8 -scale 0.2 -D CATDIR default

This will create a directory ('default') that will contain FITS files that define the tessellation of the sky. Now we can recreate the warping of chip images to warp images:

First let's create a sandbox for ourselves to put our new files into 

 mkdir -p workspace/warp workspace/stack workspace/diff

Now we'll fill a skycell. You may wonder how to know what skycell a particular chip will warp to. That is a good question that currently has an answer that's too complicated to be practical

Easy RA,Dec->skycell is a prime opportunity for a contribution. [On Feb 4 Mark Huber said he was going to work on this].

For now just take the following example: 

dvoImageExtract skycell.02879.51 -D CATDIR default -D CAMERA gpc1 -o skycell.02879.51_template.fits -D CAMERA gpc1

pswarp -file 36475/chip/o4741g0234o.36475.ch.11948.XY33.ch.fits \
       -mask 36475/chip/o4741g0234o.36475.ch.11948.XY33.ch.mk.fits \
       -weight 36475/chip/o4741g0234o.36475.ch.11948.XY33.ch.wt.fits \
       workspace/warp/o4741g0234o.36475.wrp.11948.skycell.02879.51 \
       skycell.02879.51_template.fits

Stacking

If we have a set of images that we have warped to the same skycell and we want to stack (coadd) these images together we first specify a file that defines the images, masks, weights, PSF models, etc. for the input images to the stack: 

INPUT0  METADATA
    IMAGE      STR   36475/warp/o4741g0234o.36475.wrp.1277.skycell.02879.51.fits
    MASK       STR   36475/warp/o4741g0234o.36475.wrp.1277.skycell.02879.51.mask.fits
    WEIGHT     STR   36475/warp/o4741g0234o.36475.wrp.1277.skycell.02879.51.wt.fits
    PSF        STR   36475/warp/o4741g0234o.36475.wrp.1277.skycell.02879.51.psf
    SOURCES    STR   36475/warp/o4741g0234o.36475.wrp.1277.skycell.02879.51.cmf
    WEIGHTING  F32   1.0
    SCALE      F32   1.0
END

INPUT1  METADATA
    IMAGE      STR   36476/warp/o4741g0235o.36476.wrp.1279.skycell.02879.51.fits
    MASK       STR   36476/warp/o4741g0235o.36476.wrp.1279.skycell.02879.51.mask.fits
    WEIGHT     STR   36476/warp/o4741g0235o.36476.wrp.1279.skycell.02879.51.wt.fits
    PSF        STR   36476/warp/o4741g0235o.36476.wrp.1279.skycell.02879.51.psf
    SOURCES    STR   36476/warp/o4741g0235o.36476.wrp.1279.skycell.02879.51.cmf
    WEIGHTING  F32   1.0
    SCALE      F32   1.0
END

INPUT2  METADATA
    IMAGE      STR   36477/warp/o4741g0236o.36477.wrp.1280.skycell.02879.51.fits
    MASK       STR   36477/warp/o4741g0236o.36477.wrp.1280.skycell.02879.51.mask.fits
    WEIGHT     STR   36477/warp/o4741g0236o.36477.wrp.1280.skycell.02879.51.wt.fits
    PSF        STR   36477/warp/o4741g0236o.36477.wrp.1280.skycell.02879.51.psf
    SOURCES    STR   36477/warp/o4741g0236o.36477.wrp.1280.skycell.02879.51.cmf
    WEIGHTING  F32   1.0
    SCALE      F32   1.0
END

INPUT3  METADATA
    IMAGE      STR   36478/warp/o4741g0237o.36478.wrp.1284.skycell.02879.51.fits
    MASK       STR   36478/warp/o4741g0237o.36478.wrp.1284.skycell.02879.51.mask.fits
    WEIGHT     STR   36478/warp/o4741g0237o.36478.wrp.1284.skycell.02879.51.wt.fits
    PSF        STR   36478/warp/o4741g0237o.36478.wrp.1284.skycell.02879.51.psf
    SOURCES    STR   36478/warp/o4741g0237o.36478.wrp.1284.skycell.02879.51.cmf
    WEIGHTING  F32   1.0
    SCALE      F32   1.0
END

INPUT4  METADATA
    IMAGE      STR   36479/warp/o4741g0238o.36479.wrp.1281.skycell.02879.51.fits
    MASK       STR   36479/warp/o4741g0238o.36479.wrp.1281.skycell.02879.51.mask.fits
    WEIGHT     STR   36479/warp/o4741g0238o.36479.wrp.1281.skycell.02879.51.wt.fits
    PSF        STR   36479/warp/o4741g0238o.36479.wrp.1281.skycell.02879.51.psf
    SOURCES    STR   36479/warp/o4741g0238o.36479.wrp.1281.skycell.02879.51.cmf
    WEIGHTING  F32   1.0
    SCALE      F32   1.0
END

INPUT5  METADATA
    IMAGE      STR   36480/warp/o4741g0240o.36480.wrp.1285.skycell.02879.51.fits
    MASK       STR   36480/warp/o4741g0240o.36480.wrp.1285.skycell.02879.51.mask.fits
    WEIGHT     STR   36480/warp/o4741g0240o.36480.wrp.1285.skycell.02879.51.wt.fits
    PSF        STR   36480/warp/o4741g0240o.36480.wrp.1285.skycell.02879.51.psf
    SOURCES    STR   36480/warp/o4741g0240o.36480.wrp.1285.skycell.02879.51.cmf
    WEIGHTING  F32   1.0
    SCALE      F32   1.0
END

INPUT6  METADATA
    IMAGE      STR   36481/warp/o4741g0241o.36481.wrp.1282.skycell.02879.51.fits
    MASK       STR   36481/warp/o4741g0241o.36481.wrp.1282.skycell.02879.51.mask.fits
    WEIGHT     STR   36481/warp/o4741g0241o.36481.wrp.1282.skycell.02879.51.wt.fits
    PSF        STR   36481/warp/o4741g0241o.36481.wrp.1282.skycell.02879.51.psf
    SOURCES    STR   36481/warp/o4741g0241o.36481.wrp.1282.skycell.02879.51.cmf
    WEIGHTING  F32   1.0
    SCALE      F32   1.0
END

INPUT7  METADATA
    IMAGE      STR   36625/warp/o4741g0239o.36625.wrp.1283.skycell.02879.51.fits
    MASK       STR   36625/warp/o4741g0239o.36625.wrp.1283.skycell.02879.51.mask.fits
    WEIGHT     STR   36625/warp/o4741g0239o.36625.wrp.1283.skycell.02879.51.wt.fits
    PSF        STR   36625/warp/o4741g0239o.36625.wrp.1283.skycell.02879.51.psf
    SOURCES    STR   36625/warp/o4741g0239o.36625.wrp.1283.skycell.02879.51.cmf
    WEIGHTING  F32   1.0
    SCALE      F32   1.0
END

If we called this file 'skycell.02879.51.mwv.002.mdc', then we might run the following to get the output stack 

 ppStack skycell.02879.51.mwv.002.mdc skycell.02879.51.mwv.002 -sources 36475/warp/o4741g0234o.36475.wrp.1277.skycell.02879.51.cmf

(The '-sources' might be redundant since we're specify source lists in the .mdc (meta-data configuration) file).

Subtraction

Once we have a coaddition that hopeful has enough dithers to fill in the area we will want to run subtractions of individual images against that coadd template. In this case we built the coadd from the images themselves, but that can't be helped for now as long as we have limited data. 

 set ref=skycell.02879.51.mwv.002
 set new=36625/warp/o4741g0239o.36625.wrp.1283.skycell.02879.51
 ppSub $new.fits $ref.fits sub_isis \
    -inmask  $new.mask.fits -inweight  $new.wt.fits \
    -refmask $ref.mask.fits -refweight $ref.wt.fits \
    -sources $new.cmf -type ISIS -threads 2 -photometry

or in ipp-2.8 

 set ref=skycell.02879.51.mwv.002
 set new=36625/warp/o4741g0239o.36625.wrp.1283.skycell.02879.51
 ppSub $new.fits $ref.fits sub_isis \
    -inmask  $new.mask.fits -invariance  $new.wt.fits \
    -refmask $ref.mask.fits -refvariance $ref.wt.fits \
    -sources $new.cmf -type ISIS -threads 2 -photometry

would run a subtraction of $new minus $ref using the ISIS kernels. ppSub --help will generate a somewhat detailed list of options but they probably won't all make sense on a first reading.

Since we asked 'ppSub' to do photometry it will create a difference .cmf file along with the subtraction. If we had a collection of these difference .cmf files we might use DVO to combine them and if we were looking for stationary transients, we would look for multiple detections at the same position.

Note: See TracWiki for help on using the wiki.