| Version 23 (modified by , 17 years ago) ( diff ) |
|---|
This is a quick kind to the basic processing of the IPP steps that an end user might want to redo at his or her own institution. This was written with the Jan 2009 GPC1 commissioning data release in mind. There's probably some good integration that could be done with previous such pages by Jester et al.
Executive Summary
Detailed Step-by-Step guide
I have written these with the Odyssey cluster installation at Harvard as the example. Please contact your local DRAVG representative to learn where images are kept at your institution and how to set up the IPP environment.
At Harvard we use the 'module load' system to configure things on the cluster
module load hpc/ipp psconfig default
and here's where we keep the data
cd /n/data1/panstarrs/GPC1/MD01_20081002_1
In this directory are each of the different observations. The directories are named by the observation number. These span the course of several weeks.
[wwoodvas@heroint2 MD01_20081002_1]$ ls 36475/ 36476/ 36477/ 36478/ 36479/ 36480/ 36481/ 36625/ stack/
I created the 'stack' directory as a place where we can put our coadd stacked images. The coadded images were not distributed with this data release. It wouldn't make sense to put the coadds under any particular directory because they don't belong to any particular observation so I've just created a directory for them here at the base level.
Each of the observation number directories contains directories for the raw, chip, warp, and diff images
[wwoodvas@heroint2 MD01_20081002_1]$ ls 36475 chip/ diff/ raw/ warp/
There are .log files in these directories detailing the steps and output that were used to create them. You may find referring to them a useful exercise if you want to understand all of the details of what was done to derive the images.
Since we are not interested in recreating the basic detector calibration at this point we will start with the 'chip' images.
Photometry
- Basic photometry with 'psphot'
cd MD01_20081002_1/36475 psphot -file chip/o4741g0234o.36475.ch.11948.XY55.ch.fits \ -mask chip/o4741g0234o.36475.ch.11948.XY55.ch.mk.fits \ -weight chip/o4741g0234o.36475.ch.11948.XY55.ch.wt.fits \ -F PSPHOT.OUTPUT PSPHOT.OUT.CMF.MEF \ testout
will create testout.cmf (the "-F PSPHOT.OUTPUT PSPHOT.OUT.CMF.MEF" is needed to avoid '{CHIP.NAME}' being put in the name of your output file since in normal processing psphot thinks that if it's using the GPC1 camera it should use 'recipes/filerules-split.mdc'). If you want to see what psphot is doing as it runs, you may be interested in the '-visual' option
psphot -file chip/o4741g0234o.36475.ch.11948.XY55.ch.fits \
-mask chip//o4741g0234o.36475.ch.11948.XY55.ch.mk.fits \
-weight chip/o4741g0234o.36475.ch.11948.XY55.ch.wt.fits \
-visual \
-F PSPHOT.OUTPUT PSPHOT.OUT.CMF.MEF \
testout
The main output from the psphot command will be the .cmf file, which is a binary FITS table that contains information about the objects as gleaned from the image. One of the fields that generates the most questions is the FLAGS value for objects. This is a bitmask. See IPP Detection Bitmasks for more details.
- psphot options and argument.
There are a variety of options and settings for psphot. These are controlled by command line optional arguments as well as by the "recipes" file located in the "share/ippconfig/" directory in your IPP installation. On my machine this is "/Volumes/data/PS1/code/ipp/default.darwin/share/ippconfig". The default recipes are in "share/ippconfig/recipes", with camera-specific overriding configuration files in the camera subdirectories in "share/ippconfig", e.g. "share/ippconfig/gpc1" for the PS1 GPC1 camera.
As extracted from the psphot options and meanings from psphot/src/psphotArguments.c from an early January 2009 CVS build:
-file # Input FITS file
-mask # Mask FITS file
-weight # Weight FITS file (variance)
<outbasename> # Output base name;
# The output CMF file will be <outbasename>.cmf
# The output model background file will be <outbasenamne>.mdl.fits
# (Although I'm getting {CHIP.NAME} in my output files)
-version # Version string
-threads # Number of processing threads to run simultaneously
-modeltest # run the test model (requires X,Y coordinate)
-model # specify the modeltest model
-fitmode # specify the test fit mode
-fitset # ???
-photcode # photcode : used in output to supplement header data (argument or recipe?)
-visual # visual : interactive display mode
-break # break : used from recipe throughout psphotReadout-fitmode
-fitmode # fitmode : used from recipe throughout psphotReadout
# [MWV: Seems inconsistent with previous fitmode]
-region # analysis region : overrides recipe value, used in psphotReadout/psphotEnsemblePSF
-chip # chip selection is used to limit chips to be processed
The bulk of psphot behavior is specified by the recipes. Here's an example of a recipes file for psphot. Any of the variables below can be defined on the command line with the '-F' option, as done above in our example psphot with "-F PSPHOT.OUTPUT PSPHOT.OUT.CMF.MEF".
SAVE.OUTPUT BOOL TRUE SAVE.BACKMDL BOOL FALSE SAVE.BACKMDL.STDEV BOOL FALSE SAVE.BACKGND BOOL FALSE SAVE.BACKSUB BOOL FALSE SAVE.RESID BOOL FALSE SAVE.PSF BOOL FALSE LOAD.PSF BOOL FALSE SAVE.PLOTS BOOL FALSE # the zero point is used to set a basic scale for DVO # XXX it may not currently be read : double check this (EAM) ZERO_POINT F32 25.000 # zero point used by DVO ZERO_PT F32 25.000 # zero point used by DVO OUTPUT.FORMAT STR PS1_DEV_1 # these parameter govern how the background is measured BACKGROUND.XBIN S32 128 # size of background superpixels BACKGROUND.YBIN S32 128 # size of background superpixels IMSTATS_NPIX S32 10000 # number of pixels to use for sky estimate boxes: SKY_BIAS F32 0.0 # offset applied to measured sky (FOR TESTING) SKY_FIT_ORDER S32 0 SKY_FIT_LINEAR BOOL FALSE SKY_STAT STR FITTED_MEAN_V4 # statistic used to measure background SKY_CLIP_SIGMA F32 2.0 # statistic used to measure background SKY_SIG F32 1.0 # optional sky error for # allowed values for SKY_STAT: # SAMPLE_MEAN, SAMPLE_MEDIAN, CLIPPED_MEAN, ROBUST_MEAN, ROBUST_QUARTILE, FITTED_MEAN # masking parameters (XXX EAM : rework this to use psRegion like ANALYSIS_REGION) XMIN F32 0 # minimum valid x-coord XMAX F32 0 # maximum valid x-coord YMIN F32 0 # minimum valid y-coord YMAX F32 0 # maximum valid y-coord # peak finding PEAKS_SMOOTH_SIGMA F32 1.0 # smoothing kernel sigma in pixels PEAKS_SMOOTH_NSIGMA F32 2.0 # smoothing kernel width in sigmas PEAKS_NSIGMA_LIMIT F32 25.0 # peak significance threshold PEAKS_NSIGMA_LIMIT_2 F32 5.0 # peak significance threshold PEAKS_NMAX S32 0 # on first pass, only keep NMAX peaks (0 == all) PEAKS_MIN_GAUSS F32 0.5 # Minimum valid fraction of kernel # parameters to control the selection of the peak in the Sx,Sy plane MOMENTS_SCALE F32 0.25 MOMENTS_SN_MIN F32 100.0 # min S/N to measure moments MOMENTS_SX_MAX F32 50.0 MOMENTS_SY_MAX F32 50.0 MOMENTS_AR_MAX F32 1.5 # maximum axial ratio: 1 / AR < (sx / sy) < AR # basic object statistics SKY_INNER_RADIUS F32 15 # square annulus for local sky measurement SKY_OUTER_RADIUS F32 25 # square annulus for local sky measurement PSF_MOMENTS_RADIUS F32 3 # calculate initial source moments with this radius PSF_SN_LIM F32 50 # minimum S/N for stars used for PSF model PSF_MAX_NSTARS S32 200 # limit number of stars used for PSF model PSF_CLUMP_NSIGMA F32 1.5 # region of Sx,Sy plane to use for selecting PSF stars PSF_CLUMP_GRID_SCALE F32 0.5 # size of Sx,Sy image pixel PSF_CLUMP_NX S32 1 # subdivide image in to NX x NY regions for PSF_CLUMP_NY S32 1 # selecting PSF stars
Currently 'psphot' does not output its options or have an very useful help message. This is something that is being worked on and in fact might even be a result of our efforts to understand psphot better.
DVO
- Let's say we have a set of cmf files and we want to use DVO to combine the detections from these files. A series of commands such as
cd MD01_20081002_1 addstar -D CAMERA gpc1 -D CATDIR mycatdir -accept-astrom 36475/chip/o4741g0234o.36475.ch.11948.XY55.cmf addstar -D CAMERA gpc1 -D CATDIR mycatdir -accept-astrom 36476/chip/o4741g0235o.36476.ch.11949.XY55.cmf addstar -D CAMERA gpc1 -D CATDIR mycatdir -accept-astrom 36477/chip/o4741g0236o.36477.ch.11950.XY55.cmf addstar -D CAMERA gpc1 -D CATDIR mycatdir -accept-astrom 36478/chip/o4741g0237o.36478.ch.11951.XY55.cmf
will create a DVO database and fill it with the detections in the given CMF files. Since these are the same pointings we should get multiple detections (matches) when we put together the catalog. Currently -accept-astrom is still necessary until the WCS solutions finalizes and we know our true tolerances for good WCS. Need other page reference or more notes to explore DVO further.
Warping, Stacking, and Subtraction
Warping
- To create warped images that conform to a given position on the sky, we need to define how we want to break up th sky. This tessellation is what IPP will use to map the chip images to what we call skycells.
skycells -D CAMERA gpc1 8 -scale 0.2 -D CATDIR default skycells -D CAMERA gpc1 -level 8 -scale 0.2 -D CATDIR default
This will create a directory ('default') that will contain FITS files that define the tessellation of the sky. Now we can recreate the warping of chip images to warp images:
First let's create a sandbox for ourselves to put our new files into
mkdir -p workspace/warp workspace/stack workspace/diff
Now we'll fill a skycell. You may wonder how to know what skycell a particular chip will warp to. That is a good question that currently has an answer that's too complicated to be practical
Easy RA,Dec->skycell is a prime opportunity for a contribution. [On Feb 4 Mark Huber said he was going to work on this].
For now just take the following example:
dvoImageOverlaps 36475/chip/o4741g0234o.36475.ch.11948.XY33.ch.fits -accept-astrom -D CATDIR default
(right skycell format but still wrong output with: skycells 5 -scale 0.2 -nx 8 -ny 8 -fix-ns -padding 0.05)
dvoImageExtract skycell.02879.51 -D CATDIR default -D CAMERA gpc1 -o skycell.02879.51_template.fits -D CAMERA gpc1
pswarp -file 36475/chip/o4741g0234o.36475.ch.11948.XY33.ch.fits \
-mask 36475/chip/o4741g0234o.36475.ch.11948.XY33.ch.mk.fits \
-weight 36475/chip/o4741g0234o.36475.ch.11948.XY33.ch.wt.fits \
workspace/warp/o4741g0234o.36475.wrp.11948.skycell.02879.51 \
skycell.02879.51_template.fits
Stacking
If we have a set of images that we have warped to the same skycell and we want to stack (coadd) these images together we first specify a file that defines the images, masks, weights, PSF models, etc. for the input images to the stack:
INPUT0 METADATA
IMAGE STR 36475/warp/o4741g0234o.36475.wrp.1277.skycell.02879.51.fits
MASK STR 36475/warp/o4741g0234o.36475.wrp.1277.skycell.02879.51.mask.fits
WEIGHT STR 36475/warp/o4741g0234o.36475.wrp.1277.skycell.02879.51.wt.fits
PSF STR 36475/warp/o4741g0234o.36475.wrp.1277.skycell.02879.51.psf
SOURCES STR 36475/warp/o4741g0234o.36475.wrp.1277.skycell.02879.51.cmf
WEIGHTING F32 1.0
SCALE F32 1.0
END
INPUT1 METADATA
IMAGE STR 36476/warp/o4741g0235o.36476.wrp.1279.skycell.02879.51.fits
MASK STR 36476/warp/o4741g0235o.36476.wrp.1279.skycell.02879.51.mask.fits
WEIGHT STR 36476/warp/o4741g0235o.36476.wrp.1279.skycell.02879.51.wt.fits
PSF STR 36476/warp/o4741g0235o.36476.wrp.1279.skycell.02879.51.psf
SOURCES STR 36476/warp/o4741g0235o.36476.wrp.1279.skycell.02879.51.cmf
WEIGHTING F32 1.0
SCALE F32 1.0
END
INPUT2 METADATA
IMAGE STR 36477/warp/o4741g0236o.36477.wrp.1280.skycell.02879.51.fits
MASK STR 36477/warp/o4741g0236o.36477.wrp.1280.skycell.02879.51.mask.fits
WEIGHT STR 36477/warp/o4741g0236o.36477.wrp.1280.skycell.02879.51.wt.fits
PSF STR 36477/warp/o4741g0236o.36477.wrp.1280.skycell.02879.51.psf
SOURCES STR 36477/warp/o4741g0236o.36477.wrp.1280.skycell.02879.51.cmf
WEIGHTING F32 1.0
SCALE F32 1.0
END
INPUT3 METADATA
IMAGE STR 36478/warp/o4741g0237o.36478.wrp.1284.skycell.02879.51.fits
MASK STR 36478/warp/o4741g0237o.36478.wrp.1284.skycell.02879.51.mask.fits
WEIGHT STR 36478/warp/o4741g0237o.36478.wrp.1284.skycell.02879.51.wt.fits
PSF STR 36478/warp/o4741g0237o.36478.wrp.1284.skycell.02879.51.psf
SOURCES STR 36478/warp/o4741g0237o.36478.wrp.1284.skycell.02879.51.cmf
WEIGHTING F32 1.0
SCALE F32 1.0
END
INPUT4 METADATA
IMAGE STR 36479/warp/o4741g0238o.36479.wrp.1281.skycell.02879.51.fits
MASK STR 36479/warp/o4741g0238o.36479.wrp.1281.skycell.02879.51.mask.fits
WEIGHT STR 36479/warp/o4741g0238o.36479.wrp.1281.skycell.02879.51.wt.fits
PSF STR 36479/warp/o4741g0238o.36479.wrp.1281.skycell.02879.51.psf
SOURCES STR 36479/warp/o4741g0238o.36479.wrp.1281.skycell.02879.51.cmf
WEIGHTING F32 1.0
SCALE F32 1.0
END
INPUT5 METADATA
IMAGE STR 36480/warp/o4741g0240o.36480.wrp.1285.skycell.02879.51.fits
MASK STR 36480/warp/o4741g0240o.36480.wrp.1285.skycell.02879.51.mask.fits
WEIGHT STR 36480/warp/o4741g0240o.36480.wrp.1285.skycell.02879.51.wt.fits
PSF STR 36480/warp/o4741g0240o.36480.wrp.1285.skycell.02879.51.psf
SOURCES STR 36480/warp/o4741g0240o.36480.wrp.1285.skycell.02879.51.cmf
WEIGHTING F32 1.0
SCALE F32 1.0
END
INPUT6 METADATA
IMAGE STR 36481/warp/o4741g0241o.36481.wrp.1282.skycell.02879.51.fits
MASK STR 36481/warp/o4741g0241o.36481.wrp.1282.skycell.02879.51.mask.fits
WEIGHT STR 36481/warp/o4741g0241o.36481.wrp.1282.skycell.02879.51.wt.fits
PSF STR 36481/warp/o4741g0241o.36481.wrp.1282.skycell.02879.51.psf
SOURCES STR 36481/warp/o4741g0241o.36481.wrp.1282.skycell.02879.51.cmf
WEIGHTING F32 1.0
SCALE F32 1.0
END
INPUT7 METADATA
IMAGE STR 36625/warp/o4741g0239o.36625.wrp.1283.skycell.02879.51.fits
MASK STR 36625/warp/o4741g0239o.36625.wrp.1283.skycell.02879.51.mask.fits
WEIGHT STR 36625/warp/o4741g0239o.36625.wrp.1283.skycell.02879.51.wt.fits
PSF STR 36625/warp/o4741g0239o.36625.wrp.1283.skycell.02879.51.psf
SOURCES STR 36625/warp/o4741g0239o.36625.wrp.1283.skycell.02879.51.cmf
WEIGHTING F32 1.0
SCALE F32 1.0
END
If we called this file 'skycell.02879.51.mwv.002.mdc', then we might run the following to get the output stack
ppStack skycell.02879.51.mwv.002.mdc skycell.02879.51.mwv.002 -sources 36475/warp/o4741g0234o.36475.wrp.1277.skycell.02879.51.cmf
(The '-sources' might be redundant since we're specify source lists in the .mdc (meta-data configuration) file).
Subtraction
Once we have a coaddition that hopeful has enough dithers to fill in the area we will want to run subtractions of individual images against that coadd template. In this case we built the coadd from the images themselves, but that can't be helped for now as long as we have limited data.
set ref=skycell.02879.51.mwv.002
set new=36625/warp/o4741g0239o.36625.wrp.1283.skycell.02879.51
ppSub $new.fits $ref.fits sub_isis \
-inmask $new.mask.fits -inweight $new.wt.fits \
-refmask $ref.mask.fits -refweight $ref.wt.fits \
-sources $new.cmf -type ISIS -threads 2 -photometry \
-F PSPHOT.OUTPUT PSPHOT.OUT.CMF.MEF \
-F SOURCE.PLOT.RAW.MOMENTS SOURCE.PLOT.SKY.MOMENTS \
-F SOURCE.PLOT.RAW.PSFMODEL SOURCE.PLOT.SKY.PSFMODEL \
-F SOURCE.PLOT.RAW.APRESID SOURCE.PLOT.SKY.APRESID \
-F PSPHOT.PSF.SAVE PSPHOT.PSF.SKY.SAVE \
-F PSPHOT.BACKMDL PSPHOT.BACKMDL.MEF
or in ipp-2.8
set ref=skycell.02879.51.mwv.002
set new=36625/warp/o4741g0239o.36625.wrp.1283.skycell.02879.51
ppSub $new.fits $ref.fits sub_isis \
-inmask $new.mask.fits -invariance $new.wt.fits \
-refmask $ref.mask.fits -refvariance $ref.wt.fits \
-sources $new.cmf -type ISIS -threads 2 -photometry \
-F PSPHOT.OUTPUT PSPHOT.OUT.CMF.MEF \
-F SOURCE.PLOT.RAW.MOMENTS SOURCE.PLOT.SKY.MOMENTS \
-F SOURCE.PLOT.RAW.PSFMODEL SOURCE.PLOT.SKY.PSFMODEL \
-F SOURCE.PLOT.RAW.APRESID SOURCE.PLOT.SKY.APRESID \
-F PSPHOT.PSF.SAVE PSPHOT.PSF.SKY.SAVE \
-F PSPHOT.BACKMDL PSPHOT.BACKMDL.MEF
would run a subtraction of $new minus $ref using the ISIS kernels. ppSub --help will generate a somewhat detailed list of options but they probably won't all make sense on a first reading.
Since we asked 'ppSub' to do photometry it will create a difference .cmf file along with the subtraction. If we had a collection of these difference .cmf files we might use DVO to combine them and if we were looking for stationary transients, we would look for multiple detections at the same position.
