IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Initial Version and Version 1 of Basic_GPC1_Processing_for_End_Users


Ignore:
Timestamp:
Feb 24, 2009, 4:23:53 PM (17 years ago)
Author:
trac
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • Basic_GPC1_Processing_for_End_Users

    v1 v1  
     1This is a quick kind to the basic processing of the IPP steps that an end user might want to redo at his or her own institution.  This was written with the Jan 2009 GPC1 commissioning data release in mind.  There's probably some good integration that could be done with previous such pages by Jester et al.
     2
     3
     4=== Executive Summary ===
     5
     6
     7
     8=== Detailed Step-by-Step guide ===
     9
     10I have written these with the Odyssey cluster installation at Harvard as the example.  Please contact your local DRAVG representative to learn where images are kept at your institution and how to set up the IPP environment.
     11
     12At Harvard we use the 'module load' system to configure things on the cluster
     13
     14 module load hpc/ipp
     15 psconfig default
     16
     17and here's where we keep the data
     18
     19 cd /n/data1/panstarrs/GPC1/MD01_20081002_1
     20
     21In this directory are each of the different observations.  The directories are named by the observation number.  These span the course of several weeks.
     22
     23 [wwoodvas@heroint2 MD01_20081002_1]$ ls
     24 36475/ 36476/  36477/  36478/  36479/  36480/  36481/  36625/  stack/
     25
     26I created the 'stack' directory as a place where we can put our coadd stacked images.  The coadded images were not distributed with this data release.  It wouldn't make sense to put the coadds under any particular directory because they don't belong to any particular observation so I've just created a directory for them here at the base level.
     27
     28Each of the observation number directories contains directories for the raw, chip, warp, and diff images
     29
     30<code>
     31 [wwoodvas@heroint2 MD01_20081002_1]$ ls 36475
     32 chip/  diff/  raw/  warp/
     33</code>
     34
     35There are .log files in these directories detailing the steps and output that were used to create them.  You may find referring to them a useful exercise if you want to understand all of the details of what was done to derive the images. 
     36
     37Since we are not interested in recreating the basic detector calibration at this point we will start with the 'chip' images.
     38
     39=== Photometry and DVO ===
     40
     41 * Photometry.  Basic photometry with 'psphot'
     42
     43<code>
     44cd MD01_20081002_1/36475
     45psphot -D CAMERA gpc1 -file chip/o4741g0234o.36475.ch.11948.XY55.ch.fits -mask chip/o4741g0234o.36475.ch.11948.XY55.ch.mk.fits -weight chip/o4741g0234o.36475.ch.11948.XY55.ch.wt.fits testout
     46</code>
     47will create testout.cmf.  If you want to see what psphot is doing as it runs, you may be interested in the '-visual' option
     48
     49psphot -file chip/o4741g0234o.36475.ch.11948.XY55.ch.fits -mask chip//o4741g0234o.36475.ch.11948.XY55.ch.mk.fits -weight chip/o4741g0234o.36475.ch.11948.XY55.ch.wt.fits testout -visual
     50
     51The main output from the psphot command will be the .cmf file, which is a binary FITS table that contains information about the objects as gleaned from the image.  One of the fields that generates the most questions is the FLAGS value for objects.  This is a bitmask.  See [wiki:IPP_Detection_Bitmasks IPP Detection Bitmasks] for more details.
     52
     53 * DVO.  Let's say we have a set of cmf files and we want to use DVO to combine the detections from these files.  A series of commands such as
     54
     55 cd MD01_20081002_1
     56 addstar -D CAMERA gpc1 -D CATDIR mycatdir -accept-astrom 36475/chip/o4741g0234o.36475.ch.11948.XY55.cmf
     57 addstar -D CAMERA gpc1 -D CATDIR mycatdir -accept-astrom 36476/chip/o4741g0235o.36476.ch.11949.XY55.cmf
     58 addstar -D CAMERA gpc1 -D CATDIR mycatdir -accept-astrom 36477/chip/o4741g0236o.36477.ch.11950.XY55.cmf
     59 addstar -D CAMERA gpc1 -D CATDIR mycatdir -accept-astrom 36478/chip/o4741g0237o.36478.ch.11951.XY55.cmf
     60
     61will create a DVO database and fill it with the detections in the given CMF files.  Since these are the same pointings we should get multiple detections (matches) when we put together the catalog.  Currently -accept-astrom is still necessary until the WCS solutions finalizes and we know our true tolerances for good WCS.  Need other page reference or more notes to explore DVO further.
     62
     63=== Warping, Stacking, and Subtraction ===
     64
     65==== Warping ====
     66
     67 * To create warped images that conform to a given position on the sky, we need to define how we want to break up th sky.  This tessellation is what IPP will use to map the chip images to what we call skycells.
     68
     69 skycells -D CAMERA gpc1 8 -scale 0.2 -D CATDIR default
     70
     71This will create a directory ('default') that will contain FITS files that define the tessellation of the sky.  Now we can recreate the warping of chip images to warp images:
     72
     73First let's create a sandbox for ourselves to put our new files into
     74
     75 mkdir -p workspace/warp workspace/stack workspace/diff
     76
     77Now we'll fill a skycell.  You may wonder how to know what skycell a particular chip will warp to.  That is a good question that currently has an answer that's too complicated to be practical
     78
     79'''Easy RA,Dec->skycell is a prime opportunity for a contribution'''.  [On Feb 4 Mark Huber said he was going to work on this].
     80
     81For now just take the following example:
     82
     83pswarp -file 36475/chip/o4741g0234o.36475.ch.11948.XY33.ch.fits -mask 36475/chip/o4741g0234o.36475.ch.11948.XY33.ch.mk.fits -weight 36475/chip/o4741g0234o.36475.ch.11948.XY33.ch.wt.fits workspace/warp/o4741g0234o.36475.wrp.11948.skycell.02879.51
     84
     85==== Stacking ====
     86
     87If we have a set of images that we have warped to the same skycell and we want to stack (coadd) these images together we first specify a file that defines the images, masks, weights, PSF models, etc. for the input images to the stack:
     88
     89<code><pre>
     90INPUT0  METADATA
     91    IMAGE      STR   36475/warp/o4741g0234o.36475.wrp.1277.skycell.02879.51.fits
     92    MASK       STR   36475/warp/o4741g0234o.36475.wrp.1277.skycell.02879.51.mask.fits
     93    WEIGHT     STR   36475/warp/o4741g0234o.36475.wrp.1277.skycell.02879.51.wt.fits
     94    PSF        STR   36475/warp/o4741g0234o.36475.wrp.1277.skycell.02879.51.psf
     95    SOURCES    STR   36475/warp/o4741g0234o.36475.wrp.1277.skycell.02879.51.cmf
     96    WEIGHTING  F32   1.0
     97    SCALE      F32   1.0
     98END
     99
     100INPUT1  METADATA
     101    IMAGE      STR   36476/warp/o4741g0235o.36476.wrp.1279.skycell.02879.51.fits
     102    MASK       STR   36476/warp/o4741g0235o.36476.wrp.1279.skycell.02879.51.mask.fits
     103    WEIGHT     STR   36476/warp/o4741g0235o.36476.wrp.1279.skycell.02879.51.wt.fits
     104    PSF        STR   36476/warp/o4741g0235o.36476.wrp.1279.skycell.02879.51.psf
     105    SOURCES    STR   36476/warp/o4741g0235o.36476.wrp.1279.skycell.02879.51.cmf
     106    WEIGHTING  F32   1.0
     107    SCALE      F32   1.0
     108END
     109
     110INPUT2  METADATA
     111    IMAGE      STR   36477/warp/o4741g0236o.36477.wrp.1280.skycell.02879.51.fits
     112    MASK       STR   36477/warp/o4741g0236o.36477.wrp.1280.skycell.02879.51.mask.fits
     113    WEIGHT     STR   36477/warp/o4741g0236o.36477.wrp.1280.skycell.02879.51.wt.fits
     114    PSF        STR   36477/warp/o4741g0236o.36477.wrp.1280.skycell.02879.51.psf
     115    SOURCES    STR   36477/warp/o4741g0236o.36477.wrp.1280.skycell.02879.51.cmf
     116    WEIGHTING  F32   1.0
     117    SCALE      F32   1.0
     118END
     119
     120INPUT3  METADATA
     121    IMAGE      STR   36478/warp/o4741g0237o.36478.wrp.1284.skycell.02879.51.fits
     122    MASK       STR   36478/warp/o4741g0237o.36478.wrp.1284.skycell.02879.51.mask.fits
     123    WEIGHT     STR   36478/warp/o4741g0237o.36478.wrp.1284.skycell.02879.51.wt.fits
     124    PSF        STR   36478/warp/o4741g0237o.36478.wrp.1284.skycell.02879.51.psf
     125    SOURCES    STR   36478/warp/o4741g0237o.36478.wrp.1284.skycell.02879.51.cmf
     126    WEIGHTING  F32   1.0
     127    SCALE      F32   1.0
     128END
     129
     130INPUT4  METADATA
     131    IMAGE      STR   36479/warp/o4741g0238o.36479.wrp.1281.skycell.02879.51.fits
     132    MASK       STR   36479/warp/o4741g0238o.36479.wrp.1281.skycell.02879.51.mask.fits
     133    WEIGHT     STR   36479/warp/o4741g0238o.36479.wrp.1281.skycell.02879.51.wt.fits
     134    PSF        STR   36479/warp/o4741g0238o.36479.wrp.1281.skycell.02879.51.psf
     135    SOURCES    STR   36479/warp/o4741g0238o.36479.wrp.1281.skycell.02879.51.cmf
     136    WEIGHTING  F32   1.0
     137    SCALE      F32   1.0
     138END
     139
     140INPUT5  METADATA
     141    IMAGE      STR   36480/warp/o4741g0240o.36480.wrp.1285.skycell.02879.51.fits
     142    MASK       STR   36480/warp/o4741g0240o.36480.wrp.1285.skycell.02879.51.mask.fits
     143    WEIGHT     STR   36480/warp/o4741g0240o.36480.wrp.1285.skycell.02879.51.wt.fits
     144    PSF        STR   36480/warp/o4741g0240o.36480.wrp.1285.skycell.02879.51.psf
     145    SOURCES    STR   36480/warp/o4741g0240o.36480.wrp.1285.skycell.02879.51.cmf
     146    WEIGHTING  F32   1.0
     147    SCALE      F32   1.0
     148END
     149
     150INPUT6  METADATA
     151    IMAGE      STR   36481/warp/o4741g0241o.36481.wrp.1282.skycell.02879.51.fits
     152    MASK       STR   36481/warp/o4741g0241o.36481.wrp.1282.skycell.02879.51.mask.fits
     153    WEIGHT     STR   36481/warp/o4741g0241o.36481.wrp.1282.skycell.02879.51.wt.fits
     154    PSF        STR   36481/warp/o4741g0241o.36481.wrp.1282.skycell.02879.51.psf
     155    SOURCES    STR   36481/warp/o4741g0241o.36481.wrp.1282.skycell.02879.51.cmf
     156    WEIGHTING  F32   1.0
     157    SCALE      F32   1.0
     158END
     159
     160INPUT7  METADATA
     161    IMAGE      STR   36625/warp/o4741g0239o.36625.wrp.1283.skycell.02879.51.fits
     162    MASK       STR   36625/warp/o4741g0239o.36625.wrp.1283.skycell.02879.51.mask.fits
     163    WEIGHT     STR   36625/warp/o4741g0239o.36625.wrp.1283.skycell.02879.51.wt.fits
     164    PSF        STR   36625/warp/o4741g0239o.36625.wrp.1283.skycell.02879.51.psf
     165    SOURCES    STR   36625/warp/o4741g0239o.36625.wrp.1283.skycell.02879.51.cmf
     166    WEIGHTING  F32   1.0
     167    SCALE      F32   1.0
     168END
     169</pre></code>
     170
     171If we called this file 'skycell.02879.51.mwv.002.mdc', then we might run the following to get the output stack
     172
     173 ppStack skycell.02879.51.mwv.002.mdc skycell.02879.51.mwv.002 -sources 36475/warp/o4741g0234o.36475.wrp.1277.skycell.02879.51.cmf
     174
     175(The '-sources' might be redundant since we're specify source lists in the .mdc (meta-data configuration) file).
     176
     177==== Subtraction ====
     178
     179Once we have a coaddition that hopeful has enough dithers to fill in the area we will want to run subtractions of individual images against that coadd template.  In this case we built the coadd from the images themselves, but that can't be helped for now as long as we have limited data.
     180
     181 set ref=skycell.02879.51.mwv.002
     182 set new=36625/warp/o4741g0239o.36625.wrp.1283.skycell.02879.51
     183 ppSub $new.fits $ref.fits sub_isis \
     184    -inmask  $new.mk.fits -inweight  $new.wt.fits \
     185    -refmask $ref.mk.fits -refweight $ref.wt.fits \
     186    -sources $new.cmf -type ISIS -threads 2 -photometry
     187
     188would run a subtraction of $new minus $ref using the ISIS kernels.  ppSub --help will generate a somewhat detailed list of options but they probably won't all make sense on a first reading.
     189
     190Since we asked 'ppSub' to do photometry it will create a difference .cmf file along with the subtraction.  If we had a collection of these difference .cmf files we might use DVO to combine them and if we were looking for stationary transients, we would look for multiple detections at the same position.