Basic_GPC1_Processing_IPP_Photpipe_Comparison – Pan-STARRS IPP

wiki:Basic_GPC1_Processing_IPP_Photpipe_Comparison

Context Navigation

Version 45 (modified by Mark Huber, 17 years ago) ( diff )
--

Initial/ongoing comparison of IPP to SuperMACHO/ESSENCE Photpipe reductions using GPC1 images from the 2009.01 release.

* WARNING work in progress with irregular documentation - please highlight areas lacking * (don't forget to note the IPP svn-head revision # being used if different from a default, say 23243 post-pswarp fix? also test runs with all default set options, need to address this somehow as well.) * note - many skycell.02877.31 typo'd as 02879.31 including data runs, need to fix

Action Items:

Come up with explicit test and verification of pswarp vs. SWarp, ppStack vs SWarp:stack, and apply to idealized(simtest) and real(release.200901) images
- Does the output match what we think it should?
- Verify that the actual estimation of the "best" lowest-common-denominator PSF is correct.
- Verify that the generated convolved stacked image matches what it theoretically should.
- Verify that the covariance map is correct (Paul has already done this).

Learn what ppSub expects the stacked image to have so we can pass information from SWarp-generated co-add as if it were an IPP-generated coadd
- ppSub needs SWarp:stack put into skycell dimensions (use swarp_single to setup each img first)
- ppSub needs SWarp:stack x flipped
- ppSub needs wt and mk from SWarp:stack

Develop non-convolving version of ppStack.
- Median and un/weighted sums/average combines that keeps track of the image, variance, mask, and PSF information appropriately.

Paul has added the non-convolving feature to ppStack on trunk, r23577. The behaviour is controlled by the PPSTACK recipe:
- Convolution can be turned off by setting CONVOLVE = FALSE.
- All rejection can be turned off by setting ITER = 0.

Run psphot on ppSub images from non-convolved stacks. Compare results with release.200901 diffim. Search for transients in new subtractions.
- Prototype with Rest/Huber hacks on 10 skycells
- Full run once re-developed ppStack exists

IPP revisions

r23243 - post-pswarp fix, much of the groucho data originally run with this
r23602 - post-ppStack convolution option fix

Basic Data Locations

All original data (particularly the chip level used to remake the warps) is from the release.200901 set, processed with ipp-magic-v0.

Odyessy: original data (36475/chip,warp,diff etc): /n/data1/panstarrs/GPC1/MD01_20081002_1/

workspace (swarps, ipp-rerun collections): /n/data1/panstarrs/GPC1/MD01_20081002_1/workspace/

groucho:

photpipe runs (pswarp hotpants etc): http://groucho.pha.jhu.edu/PS/smsn_workspace/

local ipp re-runs (uploaded to odyssey): http://groucho.pha.jhu.edu/PS/ippsvn/ipp_vs_photpipe/

huevon:

/data/pipe/v10.0/PS1/GPC1/workspace/

http://huevon.physics.harvard.edu/arest/smsn_GPC1/

http://huevon.physics.harvard.edu/arest/ipp_GPC1/

Skycell Layout

Same as default used for 2009.01 release and Basic_GPC1_Processing_for_End_Users

skycells -level 5 -scale 0.2 -nx 8 -ny 8 -fix-ns -padding 0.05 -D CATDIR default

odyssey:/n/data1/panstarrs/GPC1/MD01_20081002_1/workspace/su_pswarp/default

Test chip/skycell Sample

Hard to clearly tell where chip boundaries are, so make skycell set with just single chip. (these chosen because they are already mostly made and provide a center vs better focus ring sample). May need another sample skycell near chip XY22 with other chip contribution to test multi-chip psf matching for ppStack/ppSub.

skycell.02877.31: focus on chip XY33 (~50% of field, center of camera and tent region)

 RA,Dec center = 02:23:37.250 -04:20:03.53 J2000  35.90521  -4.33431
 X,Y center (size) = 2754, 2003 (5508,4006)
 platescale = 0.2

i.e. odyssey:/n/data1/panstarrs/GPC1/MD01_20081002_1/36475/chip/o4741g0234o.36475.ch.11948.XY33.ch.fits

MD01_20081002_1/36475/chip/o4741g0234o.36475.ch.11948.XY33.ch.fits
MD01_20081002_1/36476/chip/o4741g0235o.36476.ch.11949.XY33.ch.fits
MD01_20081002_1/36477/chip/o4741g0236o.36477.ch.11950.XY33.ch.fits
MD01_20081002_1/36478/chip/o4741g0237o.36478.ch.11951.XY33.ch.fits
MD01_20081002_1/36479/chip/o4741g0238o.36479.ch.11952.XY33.ch.fits
MD01_20081002_1/36480/chip/o4741g0240o.36480.ch.11953.XY33.ch.fits
MD01_20081002_1/36481/chip/o4741g0241o.36481.ch.11954.XY33.ch.fits
MD01_20081002_1/36625/chip/o4741g0239o.36625.ch.11955.XY33.ch.fits

skycell.02879.00: focus on chip XY22 (~90% of field, in better focus region around center. Note, large masked region in o4741g0237o.36478.ch.11951.XY22.ch.fits)
```
 RA,Dec center = 02:24:32.502 -05:00:06.29 J2000  36.09376  -5.00175
 X,Y center (size) = 2754, 2003 (5508,4006)
 platescale = 0.2
```

i.e. odyssey:/n/data1/panstarrs/GPC1/MD01_20081002_1/36475/chip/o4741g0234o.36475.ch.11948.XY22.ch.fits

MD01_20081002_1/36475/chip/o4741g0234o.36475.ch.11948.XY22.ch.fits
MD01_20081002_1/36476/chip/o4741g0235o.36476.ch.11949.XY22.ch.fits
MD01_20081002_1/36477/chip/o4741g0236o.36477.ch.11950.XY22.ch.fits
MD01_20081002_1/36478/chip/o4741g0237o.36478.ch.11951.XY22.ch.fits
MD01_20081002_1/36479/chip/o4741g0238o.36479.ch.11952.XY22.ch.fits
MD01_20081002_1/36480/chip/o4741g0240o.36480.ch.11953.XY22.ch.fits
MD01_20081002_1/36481/chip/o4741g0241o.36481.ch.11954.XY22.ch.fits
MD01_20081002_1/36625/chip/o4741g0239o.36625.ch.11955.XY22.ch.fits

Warping (PSwarp and SWarp)

Check SWarp and PSWarp on the same chip to same skycell level using the following

pswarps - uncompressed, single FITS files in

odyssey:/n/data1/panstarrs/GPC1/MD01_20081002_1/workspace/su_pswarp/

http://groucho.pha.jhu.edu/PS/ippsvn/ipp_vs_photpipe/ppsub_pswarps/

fully filled skycell: suo4741g0234o.36475.wrp.11948.all.skycell.02877.31.fits
                      suo4741g0234o.36475.wrp.11948.all.skycell.02879.00.fits
single chip skycell: suo4741g0234o.36475.wrp.11948.XY33.skycell.02877.31.fits
                     suo4741g0234o.36475.wrp.11948.XY22.skycell.02879.00.fits

SWarps: Armin notes some correlated noise still apparent in pswarps compared to SWarps Lanczos3 vs 4?

need to look at Lanczos w/ masking and boundaries - mask growth still different w/ photpipe
- photpipe has good/ok/poor set, pswarp?

Differencing (ppSub and hotpants)

Look at before stacking to since ppStack uses the same routines/convolution as ppSub. Try ppSub on both pswarps and swarps to look at differences.

ppSub [pswarp-pswarp, swarp-swarp, pswarp-ppStack, pswarp-swarp:stack]

odyssey:/n/data1/panstarrs/GPC1/MD01_20081002_1/workspace/meh_ppsub_pswarp/

http://groucho.pha.jhu.edu/PS/ippsvn/ipp_vs_photpipe/ppsub_pswarps/

ppSub [pswarp X-pswarp Y]:

odyssey: subisis.36478_36476.wrp.XY33.skycell.02877.31.fits

http://groucho.pha.jhu.edu/PS/ippsvn/ipp_vs_photpipe/ppsub_pswarps/subisis.36478_36476.wrp.XY33.skycell.02877.31.fits

odyssey: subisis.36477_36475.wrp.XY22.skycell.02879.00.fits

http://groucho.pha.jhu.edu/PS/ippsvn/ipp_vs_photpipe/ppsub_pswarps/subisis.36477_36475.wrp.XY22.skycell.02879.00.fits

xy33/02877.31, default settings, 36478-36476 xy22/02879.00, default settings, 36477-36475

Several other combos for skycell.02877.31, XY33 -

subisis.36475_36479.wrp.XY33.skycell.02877.31.fits
subisis.36477_36478.wrp.XY33.skycell.02877.31.fits
subisis.36477_36479.wrp.XY33.skycell.02877.31.fits tetris pattern
subisis.36479_36477.wrp.XY33.skycell.02877.31.fits reversed no pattern
subisis.36477_36625.wrp.XY33.skycell.02877.31.fits
subisis.36478_36476.wrp.XY33.skycell.02877.31.fits
subisis.36478_36478.wrp.XY33.skycell.02877.31.fits
subisis.36480_36479.wrp.XY33.skycell.02877.31.fits
subisis.36481_36479.wrp.XY33.skycell.02877.31.fits
subisis.36625_36477.wrp.XY33.skycell.02877.31.fits
subisis.36625_36479.wrp.XY33.skycell.02877.31.fits
36481-36475 can't make stamps
36477-36475 can't make stamps

ppSub [pswarp X-pswarp X]: silly test subtracting a warp from itself, 36478 for skycell.02877.31, XY33

odyssey: subisis.36478_36478.wrp.XY33.skycell.02877.31.fits

http://groucho.pha.jhu.edu/PS/ippsvn/ipp_vs_photpipe/ppsub_pswarps/subisis.36478_36478.wrp.XY33.skycell.02877.31.fits

36478-36478 for skycell.02877.31 XY33

ppSub [SWarp-SWarp:stack]: need SWarp'd images and setup for what ppSub needs added to header, weight/mask files

ppSub [pswarp-SWarp:stack]: need to learn what ppSub needs added to header, how to make weight and mask files

ppSub [pswarp-ppStack]: need to setup ppStack w+w/o convolution

http://groucho.pha.jhu.edu/PS/ippsvn/ipp_vs_photpipe/ppsub_pswarp_ppstack

Hotpants [pswarp-pswarp, pswarp-ppStack, swarp-ppStack, pswarp-SWarp:stack]

hotpants [pswarp-pswarp]:
- pswarps converted to photpipe format with convert_ippwarp2smsn.pl
  - fields are skycell.02877.31==f02877_31, 02879.00==f02879.00

test sample, fully filled skycells - will make amp 0 soon

36480-36475: http://groucho.pha.jhu.edu/PS/smsn_workspace/ut081002meh80_ut081002meh75/1/

36478-36475: http://groucho.pha.jhu.edu/PS/smsn_workspace/ut081002meh78_ut081002meh75/1/

for skycells with single chip only inclusion, amp=chip#

36477-36475: http://groucho.pha.jhu.edu/PS/smsn_workspace/ut081002meh77_ut081002meh75/22

36475_XY22 36477_XY22
ppSub hotpants

36625-26479: http://groucho.pha.jhu.edu/PS/smsn_workspace/ut081002meh625_ut081002meh79/33

36479_XY33 36625_XY33
hotpants ppSub

http://groucho.pha.jhu.edu/PS/smsn_workspace/

36479-36478 hotpants had problems
36625-36477
36625-36479
36625-36480
36476-36475
36477-36475
36477-36476
36480-36475
36480-36479
36481-36479
36481-36480

hotpants [pswarp-ppStack]: convert the ppStack file like a pswarp using convert_ippwarp2smsn.pl

http://groucho.pha.jhu.edu/PS/smsn_workspace/ut081002meh80_ut081002mehstk4a/1/

hotpants [swarp-ppStack]: need to setup ppStack w+w/o convolution

hotpants [pswarp-SWarp:stack]: need photpipe setup for SWarp:stack, weight/mask files?

Stacking (ppStack and SWarp)

ppStack [pswarps,pswarps:diffims, SWarps,SWarp:diffims]

odyssey:/n/data1/panstarrs/GPC1/MD01_20081002_1/workspace/meh_ppstack

http://groucho.pha.jhu.edu/PS/ippsvn/ipp_vs_photpipe/ppstack_pswarps/

ppStack [pswarps WXYZ]: test combination 1st 4 imgs(a), 2nd 4(b), all 8

skycell.02879.31.stk4a.fits
skycell.02879.31.stk4b.fits
skycell.02879.31.stk8.fits

ppStack [pswarps XXXX]: silly test combining 4 copies of the same image

odyssey: skycell.02879.31.stk4a78.fits

http://groucho.pha.jhu.edu/PS/ippsvn/ipp_vs_photpipe/ppstack_pswarps/skycell.02879.31.stk4a78.fits

ppStack [pswarps, r23602 test]: dupe of previous without convolution

http://groucho.pha.jhu.edu/PS/ippsvn/ipp_vs_photpipe/ppstack_pswarps_r23602/skycell.02877.31.stk4a.fits

http://groucho.pha.jhu.edu/PS/ippsvn/ipp_vs_photpipe/ppstack_pswarps_r23602/skycell.02877.31.stk4a78.fits

ppStack [SWarps]: SWarp skycells need to be built

ppStack [ppSubs:pswarps]: stack of ppSub:pswarp diffims (and for SWarps)

ppStack [hotpants:pswarps]: stack of hotpants:pswarp diffims (and for SWarps)

SWarp [pswarps, pswarp:diffims, SWarps,SWarp:diffims]

SWarp [pswarps]: more info needed from Armin and needs to be resampled into skycell boundaries.

odyssey:/n/data1/panstarrs/GPC1/MD01_20081002_1/workspace/MYDIR

http://groucho.pha.jhu.edu/PS/ippsvn/ipp_vs_photpipe/swarpstk_pswarps

Note that a set of 4 okay, doing set of has an odd result (in what way Armin?)

SWarp [SWarps]: SWarp skycells need to be built

SWarp [ppSub:pswarps]: SWarp stack of ppSub:pswarp diffims (and for SWarps)

Initial stack test stats --

PSWarp
 imstats suo4741g0234o.36475.wrp.11948.all.skycell.02877.31.fits -box 4948 2331 80 80
 STATS: bkg:1.608557 sigma_bkg:24.464390 Nused:6382 Nclipped:18 lowest_val:-85.380753 highest_val:88.475899

SWarp:stack
 imstats 36475-36478.fits -box 4948 2331 80 80
 STATS: bkg:-0.985735 sigma_bkg:12.967378 Nused:6382 Nclipped:18 lowest_val:-52.730080 highest_val:55.483871

ppStack first 4
 imstats skycell.02879.31.stk4a.fits -box 4948 2331 80 80
 STATS: bkg:-0.324937 sigma_bkg:2.308740 Nused:6225 Nclipped:175 lowest_val:-5.852595 highest_val:10.131362

ppStack last 4
 imstats skycell.02879.31.stk4b.fits -box 4948 2331 80 80
 STATS: bkg:-7.943039 sigma_bkg:6.104924 Nused:6304 Nclipped:6 lowest_val:-25.602289 highest_val:12.588408

ppStack all 8
 imstats skycell.02879.31.stk8.fits -box 4948 2331 80 80
 STATS: bkg:-2.741246 sigma_bkg:5.151842 Nused:6216 Nclipped:184 lowest_val:-17.833969 highest_val:21.132132

 -----

PSWarp, new position 
 imstats suo4741g0234o.36475.wrp.11948.all.skycell.02877.31.fits -box 2650 2180 80 80
 STATS: bkg:4.214430 sigma_bkg:28.889675 Nused:3614 Nclipped:149 lowest_val:-4511.794922 highest_val:4742.093750

ppStack all 8, new position
 imstats skycell.02879.31.stk8.fits -box 2650 2180 80 80
 STATS: bkg:0.223009 sigma_bkg:5.304905 Nused:6396 Nclipped:4 lowest_val:-14.943192 highest_val:17.027874

Photometry (psphot and dophot/pixie)

Attachments (12)

stktest.ds9.jpg (100.2 KB ) - added by Mark Huber 17 years ago.
chip33_skycell02877.31.ds9.jpg (157.2 KB ) - added by Mark Huber 17 years ago. chip XY33 pswarp'd into skycell.02877.31
chip33_all_skycell02877.31.ds9.jpg (243.6 KB ) - added by Mark Huber 17 years ago. all relevant chips pswarp'd into skycell.02877.31
chip22_skycell02879.00.ds9.jpg (198.8 KB ) - added by Mark Huber 17 years ago. chip XY22 pswarp'd into skycell.02879.00
subisis_default_n78_r76_xy33_skycell02877.31.ds9.jpg (176.4 KB ) - added by Mark Huber 17 years ago. ppsub of pswarps xy33 in 02877.31 with default settings 36478-36476
subisis_default_n77_r75_xy22_skycell02879.00.ds9.jpg (220.6 KB ) - added by Mark Huber 17 years ago. ppsub of pswarps xy22 in 02879.00 with default settings 36477-36475
ppsub_hotpants_default_n77_r75_xy22_skycell02879.00.ds9.jpg (178.4 KB ) - added by Mark Huber 17 years ago. ppsub and hotpants on pswarps 36477-36475 XY22 skycell.02879.00
ppsub_hotpants_default_n625_r79_xy33_skycell02877.31.ds9.jpg (172.2 KB ) - added by Mark Huber 17 years ago. ppsub and hotpants on pswarps 36625-36479 XY33 skycell.02877.31
subisis_default_n78_r78_xy33_skycell02877.31.ds9.jpg (160.5 KB ) - added by Mark Huber 17 years ago. ppsub on same pswarps 36478-36478 XY33 skycell.02877.31
skycell.02879.31.stk4a78.ds9.jpg (158.6 KB ) - added by Mark Huber 17 years ago. ppStack of 4 copies of pswarp 36478
zoom_chip22_skycell02879.00.ds9.jpg (253.2 KB ) - added by Mark Huber 17 years ago. sample zoomed region on same scale for skycell.02879.00, XY22
zoom_chip33_skycell02877.31.ds9.jpg (245.1 KB ) - added by Mark Huber 17 years ago. sample zoomed region on same scale for skycell.02877.31, XY33

Note: See TracWiki for help on using the wiki.

Download in other formats:

Plain Text

xy33/02877.31, default settings, 36478-36476	xy22/02879.00, default settings, 36477-36475

subisis.36475_36479.wrp.XY33.skycell.02877.31.fits
subisis.36477_36478.wrp.XY33.skycell.02877.31.fits
subisis.36477_36479.wrp.XY33.skycell.02877.31.fits	tetris pattern
subisis.36479_36477.wrp.XY33.skycell.02877.31.fits	reversed no pattern
subisis.36477_36625.wrp.XY33.skycell.02877.31.fits
subisis.36478_36476.wrp.XY33.skycell.02877.31.fits
subisis.36478_36478.wrp.XY33.skycell.02877.31.fits
subisis.36480_36479.wrp.XY33.skycell.02877.31.fits
subisis.36481_36479.wrp.XY33.skycell.02877.31.fits
subisis.36625_36477.wrp.XY33.skycell.02877.31.fits
subisis.36625_36479.wrp.XY33.skycell.02877.31.fits
36481-36475	can't make stamps
36477-36475	can't make stamps

36479-36478	hotpants had problems
36625-36477
36625-36479
36625-36480
36476-36475
36477-36475
36477-36476
36480-36475
36480-36479
36481-36479
36481-36480