Initial/ongoing comparison of IPP to SuperMACHO/ESSENCE Photpipe reductions using GPC1 images from the 2009.01 release.

(don't forget to note the IPP svn-head revision # being used if different a default, say 23243?)

* WARNING work in progress with irregular documentation - please highlight areas lacking *

Action Items:

• Come up with explicit test and verification of pswarp vs. SWarp, ppStack vs SWarp:stack, and apply to idealized(simtest) and real(release.200901) images
  • Does the output match what we think it should?
  • Verify that the actual estimation of the "best" lowest-common-denominator PSF is correct.
  • Verify that the generated convolved stacked image matches what it theoretically should.
  • Verify that the covariance map is correct (Paul has already done this).

• Learn what ppSub expects the stacked image to have so we can pass information from SWarp-generated co-add as if it were an IPP-generated coadd

• Develop non-convolving version of ppStack.
  • Median and un/weighted sums/average combines that keeps track of the image, variance, mask, and PSF information appropriately.

• Run psphot on ppSub images from non-convolved stacks. Compare results with release.200901 diffim. Search for transients in new subtractions.
  • Prototype with Rest/Huber hacks on 10 skycells
  • Full run once re-developed ppStack exists

Basic Data Locations

All original data (particularly the chip level used to remake the warps) is from the release.200901 set, processed with ipp-magic-v0.

• Odyessy: original data (36475/chip,warp,diff etc): /n/data1/panstarrs/GPC1/MD01_20081002_1/

workspace (swarps, ipp-rerun collections): /n/data1/panstarrs/GPC1/MD01_20081002_1/workspace/

• groucho:

photpipe runs (pswarp hotpants etc): ​http://groucho.pha.jhu.edu/PS/smsn_workspace/

local ipp re-runs (uploaded to odyssey): ​http://groucho.pha.jhu.edu/PS/ippsvn/ipp_vs_photpipe/

• huevon:

/data/pipe/v10.0/PS1/GPC1/workspace/

​http://huevon.physics.harvard.edu/arest/smsn_GPC1/

​http://huevon.physics.harvard.edu/arest/ipp_GPC1/

Skycell Layout

Same as default used for 2009.01 release and Basic_GPC1_Processing_for_End_Users

skycells -level 5 -scale 0.2 -nx 8 -ny 8 -fix-ns -padding 0.05 -D CATDIR default 

odyssey:/n/data1/panstarrs/GPC1/MD01_20081002_1/workspace/su_pswarp/default

Test chip/skycell Sample

Hard to clearly tell where chip boundaries are, so make skycell set with just single chip. (these chosen because they are already mostly made and provide a center vs better focus ring sample). May need another sample skycell near chip XY22 with other chip contribution to test multi-chip psf matching for ppStack/ppSub.

• skycell.02877.31: focus on chip XY33 (~50% of field, center of camera and tent region)

   RA,Dec center = 02:23:37.250 -04:20:03.53 J2000  35.90521  -4.33431
   X,Y center (size) = 2754, 2003 (5508,4006)
   platescale = 0.2
  

i.e. odyssey:/n/data1/panstarrs/GPC1/MD01_20081002_1/36475/chip/o4741g0234o.36475.ch.11948.XY33.ch.fits

MD01_20081002_1/36475/chip/o4741g0234o.36475.ch.11948.XY33.ch.fits

MD01_20081002_1/36476/chip/o4741g0235o.36476.ch.11949.XY33.ch.fits

MD01_20081002_1/36477/chip/o4741g0236o.36477.ch.11950.XY33.ch.fits

MD01_20081002_1/36478/chip/o4741g0237o.36478.ch.11951.XY33.ch.fits

MD01_20081002_1/36479/chip/o4741g0238o.36479.ch.11952.XY33.ch.fits

MD01_20081002_1/36480/chip/o4741g0240o.36480.ch.11953.XY33.ch.fits

MD01_20081002_1/36481/chip/o4741g0241o.36481.ch.11954.XY33.ch.fits

MD01_20081002_1/36625/chip/o4741g0239o.36625.ch.11955.XY33.ch.fits

• skycell.02879.00: focus on chip XY22 (~90% of field, in better focus region around center. Note, large masked region in o4741g0237o.36478.ch.11951.XY22.ch.fits)

   RA,Dec center = 02:24:32.502 -05:00:06.29 J2000  36.09376  -5.00175
   X,Y center (size) = 2754, 2003 (5508,4006)
   platescale = 0.2
  

i.e. odyssey:/n/data1/panstarrs/GPC1/MD01_20081002_1/36475/chip/o4741g0234o.36475.ch.11948.XY22.ch.fits

MD01_20081002_1/36475/chip/o4741g0234o.36475.ch.11948.XY22.ch.fits

MD01_20081002_1/36476/chip/o4741g0235o.36476.ch.11949.XY22.ch.fits

MD01_20081002_1/36477/chip/o4741g0236o.36477.ch.11950.XY22.ch.fits

MD01_20081002_1/36478/chip/o4741g0237o.36478.ch.11951.XY22.ch.fits

MD01_20081002_1/36479/chip/o4741g0238o.36479.ch.11952.XY22.ch.fits

MD01_20081002_1/36480/chip/o4741g0240o.36480.ch.11953.XY22.ch.fits

MD01_20081002_1/36481/chip/o4741g0241o.36481.ch.11954.XY22.ch.fits

MD01_20081002_1/36625/chip/o4741g0239o.36625.ch.11955.XY22.ch.fits

Warping (PSwarp and SWarp)

Check SWarp and PSWarp on the same chip to same skycell level using the following

• pswarps - uncompressed, single FITS files in

odyssey:/n/data1/panstarrs/GPC1/MD01_20081002_1/workspace/su_pswarp/

​http://groucho.pha.jhu.edu/PS/ippsvn/ipp_vs_photpipe/ppsub_pswarps/

fully filled skycell: suo4741g0234o.36475.wrp.11948.all.skycell.02877.31.fits
                      suo4741g0234o.36475.wrp.11948.all.skycell.02879.00.fits
single chip skycell: suo4741g0234o.36475.wrp.11948.XY33.skycell.02877.31.fits
                     suo4741g0234o.36475.wrp.11948.XY22.skycell.02879.00.fits

• SWarps: Armin notes some correlated noise still apparent in pswarps compared to SWarps Lanczos3 vs 4?

• need to look at Lanczos w/ masking and boundaries - mask growth still different w/ photpipe
  • photpipe has good/ok/poor set, pswarp?

Differencing (ppSub and hotpants)

Look at before stacking to since ppStack uses the same routines/convolution as ppSub. Try ppSub on both pswarps and swarps to look at differences.

ppSub [pswarp-pswarp, swarp-swarp, pswarp-ppStack, pswarp-swarp:stack]

odyssey:/n/data1/panstarrs/GPC1/MD01_20081002_1/workspace/meh_ppsub_pswarp/

​http://groucho.pha.jhu.edu/PS/ippsvn/ipp_vs_photpipe/ppsub_pswarps/

• ppSub [pswarp X-pswarp Y]:

odyssey: subisis.36478_36476.wrp.XY33.skycell.02877.31.fits

​http://groucho.pha.jhu.edu/PS/ippsvn/ipp_vs_photpipe/ppsub_pswarps/subisis.36478_36476.wrp.XY33.skycell.02877.31.fits

odyssey: subisis.36477_36475.wrp.XY22.skycell.02879.00.fits

​http://groucho.pha.jhu.edu/PS/ippsvn/ipp_vs_photpipe/ppsub_pswarps/subisis.36477_36475.wrp.XY22.skycell.02879.00.fits

xy33/02877.31, default settings, 36478-36476	xy22/02879.00, default settings, 36477-36475

Several other combos for skycell.02877.31, XY33 -

subisis.36475_36479.wrp.XY33.skycell.02877.31.fits
subisis.36477_36478.wrp.XY33.skycell.02877.31.fits
subisis.36477_36479.wrp.XY33.skycell.02877.31.fits	tetris pattern
subisis.36479_36477.wrp.XY33.skycell.02877.31.fits	reversed no pattern
subisis.36477_36625.wrp.XY33.skycell.02877.31.fits
subisis.36478_36476.wrp.XY33.skycell.02877.31.fits
subisis.36478_36478.wrp.XY33.skycell.02877.31.fits
subisis.36480_36479.wrp.XY33.skycell.02877.31.fits
subisis.36481_36479.wrp.XY33.skycell.02877.31.fits
subisis.36625_36477.wrp.XY33.skycell.02877.31.fits
subisis.36625_36479.wrp.XY33.skycell.02877.31.fits
36481-36475	can't make stamps
36477-36475	can't make stamps

• ppSub [pswarp X-pswarp X]: silly test subtracting a warp from itself, 36478 for skycell.02877.31, XY33

odyssey: subisis.36478_36478.wrp.XY33.skycell.02877.31.fits

​http://groucho.pha.jhu.edu/PS/ippsvn/ipp_vs_photpipe/ppsub_pswarps/subisis.36478_36478.wrp.XY33.skycell.02877.31.fits

36478-36478 for skycell.02877.31 XY33

• ppSub [SWarp-SWarp:stack]: need SWarp'd images and setup for what ppSub needs added to header, weight/mask files

• ppSub [pswarp-SWarp:stack]: need to learn what ppSub needs added to header, how to make weight and mask files

• ppSub [pswarp-ppStack]: need to setup ppStack w+w/o convolution

​http://groucho.pha.jhu.edu/PS/ippsvn/ipp_vs_photpipe/ppsub_pswarp_ppstack

Hotpants [pswarp-pswarp, pswarp-ppStack, swarp-ppStack, pswarp-SWarp:stack]

• hotpants [pswarp-pswarp]:
  • pswarps converted to photpipe format with convert_ippwarp2smsn.pl
    • fields are skycell.02877.31==f02877_31, 02879.00==f02879.00

• test sample, fully filled skycells - will make amp 0 soon

36480-36475: ​http://groucho.pha.jhu.edu/PS/smsn_workspace/ut081002meh80_ut081002meh75/1/

36478-36475: ​http://groucho.pha.jhu.edu/PS/smsn_workspace/ut081002meh78_ut081002meh75/1/

• for skycells with single chip only inclusion, amp=chip#

36477-36475: ​http://groucho.pha.jhu.edu/PS/smsn_workspace/ut081002meh77_ut081002meh75/22

36475_XY22	36477_XY22
ppSub	hotpants

36625-26479: ​http://groucho.pha.jhu.edu/PS/smsn_workspace/ut081002meh625_ut081002meh79/33

36479_XY33	36625_XY33
hotpants	ppSub

​http://groucho.pha.jhu.edu/PS/smsn_workspace/

36479-36478	hotpants had problems
36625-36477
36625-36479
36625-36480
36476-36475
36477-36475
36477-36476
36480-36475
36480-36479
36481-36479
36481-36480

• hotpants [pswarp-ppStack]: convert the ppStack file like a pswarp using convert_ippwarp2smsn.pl

​http://groucho.pha.jhu.edu/PS/smsn_workspace/ut081002meh80_ut081002mehstk4a/1/

• hotpants [swarp-ppStack]: need to setup ppStack w+w/o convolution

• hotpants [pswarp-SWarp:stack]: need photpipe setup for SWarp:stack, weight/mask files?

Stacking (ppStack and SWarp)

ppStack [pswarps,pswarps:diffims, SWarps,SWarp:diffims]

odyssey:/n/data1/panstarrs/GPC1/MD01_20081002_1/workspace/meh_ppstack

​http://groucho.pha.jhu.edu/PS/ippsvn/ipp_vs_photpipe/ppstack_pswarps/

• ppStack [pswarps WXYZ]: test combination 1st 4 imgs(a), 2nd 4(b), all 8

skycell.02879.31.stk4a.fits

skycell.02879.31.stk4b.fits

skycell.02879.31.stk8.fits

• ppStack [pswarps XXXX]: silly test combining 4 copies of the same image

odyssey: skycell.02879.31.stk4a78.fits

​http://groucho.pha.jhu.edu/PS/ippsvn/ipp_vs_photpipe/ppstack_pswarps/skycell.02879.31.stk4a78.fits

• ppStack [SWarps]: SWarp skycells need to be built

• ppStack [ppSubs:pswarps]: stack of ppSub:pswarp diffims (and for SWarps)

• ppStack [hotpants:pswarps]: stack of hotpants:pswarp diffims (and for SWarps)

SWarp [pswarps, pswarp:diffims, SWarps,SWarp:diffims]

• SWarp [pswarps]: more info needed from Armin and needs to be resampled into skycell boundaries.

odyssey:/n/data1/panstarrs/GPC1/MD01_20081002_1/workspace/MYDIR

​http://groucho.pha.jhu.edu/PS/ippsvn/ipp_vs_photpipe/swarpstk_pswarps

Note that a set of 4 okay, doing set of has an odd result (in what way Armin?)

• SWarp [SWarps]: SWarp skycells need to be built

• SWarp [ppSub:pswarps]: SWarp stack of ppSub:pswarp diffims (and for SWarps)

Initial stack test stats --

PSWarp
 imstats suo4741g0234o.36475.wrp.11948.all.skycell.02877.31.fits -box 4948 2331 80 80
 STATS: bkg:1.608557 sigma_bkg:24.464390 Nused:6382 Nclipped:18 lowest_val:-85.380753 highest_val:88.475899

SWarp:stack
 imstats 36475-36478.fits -box 4948 2331 80 80
 STATS: bkg:-0.985735 sigma_bkg:12.967378 Nused:6382 Nclipped:18 lowest_val:-52.730080 highest_val:55.483871

ppStack first 4
 imstats skycell.02879.31.stk4a.fits -box 4948 2331 80 80
 STATS: bkg:-0.324937 sigma_bkg:2.308740 Nused:6225 Nclipped:175 lowest_val:-5.852595 highest_val:10.131362

ppStack last 4
 imstats skycell.02879.31.stk4b.fits -box 4948 2331 80 80
 STATS: bkg:-7.943039 sigma_bkg:6.104924 Nused:6304 Nclipped:6 lowest_val:-25.602289 highest_val:12.588408

ppStack all 8
 imstats skycell.02879.31.stk8.fits -box 4948 2331 80 80
 STATS: bkg:-2.741246 sigma_bkg:5.151842 Nused:6216 Nclipped:184 lowest_val:-17.833969 highest_val:21.132132

 -----

PSWarp, new position 
 imstats suo4741g0234o.36475.wrp.11948.all.skycell.02877.31.fits -box 2650 2180 80 80
 STATS: bkg:4.214430 sigma_bkg:28.889675 Nused:3614 Nclipped:149 lowest_val:-4511.794922 highest_val:4742.093750

ppStack all 8, new position
 imstats skycell.02879.31.stk8.fits -box 2650 2180 80 80
 STATS: bkg:0.223009 sigma_bkg:5.304905 Nused:6396 Nclipped:4 lowest_val:-14.943192 highest_val:17.027874

Photometry (psphot and dophot/pixie)