Changes between Initial Version and Version 1 of SimtestVerification

Timestamp:

Feb 24, 2009, 4:23:58 PM (17 years ago)

Author:

trac

Comment:

--

SimtestVerification

@@ +1 @@
+= Aim of verification testing =
+
+Create a suite of automated tests and diagnostics for comparing IPP outputs against some external "truth", be that generated by simtest or ppSim in general, or by other pipelines such as SDSS <code>photo</code> or that of the Essence/SuperMacho project.
+
+= People =
+
+ * Eric Bell (bell at mpia.de) is working on verifying that ppSim produces consistent images and catalogs
+ * Mark Huber (mhuber at pha.jhu.edu) is interested in the Essence/SuperMacho comparison and in setting up a generic system
+ * Sebastian Jester (jester at mpia.de) is interested in the SDSS comparison and in settin gup a generic system
+ * Bill Giebink (giebink at ifa.hawaii.edu) is setting up an automated system for running simtest, and we hope he will automate the verification scripts, too
+
+= cvs =
+
+Verification-related scripts live in IPP/ippTests. At the moment (2008-09-29), there is a python script for comparing SDSS and IPP outputs which I hope can be modularized into something extendable. The script lives in a branch <code>sj_ippTests_branch_20080929</code>; to get it from cvs into a new directory IPP_ippTests, say
+
+ cvs co -r sj_ippTests_branch_20080929 -d IPP_ippTests
+
+= Comparison metrics =
+
+'''NOTE''' that in the PS1_DEV_1 format for .cmf files, there are only columns for PSF-fitting quantities. However, for objects that are recognized as extended <code>(FLAGS & 0x4000 != 0)</code>, some quantities are in fact based on the extended-model fit, ''not'' the PSF fit, e.g. the <code>X,Y_PSF_SIGMA</code> position errors. Hence, for the time being one can compare all IPP PSF quantities directly against other PSF quantities only for point-like objects (e.g. in SDSS, a PSF quantities always reports the result of a PSF fit).
+
+'''Also NOTE''' that IPP at the moment does not apply any aperture corrections by default. Somewhere post-2.6.1, IPP switched to determining a variable PSF by default.
+
+== SDSS ==
+
+For the SDSS comparison, I am looking at the differences between IPP and SDSS versions of the following columns, computing the mean, RMS and the three quartiles in each case:
+ * Object centroid & error (there is a global 0.5,0.5 offset) 
+ * PSF magnitude & error
+ * Sky level & error
+ * Pointsourceness (1 or 0)
+
+Ideas that are not yet implemented are:
+
+ * For classification columns (like pointsourceness), give comparison matrix
+ * Copy values of external parameters such as seeing, air mass to output file (for added complication, at least in the case of SDSS, these are in a different table from the object measurements)
+ * Plots:
+  * Scatter plot of object centroid offsets
+  * Magnitude histograms / "completeness" fractions
+
+
+Plots could be per-field or per-camcol or per-run.
+
+= (Re)-Implementing for automated runs = 
+
+I wrote a python script because that's what I could get started with most quickly. But in the long run, we should use whatever's most maintainable and integrated with the rest of IPP. I am imagining something that has:
+ * A reasonably flexible system for specifying which measurements to compare, and how they map between IPP and external data files (e.g. something like the camera config system, but applying to photometry tables not images - i.e. internal concepts and how they map to columns in fits tables)
+ * Re-usable input/output routines
+ * Re-usable plotting routines
+
+= Results =
+
+== SDSS ==
+
+A sample summary .fits table for the SDSS files I have IPP'ed so far is at [http://www.mpia.de/homes/jester/prod_20080815_comp.fits http://www.mpia.de/homes/jester/prod_20080815_comp.fits]
+
+== Eric's work on ppSim ==
+
+I have fitted Gaussians both to an image created by ppSim via simtest and an equivalent image that Eric created with Gaussian stars and "sky noise" (but 0 mean background), with the true counts and positions as initial guesses for the fitter. Both are supposed to represent the sources in simtest.1.00a.cmf (i.e. 1" seeing FWHM).
+
+The result: the fit to Eric's Gaussians gives counts and FWHMs that are very close to the input counts, but ppSim creates broader obects with about 10% fewer counts fitted. Eric's positions seem random within a 0.5x0.5 pix box, while the ppSim positions seem more or less Gauss-distributed over the true center (the 1,1-pix offset may be an artefact of my fitting).
+
+The input files are simtest.1.00a.cmf.fits (input object list), simtest.1.00a.fits (ppSim output) and efbsim.1.00a.fits (Eric's simulated image with Gaussians), the table with the fit results is [[Image(htdocs:/images/Gaussfit_efbsim.1.00a_simtest.1.00a.fits)]] and there are also images with the residuals for Eric's and the ppSim image. The image files are too big for this wiki and available from jester @ mpia.de
+
+The first 3 plots are comparing the fit results for 1" seeing; the rest compare 1" vs. 1.5" seeing, and fitting pixels within 9pix and 3pix of the peak:
+
+<gallery>
+Image:histo_fwhm_x_Eric_ppSim.gif|Histogram of fitted x-FWHMs
+Image:counts_Eric_ppSim.gif|Fitted vs. input counts
+Image:offsets_Eric_ppSim.gif|Fitted vs. input positional offsets
+Image:counts_loglog_Eric_ppSim_r39_FWHM1_1.5.gif|Log-log of Fitted/input vs. input counts, for Eric's version (red), fit to ppSim for all pixels within r=9pix from center for 1"/1.5" seeing (blue/yellow), and fit to ppSim for all pixels within r=3pix from center for 1"/1.5" seeing (green/magenta)
+Image:counts_Eric_ppSim_r39_FWHM1_1.5.gif|Closeup to Origin of linear version of fitted vs. input counts, for Eric's version (red), fit to ppSim for all pixels within r=9pix from center for 1"/1.5" seeing (blue/yellow), and fit to ppSim for all pixels within r=3pix from center for 1"/1.5" seeing (green/magenta)
+Image:histo_fwhm_r3pix_Eric_ppSim.gif|Fitted FWHM_X histograms for 3pix fitting radius: Eric (red/blue) and ppSim (green/grey) for 1"/1.5" seeing
+</gallery>