Index: trunk/doc/release.2015/ps1.calibration/calibration.tex =================================================================== --- trunk/doc/release.2015/ps1.calibration/calibration.tex (revision 40597) +++ trunk/doc/release.2015/ps1.calibration/calibration.tex (revision 40602) @@ -116,4 +116,7 @@ \section{Introduction}\label{sec:intro} +\note{list all ID\_IMAgE, ID\_MEAS, ID\_OBJ, ID\_SECF flags from libdvo/include/dvo.h and identify how they are set; make tables} + + From May 2010 through March 2014, the Pan-STARRS Science Consortium used the 1.8m \PSONE\ telescope to perform a set of wide-field science @@ -663,4 +666,196 @@ that into the master PV3 $3\pi$ DVO database. +The master DVO database is used to perform the full photometric and +astrometric calibration of the data. During these analysis steps, a +wide variety of conditions are noted for individual measurements, for +the objects (either as a whole or for specific filters) and for the +images. A set of bit-valued flags are used in the database to record +these conditions. +% +Table~\ref{tab:measure_mask_values} lists the flags specific to +individual measurements. These values are stored in the DVO database in the +field \code{Measure.dbFlags} and exposed in the public database \citep[PSPS][]{flewelling2017} +in the fields \code{Detection.infoFlag3}, +\code{StackObjectThin.XinfoFlag3} (where \code{X} is one of + {$grizy$}), and \code{ForcedWarpMeasurement.FinfoFlag3}. +% +Table~\ref{tab:secf_mask_values} lists the flags which are set for +each filter for individual objects in the database. These values are +recorded in the DVO database field \code{SecFilt.flags} and are +exposed in PSPS in the fields +\code{MeanObject.XFlags} and \code{StackObjectThin.XinfoFlag4}, where +\code{X} in both cases is one of {$grizy$}. +% +Table~\ref{tab:tab:object_mask_values} lists the flags specific to an +object as a whole. These values are stored in the DVO database field +\code{Average.flags} and are exposed in PSPS in +the field \code{MeanObject.objInfoFlag}. +% +Table~\ref{tab:image_mask_values} lists the flags raised for images. +These flags are stored in the DVO database field \code{Image.flags} +and are exposed in PSPS in the field \code{ImageMeta.qaFlags}. +% +The type of conditions which are recorded by these bits range from +information about the presence of external measurements (e.g., 2MASS +or WISE) to determinations of good or bad quality measurements for +astrometry or photometry. In the sections below, these flag values in +these tables are described where appropriate. Note that some of the +listed bits are either ephemeral (used internal to specific programs) +or are not relevant to the current DR2 analysis and reserved for +future use. + +\begin{table*} +\begin{center} +\footnotesize +\caption{\label{tab:measure_mask_values} Per-Measurement Flag Bit Values} % \vspace{-0.5cm} +\begin{tabular}{lcl} +\hline +\hline +{\bf Bit Name} & {\bf Bit Value} & {\bf Description} \\ +\hline +ID\_MEAS\_NOCAL & 0x00000001 & detection ignored for this analysis (photcode, time range) -- internal only \\ +ID\_MEAS\_POOR\_PHOTOM & 0x00000002 & detection is photometry outlier (not used PV3) \\ +ID\_MEAS\_SKIP\_PHOTOM & 0x00000004 & detection was ignored for photometry measurement (not used PV3) \\ +ID\_MEAS\_AREA & 0x00000008 & detection near image edge (not used PV3) \\ +ID\_MEAS\_POOR\_ASTROM & 0x00000010 & detection is astrometry outlier \\ +ID\_MEAS\_SKIP\_ASTROM & 0x00000020 & detection was ignored for astrometry measurement \\ +ID\_MEAS\_USED\_OBJ & 0x00000040 & detection was used during update objects \\ +ID\_MEAS\_USED\_CHIP & 0x00000080 & detection was used during update chips (not saved PV3) \\ +ID\_MEAS\_BLEND\_MEAS & 0x00000100 & detection is within radius of multiple objects \\ +ID\_MEAS\_BLEND\_OBJ & 0x00000200 & multiple detections within radius of object \\ +ID\_MEAS\_WARP\_USED & 0x00000400 & measurement used to find mean warp photometry \\ +ID\_MEAS\_UNMASKED\_ASTRO & 0x00000800 & measurement was not masked in final astrometry fit \\ +ID\_MEAS\_BLEND\_MEAS\_X & 0x00001000 & detection is within radius of multiple objects across catalogs \\ +ID\_MEAS\_ARTIFACT & 0x00002000 & detection is thought to be non-astronomical \\ +ID\_MEAS\_SYNTH\_MAG & 0x00004000 & magnitude is synthetic \\ +ID\_MEAS\_PHOTOM\_UBERCAL & 0x00008000 & externally-supplied zero point from ubercal analysis \\ +ID\_MEAS\_STACK\_PRIMARY & 0x00010000 & this stack measurement is in the primary skycell \\ +ID\_MEAS\_STACK\_PHOT\_SRC & 0x00020000 & this measurement supplied the stack photometry \\ +ID\_MEAS\_ICRF\_QSO & 0x00040000 & this measurement is an ICRF reference position \\ +ID\_MEAS\_IMAGE\_EPOCH & 0x00080000 & this measurement is registered to the image epoch (not tied to ref catalog epoch) \\ +ID\_MEAS\_PHOTOM\_PSF & 0x00100000 & this measurement is used for the mean psf mag \\ +ID\_MEAS\_PHOTOM\_APER & 0x00200000 & this measurement is used for the mean ap mag \\ +ID\_MEAS\_PHOTOM\_KRON & 0x00400000 & this measurement is used for the mean kron mag \\ +ID\_MEAS\_MASKED\_PSF & 0x01000000 & this measurement is masked based on IRLS weights for mean psf mag \\ +ID\_MEAS\_MASKED\_APER & 0x02000000 & this measurement is masked based on IRLS weights for mean ap mag \\ +ID\_MEAS\_MASKED\_KRON & 0x04000000 & this measurement is masked based on IRLS weights for mean kron mag \\ +ID\_MEAS\_OBJECT\_HAS\_2MASS & 0x10000000 & measurement comes from an object with 2mass data \\ +ID\_MEAS\_OBJECT\_HAS\_GAIA & 0x20000000 & measurement comes from an object with gaia data \\ +ID\_MEAS\_OBJECT\_HAS\_TYCHO & 0x40000000 & measurement comes from an object with tycho data \\ +\hline +\end{tabular} +\end{center} +\end{table*} + +\begin{table*} +\begin{center} +\footnotesize +\caption{\label{tab:secf_mask_values} Relphot Per-Filter Info Flag Bit Values} % \vspace{-0.5cm} +\begin{tabular}{lcl} +\hline +\hline +{\bf Bit Name} & {\bf Bit Value} & {\bf Description} \\ +\hline +ID\_SECF\_STAR\_FEW & 0x00000001 & Used within relphot: skip star \\ +ID\_SECF\_STAR\_POOR & 0x00000002 & Used within relphot: skip star \\ +ID\_SECF\_USE\_SYNTH & 0x00000004 & Synthetic photometry used in average measurement \\ +ID\_SECF\_USE\_UBERCAL & 0x00000008 & Ubercal photometry used in average measurement \\ +ID\_SECF\_HAS\_PS1 & 0x00000010 & PS1 photometry used in average measurement \\ +ID\_SECF\_HAS\_PS1\_STACK & 0x00000020 & PS1 stack photometry exists \\ +ID\_SECF\_HAS\_TYCHO & 0x00000040 & Tycho photometry used for synth mags \\ +ID\_SECF\_FIX\_SYNTH & 0x00000080 & Synth mags repaired with zpt map \\ +ID\_SECF\_RANK\_0 & 0x00000100 & Average magnitude uses rank 0 values \\ +ID\_SECF\_RANK\_1 & 0x00000200 & Average magnitude uses rank 1 values \\ +ID\_SECF\_RANK\_2 & 0x00000400 & Average magnitude uses rank 2 values \\ +ID\_SECF\_RANK\_3 & 0x00000800 & Average magnitude uses rank 3 values \\ +ID\_SECF\_RANK\_4 & 0x00001000 & Average magnitude uses rank 4 values \\ +ID\_SECF\_OBJ\_EXT\_PSPS & 0x00002000 & In PSPS ID\_SECF\_OBJ\_EXT is saved here so it fits within 16 bits \\ +ID\_SECF\_STACK\_PRIMARY & 0x00004000 & PS1 stack photometry includes a primary skycell \\ +ID\_SECF\_STACK\_BESTDET & 0x00008000 & PS1 stack best measurement is a detection (not forced) \\ +ID\_SECF\_STACK\_PRIMDET & 0x00010000 & PS1 stack primary measurement is a detection (not forced) \\ +ID\_SECF\_STACK\_PRIMARY\_MULTIPLE & 0x00020000 & PS1 stack object has multiple primary measurements \\ +ID\_SECF\_HAS\_SDSS & 0x00100000 & This photcode has SDSS photometry \\ +ID\_SECF\_HAS\_HSC & 0x00200000 & This photcode has HSC photometry \\ +ID\_SECF\_HAS\_CFH & 0x00400000 & This photcode has CFH photometry (mostly Megacam) \\ +ID\_SECF\_HAS\_DES & 0x00800000 & This photcode has DES photometry \\ +ID\_SECF\_OBJ\_EXT & 0x01000000 & Extended in this band \\ +\hline +\end{tabular} +\end{center} +\end{table*} + +\begin{table*} +\begin{center} +\footnotesize +\caption{\label{tab:object_mask_values} Per-Object Flag Bit Values} % \vspace{-0.5cm} +\begin{tabular}{lcl} +\hline +\hline +{\bf Bit Name} & {\bf Bit Value} & {\bf Description} \\ +\hline +ID\_OBJ\_FEW & 0x00000001 & used within relphot: skip star \\ +ID\_OBJ\_POOR & 0x00000002 & used within relphot: skip star \\ +ID\_OBJ\_ICRF\_QSO & 0x00000004 & object IDed with known ICRF quasar (may have ICRF position measurement) \\ +ID\_OBJ\_HERN\_QSO\_P60 & 0x00000008 & identified as likely QSO \citep{2016ApJ...817...73H}, $P_{\rm QSO} \geq 0.60$ \\ +ID\_OBJ\_HERN\_QSO\_P05 & 0x00000010 & identified as possible QSO \citep{2016ApJ...817...73H}, $P_{\rm QSO} \geq 0.05$ \\ +ID\_OBJ\_HERN\_RRL\_P60 & 0x00000020 & identified as likely RR Lyra \citep{2016ApJ...817...73H}, $P_{\rm RRLyra} \geq 0.60$ \\ +ID\_OBJ\_HERN\_RRL\_P05 & 0x00000040 & identified as possible RR Lyra \citep{2016ApJ...817...73H}, $P_{\rm RRLyra} \geq 0.05$ \\ +ID\_OBJ\_HERN\_VARIABLE & 0x00000080 & identified as a variable by \cite{2016ApJ...817...73H} \\ +ID\_OBJ\_TRANSIENT & 0x00000100 & identified as a non-periodic (stationary) transient \\ +ID\_OBJ\_HAS\_SOLSYS\_DET & 0x00000200 & identified with a known solar-system object (asteroid or other) \\ +ID\_OBJ\_MOST\_SOLSYS\_DET & 0x00000400 & most detections from a known solar-system object \\ +ID\_OBJ\_LARGE\_PM & 0x00000800 & star with large proper motion \\ +ID\_OBJ\_RAW\_AVE & 0x00001000 & simple weighted average position was used (no IRLS fitting) \\ +ID\_OBJ\_FIT\_AVE & 0x00002000 & average position was fitted \\ +ID\_OBJ\_FIT\_PM & 0x00004000 & proper-motion model was fitted \\ +ID\_OBJ\_FIT\_PAR & 0x00008000 & full parallax and proper-motion model was fitted \\ +ID\_OBJ\_USE\_AVE & 0x00010000 & average position used (no proper-motion or parallax) \\ +ID\_OBJ\_USE\_PM & 0x00020000 & proper motion fit used (no parallax) \\ +ID\_OBJ\_USE\_PAR & 0x00040000 & full fit with proper motion and parallax \\ +ID\_OBJ\_NO\_MEAN\_ASTROM & 0x00080000 & mean astrometry could not be measured \\ +ID\_OBJ\_STACK\_FOR\_MEAN & 0x00100000 & stack position used for mean astrometry \\ +ID\_OBJ\_MEAN\_FOR\_STACK & 0x00200000 & mean astrometry could not be measured \\ +ID\_OBJ\_BAD\_PM & 0x00400000 & failure to measure proper-motion model \\ +ID\_OBJ\_EXT & 0x00800000 & extended in Pan-STARRS data \\ +ID\_OBJ\_EXT\_ALT & 0x01000000 & extended in external data (2MASS) \\ +ID\_OBJ\_GOOD & 0x02000000 & good-quality measurement in Pan-STARRS data \\ +ID\_OBJ\_GOOD\_ALT & 0x04000000 & good-quality measurement in external data (2MASS) \\ +ID\_OBJ\_GOOD\_STACK & 0x08000000 & good-quality object in the stack ($> 1$ good stack) \\ +ID\_OBJ\_BEST\_STACK & 0x10000000 & the primary stack measurements are the ``best'' measurements \\ +ID\_OBJ\_SUSPECT\_STACK & 0x20000000 & suspect object in the stack ($> 1$ good or suspect stack, $< 2$ good) \\ +ID\_OBJ\_BAD\_STACK & 0x40000000 & poor-quality object in the stack ($< 1$ good stack) \\ +\hline +\end{tabular} +\end{center} +\end{table*} + +%% Image.flags => ImageMeta.qaFlags + +\begin{table*} +\begin{center} +\footnotesize +\caption{\label{tab:image_mask_values} Per-Image Flag Bit Values} % \vspace{-0.5cm} +\begin{tabular}{lcl} +\hline +\hline +{\bf Bit Name} & {\bf Bit Value} & {\bf Description} \\ +\hline +ID\_IMAGE\_NEW & 0x00000000 & no calibrations yet attempted \\ +ID\_IMAGE\_PHOTOM\_NOCAL & 0x00000001 & user-set value used within relphot: ignore \\ +ID\_IMAGE\_PHOTOM\_POOR & 0x00000002 & relphot says image is bad (dMcal > limit) \\ +ID\_IMAGE\_PHOTOM\_SKIP & 0x00000004 & user-set value: assert that this image has bad photometry \\ +ID\_IMAGE\_PHOTOM\_FEW & 0x00000008 & currently too few measurements for photometry \\ +ID\_IMAGE\_ASTROM\_NOCAL & 0x00000010 & user-set value used within relastro: ignore \\ +ID\_IMAGE\_ASTROM\_POOR & 0x00000020 & relastro says image is bad (dR,dD > limit) \\ +ID\_IMAGE\_ASTROM\_FAIL & 0x00000040 & relastro fit diverged, fit not applied \\ +ID\_IMAGE\_ASTROM\_SKIP & 0x00000080 & user-set value: assert that this image has bad astrometry \\ +ID\_IMAGE\_ASTROM\_FEW & 0x00000100 & currently too few measurements for astrometry \\ +ID\_IMAGE\_PHOTOM\_UBERCAL & 0x00000200 & externally-supplied photometry zero point from ubercal analysis \\ +ID\_IMAGE\_ASTROM\_GMM & 0x00000400 & image was fitted to positions corrected by the galaxy motion model \\ +\hline +\end{tabular} +\end{center} +\end{table*} + %% \note{need to describe the assignment of flags, etc, for the external data sources}. @@ -775,5 +970,5 @@ camera position. Each image which is part of the ubercal subset is marked with a bit in the field \code{Image.flags}: -\code{ID_IMAGE_PHOTOM_UBERCAL = 0x00000200} +\code{ID_IMAGE_PHOTOM_UBERCAL = 0x00000200}. \begin{table}[hb] @@ -1284,40 +1479,9 @@ Table~\ref{tab:secf_mask_values}). -\begin{table*} -\begin{center} -\footnotesize -\caption{\label{tab:secf_mask_values} Relphot Per-Filter Info Flag Bit Values} % \vspace{-0.5cm} -\begin{tabular}{lcl} -\hline -\hline -{\bf Bit Name} & {\bf Bit Value} & {\bf Description} \\ -\hline -ID\_SECF\_STAR\_FEW & 0x00000001 & Used within relphot: skip star \\ -ID\_SECF\_STAR\_POOR & 0x00000002 & Used within relphot: skip star \\ -ID\_SECF\_USE\_SYNTH & 0x00000004 & Synthetic photometry used in average measurement \\ -ID\_SECF\_USE\_UBERCAL & 0x00000008 & Ubercal photometry used in average measurement \\ -ID\_SECF\_HAS\_PS1 & 0x00000010 & PS1 photometry used in average measurement \\ -ID\_SECF\_HAS\_PS1\_STACK & 0x00000020 & PS1 stack photometry exists \\ -ID\_SECF\_HAS\_TYCHO & 0x00000040 & Tycho photometry used for synth mags \\ -ID\_SECF\_FIX\_SYNTH & 0x00000080 & Synth mags repaired with zpt map \\ -ID\_SECF\_RANK\_0 & 0x00000100 & Average magnitude uses rank 0 values \\ -ID\_SECF\_RANK\_1 & 0x00000200 & Average magnitude uses rank 1 values \\ -ID\_SECF\_RANK\_2 & 0x00000400 & Average magnitude uses rank 2 values \\ -ID\_SECF\_RANK\_3 & 0x00000800 & Average magnitude uses rank 3 values \\ -ID\_SECF\_RANK\_4 & 0x00001000 & Average magnitude uses rank 4 values \\ -ID\_SECF\_OBJ\_EXT\_PSPS & 0x00002000 & In PSPS ID\_SECF\_OBJ\_EXT is saved here so it fits within 16 bits \\ -ID\_SECF\_STACK\_PRIMARY & 0x00004000 & PS1 stack photometry includes a primary skycell \\ -ID\_SECF\_STACK\_BESTDET & 0x00008000 & PS1 stack best measurement is a detection (not forced) \\ -ID\_SECF\_STACK\_PRIMDET & 0x00010000 & PS1 stack primary measurement is a detection (not forced) \\ -ID\_SECF\_STACK\_PRIMARY\_MULTIPLE & 0x00020000 & PS1 stack object has multiple primary measurements \\ -ID\_SECF\_HAS\_SDSS & 0x00100000 & This photcode has SDSS photometry \\ -ID\_SECF\_HAS\_HSC & 0x00200000 & This photcode has HSC photometry \\ -ID\_SECF\_HAS\_CFH & 0x00400000 & This photcode has CFH photometry (mostly Megacam) \\ -ID\_SECF\_HAS\_DES & 0x00800000 & This photcode has DES photometry \\ -ID\_SECF\_OBJ\_EXT & 0x01000000 & Extended in this band \\ -\hline -\end{tabular} -\end{center} -\end{table*} +%% where do these go? analyis? +%% ID_GALPHOT_FAIL_FIT = 0x00000001, // fit failed to converge or was degenerate +%% ID_GALPHOT_TOO_FEW = 0x00000002, // not enough points to fit the model +%% ID_GALPHOT_OUT_OF_RANGE = 0x00000004, // fit minimum too far outside data range +%% ID_GALPHOT_BAD_ERROR = 0x00000008, // invalid error (nan or inf) \subsubsection{Iteratively Reweighted Least Squares Fitting} @@ -1433,8 +1597,8 @@ In addition to the these relatively rare failure cases, the objects -detected in the stacks are more likely to have multiple measurements +detected in the stacks may also have multiple measurements due to the overlap between neighboring stack images. The skycells (within which the stacks are generated) for a given projection cell -are defined to have significant overlap between neighbors to ensure a +are defined to have significant overlap between neighbors to ensure that a modestly-extended object can be measured completely on the pixels in a single skycell image. For the \ippmisc{RINGS.V3} skycell tessellation @@ -1480,5 +1644,9 @@ the skycell ``primary'' boundary. Thus, for a given object in the database, we expect all 5 filters to provide a ``primary'' measurement -from the same skycell for each object. +from the same skycell for each object. Also note that a faint object, +near the detection limit of the stack, may be detected on a +secondary skycell but not (due to statistical flucuations) be detected +on the corresponding primary skycell. Thus it is expected that some +objects may be lacking any primary detections. Since the ``primary'' identification is purely based on the skycell @@ -1587,11 +1755,11 @@ First, the astrometric calibration has a larger number of systematic effects which must be performed. These consist of: 1) the -Koppenh\"offer Effect, 2) Differential Chromatic Refraction, 3) Static +Koppenh\"ofer Effect, 2) Differential Chromatic Refraction, 3) Static deviations in the camera. We discuss each of these in turn below. -\subsubsection{Koppenh\"offer Effect} - -The Koppenh\"offer Effect was first identified in February 2011 by -Johannes Koppenh\"offer (MPE) as part of the effort to search for +\subsubsection{Koppenh\"ofer Effect} + +The Koppenh\"ofer Effect was first identified in February 2011 by +Johannes Koppenh\"ofer (MPE) as part of the effort to search for planet transists in the Stellar Transit Survey data. He noticed that the astromety of bright stars and faint stars disagreed on overlapping @@ -1619,5 +1787,5 @@ % \note{was there is significant difference using a surface brightness version?} -We measured the Koppenh\"offer Effect by accumulating the residual +We measured the Koppenh\"ofer Effect by accumulating the residual astrometry statistics for stars in the database. For each chip, we measured the mean X and Y displacements of the astrometric residuals @@ -1860,5 +2028,5 @@ Measure.XoffDCR,YoffDCR, Measure.XoffCAM,YoffCAM}. The offsets are calculated for each measurement based on the observed instrumental -chip magnitudes and FWHM for the Koppenhoffer Effect, on the average +chip magnitudes and FWHM for the Koppenh\"ofer Effect, on the average chip colors and the altitude \& azimuth of each measurement for the DCR correction, and on the chip coordinates for the astrometric @@ -2219,5 +2387,5 @@ * zero point history, including / excluding ubercal? (from Eddie) * applied flat-field images [FITS -> png] -* Koppenhoffer plots [from presentations] +* Koppenhofer plots [from presentations] * DCR plots [exist] * astrometric flat fields [FITS -> png]