Index: trunk/doc/modules/ModulesSDRS.tex =================================================================== --- trunk/doc/modules/ModulesSDRS.tex (revision 4982) +++ trunk/doc/modules/ModulesSDRS.tex (revision 5022) @@ -1,3 +1,3 @@ -%%% $Id: ModulesSDRS.tex,v 1.55 2005-09-09 18:30:32 eugene Exp $ +%%% $Id: ModulesSDRS.tex,v 1.56 2005-09-13 03:26:45 price Exp $ \documentclass[panstarrs]{panstarrs} @@ -367,10 +367,10 @@ DEFAULTS METADATA CELL.BAD S32 0 - CELL.YPARITY.DEPEND STR CHIP.NAME - CELL.YPARITY METADATA - ccd00 S32 -1 - ccd01 S32 -1 - ccd02 S32 -1 - ccd03 S32 -1 + CELL.PARITY.DEPEND STR CHIP.NAME + CELL.PARITY METADATA + amp00 S32 1 + amp01 S32 -1 + amp02 S32 1 + amp03 S32 -1 END END @@ -440,9 +440,8 @@ (for want of a better word). These are values corresponding to general quantities and qualities relevant to the IPP such as airmass, -date, read noise and filter. These concepts are not always known by -the same name, or are obtained in the same manner for all cameras, and -so their source or value must be specified in the camera configuration -file. Some of these concepts make most sense to be defined at the FPA -level, while others are logically defined at the cell level. +date, read noise and filter. The values of each of the below concepts +shall be determined when the FPA is read into memory (via +\code{pmFPARead}), and stored at the appropriate level in the focal +plane hierarchy. Below is a list of concepts that the IPP requires, with the @@ -450,16 +449,14 @@ \begin{itemize} -\item \code{FPA.AIRMASS} (F32): Airmass at which the observation is made - (boresight). +\item \code{FPA.AIRMASS} (F32): Airmass at which the observation is made (boresight). \item \code{FPA.FILTER} (STR): Filter used in observation \item \code{FPA.POSANGLE} (F32): Position angle for camera -\item \code{FPA.RA}: Right Ascension of boresight -\item \code{FPA.DEC}: Declination of boresight +\item \code{FPA.RA} (F64): Right Ascension of boresight in radians +\item \code{FPA.DEC} (F64): Declination of boresight in radians \item \code{FPA.RADECSYS} (STR): System of RA,Dec (e.g., J2000 or ICRS) \item \code{FPA.NAME} (STR): An identifier (e.g., observation number) for the FPA instance \item \code{CHIP.NAME} (STR): The name of the chip (unique within the FPA) --- set at FITS read \item \code{CELL.NAME} (STR): The name of the cell (unique within the parent chip) --- set at FITS read -\item \code{CELL.TIME}: Time of observation start -\item \code{CELL.TIMESYS}: Time system in use (e.g., UTC) +\item \code{CELL.TIME} (\code{psTime}): Time of observation start \item \code{CELL.READDIR} (S32): Read direction: line (1) or column (2) \item \code{CELL.BIASSEC} (STR): Overscan region(s) @@ -469,28 +466,34 @@ \item \code{CELL.SATURATION} (F32): CCD saturation point (ADU) \item \code{CELL.BAD} (F32): CCD bad pixel point (ADU) -\item \code{CELL.BINNING}: CCD Binning -\item \code{CELL.PARITY} (S32): Direction of CCD readout -\item \code{READOUT.EXPOSURE} (F32): Exposure time of image (sec) -\item \code{READOUT.DARKTIME} (F32): Dark time for image (sec) +\item \code{CELL.XBIN} (S32): CCD binning in x +\item \code{CELL.YBIN} (S32): CCD binning in y +\item \code{CELL.XPARITY} (S32): Direction of CCD readout in x relative to the rest of the chip +\item \code{CELL.YPARITY} (S32): Direction of CCD readout in y relative to the rest of the chip +\item \code{CELL.EXPOSURE} (F32): Exposure time of image (sec) +\item \code{CELL.DARKTIME} (F32): Dark time for image (sec) \end{itemize} -The value of a concept shall be found by searching in the following -order: +\tbd{Note that \code{CELL.EXPOSURE}, \code{CELL.DARKTIME} and +\code{CELL.TIME} should actually be specified at the readout level. +However, at this present time, we're not sure how these should be +specified, and so we move them up to the cell level and assume that +all readouts are of the same exposure and dark time.} + +These concepts are not always known by the same name, nor are they +generally obtained in the same manner, in different camera systems, +and so their source or value must be specified in the camera +configuration file. The value of a concept shall be found by +searching in the following order: \begin{itemize} -\item A cache of values. \item The FITS header via the \code{TRANSLATION} table. \item The \code{DATABASE} lookup. \item The \code{DEFAULTS} value. \end{itemize} -When a concept is retrieved, it shall be stored in a cache (a -\code{psMetadata} within the FPA/chip/cell structures) to optimise -future retrieval. We have specified a cache and FITS header storage -in the various focal plane structures for the purposes of concept -retrieval. - -Because of the variety of methods for specifying these concepts -(especially in FITS headers), we must also specify additional -information in the camera configuration that specifies how to -interpret the data provided. + +After ingest, the user may safely assume that all of the above +concepts exist (defined at the appropriate level), is of the specified +type and in the specified format. + +\subsubsection{Dependencies for defaults} In the \code{DEFAULTS} table in the camera configuration, we allow the @@ -516,8 +519,13 @@ \end{verbatim} -In the FITS \code{TRANSLATION} table in the camera configuration, for -certain concepts we allow the specification of the concept with an -additional suffix, \code{FORMAT} which specifies the format of the -FITS header. The value is dependent upon the particular concept. +\subsubsection{FORMATS} + +Because of the variety of methods for specifying these concepts +(especially in FITS headers), we must also specify additional +information in the camera configuration that specifies how to +interpret the data provided. These are provided in an entry +\code{FORMATS} (of type \code{METADATA}) in the camera configuration. +Within the \code{FORMATS} metadata, there is a string for each of the +concepts that requires a format to be specified. \paragraph{CELL.TIME} @@ -537,6 +545,6 @@ \item \code{SEPARATE}: The date and time are specified separately, and the \code{CELL.TIME} contains the headers for the date and the time - separated by whitespace or a comma. Then it might be necessary to - add additional qualifiers to specify the formats of these: + separated by whitespace or a comma. Then it is necessary to add + additional qualifiers to specify the formats of these: \begin{itemize} \item \code{PRE2000}: The year is in the old style two-digit format @@ -554,18 +562,5 @@ based on someone foolishly putting the end- or mid-time in the header.} -\paragraph{CELL.BINNING} - -The binning is usually specified in FITS headers either as separate -headers for the x and the y, or in the same FITS header separated by a -space or a comma. Permitted values of \code{CELL.BINNING.FORMAT} are: - -\begin{itemize} -\item \code{SEPARATE}: The \code{CELL.BINNING} contains the header - keywords for the x and the y binning separated by whitespace or a - comma. -\item \code{TOGETHER}: \code{CELL.BINNING} contains the keyword for - which the x and y binning are separated by whitespace and/or a - comma. -\end{itemize} +\tbd{Should we move \code{CELL.TIMESYS} into the format as well?} \paragraph{FPA.RA and FPA.DEC} @@ -591,26 +586,31 @@ that it is in decimal format. -\subsubsection{CELLS} - -The \code{CELLS} entry in the camera configuration contains data -appropriate to each cell. These will generally consist of the -\code{CELL.BIASSEC} and \code{CELL.TRIMSEC} concepts, though it might -contain \code{CELL.GAIN} and \code{CELL.READNOISE} values as well -instead of going to the trouble of specifying these in the -\code{DEFAULTS} with a long \code{.DEPENDS} listing. - -However, in some cases, we need to specify for these where the value -comes from. We declare the following rule: +\subsubsection{Implicit format information} + +While details like the units of the right ascension in the header must +be specified explicitly, some other details can be determined from +implicit information. \begin{itemize} -\item If the type is other than \code{STR}, then the concept has that - value for the cell. -\item If the type is \code{STR}, then the value might potentially be - confused between being a header keyword or an actual value. For - this reason, we introduce an additional concept suffix, - \code{.SOURCE} (of type \code{STR}) which may be either - \code{HEADER} or \code{VALUE}. +\item \code{FPA.RA} and \code{FPA.DEC}: if the value on ingest is of +type \code{STRING}, then it may be interpreted as sexagesimal +notation, ``\code{dd:mm:ss.ss}'', or ``\code{dd:mm.mmm}''. A space +may be used instead of a colon to separate the values. Otherwise, if +the value is of a numerical type (\code{F32} or \code{F64}), then that +is the appropriate value. +\item \code{CELL.XBIN} and \code{CELL.YBIN}: if the value on ingest is +of type \code{STRING}, then it may be interpreted as ``\code{x,y}'', +where \code{x} is the binning in x, and \code{y} is the binning in y. +A space may be used instead of a comma, and there may even be a space +before or after the comma (or both). Otherwise, if the value is of a +numerical type (\code{S32}, etc), then that is the appropriate value. +\item \code{CELL.BIASSEC} and \code{CELL.TRIMSEC}: These values on +ingest should always be of type \code{STRING}. If they contain a +square bracket, then they may be interpreted as a list of standard +region specifications, ``\code{[x0:x1,y0:y1];[x2:x3,y2:y3];...}'', +where the semi-colon may be replaced by spaces. Otherwise, the string +may be interpreted as a FITS header (or headers, separated by spaces, +commas or semi-colons) that contains the appropriate values. \end{itemize} - \subsection{Configuration APIs} @@ -718,56 +718,4 @@ \end{verbatim} -\subsubsection{Lookups} - -We first specify a general concept lookup function, which shall return the -appropriate \code{psMetadataItem} for the value of the given \code{concept} -for the specified \code{cell}, \code{chip} or \code{fpa}: -\begin{prototype} -psMetadataItem *pmCellGetConcept(pmCell *cell, const char *concept); -psMetadataItem *pmChipGetConcept(pmChip *chip, const char *concept); -psMetadataItem *pmFPAGetConcept(pmFPA *fpa, const char *concept); -\end{prototype} - -We next specify a series of specific functions for concept lookups. -These will generally be what the user utilises, so the goal is to -provide a simple interface providing a single type back, so the user -doesn't have to go to the trouble of checking types, etc. These -functions should employ the above three general lookup functions and -deal with the result appropriately. - -\begin{prototype} -float pmFPAGetAirmass(pmFPA *fpa); // FPA.AIRMASS -psString pmFPAGetFilter(pmFPA *fpa); // FPA.FILTER -float pmFPAGetPosAngle(pmFPA *fpa); // FPA.POSANGLE -double pmFPAGetRA(pmFPA *fpa); // FPA.RA -double pmFPAGetDec(pmFPA *fpa); // FPA.DEC -psString pmFPAGetRADecSys(pmFPA *fpa); // FPA.RADECSYS -psString pmFPAGetName(pmFPA *fpa); // FPA.NAME -psString pmChipGetName(pmChip *chip); // CHIP.NAME -psString pmCellGetName(pmCell *cell); // CELL.NAME -psTime *pmCellGetTime(pmCell *cell); // CELL.TIME -psList *pmCellGetBiasSec(pmCell *cell); // CELL.BIASSEC -psRegion pmCellGetTrimSec(pmCell *cell); // CELL.TRIMSEC -int pmCellGetReaddir(pmCell *cell); // CELL.READDIR -float pmCellGetGain(pmCell *cell); // CELL.GAIN -float pmCellGetReadNoise(pmCell *cell); // CELL.READNOISE -float pmCellGetSaturation(pmCell *cell); // CELL.SATURATION -float pmCellGetBad(pmCell *cell); // CELL.BAD -psPixelCoord pmCellGetBin(pmCell *cell); // CELL.BIN -psPixelCoord pmCellGetParity(pmCell *cell); // CELL.PARITY -float pmReadoutGetExposure(pmReadout *readout); // READOUT.EXPOSURE -float pmReadoutGetDarkTime(pmReadout *readout); // READOUT.DARKTIME -\end{prototype} - -Most of these are straight-forward, but some need some explanation. - -\code{pmCellGetBiasSec} shall return a list of \code{psRegion}s, one -for each bias section. - -\code{pmCellGetBin} shall return a \code{psPixelCoord} with the -binning factors appropriately set in the \code{x} and \code{y} -members. Similarly with \code{pmCellGetParity}. - - \input{CameraImages.tex} @@ -878,8 +826,7 @@ The API shall be the following: \begin{prototype} -pmReadout *pmSubtractBias(pmReadout *in, void *fitSpec, const psList *overscans, - pmOverscanAxis overscanAxis, const psStats *stat, - int nBin, pmFit fit, const pmReadout *bias, - const pmReadout *dark); +pmReadout *pmSubtractBias(pmReadout *in, void *fitSpec, pmFit fit, bool overscan, + const psStats *stat, int nBin, + const pmReadout *bias, const pmReadout *dark); \end{prototype} @@ -894,42 +841,15 @@ The input image may be of type U16, S32, or F32. -The type of the overscan fit function, \code{fitSpec}, shall be -dependent upon the value of \code{fit}, which specifies the type of -fit to be employed and is described below. - -The prescan and/or overscan regions to be used are specified in -\code{overscans}, which is a linked list of subimages. In cases where -\code{overscans} is \code{NULL} and \code{overscanAxis} is not -\code{PM_OVERSCAN_NONE}, the function shall generate an error. If -\code{overscans} is non-\code{NULL} and \code{overscanAxis} is -\code{PM_OVERSCAN_NONE}, then the function shall generate a warning, -no overscan subtraction shall be performed, and the function shall -proceed to the full-frame bias subtraction. - -The \code{overscanAxis} specifies how the prescan/overscan subtraction -is to be performed. It is an enumerated type: -\begin{datatype} -/** Overscan axis */ -typedef enum { - PM_OVERSCAN_NONE, ///< No overscan subtraction - PM_OVERSCAN_ROWS, ///< Subtract rows - PM_OVERSCAN_COLUMNS, ///< Subtract columns - PM_OVERSCAN_ALL ///< Subtract the statistic of all pixels in overscan region -} pmOverscanAxis; -\end{datatype} - -If the \code{overscanAxis} is \code{PM_OVERSCAN_NONE}, then the -function shall not perform any overscan subtraction, but proceed to -the full-frame bias subtraction. If the \code{overscanAxis} is -\code{PM_OVERSCAN_ALL}, then all the overscan regions shall be used to -generate a single statistic (specified by \code{stat}) which shall be -subtracted from the entire image. A warning shall be generated if the -\code{overscanAxis} is \code{PM_OVERSCAN_NONE} or -\code{PM_OVERSCAN_ALL} and the \code{fit} is not \code{PM_FIT_NONE}. - -If the \code{overscanAxis} is \code{PM_OVERSCAN_ROWS} or -\code{PM_OVERSCAN_COLUMNS}, then the overscan shall be reduced to a -single vector (in the specified dimension) using the specified -statistic (\code{stat}). +Overscan subtraction is controlled by the \code{overscan} boolean. If +it is \code{true}, then overscan subtraction is performed. The +prescan and/or overscan regions to be used are specified in +\code{in->bias} (a linked list of subimages). These shall be reduced +to a single vector (in the dimension specified by \code{CELL.READDIR}, +which is accessed via the parent of the \code{pmReadout}). + +If the overscan is not defined for each row/column, then the function +shall generate a warning and then interpolate using the provided +functional form if \code{fit} is not \code{PM_FIT_NONE} (see below); +otherwise, the function shall generate an error. The statistic to use in combining multiple pixels in the @@ -941,17 +861,17 @@ according to the following priority order: \code{PS_STAT_SAMPLE_MEAN}, \code{PS_STAT_SAMPLE_MEDIAN}, \code{PS_STAT_CLIPPED_MEAN}, -\code{PS_STAT_ROBUST_MEAN},\ \code{PS_STAT_ROBUST_MEDIAN}, +\code{PS_STAT_ROBUST_MEAN}, \code{PS_STAT_ROBUST_MEDIAN}, \code{PS_STAT_ROBUST_MODE}. If \code{nBin} is positive and less than the size of the vector, then -the vector shall subsequently be binned into bins that are a relative -size of \code{nBin} compared to the original pixels, again using the -specified statistic (\code{stat}). If \code{fit} is -\code{PM_FIT_SPLINE}, then \code{nBin} also serves as the number of -spline pieces. - -\code{fit} is an enumerated type: +the vector shall subsequently be binned into \code{nBin} bins, using +the specified statistic (\code{stat}). For example, the whole +overscan region can be taken as a unity if \code{nBin=1}. If +\code{fit} is \code{PM_FIT_SPLINE}, then \code{nBin} also serves as +the number of spline pieces. + +\code{fit} is an enumerated type which specifies the type of fit to +employed on the overscan vector (and hence the type of \code{fitSpec}): \begin{datatype} -/** Fit types */ typedef enum { PM_FIT_NONE, ///< No fit @@ -961,32 +881,26 @@ \end{datatype} -If \code{fitSpec} is \code{NULL}, or \code{fit} is \code{PM_FIT_NONE}, -then no fit shall be performed to the overscan. Otherwise, -\code{fitSpec} shall be interpreted to be a structure of the +If \code{fit} is \code{PM_FIT_POLYNOMIAL} or \code{PM_FIT_SPLINE}, +then \code{fitSpec} shall be interpreted to be a structure of the appropriate type (\code{psPolynomial1D} for \code{PM_FIT_POLYNOMIAL}, and \code{psSpline1D} for \code{PM_FIT_SPLINE}), and the overscan -shall (after reduction of the vector and binning) be fit using the -specified functional form. Upon return, the \code{fitSpec} shall -contain the coefficients of the overscan fit. If \code{fit} is -\code{PM_FIT_SPLINE}, then the \code{fitSpec} may be \code{NULL}, -in which case a new \code{psSpline} is allocated; in any case, the -number of spline pieces shall be set to \code{nBin}. - -If the overscan is not defined for each row/column, then the function -shall generate a warning and then interpolate using the provided -functional form if \code{fit} is not \code{PM_FIT_NONE}; otherwise, -the function shall generate an error. - -Following any binning, the vector shall be fit by the functional form -specified by \code{fit}. Then the overscan shall be subtracted from -the image, using values from the fit if \code{fit} is not -\code{PM_FIT_NONE}; otherwise using values from the overscan vector if -\code{overscanAxis} is not \code{PM_OVERSCAN_ALL}; otherwise using the -appropriate statistic applied to all the prescan/overscan pixels. - -A bias image shall be subtracted pixel-by-pixel from the input image -if \code{bias} is non-NULL. A dark image, multiplied by the -\code{darkTime}, shall be subtracted pixel-by-pixel from the input -image if \code{dark} is non-NULL. Note that the input image, +vector shall be fit using the specified functional form. + +In cases where \code{fitSpec} is \code{NULL} and \code{fit} is not +\code{PM_FIT_NONE} or \code{overscan} is \code{true}, the function +shall generate an error. If \code{overscan} is \code{false} and +\code{fit} is not \code{PM_FIT_NONE}, then the function shall generate +a warning, and no overscan subtraction shall be performed. Upon +return, the \code{fitSpec} shall contain the coefficients of the +overscan fit. + +If \code{fitSpec} is \code{NULL}, or \code{fit} is \code{PM_FIT_NONE}, +then no fit shall be performed to the overscan. + +A bias frame shall be subtracted pixel-by-pixel from the input image +if \code{bias} is non-NULL. If \code{dark} is non-\code{NULL}, then +the dark image, scaled by the ratio of dark times (from +\code{CELL.DARKTIME}) shall be subtracted pixel-by-pixel from the +input image. Note that the input image, \code{in}, and the \code{bias} and \code{dark} frames need not be the same size, but the function shall use the offsets in the image @@ -999,5 +913,5 @@ images may be copied to the same type as the input image if required. -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Non-linearity} @@ -1037,11 +951,14 @@ type U16, S32, or F32. +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + \subsection{Flat-fielding} Given an input image and a flat-field image, \code{pmFlatField} shall divide the input image by the flat-field image and return it in place, -updating the mask as appropriate. The API shall be the following: -\begin{prototype} -bool pmFlatField(pmReadout *in, pmReadout *mask, const pmReadout *flat); +updating the mask contained within the input image as appropriate. +The API shall be the following: +\begin{prototype} +bool pmFlatField(pmReadout *in, const pmReadout *flat); \end{prototype} @@ -1071,10 +988,11 @@ is left to the caller. This function is basically equivalent to a divide (with \code{psImageOp}), but with care for the region that is -divided, checking for negative pixels, and copying of the mask from -the \code{flat} to the output. - -The input and flat-field images must both be of type F32. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +divided, checking for zero and negative pixels, and copying of the +mask from the \code{flat} to the output. + +The images in the input and flat-field readouts must both be of type +F32. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Masking} @@ -1121,5 +1039,5 @@ the \code{growVal}). -\tbd{In the future, may change grow to a convolution kernel}. +\tbd{In the future, may change \code{grow} to a convolution kernel}. Note that the input image, \code{in}, and the \code{mask} need not be @@ -1136,4 +1054,6 @@ \subsection{Subtract sky} + +\tbd{This may be deferred.} Given an input image, a polynomial or spline specifying the order of a @@ -3503,4 +3423,7 @@ \end{document} +%%% All this is here for storage only --- it's not part of the document. + + \subsection{Cosmic rays} @@ -3648,2 +3571,43 @@ \code{in}. +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Fixed Pattern} + +The fixed pattern is a correction to the general astrometric solution +formed by summing the residuals from many observations. The intent is +to correct for higher-order distortions in the camera system on a +coarse grid (larger than individual pixels, but smaller than a single +cell). Hence, in addition to the offsets, we need to specify the size +and scale of the grid in $x$ and $y$, as well as the origin of the +grid. + +\begin{datatype} +typedef struct { + int nX; ///< Number of elements in x + int nY; ///< Number of elements in y + double x0; ///< Position of 0,0 corner on focal plane + double y0; ///< Position of 0,0 corner on focal plane + double xScale; ///< Scale of the grid + double yScale; ///< Scale of the grid + double **x; ///< The grid of offsets in x + double **y; ///< The grid of offsets in y +} psFixedPattern; +\end{datatype} + +The constructor for \code{psFixedPattern} shall be: +\begin{prototype} +psFixedPattern *psFixedPatternAlloc(double x0, double y0, + double xScale, double yScale, + const psImage *x, const psImage *y); +\end{prototype} +Here, \code{x0}, \code{y0}, \code{xScale} and \code{yScale} have the +same meaning as in the \code{psFixedPattern} structure. Note that the +values of the fixed pattern offsets are specified as images, the +values from which need to be copied into the \code{double **x} and +\code{double **y} of \code{psFixedPattern}, and that the number of +elements may be derived from the size of the images. + +\tbd{Usage of this type is not clear, and awaits prototyping --- do not +worry about coding this in detail yet.} +