PS1 Science Processing Status (20090705)

(up to IPP for PS1)

The PS1 Science Consortium has directed us to process a substantial amount of data for fast release, regardless of the data loss rate due to Magic mask. We have started with the 3 Medium Deep fields observed heavily since April: MD06, MD07, and MD08.

See Known data quality issues in the 2009.07.07 Release

Summary of data transfers for the 2009.07.07 Release

Some investigation of MDS image quality by Durham

Questions arising from the above MDS image analysis

from Photpipe - Harvard/JHU

Data in First Release

The first big batch of data from 3 Medium Deep fields should be available for download a bit later today (2009.07.07). This first set will consist of the June exposures from MD06, MD07, and MD08 (raw, chip, warp, diff) and the stacks used to generate the diffs. For this set of data, Paul Sydney has examined the magic results and certified the data to be released. There is a small number of images (7 or 8) which did not pass -- streaks leaked through -- and we will hold off on those until we can understand what happened (and mask them). The April and May data for these fields will be available soon -- the data has been processed, but we need a pass from Paul Sydney.

The June data consists of about 420 exposures. April and May together add up to about 300 more exposures.

For the purpose of managing the data transfer, Consortium sites need to register with the IPP team what portions of the data they would like to download.

Consortium sites may request data from people request any of the following 4 image sets, with the given typical sizes for the image and variance (does not exist for the raw data, of course):

Processing Stage	Image Data per Exposure	Data Products
Raw Data	~1.5GB	Raw images + rawExp, rawImfile DB table entries
Chip (Detrended)	~3.6GB (images + variance)	Detrend images, mask, variance, log, DB table entries
Camera (Calibrated)	~0.0GB (masks only)	Calibrated detection files, logs, DB table entries
Warp	~6.0GB (images + variance)	Warped images, mask, variance, log, DB table entries
Diff	~5.6GB (images + variance)	Difference images, mask, variance, detection tables, log, DB table entries
Stack	~6.0GB per field per filter	Stack images, mask, variance, detection tables, log, DB table entries

Except for the images and the variance images, the other data products (including the masks) are a small fraction of the image data value (< 5%).

The full June data set is about 420*(1.8 + 3.6 + 6 + 5.6) = 7TB. April + May will be another 5TB.

Institutions may register for subsets of the data, and may request (for example) to receive the detection files from the detrend chip stage, but no pixels, and all detections and pixels for the difference stage. At the moment, choices may be controlled independently for each MD field (and filter?).

A few other notes related to the data products for different stages:

• the detection tables generated at the chip stage (with names of the form o4984g0109o.72894.ch.13035.XY33.cmf) are intermediate, uncalibrated data products. Are we distributing these still? The calibrated detections are rolled up into a single file (with many FITS extensions) with names of the form o4984g0109o.72894.cm.13035.smf.

• the detection tables from the warp stage are generated by transforming the detections from the chip stage, not by performing a full re-detection stage. The PSF is remeasured to determine the impact of warping on the image quality.

• the detection tables generated in the stack creation stage are (a) only going to limited depth and (b) only performing PSF analysis. These tables do not represent the full analysis of the stack images, but are used at this stage to check the output PSF model.

Summary of Data

field	filter	Nexp	Ncam	Nstack	Stack Max(FWHM)
MD06	g.00000	25	25	15	5.7
""	r.00000	56	56	31	5.7
""	i.00000	105	101	44	5.0
""	z.00000	95	95	30	5.0
""	y.00000	14	14	14	5.0
""	all	295	291
MD07	g.00000	33	32	16	5.7
""	r.00000	46	46	21	6.5
""	i.00000	100	97	36	5.0
""	z.00000	76	76	17	5.2
""	y.00000	23	23	18	5.0
""	all	278	274
MD08	g.00000	36	35	19	6.0
""	r.00000	62	62	25	5.7
""	i.00000	81	80	20	5.0
""	z.00000	66	66	24	5.7
""	y.00000	8	8	8	5.0
""	all	253	251

The table above lists the number of exposures (Nexp) in each of the filters available for the three MD fields for the period April-June 2009. The column labeled Ncam lists the number of exposures processed at the camera stage (a small number failed on the chip analysis). The column labeled Nstack lists the number of exposures nominally used for each stack. Note that the exact number per stack depends on the skycell because of dithers and rotations. The last column gives the maximum seeing accepted for each stack (in pixels). Note that for MD06 and M08, since the y-band exposures all passed this limit, the actual max may be somewhat smaller than 5.0 pixels in these cases.

Data Requests

Institution	Fields	Filters	Stages	Mode	Priority	Notes
MPG	MD06, MD07, MD08,	grizy	raw	full	1	when possible
MPG	MD06, MD07, MD08,	grizy	chip	full	1
MPG	MD06, MD07, MD08,	grizy	camera, warp	full	2
MPG	MD06, MD07, MD08,	grizy	stack, diff	full	3
Durham, UK	MD08	grizy	stack	full	1
Durham, UK	MD07	grizy	stack	full	2
Durham, UK	MD08	gz	warp	full	3
QUB, UK	MD06, MD07, MD08	r	warp,stack,diff	full	1
QUB, UK	MD06, MD07, MD08	griz	diff	cleaned
Edinburgh, UK	MD06, MD07, MD08	ri	stack,chip,raw,warp	full
Edinburgh, UK	MD06, MD07, MD08	grizy	stack	full
Edinburgh, UK	MD06	grizy	camera, stack	full
Harvard,JHU	MD06: skycell.077	grizy	warp,stack,diff	full	1	NGC 4258
Harvard	MD08	grizy	warp	full	1	Overlaps with SDSS 82
Harvard	MD08	grizy	stack,diff	full	1	Overlaps with SDSS 82
Harvard	MD06, MD07, MD08	grizy	warp	full	2
Harvard	MD06, MD07, MD08	grizy	chip,camera	full	2
Harvard	MD06, MD07, MD08	grizy	stack, diff	full	3
JHU	MD06, MD07, MD08	grizy	chip,camera,warp,stk,diff	cleaned	1
JHU	MD08	gri	warp,stk,diff	full	2
JHU	MD06	ri	warp	full	3
NCU, Taiwan	MD07	grizy	warp,diff,stack	full	1
NCU, Taiwan	MD06, MD08	giz	warp,diff,stack	cleaned	1
NCU, Taiwan	MD06, MD08	y	diff	cleaned	1
NCU, Taiwan	MD06, MD08	y	warp,stack	full	2
NCU, Taiwan	MD06, MD08	r	warp,diff,stack	full	3

[Mode can be 'full': all products, or 'cleaned': only non-image products .]

A list of the various run ids can be found here Run ids for 200907 Release. A similar list for the stacks is here Stack ids for 200907 Release.

Launch Scripts

For the IPP Team's reference, the scripts used to queue the data for the analysis stages can be found in ~ipp/stdscience/stdsci.v1. For the chip analysis, there is as script for each field of with names like md06.v1.chip. These scripts select the exposures for each of the nights in this period and set up the labels and the workdirs (along with other defaults). The scripts used to launch the stack, diff, and magic runs have names like md08.v1.stk. The 'mode' variable for each of these scripts must be set manually to one of 'stack', 'diff', or 'magic'. For the moment, these scripts need to be run serially since they specify 'rerun' for the diff and magic stages. Eventually, we should convince ourselves that difftool and magictool correctly generate only a single run for a given input set, in which case these could be run automatically.

The stacks are all build using a good seeing subset of the data from the entire 3 month period. These selections were fairly subjective; in a future re-analysis pass, we can make a more educated choice for the inputs to maximize the S/N of the stack for point source detections (or other criterion as advised). Each of the stacks has a label of the form MD0n.200906.v1.