Back to GPC1 Data Distribution

DRAVG members, please give here a summary of mirroring requirements at your institution, with a link to a more detailed page if necessary.

Some numbers as background info

Estimates made by Gene:

• 1 copy of the static sky images is ~350 TB
• At the end of the survey, the dvo database of the 3pi sky will contain 100e9 distinct measurements of 5e9 distinct objects, making up 50-60 TB of data.
• 90% of the objects are in the 10% of sky that is in the plane. I.e., if you are not interested in the galactic plane, you "only" need ~6 TB for storing all of the catalogs that interest you.

See Mirroring for more (rough) estimates, in particular of bandwidth required.

Requirements

• MPIA/MPE
  • Andromeda, Milky Way, brown dwarfs and quasars at MPIA (M31, 3pi): cmf files and dvo catalogs and corresponding postage stamps images ASAP.
  • For brown dwarfs & quasars: initially, the Galactic plan is not necessary; the brown dwarfs want the entire sky once proper-motions can be determined internally.
  • For quasars (and maybe brown dwarfs, too?):
    • izy stacks as soon as possible after data are taken
    • forced photometry in all other bands at the locations of all z and y-band detections
  • 3pi: only data from the small area survey (SAS): raw, chip, chip background restored, camera, warp, warp background restored, stack (convolved and unconvolved)
  • MDS: raw, chip, chip background restored, camera, warp, warp background restored, stack (convolved and unconvolved), diffs
  • Sweetspot: raw, chip, camera, warp, stack
  • Pan-Planets: raw, chip, chip background restored, camera, warp, warp background restored and diffs for all images of the crowded STS replacement field taken in demo month
  • PAndromeda: raw, chip (background restored), camera, warp (background resored), 1 single deep r-band stack (using all good images from all of the old data), diffs (of all r-band images with respect to the stack mentioned before) and diffs for all existing M31 observations + master and raw detrend images (darks, flats)
• Belfast
  • 3pi survey from MHPCC: cmf files, postage stamp images,
  • from Garching: some detrend and difference images;
  • MDS from Garching: detrend images
  • UPDATED (2011-08-12):
    • 3pi survey: Static sky diff catalogs if possible or "off-night" diffs if not, postage stamp images (target, reference, difference)
    • REPROCESSED DATA:
      • MD Stack-Stack Diff Catalogs, postage stamp images (target, reference, difference)
• Edinburgh (see ​Andy Taylor's message to ps-ipp-users - list subscription required to see it)
  • We would encourage the suggestion that the MDS fields be sent in a "full" state, ie, clean and update does not have to be run, and they contain all data products.
  • To clarify what we think we need for KP11 is detrended, unwarped images (with warp information) for each exposure (ie unstacked) for the full field.
  • We need the full raw (flat-fielded, bias subtracted but nothing else) data for the full MDS fields, not postage stamps.
  • We need other IPP products such as the CMF file, star and galaxy catalogues.
• Durham
  • GPC1 database (downloaded daily)
  • full datasets of all MD nightly and reference stacks (downloaded daily)
  • full stack datasets for all test surveys (e.g. SAS, SAS2, LAP)
• CfA
  • Full catalogs. 50-100% of Medium Deep Survey; detrending images only. Rapid-response postage stamp images.
• JHU
  • Full catalogs, rapid-response postage stamp images all surveys. Access to sub-set MDS images (will work with CfA).
• LCOGT/UCSB
• NCU/TW
  • Full catalogs; Medium Deep Survey; detrending images only; Rapid-response postage stamp images; cmf files.

Postage-Stamp Server needs

• The Transient Classification Server requirements document specifies a worst-case of 100,000 request of 100x100 pixel stamps per day.