Started by Sebastian Jester

More info

See dvo links on IPPwiki front page, and in particular DVO Challenge for examples of dvo command usage, and ​dvo.pdf for background (even though dvo.pdf is quite outdated, it's the only place where you can read about dvo's design principles...).

dvo concepts

Skycells, Objects, measurements, averages, photcodes, CATDIRs

dvo has a spatial index

dvo stores data in subdirectories of a CATDIR that correspond to different parts of the sky. This naturally allows comparatively fast spatial indexing.

Correspondingly, dvo operations are typically done on a given region of the sky.

Objects, measurements, averages

dvo uses two types of tables

• Measurement tables contain all individual measurements from different epochs and filters in a giant table. There is originally no notion of which measurements correspond to the same astronomical object, and which to different ones.

• Average tables know which measurements belong to the same objects, and record the average value, scatter, and error in the mean of the same object over different epochs, for each filter. They also record the proper motion and parrallax (if it has been determined).

Both sets of tables, the links between them, and derived quantities are created or computed by a set of commands outside of dvo (addstar, relphot, relastro). Once you start dvo, everything you need should already be there...

photcodes

A photcode is an internal identifier of a photometric system. dvo distinguishes between different kinds of photcodes, depending on which kind of information is accessible in addition to the photometric catalog, and how the photometric data were computed. dvo allows conversion and cross-calibration between different systems.

• ref (reference) photcodes are for external catalogs. Typically, only the photometric data and a (rough) epoch are available, but it is not possible to access the original images or instrumental magnitudes.
• dep (dependent) photcodes are for measurements where all input data are accessible, i.e. images and instrumental magnitudes, zero points, air mass, observing time etc. Measurements are reported in dep photcodes. There can be multiple dep photcodes for a given bandpass, eg. in PS1, every chip/filter combination (300 of them) gets its own dep photcode.
• average photcodes are for derived quantities such as averages of dep measurements, where there isn't an image that can be identified as the origin of the photometric quantity. These are meant to be generic filters, e.g. the (abstract) average PS1 r-band filter.

Existing dep and ref photcodes can be averaged into average ones.

Installing dvo

dvo comes as part of IPP - see Installation-related links on IPPwiki main page. Talk to your local IPP guru about where it's installed. We'll also be aiming to make a standalone dvo-only tarball for use on laptops.

In Heidelberg, see /IPP/README on the IPP cluster (and read all of it!)

Using dvo

Startup

 dvo -D CATDIR /path/to/catdir

e.g. in Heidelberg:

 dvo -D CATDIR /IPP/data/GPC1/flatcorr_20081029/catdir.flatcorr
 #or for the GPC1 data overlapping SDSS and UKIDSS on stripe 82
 dvo -D CATDIR /a773/jester/PS1/catdir.20090327 -D CAMERA gpc1

Getting help

Most dvo commands have a help text that can be read with

 help command

The dvo shell allows tab completion of commands and filenames.

Most commands will also com/explain what they expect if called without arguments.

Find out what regions of sky are in the database

 # set the entire sky as plot region and plot coordinate grid
 region 0 0 180 AIT
 # syntax is ra dec radius [projection]
 cgrid
 # Show outlines of images whose objects are in the db
 images
 # Find out coordinates of interesting regions interactively
 cursor
 # this brings up a graphics cursor - move it to an interesting point,
 # hit any letter key, (e.g. a) to store the coordinates of that point 
 # in variables $Ra $Da (for "RA of a, DEC of a").
 # Or just click the mouse button to see coordinates.
 # Get out of cursor mode by hitting 'q' (also stores $Rq $Dq), 
 # then zoom in by resetting the 'region' and replotting
 region $Ra $Da 1
 cgrid
 images

Setting the region for plotting

 # for the flatcorr catdir
 
 Set the region for plotting and plot a coordinate grid:
 region 314.7428 37.1718 3
 cgrid
 #Set the region for catalog extraction:
 skyregion 310 320 33 40
 
 # for the 20090327 catdir
 region 60 0 5
 cgrid
 skyregion 55 65 -2 2

Loading data

Show which images are present in the current skyregion:

 images

Extracting average quantities into vectors (similar to those in sm, or arrays in IDL or C etc.) is done via avextract; from the currently defined plotting region by giving the -region argument:

 avextract -region RA DEC g r i y
 [... takes a while for big regions with lots of measurements ...]

The syntax for regions in avextract is:

 -region # take ra,dec limits from currently defined plotting 'region'
 -skyregion ramin ramax decmin decmax # use ra,dec limits as given
 no -region or -skyregion command # use previously defined skyregion

However, with the -region setting, the output list does not actually seem to be affected by changing the region. I.e. better use the skyregion variant!

Also note that there is only ever one active skyregion, ie. avextract -skyregion changes the skyregion in the same way as the skyregion command. You can see the currently active one with the

 skyregion

command.

In addition to the quantities themselves, there is both an uncertainty and scatter; I think, usually the scatter is reported in the :err version of quantities; for RA and DEC, the error is in dRA, dDEC (see avextract --help fields:

  avextract -region RA dRA DEC dDEC g g:err r r:err i i:err y y:err

Note there have been problems with the calculations of some scatter/error columns, so not all of them may be there. To get also the number of measurements that went into each average, use :nphot:

  avextract -region RA DEC g g:err r r:err i i:err z z:err y y:err g:nphot r:nphot i:nphot z:nphot y:nphot

To see which vectors you've loaded, say

 vectors

There are two more extract commands:

• mextract extracts measurements; however, one can't compute colours from these (e.g.) because they are returned without any notion of which entry in the vector belongs to which object
• mmextract extracts pairs of measurements belonging to the same object

Searching

Restricting output of avextract etc. to interesting objects

The avextract, mextract and mmextract commands allow restricting the rows that are returned by adding a SQL-like where clause:

 avextract -region RA DEC g r i y where g-r < 0

Find out which images contain a given ra, dec

 gimages 314.7428 37.1718 -region

Plotting and computing

To plot two vectors against each other, say

 limits g r
 box
 plot g r

To compute and plot new vectors, say

 set gr = g - r
 clear
 limits gr r
 box
 plot gr r

To plot things in sky coordinates, use cplot:

 cplot RA DEC

Make a colour-colour plot to find a g-dropout quasar

# On a command line, type
dvo -D CATDIR /a773/jester/PS1/ThreePi.Run2.v0 -D CAMERA gpc1
# to start the dvo shell.

# The remaining commands are typed in the dvo shell.
# Set plot region
region 14.84 0.05 1
# Show coordinate grid
cgrid
# Show image outlines in db in this region
images
# (note: will plot 0 images if region is entirely within one or more image outlines!)

skyregion -1 -1 1 1
avextract -region RA DEC g g:err r r:err i i:err z z:err y y:err flag
# Not extended, not CR, not defect, not saturated, not bad PSF
set notext = (flag & 0x7c00) == 0
# Small errors only
set in = r:err < 0.1
# Select subsets satisfying both criteria
subset ri_psf = ri if (notext && (in))
subset gr_psf = gr if (notext && in)
# can't do && !in ???
# plot them
limits gr_psf ri_psf
clear;box;plot gr_psf ri_psf

See source:trunk/psModules/src/objects/pmSourceMasks.h​ for possible flag values. NB: At the moment, the flags are only defined for individual measurements, not for averages! For those, all the flags are 0, so everything is good and nothing is extended after an avextract.

Make a colour-colour and colour-magnitude diagram of a stellar locus

 region 17 35 2
 avextract RA DEC g g:err r r:err z z:err 
 # Define a logical vector to filter out "good" objects 
 # Normally want to avextract & check the 'flag' vector, too, but 
 # at the moment flag seems to be all 0
 set in = g<20 && g:err<0.2 && r:err < 0.2 && z:err & 0.2
 # Compute colours
 set gr = g-r
 set rz = r-z
 # Filter subsets
 subset gr_good = gr if (in)
 subset rz_good = rz if (in)
 # Plot a colour-colour diagram
 limits -1 3 -1 2
 box
 plot gr_good rz_good
 # Write vectors to file
 write -f "%9.6f %9.6 %6.3f %6.3f %6.3f %6.3f %6.3f %6.3f" PS1_17_35.dat RA DEC g g:err r r:err z z:err
 # Write a column header to a new file using a command shell
 exec echo ra dec g g_err r r_err z z_err > PS1_17_35_clean.dat
 # Filter out nans
 exec grep -v 'nan' PS1_17_35.dat >> PS1_17_35_clean.dat
 # NB: In the dvo shell, you can replace 'exec' by '!', e.g.
 ! ls 

You can save all these commands to a file makecmd.dvo (e.g.) and then execute them in a dvo session via

 input makecmd.dvo

To get a control sample of the same region from SDSS, run this SQL query via the ​SDSS sql search (the flag-checking part is the official recommendation from ​SDSS sample query)

select psfmag_g, psfmagerr_g, psfmag_r, psfmagerr_r, psfmag_z, psfmagerr_z into mydb.PS1_compare_CMD from star
where ra between 15 and 19
and dec between 33 and 37
AND ((flags_r & 0x10000000) != 0) 
AND ((flags_r & 0x8100000c00a4) = 0) 
AND (((flags_r & 0x400000000000) = 0) or (psfmagerr_r <= 0.2)) 

AND (((flags_r & 0x100000000000) = 0) or (flags_r & 0x1000) = 0)

Things we want to know/do

M31 / local group

• Load M31 etc. data into dvo

   cd /disk1/martin/GPC1/M31_Nov2008_mgc/catdir
   addstar -D CAMERA gpc1 -D CATDIR /disk1/martin/GPC1/M31_Nov2008_mgc/catdir -accept-astrom ../47074/chip/o4776g0144o.47074.ch.11260.XY11.cmf 
   # I think relphot then needs to be run like so:
   # If you add more images, add -update to the addstar arguments
   # and then when all the addstars are finished, run one more addstar 
   # to resort the catalog:
   addstar -D CATDIR . -D CAMERA gpc1 -resort 
   relphot -D CATDIR . -D CAMERA gpc1 -averages -update
  

  If you have gpc1 as your default camera in your ~/.ptolemyrc, you can omit the -D CAMERA gpc1.

The Relphot command comes from IPPstripe82

• Pull out star/galaxy separation flag, restrict sample to stars
• Filter on magnitude errors
• Dump ra dec mag magerr stargalflag into ASCII file for customized analysis