
2008.01.31

  For testing the quality of the extended source measurements, I need
  to include them somehow within DVO.  My plan is to have a set of
  extended source tables linked to the average tables, parallel to the
  measure tables.  One issue is: how many rows in this table?  I think
  I need to have room for all fields measured by psphot  Another
  question: do I use a fixed layout I/O file, or do I use the header
  to read only the columns of interest.  I can do the latter, and it
  is only a little slower.

2007.02.22

I have several DVO improvements to implement.  I need to plan them a
bit carefully.  Here are my thoughts on these updates.

- add chip X,Y to measure table

  this is a fairly simple addition.  In addstar/find_matches, I just need
  to assign the X,Y value from the incoming star data.  For loading
  old catalogs which don't include X,Y in the table, I need to
  calculate the X,Y position based on the R,D after finding the
  matching image.  I have code to do this calculation currently in
  dvo/photometry.c for looking up X,Y on the fly.  once this is moved
  into the load functions, it will not be needed in photometry.c

- remove PRI/SEC and only use secfilt table for mags

  this is not a very difficult change, conceptually, but it may be a
  fair amount of work.  all of the functions which currently switch on
  the case of PRI/SEC just need to look in the secfilt location.  the
  value of Nsec needs to increase by 1.  the load from tables which
  used PRI/SEC need to move the PRI values to the correct location in
  secfilt

  eventually, rename 'secfilt' to a better name choice

- add image, average, measure IDs

  for PS1, the image and measure need to be generated by the external
  software.  they would just be part of the input stream.  for the
  case were the IDs are not supplied, DVO needs to generate them in a
  unique way.  I probably need to understand / define that mechanism
  before tackling this problem.

  one point: I need to keep the old averef index.  how do this index
  and the objID work together?

- link measure to image

  temporariliy use an index like averef?

- move photcode table to catdir (also zero points)

  should be fairly easy: just like the SkyTable, look in the catdir
  first, then generate from the default text table if it is missing.

- color term as a function of mosaic position
 
- mextract field,field,field 

  this should not be a very difficult job.  just add a loop to the
  avextract / mextract functions.

- mextract field(s) where condition

  this is a bit trickier, and could make use of the code in the dvo
  math functions

- dvo select from mysql

  not very difficult: need to define the commands to select the
  database and set up a connection, then parse the select line into
  the appropriate format.  just need to as the db for the type of the
  fields that are requested: must be numerical.

  select field,field,field from table where condition

- change vectors to doubles

  probably not a terrible job.  this will depend on how the union is
  defined.  I don't want this to explode the size of an image!

---------

I now have the ability to load and save DVO databases in old formats,
with automatic conversion to the current 'internal' representations
for the average, measure, and image tables.  an important concern with
this process is ensuring that we track the old layout names and avoid
breaking existing databases.  Here is a list of old databases, some of
which may not have been written with the right names, or with
inaccurate names.  these can be fixed by setting the FORMAT and MODE
header keywords with the 'fits_insert' command:

- LONEOS : old-style cmp files (ie, RAW), with the 'LONEOS' format.
- CFHT Elixir databases: cmp files, ELIXIR format
- Brandon's Taurus db: incorrect 'PANSTARRS' format, should be called
  'PSTEST1'.  drop support for this eventually? have brandon migrate to
  the new panstarrs formats?

I am going to use the following naming convention for future db table
updates:

- PANSTARRS.DEV.0, PANSTARRS.DEV.1, etc
- PANSTARRS.PS1.0, PANSTARRS.PS1.1, etc
- PANSTARRS.PS4.0, PANSTARRS.PS4.1, etc

The next version of the DVO format needs several major changes:

  - unique IDs in the measure and average tables.
  - reference to the source image in the measure table.
  - chip X,Y coordinates and errors in the measure table

  - remove primary average magnitude from average.d and only use
    the secfilt.d table for the average magnitudes

These are fairly substantial modifications.  Can I do these changes on
an appropriate timescale to get Dave Monet working with relastro?  Is
it necessary to do these all at once?

------

DVO has undergone at least two bit-incompatible versions of the data
structures to date: the 'loneos' version and the 'elixir' version.  

the 'loneos' version was used for the loneos database currently still
in my dataspace.

the 'elixir' version, which extended some of the structures and added
the Secfilt concept, was added at CFHT and is used for all of the
existing CFHT elixir databases.

For DVO-2 development, I will need to make a few modifications to
those tables.  It will be useful to allow versions of DVO which
interact with these different versions correctly.  At one level, this
simply means changing the structures in the included file.  More
difficult is that certain functions use elements of the structures
from one version which are not available in the other versions.  

I am going to explore this process briefly to get DVO-2 started,
particularly so I can solve the astrometric problems which are
currently limited by the use of floats to represent RA & DEC.  

I am going to make the selection of the structures depend on #define
values in the header files.  Currently, everybody which works with the
dvo files includes loneos.h.  This should be broken out to a loneos.h,
elixir.h, panstarrs.h, etc, all independently included from within
dvo.h, which will be the include file called within the user-level
programs.

Translations

/* average data as stored in the LONEOS database -- must be translated on load */

Loneos:Measure -> Elixir:Measure
/* OLD LONEOS VERSION.  NOT BYTE COMPATIBLE WITH NEW VERSION 
   (can be loaded with translation, losing the exptime,
   and limited the number of Naverage to 0xffffff - in load_catalog) */

------

the DVO structure Catalog contains the data from a single sky patch, including:
    average, measure, missing, and secfilt.  

the catalog may be saved in one of 4 modes (catalog.catmode):
    raw   : the old Elixir style which is a header + 4 binary tables
    mef   : catalog is a single file with header and 4 fits tables
    split : catalog is 5 files, one with the header and one for each table
    mysql : catalog is 4 (or 5?) mysql tables

    in split mode, the filename is the file of the descriptive header
    and the averages, the measure, missing, and secfilt data are
    stored in additional files referenced in the header and named
    based on the filename (replacing extension .cpa with .cpb, .cpc,
    .cpd, for example)

    
