Changes between Initial Version and Version 1 of Other_DVO_Tools

Timestamp:

Dec 8, 2010, 9:57:17 AM (16 years ago)

Author:

eugene

Comment:

--

Other_DVO_Tools

@@ +1 @@
+== Other DVO Tools ==
+
+dvomerge
+
+dvomerge is used to merge dvo databases.  there are two main modes of operation:
+
+* merge two databases into a third, previously absent, database: dvomerge (input1) and (intput2) to (output)
+* merge one database into an existing database : dvomerge (input) into (output)
+
+Note that all dvomerge options may include the -region (RAmin) (RAmax) (DECmin) (DECmax) options
+  to restrict the merge to a portion of the sky.  
+
+The second mode of operation has two minor variants:
+
+* dvomerge (input) into (output) continue
+  the 'continue' option allows the dvomerge process to operate on a
+  fraction of the sky, in the assumption that an initial merge was
+  performed (with the 'into' mode).  The 'continue' is needed to tell
+  dvomerge that the images of the two databases have already been
+  merged; in this case, dvomerge will examine the image tables and
+  determine the mapping of image IDs from the state of the output
+  database. 
+
+* dvomerge (input) into (output) from (list)
+  the 'from' variant is equivalent to the 'continue' variant, but in
+  this case the catalog tables to be merged are specified in the list
+  file.  The file contains one catalog file per line, with the name
+  giving the directory and filename of the cpt files, without the
+  catdir component (eg, n0000/0001.01.cpt).  
+
+Threads : threading support is available for dvomerge .. into, except
+for the 'from' mode.  Add the option '-threads N' to specify the
+number of threads to be used.  Each thread handles a separate catalog
+table (read from disk, calculate the merge, write to disk).  With only
+a few threads, it is easy to saturate the disk I/O system, at which
+point the threading does not improve the speed of the merge process.
+
+Some other words of warning: 
+
+* dvomerge does not attempt to check for previous merges (successful
+  or not).  A re-run of dvomerge will simply duplicate the data
+  previously merged.  You must track your own results: which cpt files
+  were merged in a given run.
+
+* the region option selects the cpt files which touch the region.  to
+  select exactly the cpt files you want, reduce the region by a small
+  amount to avoid overlapping neighboring regions.  For example, if
+  you want to include only a specific cpt file with RA,DEC bounds of
+  0.0 - 1.0, 2.5 - 3.5, you might use -region 0.01 0.99 2.51 3.49
+
+* the cpt headers define their boundaries (RA0, RA1, DEC0, DEC1).
+
+* the merged dvo database is not sorted (indexed).  To use it with the
+  dvo shell or programs like relphot or relastro, it is necessary to
+  re-index with addstar -resort -D CATDIR (catdir).  (Note that
+  addstar -resort is also now able to use multiple threads: add the
+  -threads N option).
+
+dvorepair
+
+dvorepair can be used to fix damaged dvo databases.  Repair is
+possible because the measurement tables (cpm) and the image table
+together contain all of the information needed to regenerate the
+average (cpt) and secfilt (cps) tables.  
+
+dvorepair has several modes of operation:
+
+* dvorepair -images-vs-measures (catdir) (Ntol)
+
+  this mode scans the database and compares the number of detections
+  tied to a specific image (in the measure tables) with the number of
+  detections claimed by that image (in the average table).  Images
+  which have too few detections are reported to standard out.  The
+  value of Ntol specifies the number of detections which can be
+  dropped -- a few missing detections for a given image is not
+  catastrophic.  
+
+* dvorepair -delete-image-list (catdir) (list) 
+
+  this mode accepts the list of bad images as reported by the
+  -images-vs-measures mode.  It then scans the database and deletes
+  all detections associated with the deleted images.  It also repairs
+  the average (cpt) and secfilt (cps) tables so they are consistent
+  with the repaired measurement tables.  The repaired output tables do
+  not overwrite the existing tables; instead they are written with the
+  suffix ".fixed".  It is up to the user to find and replace the
+  original files with the .fixed version.
+
+  Note that the user may edit the output list from -images-vs-measures
+  and choose not to delete images as desired.  But be careful: the
+  format of the lines of the input file must match the original output
+  format.
+
+  Note that a repaired database is not indexed: you must run addstar
+  -resort to re-index the database.
+
+* dvorepair -fix-images (catdir) (list)
+
+  This mode accepts the same for list file as -delete-image-list, and
+  in this case, deletes the images from the image table (as opposed to
+  the detections).
+
+* dvorepair -fix-cpt (images) (list)
+
+  This mode accepts a list of corrupted cpt files and rebuilds the
+  average and secfilt tables from the measurements.  Instead of a
+  (catdir), the inputs are the full path or relative path to the image
+  table and a list of the full or relative paths to the cpt files
+  which are broken, ** with the .cpt extension removed**.
+
+  Again, repaired cpt, cps, cpm files are saved with the extension .fixed.
+
+dvoverify
+
+dvoverify will scan a dvo database or subset (by region) and verify
+the database integrity.  At the moment, the only check it makes is to
+verify that the tables are FITS compliant.  (ie, no check is made for
+internal consistency of the number of average, measure, or secfilt
+entries).
+
+
+