IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changeset 5234


Ignore:
Timestamp:
Oct 7, 2005, 6:33:25 AM (21 years ago)
Author:
eugene
Message:

working on the update concept, added split table support

Location:
trunk/Ohana/src
Files:
13 edited

Legend:

Unmodified
Added
Removed

  • trunk/Ohana/src/addstar/doc/Changes.txt

    r5229 r5234  
     1
     22005.10.06
     3        split / nosort / update
     4        I have added a few new concepts to addstar recently: split
     5        catalog files, nosort for the measurement table, and
     6        update-only.
     7
     8        split mode
     9
     10        The split mode is quite straightforeward.  In this mode, each
     11        catalog is represented by a set of four files: *.cpt, *.cpm,
     12        *.cpn, *.cps.  Each file contains only one FITS table of the
     13        data, along with basic header and empty matrix.  Having
     14        individual tables for each component of the database lets me
     15        add entries without re-writing the entire table.  This should
     16        save on I/O operations in the long run. 
     17
     18        The first file contains the table of averages, and is the file
     19        normally identified in the table lookup functions.  The header
     20        of this file contains the names of the other table files
     21        (paths relative to the directory containing the cpt file).
     22        The names and extensions are specified in 'mkcatalog.c'; all
     23        other functions use the defined filename references, rather
     24        than expecting a naming convention.
     25
     26        The additional files contain the measures (cpm), missings
     27        (cpn), and secfilt (cps) elements of the catalog tables. 
     28
     29        To facilitate the handling of the additional filenames, file
     30        pointers, and headers, Catalog was extended to include
     31        pointers to the measure, missing, and secfilt files as
     32        additional catalogs.  When the data are loaded into memory,
     33        these catalogs are locked (as usual), and file information is
     34        stored in the individual Catalog entries; the data segements
     35        are all loaded into the main catalog pointers (eg, measures
     36        are loaded into catalog[0].measure, rather than
     37        catalog[0].measure_catalog[0].measure).
     38       
     39        The function 'load_catalog' auto-recognizes the SPLIT format
     40        by looking for the header keyword MEASURE, identifying the
     41        file containing the measures.  The identification of the RAW
     42        format and the SPLIT format are not cross-checked: if the
     43        NAXIS keyword is set to 2, the file is assumed to be RAW, even
     44        if the MEASURE keyword is present.  Careful with this (though
     45        there is no reason the main matrix should be used in a basic
     46        database table).
     47
     48        nosort
     49
     50        the nosort option by itself provides a minor processing
     51        speed-up by deferring the re-sorting of the measurement table
     52        until after multiple addstar processes are run.  addstar
     53        should not require the measurements to be sorted, so this step
     54        can be safetly deferred if only addstars are being performed.
     55        the other DVO operations require the sorted table, so the sort
     56        must be performed before they are run (either as part of the
     57        catalog load, not implemented yet, or with a call to addstar
     58        without the -nosort option set.  the real goal of the nosort
     59        option is to enable the -update concept in addstar, in which
     60        only the new rows are written out; this will only work if
     61        addstar can handle unsorted measures.
     62
     63        the nosort option required the addition of a 'sorted' element
     64        in the Catalog structure to track if the data are sorted or
     65        not.  On load, this flag is set based on the value of the
     66        header keyword SORTED; if the data is sorted during addstar,
     67        the flag is appropriately set, otherwise it is set FALSE be
     68        default. 
     69
     70        The nosort option requires a function which can generate the
     71        'next_meas' link sequence based on the measure table.  there
     72        is now a function called 'build_meas_link' which generates a
     73        correct link list; there is also the pair of functions
     74        'init_meas_links' and'init_miss_links' to generate the links
     75        in the event that the table is sorted (should be must
     76        quicker).
     77
     78        The 'missing' table is problematic: the LONEOS and ELIXIR
     79        formats do not carry an averef entry, thus they do not have
     80        enough information to define the links based only on the
     81        missing table.  This means we are forced to write out a sorted
     82        missing table; the nosort option is invalid for the missing
     83        table.  One future upgrade path is to add the averef entry to
     84        the PANSTARRS format and then only require the missing table
     85        to be sorted if the format is old and does not support
     86        -nosort.  (Note also that, for the moment, the missing table
     87        has only a single valid format).
     88
     89        In the process of defining the nosort option, I also cleaned
     90        up a bit the find_matches functions to use clearer functions
     91        for the links. 
     92       
     93        update
     94
     95        The 'update' process in principle allows addstar to
     96        substantially reduce the amount of I/O it needs to perform by
     97        only requiring addstar to write out new measures and new
     98        average/secfilt entries.
     99
     100        The 'missing' table is problematic: since the format does not
     101        support the 'nosort' option, it is not possible to use update
     102        with the missing table.  This means we are forced to write out
     103        a complete, sorted missing table.  This is currently
     104        implemented in update_catalog_split by simply writing out the
     105        complete missing table.  In fact, this choice is still flawed
     106        because the average table, since it is not written out in full
     107        each time, is inconsistent with the missing table: the Nn
     108        entries for each average, which identifies the number of
     109        missing entries, are not updated.  In practice, this means
     110        that the -update option forces the use of the -missed option,
     111        though at the moment, this is not forced or checked in any
     112        way.
     113
     114        Note that the 'missed' table contains duplicate information
     115        and can, in principle, be completely regenerated at any time.
     116        This should be an addstar option: to re-construct the missing
     117        table, potentially with constraints on the images which are
     118        searched for matches. 
    1119
    21202005.10.04

  • trunk/Ohana/src/addstar/doc/notes.txt

    r5229 r5234  
    11
    2 - measure <-> average matches and reordering
     22005.10.06
    33
    4   next[i] = i + 1
    5  
    6 
    7  
    8 
     4  I am getting seg faults from find_matches_closest, apparently related to the
     5  partial Measure load.  They occur when trying to realloc secfilt.  I
     6  tried to debug using the ohana_allocate stuff, but it causes
     7  problems because fitsio does not fall under the ohana memory
     8  system.  This is fairly bad; I should probably deal with that issue
     9  before pursuing the other stuff.  Do I have to make libfits depend
     10  on libohana, in which case I should probably split off libdvo from
     11  libohana.  That is probably not a bad plan in any case...
    912
    10132005.03.07 : notes related to new version of addstar

  • trunk/Ohana/src/addstar/include/addstar.h

    r5229 r5234  
    103103int ONLY_MATCH;
    104104int CLOSEST;
    105 int REORDER;
     105
     106int NOSORT;
     107int UPDATE;
    106108
    107109int MODE;
    … …  
    190192int add_miss_link (Average *average, int *next, int Nmissing);
    191193int *build_measure_links (Average *average, int Naverage, Measure *measure, int Nmeasure);
    192 Measure *reorder_measure (Average *average, int Naverage, Measure *measure, int Nmeasure, int *next);
    193 Missing *reorder_missing (Average *average, int Naverage, Missing *missing, int Nmissing, int *next_miss);
     194Measure *sort_measure (Average *average, int Naverage, Measure *measure, int Nmeasure, int *next);
     195Missing *sort_missing (Average *average, int Naverage, Missing *missing, int Nmissing, int *next_miss);
    194196
    195197/**

  • trunk/Ohana/src/addstar/src/args.c

    r5229 r5234  
    2323  fprintf (stderr, "  -replace                    : replace time/photcode measurements (no duplication)\n");
    2424  fprintf (stderr, "  -closest                    : use closest-star algorith\n");
     25  fprintf (stderr, "  -nosort                     : don't re-sort the measure entries (improves speed)\n");
     26  fprintf (stderr, "  -update                     : only update the new rows (foreces -nosort)\n");
    2527  fprintf (stderr, "  -image                      : only insert image data\n");
    2628  fprintf (stderr, "  -cal                        : perform zero-point calibration\n");
    … …  
    151153    remove_argument (N, &argc, argv);
    152154  }
    153   /* only add new rows (-update) or re-write complete measure table */
    154   REORDER = TRUE;
     155
     156  /* don't re-sort the measure sequence */
     157  NOSORT = FALSE;
     158  if ((N = get_argument (argc, argv, "-nosort"))) {
     159    NOSORT = TRUE;
     160    remove_argument (N, &argc, argv);
     161  }
     162  /* only add new rows (-update) or re-write complete measure table (forces -nosort) */
     163  UPDATE = FALSE;
    155164  if ((N = get_argument (argc, argv, "-update"))) {
    156     REORDER = FALSE;
    157     remove_argument (N, &argc, argv);
    158   }
     165    UPDATE = TRUE;
     166    NOSORT = TRUE;
     167    remove_argument (N, &argc, argv);
     168  }
     169
    159170  /* only add image potion to image table */
    160171  ONLY_IMAGES = FALSE;

  • trunk/Ohana/src/addstar/src/build_links.c

    r5229 r5234  
    11# include "addstar.h"
    22
    3 /* build the initial links assuming the table is re-ordered */
     3/* build the initial links assuming the table is sorted */
    44int *init_measure_links (Average *average, int Naverage, Measure *measure, int Nmeasure) {
    55
    … …  
    2323}
    2424
    25 /* build the initial links assuming the table is re-ordered */
     25/* build the initial links assuming the table is sorted */
    2626int *init_missing_links (Average *average, int Naverage, Missing *missing, int Nmissing) {
    2727
    … …  
    9494      m = next[m];
    9595    }
    96     if (m == Nm) continue;
    97     next[m] = Nm;
     96    if (m == Nm) {
     97      average[averef].Nm = k + 1;
     98    } else {
     99      average[averef].Nm = k + 2;
     100      next[m] = Nm;
     101    }
    98102  }
    99103  return (next);
    100104}
    101105
    102 /* Missing does not carry enough information to reconsruct the links
     106/* Missing does not carry enough information to reconstruct the links
    103107   we must always save the missing table, if it exists */
    104108
    105 Measure *reorder_measure (Average *average, int Naverage, Measure *measure, int Nmeasure, int *next) {
     109Measure *sort_measure (Average *average, int Naverage, Measure *measure, int Nmeasure, int *next) {
    106110
    107111  int i, k, n, N;
    … …  
    124128}
    125129
    126 Missing *reorder_missing (Average *average, int Naverage, Missing *missing, int Nmissing, int *next) {
     130Missing *sort_missing (Average *average, int Naverage, Missing *missing, int Nmissing, int *next) {
    127131
    128132  int i, k, n, N;

  • trunk/Ohana/src/addstar/src/find_matches.c

    r5229 r5234  
    88  float dX, dY, dR;
    99  int *N1, *N2,  *next_meas, *next_miss;
    10   int Nave, NAVE, Nmeas, NMEAS, Nmiss, NMISS, Nmatch;
     10  int Nave, NAVE, Nmeas, NMEAS, Nmiss, NMISS, Nmatch, offset;
    1111  Coords tcoords;
    1212  int Nsecfilt, Nsec;
    … …  
    4141  NMEAS = Nmeas = catalog[0].Nmeasure;
    4242  NMISS = Nmiss = catalog[0].Nmissing;
     43  offset = catalog[0].Nmeas_off;
    4344 
    4445  /* project onto rectilinear grid with 1 arcsec pixels */
    … …  
    7677
    7778  /* set up pointers for linked list of measure, missing */
    78   if (catalog[0].ordered) {
     79  if (catalog[0].sorted) {
    7980    next_meas = init_measure_links (catalog[0].average, Nave, catalog[0].measure, Nmeas);
    8081  } else {
    … …  
    8283  }   
    8384  next_miss = init_missing_links (catalog[0].average, Nave, catalog[0].missing, Nmiss);
    84   /* missing MUST be written ordered, or not at all */
     85  /* missing MUST be written 'sorted', or not at all */
    8586
    8687  /* choose a radius for matches */
    … …  
    142143      catalog[0].measure[Nmeas].Mcal_PS     = image[0].Mcal_PS;
    143144      catalog[0].measure[Nmeas].t           = image[0].tzero + 1e-4*stars[N].Y*image[0].trate;  /* trate is in 0.1 msec / row */
    144       catalog[0].measure[Nmeas].averef      = n;
     145      catalog[0].measure[Nmeas].averef      = n;              /* this must be an absolute sequence number, if partial average is loaded */
    145146      catalog[0].measure[Nmeas].source      = stars[N].code;  /* photcode */
    146147      catalog[0].measure[Nmeas].dophot      = stars[N].dophot; 
    … …  
    159160      Mval = (Nsec == -1) ? &catalog[0].average[n].M_PS : &catalog[0].secfilt[n*Nsecfilt+Nsec].M_PS;
    160161      if (*Mval == NO_MAG) *Mval = Mcat;
     162      /* in UPDATE mode, this value is not saved; use relphot to recalculate */
    161163
    162164      /* adds the measurement to the calibration if appropriate color terms are found */
    … …  
    184186        catalog[0].found[n] = Nmeas;
    185187      }
     188      /* Nm is updated, but not written out in -update mode (for existing entries)
     189         Nm is recalculated in build_meas_links if loaded table is not sorted */
    186190      catalog[0].average[n].Nm ++;
    187191      Nmeas ++;
    188192
    189       update_coords (&catalog[0].average[n], &catalog[0].measure[0], next_meas);
     193      if (!UPDATE) {
     194        /* in UPDATE mode, newly calculated coordinates are not saved */
     195        update_coords (&catalog[0].average[n], &catalog[0].measure[0], next_meas);
     196      }
    190197    }
    191198    i++;
    … …  
    217224
    218225  /* incorporate unmatched image stars, if this star is in field of this catalog */
     226  /* these new entries are all written out in UPDATE mode */
    219227  for (i = 0; (i < Nstars) && !ONLY_MATCH; i++) {
    220228    /* make sure there is space for next entry */
    … …  
    245253    catalog[0].average[Nave].Xm        = NO_MAG;
    246254    catalog[0].average[Nave].Xg        = NO_MAG;
    247     catalog[0].average[Nave].offset    = Nmeas;
     255    catalog[0].average[Nave].offset    = Nmeas + offset;   /* this needs to be an absolute reference */
    248256    catalog[0].average[Nave].missing   = -1;
    249257    catalog[0].average[Nave].code      = 0;
    … …  
    261269    catalog[0].measure[Nmeas].Mcal_PS     = image[0].Mcal_PS;
    262270    catalog[0].measure[Nmeas].t           = image[0].tzero + 1e-4*stars[N].Y*image[0].trate; /* trate is in 0.1 msec / row */
    263     catalog[0].measure[Nmeas].averef      = Nave;
     271    catalog[0].measure[Nmeas].averef      = Nave;           /* XXX EAM : must be absolute Nave if partial read */
    264272    catalog[0].measure[Nmeas].source      = stars[N].code;  /* photcode */
    265273    catalog[0].measure[Nmeas].dophot      = stars[N].dophot; 
    … …  
    296304    }
    297305
     306    /* next[Nmeas] should always be -1 in this context (it is always the only
     307       measurement for the star) */
    298308    stars[N].found = Nmeas;
    299309    next_meas[Nmeas] = -1;
    … …  
    316326  } */
    317327
    318   if (REORDER) {
    319     catalog[0].measure = reorder_measure (catalog[0].average, Nave, catalog[0].measure, Nmeas, next_meas);
    320     catalog[0].ordered = TRUE;
     328  if (NOSORT) {
     329    catalog[0].sorted = FALSE;
    321330  } else {
    322     catalog[0].ordered = FALSE;
    323   }
    324   catalog[0].missing = reorder_missing (catalog[0].average, Nave, catalog[0].missing, Nmiss, next_miss);
     331    catalog[0].sorted = TRUE;
     332    catalog[0].measure = sort_measure (catalog[0].average, Nave, catalog[0].measure, Nmeas, next_meas);
     333  }
     334  catalog[0].missing = sort_missing (catalog[0].average, Nave, catalog[0].missing, Nmiss, next_miss);
    325335  /* missing is REQUIRED to be sorted */
    326336

  • trunk/Ohana/src/addstar/src/find_matches_closest.c

    r5229 r5234  
    33void find_matches_closest (GSCRegion *region, Stars *stars, int Nstars, Catalog *catalog, Image *image, Image *overlap, int Noverlap) {
    44
    5   int i, j, n, N, J;
     5  int i, j, n, N, J, Jmin;
    66  double X, Y, RADIUS, RADIUS2, Rmin, secz;
    77  float *X1, *Y1, *X2, *Y2;
    88  float dX, dY, dR;
    99  int *N1, *N2,  *next_meas, *next_miss;
    10   int Nave, NAVE, Nmeas, NMEAS, Nmiss, NMISS, Nmatch, Jmin;
     10  int Nave, NAVE, Nmeas, NMEAS, Nmiss, NMISS, Nmatch, offset;
    1111  Coords tcoords;
    1212  int Nsecfilt, Nsec;
    … …  
    4141  NMEAS = Nmeas = catalog[0].Nmeasure;
    4242  NMISS = Nmiss = catalog[0].Nmissing;
     43  offset = catalog[0].Nmeas_off;
    4344 
    4445  /* project onto rectilinear grid with 1 arcsec pixels */
    … …  
    7677
    7778  /* set up pointers for linked list of measure, missing */
    78   if (catalog[0].ordered) {
     79  if (catalog[0].sorted) {
    7980    next_meas = init_measure_links (catalog[0].average, Nave, catalog[0].measure, Nmeas);
    8081  } else {
    … …  
    8283  }   
    8384  next_miss = init_missing_links (catalog[0].average, Nave, catalog[0].missing, Nmiss);
    84   /* missing MUST be written ordered, or not at all */
     85  /* missing MUST be written 'sorted', or not at all */
    8586
    8687  /* choose a radius for matches */
    … …  
    174175    Mval = (Nsec == -1) ? &catalog[0].average[n].M_PS : &catalog[0].secfilt[n*Nsecfilt+Nsec].M_PS;
    175176    if (*Mval == NO_MAG) *Mval = Mcat;
     177    /* in UPDATE mode, this value is not saved; use relphot to recalculate */
    176178
    177179    /* adds the measurement to the calibration if appropriate color terms are found */
    … …  
    183185    /* if we choose to calculate RA,DEC averages, see update_coords.c */
    184186
     187    /* Nm is updated, but not written out in -update mode (for existing entries)
     188       Nm is recalculated in build_meas_links if loaded table is not sorted */
    185189    stars[N].found = Nmeas;
    186190    catalog[0].found[n] = Nmeas;
    … …  
    189193    i++;
    190194  }
     195
     196  ohana_memcheck (0);
    191197
    192198  /** add reference for undetected catalog stars **/
    … …  
    215221
    216222  /** incorporate unmatched image stars, if this star is in field of this catalog **/
     223  /* these new entries are all written out in UPDATE mode */
    217224  for (i = 0; (i < Nstars) && !ONLY_MATCH; i++) {
    218225    /* make sure there is space for next entry */
    … …  
    243250    catalog[0].average[Nave].Xm        = NO_MAG;
    244251    catalog[0].average[Nave].Xg        = NO_MAG;
    245     catalog[0].average[Nave].offset    = Nmeas;
     252    catalog[0].average[Nave].offset    = Nmeas + offset;
    246253    catalog[0].average[Nave].missing   = -1;
    247254    catalog[0].average[Nave].code      = 0;
    … …  
    316323  } */
    317324
    318   if (REORDER) {
    319     catalog[0].measure = reorder_measure (catalog[0].average, Nave, catalog[0].measure, Nmeas, next_meas);
    320     catalog[0].ordered = TRUE;
     325  if (NOSORT) {
     326    catalog[0].sorted = FALSE;
    321327  } else {
    322     catalog[0].ordered = FALSE;
    323   }
    324   catalog[0].missing = reorder_missing (catalog[0].average, Nave, catalog[0].missing, Nmiss, next_miss);
     328    catalog[0].sorted = TRUE;
     329    catalog[0].measure = sort_measure (catalog[0].average, Nave, catalog[0].measure, Nmeas, next_meas);
     330  }
     331  catalog[0].missing = sort_missing (catalog[0].average, Nave, catalog[0].missing, Nmiss, next_miss);
    325332  /* missing is REQUIRED to be sorted */
    326333
    … …  
    342349  gettimeofday (&stop, NULL);
    343350  dtime = DTIME (stop, start);
    344   fprintf (stderr, "  match time %8.3f sec for %5d stars, %6d average\n", dtime, Nstars, Nave);
     351  fprintf (stderr, "  match time %9.4f sec for %5d stars, %6d average\n", dtime, Nstars, Nave);
    345352
    346353  return;

  • trunk/Ohana/src/addstar/src/gcatalog.c

    r5229 r5234  
    99
    1010  /* read catalog header */
    11   mode = LOAD_AVES | LOAD_MEAS | LOAD_MISS | LOAD_SECF;
     11  if (UPDATE) {
     12    mode = LOAD_AVES | LOAD_MEAS_META | LOAD_MISS | LOAD_SECF;
     13  } else {
     14    mode = LOAD_AVES | LOAD_MEAS | LOAD_MISS | LOAD_SECF;
     15  }
    1216  if (!load_catalog (catalog, mode, VERBOSE)) {
    1317    fprintf (stderr, "ERROR: failure loading catalog\n");
    … …  
    1620
    1721  /* should this be moved into save_catalog?? */
    18   status = fits_scan (&catalog[0].header, "ORDERED", "%t", 1, &catalog[0].ordered);
    19   if (!status) catalog[0].ordered = TRUE;
     22  status = fits_scan (&catalog[0].header, "SORTED", "%t", 1, &catalog[0].sorted);
     23  if (!status) catalog[0].sorted = TRUE;
    2024  /* XXX EAM - is this a good choice?  should the default be 'FALSE'? */
    2125

  • trunk/Ohana/src/addstar/src/mkcatalog.c

    r5229 r5234  
    111111  ALLOCATE (catalog[0].secfilt, SecFilt, 1);
    112112  catalog[0].Naverage = catalog[0].Nmeasure = catalog[0].Nmissing = 0;
     113  catalog[0].Nave_disk = catalog[0].Nmeas_disk = catalog[0].Nmiss_disk = 0;
     114  catalog[0].Nmeas_off = 0;
    113115  catalog[0].Nsecfilt = GetPhotcodeNsecfilt ();
    114116

  • trunk/Ohana/src/addstar/src/wcatalog.c

    r5229 r5234  
    77
    88  /* should this be moved into save_catalog?? */
    9   fits_modify (&catalog[0].header, "ORDERED",  "%t", 1, catalog[0].ordered);
     9  fits_modify (&catalog[0].header, "SORTED",  "%t", 1, catalog[0].sorted);
    1010
    11   if (!save_catalog (catalog, VERBOSE)) {
    12     fprintf (stderr, "ERROR: failure saving catalog\n");
    13     exit (1);
     11  if (UPDATE) {
     12    if (!update_catalog (catalog, VERBOSE)) {
     13      fprintf (stderr, "ERROR: failure updating catalog\n");
     14      exit (1);
     15    }
     16  } else {
     17    if (!save_catalog (catalog, VERBOSE)) {
     18      fprintf (stderr, "ERROR: failure saving catalog\n");
     19      exit (1);
     20    }
    1421  }
    1522

  • trunk/Ohana/src/libohana/include/dvo.h

    r5228 r5234  
    2424
    2525/* catalog values to be loaded */
    26 # define LOAD_AVES 0x01
    27 # define LOAD_MEAS 0x02
    28 # define LOAD_MISS 0x04
    29 # define LOAD_SECF 0x08
     26# define LOAD_AVES      0x01
     27# define LOAD_MEAS      0x02
     28# define LOAD_MISS      0x04
     29# define LOAD_SECF      0x08
     30# define LOAD_MEAS_META 0x10
    3031
    3132/* invalid mag value */
    … …  
    128129  Missing *missing;
    129130  SecFilt *secfilt;
    130   int Naverage, Nmeasure, Nmissing, Nsecfilt;
     131  int Naverage, Nmeasure, Nmissing, Nsecfilt;   /* current number of each component */
     132  int Nave_disk, Nmeas_disk, Nmiss_disk;        /* number of component on disk */
     133  int Nmeas_off;                                /* dist seq of first loaded data value */
     134  /* note the different counting for Nsecfilt */
    131135
    132136  /* pointers to split data files */
    … …  
    139143  int catmode;                          /* storage mode (raw, mef, split, mysql) */
    140144  int catformat;                        /* storage format (elixir, panstarrs, etc) */
    141   int ordered;                          /* is measure table average-ordered? */
     145  int sorted;                           /* is measure table average-sorted? */
    142146 
    143147  /* pointers for data manipulation */
    … …  
    177181int load_catalog (Catalog *catalog, char mode, int VERBOSE);
    178182int save_catalog (Catalog *catalog, char VERBOSE);
     183int update_catalog (Catalog *catalog, char VERBOSE);
    179184
    180185int FindMosaicForImage (Image *images, int Nimages, int entry);
    … …  
    291296int load_catalog_split (Catalog *catalog, char mode, int VERBOSE);
    292297int save_catalog_split (Catalog *catalog, char VERBOSE);
     298int update_catalog_split (Catalog *catalog, char VERBOSE);
    293299
    294300/*** DVO image db I/O Functions ***/

  • trunk/Ohana/src/libohana/src/dvo_catalog.c

    r5228 r5234  
    110110      break;
    111111    case DVO_MODE_SPLIT:
    112       update_catalog_split (catalog, VERBOSE);
     112      /* new file needs to use save_catalog_split */
     113      if (catalog[0].Nave_disk == 0) {
     114        save_catalog_split (catalog, VERBOSE);
     115      } else {
     116        update_catalog_split (catalog, VERBOSE);
     117      }
    113118      break;
    114119    default:

  • trunk/Ohana/src/libohana/src/dvo_catalog_split.c

    r5228 r5234  
    6565
    6666  measure = NULL;
     67
     68  /* (Full Load) */
    6769  if (mode & LOAD_MEAS) {
    6870    ALLOCATE (measure, Catalog, 1);
    … …  
    104106      fprintf (stderr, "Warning: mismatch between Nmeasure in PHU and Table headers (%d vs %d)\n", Nmeasure, catalog[0].Nmeasure);
    105107    }
    106     fits_free_header (&header);
    107     fits_free_matrix (&matrix);
     108    catalog[0].Nmeas_off = 0;
     109    fits_free_header (&header);
     110    fits_free_matrix (&matrix);
     111  }
     112
     113  /* (Meta Load) */
     114  if (mode & LOAD_MEAS_META) {
     115    ALLOCATE (measure, Catalog, 1);
     116
     117    /* get split filename from main header (paths relative to cpt file) */
     118    if (!fits_scan (&catalog[0].header, "MEASURE",  "%s", 1, string)) return (FALSE);
     119    ALLOCATE (measure[0].filename, char, strlen(path) + strlen(string) + 2);
     120    sprintf (measure[0].filename, "%s/%s", path, string);
     121
     122    /* lock & open catalog file */
     123    if (lock_catalog (measure, catalog[0].lockmode) != 1) {
     124      fprintf (stderr, "cannot access measure file %s\n", measure[0].filename);
     125      exit (2);
     126    }
     127
     128    /* read PHU */
     129    if (!fits_load_header (measure[0].f, &measure[0].header)) {
     130      if (VERBOSE) fprintf (stderr, "catalog file does not exist: %s\n", measure[0].filename);
     131      return (FALSE);
     132    }
     133
     134    /* set up the table size information */
     135    catalog[0].Nmeasure  = 0;        /* no rows loaded */
     136    catalog[0].Nmeas_off = Nmeasure; /* start of new entries */
    108137  }
    109138  catalog[0].measure_catalog = measure;
    … …  
    203232  catalog[0].secfilt_catalog = secfilt;
    204233
     234  /* save the current number so we can do partial updates */
     235  catalog[0].Nave_disk  = Naverage;
     236  catalog[0].Nmeas_disk = Nmeasure;
     237  catalog[0].Nmiss_disk = Nmissing;
     238
    205239  return (TRUE);
    206240}
    … …  
    275309    measure = catalog[0].measure_catalog;
    276310
     311    /* XXX EAM : warn about this condition; add code to handle? */
     312    if (catalog[0].Nmeas_off != 0) {
     313      fprintf (stderr, "WARNING: LOAD_MEAS_META mixed with save??\n");
     314      fprintf (stderr, "WARNING: this should not be allowed to happen!\n");
     315    }
     316
    277317    /* rewind file pointers and truncate (file is still open) */
    278318    fseek (measure[0].f, 0, SEEK_SET);
    … …  
    394434int update_catalog_split (Catalog *catalog, char VERBOSE) {
    395435
    396   int Nitems;
     436  int i, Nx, Ny, Nlines;
     437  int Nitems, Nskip, Nout, Ndisk, Nstart;
    397438  Matrix matrix;
    398439  Header header;
    399440  FTable ftable;
     441  VTable vtable;
    400442  Catalog *measure, *missing, *secfilt;
    401443
    402444  ftable.header = &header;
     445  vtable.header = &header;
    403446
    404447  if (catalog[0].Naverage == 0) {
    … …  
    409452  /* make sure header is consistent with data */
    410453  fits_modify (&catalog[0].header, "NSTARS",   "%d", 1, catalog[0].Naverage);
    411   fits_modify (&catalog[0].header, "NMEAS",    "%d", 1, catalog[0].Nmeasure);
     454  fits_modify (&catalog[0].header, "NMEAS",    "%d", 1, catalog[0].Nmeasure + catalog[0].Nmeas_off);
    412455  fits_modify (&catalog[0].header, "NMISS",    "%d", 1, catalog[0].Nmissing);
    413456  fits_modify (&catalog[0].header, "NSECFILT", "%d", 1, catalog[0].Nsecfilt);
    … …  
    424467  }
    425468
    426   /* in split mode, we can save only part of the data */
    427 
    428469  /*** Average Table ***/
    429470
    430471  if (catalog[0].average != NULL) {
    431     ftruncate (fileno (catalog[0].f), catalog[0].header.size);
    432 
    433     /* this is probably a NOP, do I have to keep it in? */
    434     fits_create_matrix (&catalog[0].header, &matrix);
    435     if (!fits_fwrite_matrix  (catalog[0].f, &matrix)) {
    436       fprintf (stderr, "can't write primary matrix");
    437       return (FALSE);
    438     }
    439     fits_free_matrix (&matrix);
     472
     473    /* skip past matrix (already at end of header) */
     474    Nskip = fits_matrix_size (&catalog[0].header);
     475    fseek (catalog[0].f, Nskip, SEEK_CUR);
     476
     477    /* how many lines to write out? */
     478    Nout = catalog[0].Naverage - catalog[0].Nave_disk;
    440479
    441480    /* write out Average table (convert to FITS table format) */
    442481    AverageToFtable (&ftable, catalog[0].average, catalog[0].Naverage, catalog[0].catformat);
     482    /* convert only output rows to vtable */
     483    fits_table_to_vtable (&ftable, &vtable, catalog[0].Nave_disk, Nout);
     484
    443485    if (!fits_fwrite_Theader (catalog[0].f, &header)) {
    444486      fprintf (stderr, "can't write table header");
    445487      return (FALSE);
    446488    }
    447     if (!fits_fwrite_table (catalog[0].f, &ftable)) {
    448       fprintf (stderr, "can't write table data");
    449       return (FALSE);
    450     }
     489    if (!fits_fwrite_vtable (catalog[0].f, &vtable)) {
     490      fprintf (stderr, "can't write table data");
     491      return (FALSE);
     492    }
     493    fits_free_vtable (&vtable);
    451494    fits_free_table (&ftable);
    452495    fits_free_header (&header);
    … …  
    459502    measure = catalog[0].measure_catalog;
    460503
    461     /* rewind file pointers and truncate (file is still open) */
    462     fseek (measure[0].f, 0, SEEK_SET);
    463     ftruncate (fileno (measure[0].f), 0);
    464 
    465     /* write table PHU header */
    466     if (!fits_fwrite_header  (measure[0].f, &measure[0].header)) {
    467       fprintf (stderr, "can't write primary header");
    468       return (FALSE);
    469     }
    470 
    471     /* this is probably a NOP, do I have to keep it in? */
    472     fits_create_matrix (&measure[0].header, &matrix);
    473     if (!fits_fwrite_matrix  (measure[0].f, &matrix)) {
    474       fprintf (stderr, "can't write primary matrix");
    475       return (FALSE);
    476     }
    477     fits_free_matrix (&matrix);
    478 
    479     /* write out Measure table (convert to FITS table format) */
    480     MeasureToFtable (&ftable, catalog[0].measure, catalog[0].Nmeasure, catalog[0].catformat);
     504    /* skip past PHU header and matrix */
     505    Nskip = measure[0].header.size + fits_matrix_size (&measure[0].header);
     506    fseek (measure[0].f, Nskip, SEEK_SET);
     507
     508    Ndisk  = catalog[0].Nmeas_disk;
     509    Nstart = catalog[0].Nmeas_disk - catalog[0].Nmeas_off; /* where is first new line? */
     510    Nout   = catalog[0].Nmeasure - Nstart;                   /* how many lines to write out? */
     511    Nlines = catalog[0].Nmeasure + catalog[0].Nmeas_off;   /* how many lines total in file */
     512
     513    /* convert to output format FITS table (only rows for output : 0 - Nout) */
     514    MeasureToFtable (&ftable, &catalog[0].measure[Nstart], Nout, catalog[0].catformat);
     515
     516    fits_scan (&header, "NAXIS1", "%d", 1, &Nx);
     517    fits_scan (&header, "NAXIS2", "%d", 1, &Ny);
     518
     519    /* convert all output rows to vtable */
     520    ALLOCATE (vtable.row, int, MAX (1, Nout));
     521    ALLOCATE (vtable.buffer, char *, MAX (1, Nout));
     522    for (i = 0; i < Nout; i++) {
     523      ALLOCATE (vtable.buffer[i], char, MAX (1, Nx));
     524      memcpy (vtable.buffer[i], &ftable.buffer[i*Nx], Nx);
     525      vtable.row[i] = i + Ndisk;
     526    }
     527
     528    /* modify vtable to represent full disk table */
     529    fits_modify (&header, "NAXIS2", "%d", 1, Nlines);
     530    header.Naxis[1] = Nlines;
     531
     532    vtable.size = fits_matrix_size (&header);
     533    vtable.Nrow = Nout;
     534    vtable.pad = vtable.size - Nx*Ny;
     535
    481536    if (!fits_fwrite_Theader (measure[0].f, &header)) {
    482537      fprintf (stderr, "can't write table header");
    483538      return (FALSE);
    484539    }
    485     if (!fits_fwrite_table (measure[0].f, &ftable)) {
    486       fprintf (stderr, "can't write table data");
    487       return (FALSE);
    488     }
     540    if (!fits_fwrite_vtable (measure[0].f, &vtable)) {
     541      fprintf (stderr, "can't write table data");
     542      return (FALSE);
     543    }
     544    fits_free_vtable (&vtable);
    489545    fits_free_table (&ftable);
    490546    fits_free_header (&header);
    … …  
    492548
    493549  /*** Missing Table ***/
     550  /* missing table CANNOT be written unsorted, thus it is always written
     551     out in full */
    494552
    495553  if (catalog[0].missing != NULL) {
    … …  
    535593    secfilt = catalog[0].secfilt_catalog;
    536594
    537     /* rewind file pointers and truncate (file is still open) */
    538     fseek (secfilt[0].f, 0, SEEK_SET);
    539     ftruncate (fileno (secfilt[0].f), 0);
    540 
    541     /* write table PHU header */
    542     if (!fits_fwrite_header  (secfilt[0].f, &secfilt[0].header)) {
    543       fprintf (stderr, "can't write primary header");
    544       return (FALSE);
    545     }
    546 
    547     /* this is probably a NOP, do I have to keep it in? */
    548     fits_create_matrix (&secfilt[0].header, &matrix);
    549     if (!fits_fwrite_matrix  (secfilt[0].f, &matrix)) {
    550       fprintf (stderr, "can't write primary matrix");
    551       return (FALSE);
    552     }
    553     fits_free_matrix (&matrix);
    554 
    555     /* write out SecFilt table (convert to FITS table format) */
     595    /* skip past PHU header and matrix */
     596    Nskip = secfilt[0].header.size + fits_matrix_size (&secfilt[0].header);
     597    fseek (secfilt[0].f, Nskip, SEEK_SET);
     598
     599    /* how many lines to write out? */
     600    Nout  = catalog[0].Nsecfilt * (catalog[0].Naverage - catalog[0].Nave_disk);
     601    Ndisk = catalog[0].Nsecfilt * catalog[0].Nave_disk;
     602
     603    /* convert to output format FITS table */
    556604    Nitems = catalog[0].Naverage * catalog[0].Nsecfilt;
    557605    SecFiltToFtable (&ftable, catalog[0].secfilt, Nitems, catalog[0].catformat);
     606    /* convert only output rows to vtable */
     607    fits_table_to_vtable (&ftable, &vtable, Ndisk, Nout);
     608
    558609    if (!fits_fwrite_Theader (secfilt[0].f, &header)) {
    559610      fprintf (stderr, "can't write table header");
    560611      return (FALSE);
    561612    }
    562     if (!fits_fwrite_table (secfilt[0].f, &ftable)) {
    563       fprintf (stderr, "can't write table data");
    564       return (FALSE);
    565     }
     613    if (!fits_fwrite_vtable (secfilt[0].f, &vtable)) {
     614      fprintf (stderr, "can't write table data");
     615      return (FALSE);
     616    }
     617    fits_free_vtable (&vtable);
    566618    fits_free_table (&ftable);
    567619    fits_free_header (&header);
    … …  
    578630
    579631/* XXX EAM : this file needs work on the error exit conditions and memory leaks, esp under errors */
     632
     633/* XXX EAM : update is not efficient.  MeasureToFtable should only
     634   convert the new rows (Nmeas_disk to Nmeasure). the resulting
     635   table represents the end rows of the vtable.  we need to define
     636   the vtable based on the ftable, but with Ny = Nmeasure */ 
     637 
Note: See TracChangeset for help on using the changeset viewer.