
2013.01.31
  
Still working on the dvo ingest triggers:

 * setSkyArea generates a list of the regions to be ingested, based on
   file size / date tests -- we need to override these tests

 * 

2013.01.29

I'm trying to re-work the operational model.  First, make sure we have correct names for concepts:

* region : unit of spatial partition within DVO : a single region (eg,
  n0000/0020.01) has bounds of lines of constant RA & DEC, and is
  associated with 4 files containing dvo database tables for that part
  of the sky.

* batch : the unit of data sent to psps (1 camera-stage smf, 1 stack-stack cmf, 1 region-worth for obj, 1 init batch)

* box: an area on the sky for which we are generating / have generated
  batches.

* config : an area on the sky for which we are generating batches in boxes
  XXX : this name is somewhat confused & overloaded in ipptopsps -- it
  encompases both the static configuration information and the
  information about a processing chunk.

  I propose a new split of config -- put only the global system
  information in 'config' and define a new concept 'chunk', which
  corresponds to the region of the sky currently being processed

* client : an instance of an ipptopsps running proccess

Example sequence:

* ippjython queue.py edit : create a new 'config'


2013.01.27

remaining issues:

* operational sequence:

  I need to change the way ippToPsps works at the operational level.
  currently, the concept is that a 'config' is created once and the
  boundaries / aspect changed to trigger different processing.  this
  is poor because it leaves no tracking, and it means the current
  situation can change too easily.  in addition, the rules for
  triggering dvo ingest are not completely explicit, and probably not
  enforceable from within ippToPsps for parallel DVOs

  alternatively, I'd like to have the following sequence:

  * init batch should not require a config -- it simply passes along
    the statically defined tables from tables.IN.vot.  it should be
    required first, and perhaps depends on the associated dvo database
    (eg, for skytable?).

  * queue config (probably to include P2,ST,OB batches in general
    operations) -- should only be done for a region when the items for
    that region are done.

  * trigger dvo ingest for detections (and image / skytable?)
    (allow for rerunning ingest operations which failed)

  * run loader once to bundle the batches for the pending items
    - do not load dvo tables here
  
* modify object ingest to happen at same times as the detections

* seems to be some odd behavior -- failure to ingest, duplicate keys?

2013.01.26

I have modified the methods which create the StackDetectionCalib and
DetectionCalib tables to change the sql from "INSERT / UPDATE" to
"SELECT / LOAD" operations.  these need a target tmp directory
different from /tmp.  the trick is that /tmp has permissions which
prevent my (joe-user) from removing the temp files in the jython code,
but mysql will not write over an existing file, nor will it write to
other random directories.  the easy solution is to create a subdir in
/tmp for the output files.

This simple modification speeds up the processing of the smfs by a
factor of nearly 2x!

2013.01.24 : Status of the ippToPsps Upgrade

Here are the modifications I have made to ippToPsps:

* set up autoconfig, created a new script 'ippjython' so the code can run from any directory
  - config/settings.xml : needs a more flexible definition of the database names
  - on the MHPCC cluster, users need to set the env JAVA to point to the working version in ~ipp
  - installation does not yet include the installation of the jython.jar file (needs to go into PSCONFIGDIR/share/jar)
  - I shrunk the size of the SkyCell init table in IN/tables.vot (now tables.IN.vot)
    - this table was extremely slow to load
    - I'm not convinced this is used for anything other than init
    - probably should be inserted into the db with native dvo code?
  - ditto for ProjetionCell

* created test suites to inject (a) fake data and (b) real gpc1 data into a test db
  - choice of database should be different for test and for the operational system : see config/settings.xml above

* replaced the stilts-based ingest of the SkyTable.dat file with a native dvo operation (dvopsps skytable)

* added native dvo-based ingest of dvoDetections from dvo database (dvopsps detections)

* added options to switch between native and stilts versions of dvo ingest

* some modifications needed to support the pre-load option (needed regardless of choice of dvopsps or stilts)

* disabled the build of dvograbber (requires libxml which may not be installed, and is basically deprecated)

* allow detections to come from chips not named "XYnn" (eg, for nightly stacks and for tests)

* other fixes
  * fixed header keyword regex to support values of the form 1.0E+01
  * fixed getClientsColumnBoolean to return False if no rows are returned
  * do not crash if FWHM_MAJ, FW_MJ_UQ, PSFMODEL are not in header
  * missing astrometry (or missing chip) causes chip to be skipped 
  * added a KEY to dvoDetectionFull on objID, detectID
  * removed uniqueness requirement for objID on dvoDetectionFull
  * added capability for stacks to be in non-nebulous locations
  * removed looped search for stack cmfs (since they now come from skycal with unique names)
  * added ability to run loader once for a given config
  * fix the sql used to update the cpm tables (for stilts-based detection ingest)
  * drop ImageMeta_ tables before reading headers

* some test settings to watch for
  * ipptopsps and scratch dbs are set to test names by default (config/settings.xml, loader.py, dvo.py)
  * config.force is set to True (ipptopspsdb.py)

* concerns / fixes still needed:
  * the rules for ingesting the dvo detections need to be adjusted to be consistent with the dvopsps paradigm
  * it is not always clear when the scratch db or dvo tables are created
    * I added a createDvoTables call in setupScratchDb.py, but unclear if it should be there.
  * the names for the scratch DBs are in multiple places (config/settings.xml, loader.py, dvo.py)
  * the code is somewhat fragile to unexpected conditions.  for
    example, failure to select any inputs results in a java exception
    instead of a clear error message.  there are other cases where
    selects which return 0 lines cause crashes instead of informative
    output.
  * layout of smfs is fairly hard-wired and gpc1-specific (I've
    removed some of this, but more generality would be good)

  * need to enable -parallel option to dvopsps (config option?)
  * add multi-row ingest to stilts (maybe not immediately -- the
    select/load fix helped quite a bit)
  * need to fix test obj ingest (DONE)
  * file handles are not promptly closed -- is this a garbage
    collection problem?

* operational changes / thoughts
  * ippToPsps modules are now run with the command 'ippjython (module.py) [options]' from any directory as desired.
  * all P2 and ST batches use the dvopsps ingest method (assuming dvo.useStilts is left as False)

    This second point affects the choice of config region & box somewhat.  

    The script 'loader.py' loops over the region split into boxes
    defined by the 'box size' argument of the config.  This loop
    chooses both the gpc1 exposures (or stacks) to ingest, but also
    the dvo detctions loaded into the mysql database.

    In the past, the box size was chosen to be similar to a gpc1 footprint.  

    Now (at least for the moment), dvopsps is run for each box,
    ingesting that area (plus some buffer zone).  This says that a
    given config should use a single box to avoid inefficiency with
    dvopsps.

    It is critical that the loaded region overlap all detections from
    all gpc1 exposures in the current seletion.

    If the region chosen is too small, then detections on the edges
    may be missed (this probably forces the buffer zone to be 0.5*gpc1
    diameter -- but NOTE that the width is not adjusted by cos(DEC),
    and this can be a big error).

    Also, if the region is too small, then the dvopsps ingest
    operation is performed for too many overlapping areas, and this is
    inefficient.

    However, if the region is too big, we may run into non-linear
    speed effects on the mysql database (either for ingest or for
    queries using those detections to generate the batches).  

    In my tests, the entire SAS region was successfully ingested into
    mysql with dvopsps with no problem (** after I changed the code to
    have a certain key added to the table definition **).  

    *** I recommend running some tests with the galactic center to see
        the impact of large number of sources, and to see if / where
        there is a point of diminishing returns.  Ideally, we should
        use a config region which is large fairly large, at least
        comparable to SAS or larger.  

* timing information for non-parallel dvo:
  * SAS ingest to mysql with dvopsps (78M det) : ~100 min (~12k det / sec)
  * SAS region smfs (read, join, write) : ~110 sec (~1.2k det / sec)

  * extrapolating these to the full 3pi : 
    * ingest with dvopsps : 29 days
    * make batch for smf  : 290 days

  ** we need to use the multi-row ingest for the smfs to improve that
     portion of the batch creation, but it is not certain how much
     improvement we will see from that!

  ** clearly, if we use a single thread, we will be too slow, even with the above fix.
     * we can run more than one loader with the same parallel dvo db into multiple mysql dbs
