Changes between Version 261 and Version 262 of ippToPsps

Timestamp:

Apr 4, 2012, 12:21:42 PM (14 years ago)

Author:

rhenders

Comment:

--

ippToPsps

@@ -115 +115 @@
 == DVO ==
 
-{{{ippToPsps}}} needs to access IPP DVO databases in order to attain various IDs that are assigned within DVO. This is done is two different ways, depending on survey.
-
-DVO has an API written in C that can be used to extract detection records for a give 'IMAGEID' found in cmf and smf file headers (see note below regarding IMAGEID confusion). The usual case is that {{{ippToPsps}}} uses this form of DVO access using a small C program called {{{dvograbber}}}, the source for which is in the {{{ippToPsps/src}}} directory.
-
-{{{ippToPsps}}} has a second way of accessing DVO, which is only intended to be used in the special case of surveys (like MD4), or areas of surveys, have very high coverege in a certain area of sky meaning the underlying DVO FITS grow to a very large large size. For such areas, it becomes unfeasible to access DVO via the C api as repeatedly reading FITS files as large as 2GB causes read times of up to hours per frame.
-
-Instead, for these regions, {{{ippToPsps}}} can pre-ingest a region of DVO into a scratch MySQL database.
+{{{ippToPsps}}} needs to access IPP DVO databases in order to attain various IDs that are assigned within DVO. This is done in two different ways, depending on survey.
+
+DVO has an API written in C that can be used to extract detection records for a give 'IMAGEID' found in cmf and smf file headers (see note below regarding IMAGEID confusion). The usual case is that {{{ippToPsps}}} uses this form of DVO access using a small C program called {{{dvograbber}}}, the source for which is in the {{{ippToPsps/src}}} directory, [browser:trunk/ippToPsps/src here].
+
+{{{ippToPsps}}} has a second way of accessing DVO, which is only enabled when parts of the sky are encountered where there is especially large coverage, eg a medium deep field. In such regions, the underlying DVO FITS grow to a very large size and it becomes unfeasible to access DVO via the C api as repeatedly reading FITS files as large as 2GB causes read times of up to hours per frame.
+
+So, for these regions, {{{ippToPsps}}} can pre-ingest a region of DVO into a scratch MySQL database.
 
 It can take a long time to convert a relatively small DVO database to MySQL, however, querying the MySQL database is hugely faster than accessing DVO directly, especially for regions of sky with a high density of detections such as the medium deep fields. (This was seen when loading MD4 prior to the Boston meeting in May 2011. DVO access per exposure was 40 minutes, whereas, once imported to MySQL, query time was roughly 30 seconds.)
+
+The decision of when to pre-ingest DVO into MySQL is made in the {{{loader.py}}}. The {{{queue.py}}} program has already queued up smfs to load by box on the sky. {{{loader.py}}} uses the ratio of smfs files to size of DVO files on disk to determine if it is worthwhile pre-ingesting, or else importing DVO data the usual (C-interface) way. When the loader progresses to the next, neighboring box, the same ratio is calculated. If the decision is made again to pre-ingest from DVO, then any DVO regions overlapping both boxes are retained, while the remainder are purged before importing the new ones.
 
 '''Image ID confusion'''