Changes between Initial Version and Version 1 of AddStarUpgradeNov2011

Timestamp:

Nov 15, 2011, 4:47:59 PM (15 years ago)

Author:

heather

Comment:

--

AddStarUpgradeNov2011

@@ +1 @@
+== Introduction == 
+
+We have had problems with addstar + merging in the past, and we need to fix them.  There are several failure modes which have caused problems in the past, or which I imagine could cause problems:
+
+ * the addstar machine (ipp004 and/or ipp005) fills up
+ * there are nfs related issues (this may cause problems with dvomerge?)
+ * if the merge gets corrupted, we do not stop and this causes further problems
+ * we do not back up the minidvodbs often enough
+ * we do not back up the merge 
+ * we merge the minidvodbs again, causing database chaos
+
+== The way addstar works now == 
+
+  === Queueing ===
+
+We have several scripts and survey tasks to queue up the camera stage and the staticsky stage for running addstar.  New addRuns are queued and grouped by a minidvodb_group. This minidvodb_group is used to prevent multiple camRuns for the same exp_id from being queued.  The first one in is the first one used. 
+
+  === Ingesting addstar runs === 
+
+Initially, addRuns are queued with a minidvodb_group, but no minidvodb_name. When addstar is running, it looks for the current (active state) minidvodb for a given minidvodb_group, and ingests that add_id into that minidvodb.  Once completed, the addRun is updated to reflect where the cam/staticsky was ingested.
+
+  === Minidvodb Tasks ===
+
+ * Creation 
+   * When certain criteria are met, a new minidvodb is created. Currently a new minidvodb is created when the current active minidvodb has more than 500 add_ids or is older than 1 day. When a new minidvodb is created, the old active minidvodb is retired (set to a state of "to_be_merged".
+
+ * Verification and Merging
+   * When the tasks detect a new minidvodb in a state of "to_be_merged", the following happens:
+     * relphot  -- run relphot on the minidvodb -- fail if it doesn't relphot
+     * addstar -resort -- resort the minidvodb -- fail if it doesn't resort
+     * verify that it is resorted -- fail if it isn't
+     * dvomerge the minidvodb into the big dvodb -- if there is a failure here, it unfortunately blindly goes onto the next minidvodb to merge, which causes corruption.  
+
+  === MinidvodbCopy ===
+    
+There are a set of tasks and db tables to make backup copies of the minidvodbs.  More details can be seen here (http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/MinidvodbCopy).  MinidvodbCopy takes a minidvodb_group, host, and destination path to copy minidvodbs somewhere.  I haven't run into problems with it yet. It has checks for diskspace - it doesn't transfer unless there is enough room.
+
+  === Backups of the big dvodb ===
+
+These are done by hand, and very sporadically.  We often merge hundreds of minidvodbs before making a backup. We often lose track of where the backups go (or they get modified and thus can't be used).  This is a horribly inefficient way to do this, and most recoveries take forever because we have to merge back together all the minidvodbs (this takes a month).
+
+== What I would like to change == 
+
+  I would like to split up the minidvodb tasks and tables to something like this:
+
+
+  The minidvodb tasks/tables/scripts
+
+  *  minidvodbRun
+  *  minidvodbProcesed
+  *  minidvodbCopy 
+ 
+  These would be mostly unchanged, with the exception that they are NOT merged into the big database.  Once a minidvodb is relphot'd/resorted/verified, it is now ready for merging and copying a backup to ippmanoa. 
+
+  Mergedvodb tasks/tables/scripts 
+
+  * mergedvodbRun
+  * mergedvodbProcessed
+  * mergedvodbCopy 
+
+  These would be for tracking/queueing/backing up the big dvodbs.  Some ideas:
+
+   * no queuing up a merge if there is a previously failed merge for a given big dvodb (this makes sense- allows us to fix it).  
+   * mergedvodbCopy - my idea is after each successful merge, do an rsync backup of this.  I don't know how long this takes, but we can investigate this
+   * we can easily create one-off smaller dvodbs if necessary this way, and track it and have it in the database (handy for roy and probably others)
+
+  I would also like to run some cron job that detects full disk space on ipp004/ipp005, stops the addstars if necessary, and emails me (or the group) to notify us. 
+
+
+
+== How this would address the addstar problems ==
+
+
+   *  the addstar machine (ipp004 and/or ipp005) fills up  -- solved with a cron job
+   *  there are nfs related issues (this may cause problems with dvomerge?) -- we will have a backup of the old dvodb to recover from
+   *  if the merge gets corrupted, we do not stop and this causes further problems -- we will now stop (and we have a backup).
+   *  we do not back up the minidvodbs often enough -- we will back up minidvodbs on ippb03
+   *  we do not back up the merge -- we will back this up now
+   *  we merge the minidvodbs again, causing database chaos -- this would be done as a new mergedvodbRun (instead of hacking at the minidvodb tables)
+
+   * what if we have some crazy corruption (like the wrong data in the db?) -- I don't have any good way for this, except to keep a spare older *good* dvodb to rebuild from.