Changes between Initial Version and Version 1 of fullforce_intro

Timestamp:

Dec 10, 2013, 3:07:59 PM (13 years ago)

Author:

bills

Comment:

--

fullforce_intro

@@ +1 @@
+
+== This page discusses the processing stages and database tables to be used to manage IPP full force photometry ==
+
+Notes on the new "full force" stages of the pipeline
+
+The current terminis of staticsky processing is the skycal results. The
+granulatity of these results are one per filter per skycell per processing
+version.
+
+The new full force stages of the pipeline will take these results
+back to the input exposures and measure psf and extended source photometry
+using the staticsky measurements as calibrated by skycal.
+
+The input paramters for the full force photometry is a 'cff' file.
+For efficiency these will be generated during skycal processing.
+An optional processing stage is outlined here which may be used to
+generate cff files post skycal (in case algorithms change of tweaks to
+the processing are necessary).
+
+After full force has been run for all of the  warps for a given skycell
+and filter, the results are then summarized though a TBD process.
+
+
+Tables
+======
+
+{{{
+fullForceWarpRun
+    ffw_id
+    warp_id
+    skycal_id           (sets sky_id and through stack_id, filter, 
+                         tess_id, skycell_id)
+    input_path_base     path that leads to cff file 
+                        Note: this is independent of the method
+                        used to generate the data. 
+                        The expectation is that the cffs will be generated
+                        during skycal processing. The data has been
+                        read and is available. zero point is done.
+
+    workdir
+    reduction           as usual specifies the recipe
+    label
+    data_group
+    dist_group
+}}}
+
+
+This structure has to a separate run for each skycell. This will lead to a 
+Nstack (~2Million) * Ninputs (~12) runs for a processing version (PVn)
+
+Another option would be to structure like diffRun with a input table.
+In that case skycal_id, and input_path_base would move there.
+
+At this level we're really doing the processing on the skycell, so the
+grouping by exposure might be less interesting. Avoiding the extra
+table simplifies things.
+
+Is the 'Warp' in the table name needed? Would we ever do fullforce on
+a diff or stack? I'm hesitant to use the stage, stage_id style structure
+(addRun, distribution) since that makes everything more complicated.
+The code sharing ends up being somewhat illusory.
+
+
+----
+
+We might as well anticipate that we will need to generate cffs as a
+post skycal step. We don't want to require this because if we can generate
+it at the skycal stage we will save an awful lot of I/O.
+
+
+{{{
+cffGenerateRun
+    cffg_id
+    skycal_id
+    label
+    reduction
+    etc
+
+}}}
+
+When processed reads the skycal cmf and generates a cff file. Ehh why bother.
+Why not just create a new skycalRun? Because the skycalRun might have already
+been used to build psps, etc and so we probably shouldn't rerun calibration.
+
+===========
+Once all of the fullforce warp processing is done we need to somehow collect
+the results
+
+
+{{{
+ffullForceSummaryRun
+    ffs_id
+    skycal_id       (sets stack_id, filter, skycell_id)
+
+    The inputs are defined in the secondary table
+  fullForceResultsInput
+    ffw_id  (all inputs are linked back to the same skycal_id insured by query)
+
+}}}
+
+When processed this thing goes and generates a summary based on all of the
+results. (Kind of like addstar and dvo)
+
+XXX: At this stage should we be combining the 5 filter results for each
+skycell? 
+
+TOOL
+----
+
+{{{
+
+fftool  - full force tool
+
+modes
+-definebyquery      define fullforce runs
+    usual args to select warps
+    
+    args to select input from skycal run or cffgenerate run
+    -select_skycal_label
+    -select_skycal_id
+
+    -select_cffgenerate_label   if neither of these are specified assume
+    -select cffg_id             input is in the skycal results
+
+    -set_ arguments for all other parameters
+-addresult
+-revert
+-updateffrun
+
+
+-definecffgenerate  define a cffGenerateRun which will make a
+                    cff file for a stack from the skycal outputs
+
+    -select_skycal_label
+    -select_skycal_id
+    -set_ arguments for all parameters
+
+-addcffgresult
+
+
+}}}