This page discusses the processing stages and database tables to be used to manage IPP full force photometry

Notes on the new "full force" stages of the pipeline

The current terminis of staticsky processing is the skycal results. The granulatity of these results are one per filter per skycell per processing version.

The new full force stages of the pipeline will take these results back to the input exposures and measure psf and extended source photometry using the staticsky measurements as calibrated by skycal.

The input paramters for the full force photometry is a 'cff' file. For efficiency these will be generated during skycal processing. An optional processing stage is outlined here which may be used to generate cff files post skycal (in case algorithms change of tweaks to the processing are necessary).

After full force has been run for all of the warps for a given skycell and filter, the results are then summarized though a TBD process.

Tables ======

fullForceWarpRun
    ffw_id
    warp_id
    skycal_id           (sets sky_id and through stack_id, filter, 
                         tess_id, skycell_id)
    input_path_base     path that leads to cff file 
                        Note: this is independent of the method
                        used to generate the data. 
                        The expectation is that the cffs will be generated
                        during skycal processing. The data has been
                        read and is available. zero point is done.

    workdir
    reduction           as usual specifies the recipe
    label
    data_group
    dist_group

This structure has to a separate run for each skycell. This will lead to a Nstack (~2Million) * Ninputs (~12) runs for a processing version (PVn)

Another option would be to structure like diffRun with a input table. In that case skycal_id, and input_path_base would move there.

At this level we're really doing the processing on the skycell, so the grouping by exposure might be less interesting. Avoiding the extra table simplifies things.

Is the 'Warp' in the table name needed? Would we ever do fullforce on a diff or stack? I'm hesitant to use the stage, stage_id style structure (addRun, distribution) since that makes everything more complicated. The code sharing ends up being somewhat illusory.

---

We might as well anticipate that we will need to generate cffs as a post skycal step. We don't want to require this because if we can generate it at the skycal stage we will save an awful lot of I/O.

cffGenerateRun
    cffg_id
    skycal_id
    label
    reduction
    etc

When processed reads the skycal cmf and generates a cff file. Ehh why bother. Why not just create a new skycalRun? Because the skycalRun might have already been used to build psps, etc and so we probably shouldn't rerun calibration.

=========== Once all of the fullforce warp processing is done we need to somehow collect the results

ffullForceSummaryRun
    ffs_id
    skycal_id       (sets stack_id, filter, skycell_id)

    The inputs are defined in the secondary table
  fullForceResultsInput
    ffw_id  (all inputs are linked back to the same skycal_id insured by query)

When processed this thing goes and generates a summary based on all of the results. (Kind of like addstar and dvo)

XXX: At this stage should we be combining the 5 filter results for each skycell?

TOOL

---

fftool  - full force tool

modes
-definebyquery      define fullforce runs
    usual args to select warps
    
    args to select input from skycal run or cffgenerate run
    -select_skycal_label
    -select_skycal_id

    -select_cffgenerate_label   if neither of these are specified assume
    -select cffg_id             input is in the skycal results

    -set_ arguments for all other parameters
-addresult
-revert
-updateffrun


-definecffgenerate  define a cffGenerateRun which will make a
                    cff file for a stack from the skycal outputs

    -select_skycal_label
    -select_skycal_id
    -set_ arguments for all parameters

-addcffgresult