This page discusses the processing stages and database tables to be used to manage IPP full force photometry

Notes on the new "full force" stages of the pipeline

The current terminis of staticsky processing is the skycal results. The granulatity of these results are one per filter per skycell per processing version.

The new full force stages of the pipeline will take certain static sky parameters back to the input exposures and measure psf and extended source photometry.

The input parameters for the full force photometry are contained in a 'cff' file. For efficiency these will be generated during skycal processing. An optional processing stage is outlined here which may be used to generate cff files post skycal (in case algorithms change of tweaks to the processing are necessary).

After full force has been run for all of the warps for a given skycell and filter, the results are then summarized though a TBD process.

Tables

Here are my ideas for the structure of the tables.

CREATE TABLE fullForceRun (
    ff_id           BIGINT NOT NULL AUTO_INCREMENT,
    warp_id         BIGINT,
    skycal_id       BIGINT,
    sources_path_base VARCHAR(255),
    state           VARCHAR(64),
    workdir         VARCHAR(255),
    label           VARCHAR(64),
    data_group      VARCHAR(64),
    dist_group      VARCHAR(64),
    reduction       VARCHAR(64),
    registered TIMESTAMP DEFAULT CURRENT_TIMESTAMP, -- time run was registered
    PRIMARY KEY(ff_id),
    KEY(state),
    KEY(label),
    KEY(data_group),
    KEY(warp_id),
    KEY(skycal_id),
    FOREIGN KEY(warp_id) REFERENCES warpRun(warp_id),
    FOREIGN KEY(skycal_id) REFERENCES skycalRun(skycal_id)
) ENGINE=innodb DEFAULT CHARSET=latin1;

This structure has to a separate run for each skycell. This will lead to a Nstack (~2Million) * Ninputs (~12) runs for a processing version (PVn)

Another option would be to structure like diffRun with a input table. In that case skycal_id, and input_path_base would move there.

At this level we're really doing the processing on skycells, so the grouping by exposure might be less interesting. Avoiding the extra table simplifies code.

Note that this structure ties the full force run to a warp. One could imagine doing running it on a chip, diff, or stack. I'm hesitant to use the stage, stage_id style structure (addRun, distribution) because that makes everything more complicated. The code sharing ends up being somewhat illusory. Are five 10 line functions really better than one 50 line function with lots of cases?

---

We might as well anticipate that we will need to generate cffs as a post skycal step. We don't want to require this because if we can generate it at the skycal stage we will save an awful lot of I/O.

cffGenerateRun
    cffg_id
    skycal_id
    label
    reduction
    etc

When processing this run the skycal cmf and generates a cff file. Why bother. Why not just create a new skycalRun? Because the skycalRun might have already been used to build psps, etc and so we probably shouldn't confuse things by reruning calibration.

---

Once all of the fullforce warp processing is done we need to somehow collect the results

ffullForceSummaryRun
    ffs_id
    skycal_id       (sets stack_id, filter, skycell_id)

    The inputs are defined in the secondary table
  fullForceResultsInput
    ffw_id  (all inputs are linked back to the same skycal_id insured by query)

When this thing is processed the results from the multiple warps are analyzed and we magically get the parameters for any object in the visible universe ;) (Kind of like addstar and dvo).

---

TOOL

As usual we will define a tool for managing the tables.

fftool  - full force tool

modes
 -definebyquery      define fullforce runs
    usual args to select warps
    
    args to select input from skycal run or cffgenerate run
    -select_skycal_label
    -select_skycal_id

    -select_cffgenerate_label   if neither of these are specified assume
    -select cffg_id             input is in the skycal results

    -set_ arguments for all other parameters
 -addresult
 -revert
 -updateffrun


 -definecffgenerate  define a cffGenerateRun which will make a
                    cff file for a stack from the skycal outputs

    -select_skycal_label
    -select_skycal_id
    -set_ arguments for all parameters

-addcffgresult