= Large Area Processing =

= Czaring hints =

 * [wiki:LAP_Czar LAP specific czaring issues]

= Scheduling =

 * [wiki:LAP_Science_Scheduling Current planned scheduling of LAP science] 

= Database tables =

The '''lapRun''' table defines and groups the reprocessing of a single projection cell

|| '''name'''      || '''type'''   || '''note''' ||
|| lap_id          || int    ||             ||
|| seq_id          || int    || ||
|| tess_id         || string || ||
|| projection_cell || string || ||
|| filter          || string || ||
|| state           || string || 'new' if still running, 'full' if complete ||
|| registered      || date   || date lapRun was defined ||
|| fault           || int    || ||
|| label           || string || ||
|| quick_sass_id   || int    || stackAssociation id of the quickstack for this run. No longer needed or used.||
|| final_sass_id   || int    || stackAssociation id of the final stack for this run ||
|| dist_group      || string || distribution group to give to all dependents ||

The '''lapExp''' table defines the pool of exposures, and associates these with the lapRuns that use them

|| '''name'''       || '''type'''   || '''note''' ||
|| lap_id     || int    || ||
|| exp_id     || int    || ||
|| chip_id    || int    || chip_id of the processing of exp_id used in this run ||
|| pair_id    || int    || chip_id of the companion exposure for a pairwise diff ||
|| private    || bool   || this exp_id/chip_id set is split across projection cells ||
|| pairwise   || bool   || this chip_id will be pairwise diffed against pair_id ||
|| active     || bool   || this exp_id/chip_id is currently in use by this run ||
|| data_state || string || ||

The '''lapSequence''' table groups a set of lapRuns together into a complete sample of the sky.

|| '''name'''        || '''type'''   || '''note''' ||
|| seq_id      || integer ||             ||
|| description || string  ||             ||

= Processing Steps =

== Sequence definition ==

A `lapSequence` is defined by calling 
{{{
laptool -definesequence -name "unique name" -description "long description explaining what this sequence is"
}}}
`laptool -listsequence` prints the information about sequences in the metadata format. 

== Run definition ==

A new `lapRun` is defined by calling 
{{{
laptool -definerun -seq_id S -projection_cell PC_NAME -ra RA_CENTER -decl DEC_CENTER -radius R -filter F -label L
}}}
This adds a row to `lapRun`, and then creates a row in `lapExp` for each exposure that falls within the area on the sky defined by the -ra/-decl/-radius options.  If these exposures have been previously processed (and therefore have a row in `lapExp` associated with a different `lapRun`), the `chip_id` for the new lapExp is copied from that previous entry, ensuring that each exposure is processed only a single time in each lapSequence.  

== Chip-Warp processing ==

Defining the run does not queue chipRuns for those exposures that have not been processed previously.  This is done by 
{{{
lap_science.pl --chip_mode --lap_id LAP
}}}
This script calls `laptool -pendingexp -lap_id LAP` which returns a list of all the exposures included in this `lapRun`.  Each exposure is checked, and a `chipRun` queued if the `chip_id` entry is blank.  If the associated chipRun has been cleaned, the appropriate update calls are made to update the chip, destreak, and warp stages for this data.

This stage also defines the pairs for differencing, using the object and comment fields to organize the pairs as is done in nightly science.  If a pair is found, the associated `pair_id` fields point at the `chip_id` of the companion exposure, and the `pairwise` flag is set to true.  If a pair is not found, `pair_id` is left as NULL, `pairwise` is set to false, and `private` is set to true.  This ensures that the differencing will be postponed until all the final stacks that cover this exposure have been generated.  After queuing the chipRuns, `lapRun.state` is set to `run` to signal that processing has begun, but has not yet finished.

Once all the chips have been queued, the state of the `lapRun` will be polled using the 
{{{
lap_science.pl --monitor_mode --lap_id LAP
}}}
This calls `laptool -exposures -lap_id LAP` which returns the status of each exposure at the chip, camera, fake, warp, diff, and magic stages.  In addition to gauging completion, exposure pairs are checked against each other to confirm that no faults have occurred.  If one member of a pair has an unrecoverable fault, that entry is marked with `data_state` = 'drop', and is no longer used.  The companion of that unrecoverable exposure is set to be private so it will be diffed against the stack.  There are a number of informative data_state values, that have no impact on the processing, but serve to make monitoring the progress of a lapRun easier:

|| data_state || meaning ||
|| drop       || Do not use this exposure for any further processing in this lapRun ||
|| new        || Not all processing is finished for this exposure ||
|| full       || All required processing is done ||
|| exists     || A previous processing was done for this exposure, and an update will be triggered ||
|| to_diff    || A diff has not been triggered, but warp processing is complete ||
|| to_magic   || The warp stage has not been destreaked, but the diff is complete ||


The script will also check the status of any stacks via the `laptool -stacks -lap_id LAP` which returns the status of all associated stacks for this `lapRun`.  This information is used to set a pair of flags for both the quick and final stacks: the `defined_`flag notes that stacks have been queued, and the `have_` flag notes that the stacks have completed (or attempted and failed with a fault) and may be used.

Using the information about the exposures and stacks for this `lapRun`, `lap_science.pl` will decide what the next step of the processing will be.  After all exposures have a `warpRun.state` of full (or have been dropped due to quality), `lap_science.pl` checks which step is most appropriate:

=== Quick stacks ===

We no longer make quick stacks.

=== Diffs ===

If all `lapExp` have completed the warp stage (or been dropped along the way), diffs may be queued.  If no `lapExp` has `private = true`, then the pairwise diffs are performed.  If a `lapExp` is set to be private, a check is made to see if all the stacks that overlap this exposure exist.  If not, then the exposure is dropped and the diff postponed until a future lapRun generates the appropriate stacks for the diff.

=== Final stacks ===

Once the warps have completed, an initial pass of stacks are created that have at least a minimum of six inputs.  Once these have completed, the diff/magic/destreak stage is triggered, and then a second pass of stacking is performed to fill in any missing stacks with destreaked inputs.

=== Cleanup ===

Once the final stacks have completed, the `lapRun` is also complete.  If have_final is true, then `lapRun.state` is set to 'full', and all `lapExp.active` is set to false.  Periodically, `lap_science.pl --cleanup_mode` will poll all `lapExp`, and if all `lapExp` with a given `chip_id` have `active` = false, then the products of all stages generated from that `chip_id` will be marked for cleanup (`STAGE.label = 'goto_cleaned'`, `STAGE.state = 'goto_cleaned'`).

`lap_science.pl --cleanup_mode` also can be passed a queuefile, containing a list of `laptool -definerun` commands.  Each potential run is checked in the database, and the first run that has not been previously launched is executed.  This ensures that the number of lapRuns currently active is a constant.

== Proposed Changes ==

None.

= Command Reference =

Define a new sequence of lapRuns.
{{{
 laptool -definesequence -name "unique name" -description "long description explaining what this sequence is"
}}}

List the currently defined sequences.
{{{
 laptool -listsequence
}}}

Define a new lapRun for a given projection_cell/filter.  By default selects exposures that match an obs_mode of 3PI.  A future script will allow the definition of large areas, and execute the appropriate `-definerun` commands to queue all the lapRuns that make up that large area.
{{{
 laptool -definerun -seq_id SEQ_ID -projection_cell PROJECTION_CELL -tess_id TESS_ID -ra RA -decl DEC -radius R -filter FILTER -label label -dist_group dist_group
}}}

List runs that are defined, but have not been queued.
{{{
 laptool -pendingrun -state new
}}}

List runs that are processing.
{{{
 laptool -pendingrun -state run
}}}

List runs that have completed.
{{{
 laptool -pendingrun -state full
}}}

The -listrun option is an alias for -pendingrun, and can be used in its place to query for specific lapRuns:
{{{
 laptool -listrun -projection_cell skycell.ABCD -filter g.00000 -label TEST_RUNS
}}}


List the exposures that comprise this lapRun with details needed for queuing chipRuns.
{{{
 laptool -pendingexp -lap_id LAP_ID
}}}

List the exposures that comprise this lapRun with details about the state of their processing.
{{{
 laptool -exposures -lap_id LAP_ID
}}}

List the stacks that are created from this lapRun.
{{{
 laptool -stacks -lap_id LAP_ID
}}}

List the exposures that comprise this lapRun with details about whether they are still in use.
{{{
 laptool -inactiveexp
}}}

Queue chipRuns for a lapRun with state = new.
{{{
  lap_science.pl --chip_mode --lap_id LAP_ID
}}}

"Do the right thing" for a lapRun with state = run.
{{{
  lap_science.pl --monitor_mode --lap_id LAP_ID
}}}

See if any exposures can be cleaned for a lapRun with state = full.
{{{
  lap_science.pl --cleanup_mode --lap_id LAP_ID
}}}