Changes between Version 15 and Version 16 of LAP_Science
- Timestamp:
- Oct 11, 2011, 3:36:26 PM (15 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
LAP_Science
v15 v16 9 9 The '''lapRun''' table defines and groups the reprocessing of a single projection cell 10 10 11 || '''name''' 11 || '''name''' || '''type''' || '''note''' || 12 12 || lap_id || int || || 13 13 || seq_id || int || || … … 19 19 || fault || int || || 20 20 || label || string || || 21 || quick_sass_id || int || stackAssociation id of the quickstack for this run 21 || quick_sass_id || int || stackAssociation id of the quickstack for this run. No longer needed or used.|| 22 22 || final_sass_id || int || stackAssociation id of the final stack for this run || 23 23 || dist_group || string || distribution group to give to all dependents || 24 24 25 The '''lapExp''' table defines the pool of exposures, and associates the mwith the lapRuns that use them25 The '''lapExp''' table defines the pool of exposures, and associates these with the lapRuns that use them 26 26 27 27 || '''name''' || '''type''' || '''note''' || … … 30 30 || chip_id || int || chip_id of the processing of exp_id used in this run || 31 31 || pair_id || int || chip_id of the companion exposure for a pairwise diff || 32 || private || bool || this exp_id/chip_id set is only available to this run||32 || private || bool || this exp_id/chip_id set is split across projection cells || 33 33 || pairwise || bool || this chip_id will be pairwise diffed against pair_id || 34 34 || active || bool || this exp_id/chip_id is currently in use by this run || … … 57 57 laptool -definerun -seq_id S -projection_cell PC_NAME -ra RA_CENTER -decl DEC_CENTER -radius R -filter F -label L 58 58 }}} 59 This adds a row to `lapRun`, and then creates a row in `lapExp` for each exposure that falls within the area on the sky defined by the -ra/-decl/-radius options. If these exposures have been previously processed (and therefore have a row in `lapExp` associated with a different `lapRun`), the `chip_id` for the new lapExp is copied from that previous entry, provided that the `private` flag is not true.59 This adds a row to `lapRun`, and then creates a row in `lapExp` for each exposure that falls within the area on the sky defined by the -ra/-decl/-radius options. If these exposures have been previously processed (and therefore have a row in `lapExp` associated with a different `lapRun`), the `chip_id` for the new lapExp is copied from that previous entry, ensuring that each exposure is processed only a single time in each lapSequence. 60 60 61 61 == Chip-Warp processing == … … 65 65 lap_science.pl --chip_mode --lap_id LAP 66 66 }}} 67 This script calls `laptool -pendingexp -lap_id LAP` which returns a list of all the exposures included in this `lapRun`. Each exposure is checked, and a `chipRun` queued if the `chip_id` entry is blank. 67 This script calls `laptool -pendingexp -lap_id LAP` which returns a list of all the exposures included in this `lapRun`. Each exposure is checked, and a `chipRun` queued if the `chip_id` entry is blank. If the associated chipRun has been cleaned, the appropriate update calls are made to update the chip, destreak, and warp stages for this data. 68 68 69 This stage also defines the pairs for differencing, using the object and comment fields to organize the pairs as is done in nightly science. If a pair is found, the associated `pair_id` fields point at the `chip_id` of the companion exposure, and the `pairwise` flag is set to true. If a pair is not found, `pair_id` is left as NULL, `pairwise` is set to false, and `private` is set to true. This ensures that the differenc e against the quickstack will not be used by other `lapRun`s. After queuing the chipRuns, `lapRun.state` is set to `run` to signal that processing has begun, but has not yet finished.69 This stage also defines the pairs for differencing, using the object and comment fields to organize the pairs as is done in nightly science. If a pair is found, the associated `pair_id` fields point at the `chip_id` of the companion exposure, and the `pairwise` flag is set to true. If a pair is not found, `pair_id` is left as NULL, `pairwise` is set to false, and `private` is set to true. This ensures that the differencing will be postponed until all the final stacks that cover this exposure have been generated. After queuing the chipRuns, `lapRun.state` is set to `run` to signal that processing has begun, but has not yet finished. 70 70 71 71 Once all the chips have been queued, the state of the `lapRun` will be polled using the … … 73 73 lap_science.pl --monitor_mode --lap_id LAP 74 74 }}} 75 This calls `laptool -exposures -lap_id LAP` which returns the status of each exposure at the chip, camera, fake, warp, diff, and magic stages. In addition to gauging completion, exposure pairs are checked against each other to confirm that no faults have occurred. If one member of a pair has an unrecoverable fault, that entry is marked with `data_state` = 'drop', and is no longer used. The companion of that unrecoverable exposure is set to be private so it will be diffed against the quickstack.75 This calls `laptool -exposures -lap_id LAP` which returns the status of each exposure at the chip, camera, fake, warp, diff, and magic stages. In addition to gauging completion, exposure pairs are checked against each other to confirm that no faults have occurred. If one member of a pair has an unrecoverable fault, that entry is marked with `data_state` = 'drop', and is no longer used. The companion of that unrecoverable exposure is set to be private so it will be diffed against the stack. There are a number of informative data_state values, that have no impact on the processing, but serve to make monitoring the progress of a lapRun easier: 76 76 77 Setting the `lapExp` to private "steals" this processing run away from any other `lapRun` that may want to use it. If `lap_science.pl` detects that another `lapExp` with this `exp_id`/`chip_id` set has `private` = true, then it will requeue a `chipRun` for that `exp_id`, and record the new `chip_id` for itself, and set `private` = true. 77 || data_state || meaning || 78 || drop || Do not use this exposure for any further processing in this lapRun || 79 || new || Not all processing is finished for this exposure || 80 || full || All required processing is done || 81 || exists || A previous processing was done for this exposure, and an update will be triggered || 82 || to_diff || A diff has not been triggered, but warp processing is complete || 83 || to_magic || The warp stage has not been destreaked, but the diff is complete || 84 78 85 79 86 The script will also check the status of any stacks via the `laptool -stacks -lap_id LAP` which returns the status of all associated stacks for this `lapRun`. This information is used to set a pair of flags for both the quick and final stacks: the `defined_`flag notes that stacks have been queued, and the `have_` flag notes that the stacks have completed (or attempted and failed with a fault) and may be used. … … 83 90 === Quick stacks === 84 91 85 If any `lapExp` has `private` = true, then a quickstack is needed. If `defined_quick` is false, the quickstack is queued, and the appropriate `stackAssociation.sass_id` is recorded as `lapRun.quick_sass_id`. 92 We no longer make quick stacks. 86 93 87 94 === Diffs === 88 95 89 If all `lapExp` have completed the warp stage (or been dropped along the way), diffs may be queued. If no `lapExp` has `private = true`, then the pairwise diffs are performed. If a `lapExp` is set to be private, a nd the `have_quick = true` (marking that the quickstack has finished), then the private `lapExp`s are diffed against the quickstack, and the others diffed as a pair.96 If all `lapExp` have completed the warp stage (or been dropped along the way), diffs may be queued. If no `lapExp` has `private = true`, then the pairwise diffs are performed. If a `lapExp` is set to be private, a check is made to see if all the stacks that overlap this exposure exist. If not, then the exposure is dropped and the diff postponed until a future lapRun generates the appropriate stacks for the diff. 90 97 91 98 === Final stacks === 92 99 93 Once the diffs have completed, the regular stdscience/distribution tools will be used to process all data through the magic and destreak steps. Once these have completed, the `warpRun.magicked` value is used to determine if a `lapExp` is ready for the final stacking stage. If all `lapExp`s have a non-zero `magicked` value, or have a data_state = 'drop', and `defined_final` is false, then the final stack is queued, and the appropriate `stackAssociation.sass_id` is recorded as `lapRun.final_sass_id`.100 Once the warps have completed, an initial pass of stacks are created that have at least a minimum of six inputs. Once these have completed, the diff/magic/destreak stage is triggered, and then a second pass of stacking is performed to fill in any missing stacks with destreaked inputs. 94 101 95 102 === Cleanup === … … 97 104 Once the final stacks have completed, the `lapRun` is also complete. If have_final is true, then `lapRun.state` is set to 'full', and all `lapExp.active` is set to false. Periodically, `lap_science.pl --cleanup_mode` will poll all `lapExp`, and if all `lapExp` with a given `chip_id` have `active` = false, then the products of all stages generated from that `chip_id` will be marked for cleanup (`STAGE.label = 'goto_cleaned'`, `STAGE.state = 'goto_cleaned'`). 98 105 106 `lap_science.pl --cleanup_mode` also can be passed a queuefile, containing a list of `laptool -definerun` commands. Each potential run is checked in the database, and the first run that has not been previously launched is executed. This ensures that the number of lapRuns currently active is a constant. 107 99 108 == Proposed Changes == 100 109 101 === Update === 102 103 Even after cleanup, `lapExp` still records whether an exposure has been fully processed previously. Therefore, instead of queuing a new `chipRun` for this exposure, the data products may be updated instead. This will be faster than starting from scratch. Implementing this is not significantly difficult, as it requires only relaxing the state restriction in `laptool -definerun` and having `lap_science.pl` know how to update a `lapExp` with `STAGE.state = 'cleaned'`. 104 105 === Complete Private diffs === 106 107 If an unpaired `lapExp` spans multiple `projection_cell`s, then if we make the requirement that the quickstacks associated with all of the `projection_cell`s are complete before diffing, then we do not need to make that `lapExp` private. The difference can then be done across the entire image, instead of only over the overlapping area. This will reduce the amount of processing required, but will likely increase the mask fraction for that `lapExp`. 110 None. 108 111 109 112 = Command Reference =
