Nightly Science v2

Steps

1. At UTC midnight, the nightly science tasks in the stdscience pantasks initialize books for a new day, and clean up old data according to the rules in nightly_science.config.
2. An exposure it observed at the summit.
3. The summitcopy pantasks identifies these exposures and populates the summitExp/summitImfile tables with the list of exposures available at the summit. It then downloads the fits files to MHPCC, and populates the newExp/newImfile tables.
4. The registration pantasks examines the downloaded fits images, and (using register_imfile.pl) registers an entry in the rawImfile table for each imfile (setting data_state to pending_burntool and burntool_state to 0). It then examines the series of previous exposures, and determines if this newly registered imfile can be run through burntool. This requires all previous imfiles known at the summit with the same class_id as the one under examination to have been registered and burntooled. If burntool cannot be run at this time (most likely due to a previous imfile not being downloaded yet), register_imfile.pl sets the data_state to pending_burntool and exits. If burntool can be run, it runs ipp_apply_burntool_single.pl and continues as below.
5. The registration pantasks periodically runs regtool -pendingburntoolimfile -date YYYY-MM-DD -valid_burntool 14 to examine the imfiles with data_state = pending_burntool and decide if they can now be burntooled. If so, the appropriate ipp_apply_burntool_single.pl command is constructed and run.
6. ipp_apply_burntool_single.pl takes the uris for the imfile under consideration and the previous uri, runs burntool and if successful, sets data_state = full and burntool_state = -14.
7. regtool -pendingexp refuses to complete the registration of the full exposure and populate rawExp until all component imfiles have data_state = full. This blocks the exposure level registration until all imfiles have been burntooled. register_exp.pl constructs the regtool command to populate rawExp, and decides if this exposure is a science exposure, based on the rules defined in the nightly_science.config file. If it is not a science exposure, then the end_stage is set to reg, and nightly science is finished with it. Science exposures have the appropriate values defined for the label, data_group, dist_group, end_stage, tess_id, workdir, and reduction, and after populating rawExp, an entry is written in chipRun queuing science processing on that exposure.
8. While exposures are taken, the nightly science tasks periodically run to examine whether stacks and diffs can be created.
9. For each stackable survey target defined in nightly_science.config, the number of chip and warp stage exposures is checked. For a stack to be queued it must satisfy the following criteria:
   1. No stack already exists with that set of data_group,skycell_id,filter,tess_id.
   2. The number of chip stage exposures equals the number of warp stage exposures with state = 'full' or with a quality value listed as acceptable in the configuration.
   3. If we have not yet reached end of night, then there must be at least the minimum number of warp exposures listed in the configuration. This prevents premature queuing of stacks in the event of a delay in summitcopy.
10. For each diffable survey target defined in nightly_science.config, all completed warps are grouped by object. For each object-group, the comment strings are compared, and any duplicates are resolved by taking the exposure with the latest dateobs value. Exposures are then grouped into pairs by dateobs and a diff is queued for each pair provided that no diff already exists with that pair of exposures in either direction.
11. This proceeds until any criterion for end of night is reached, as described below. Once nightly science reaches the end of night, it starts checking whether stacks and diffs have finished. It does this by comparing the number of potential runs to be queued and the number of runs that have been queued. When these are equal, it sets the states to the appropriate FINISHED_ value.

Configuration

The configuration is stored in ippconfig/recipes/nightly_science.config and has a standard IPP metadata format.

CLEAN_MODES

These entries list how the cleanup of old data is handled. Each CLEAN_MODE rule has three parts, a MODE which describes the name of the rule, COMMAND which lists the command to run to clean up data under this rule, and RETENTION_TIME which defines how long data is kept before this rule is invoked. The COMMAND entries allow for string replacement. The current values that can be replaced are:

• @DBNAME@
• @LABEL@
• @WORKDIR@
• @OBSMODE@
• @OBJECT@
• @TESS_ID@
• @DIST_GROUP@
• @DATA_GROUP@
• @REDUCTION@
• @CURRENT_DATE@

The values of RETENTION_TIME can be overruled for a given survey target if the MODE name is listed in a TARGET entry described below.

END_OF_NIGHT

These entries describe a set of rules used to match exposures to determine if observing has ended. Even if these rules are not met, end of night will always be true if the local time (HST) is later than 6:30AM. The rules must have a NAME, and can contain any of OBJECT,OBSMODE,EXPTYPE,COMMENT to define the rule.

FILTERS

This is simple a list of the filters to examine.

UNRECOVERABLE_QUALITY

This lists quality values that are acceptable for a warp stage exposure to have with respect to stack queuing. If a warp has one of these values, then it is counted as full, even if not.

MACRO_DEFINITIONS

These define a macro to be executed by pantasks upon successful completion of a task. The PROC_MODE describes the stage of nightly_science.pl the macro defined in MACRO should be executed. This is not well tested and is not used at the moment.

TARGETS

These rules define the properties of exposures for each survey target. Each target must have a NAME. Exposures that comprise this target are defined by one or more of the the OBSMODE,OBJECT,COMMENT fields, which accept SQL-like % wildcards. The DISTRIBUTION,TESS,REDUCTION fields allow the dist_group,tess_id,reduction to be set for the science processing. The two booleans STACKABLE,DIFFABLE define whether this target should be examined for stacks and diffs. The MIN_STACK sets the minimum number of warps needed to make a stack while observing is still happening. Finally, any of the CLEAN_MODES listed above may be set here to change the retention time for this data.

DETRENDS

These rules define how the detrend verify runs are queued. Each has a NAME and use DETTYPE,EXPTYPE to determine the type of verify run to generate. The reference detrend is set using the REF_ID,REF_ITER values. If a FILTER value is set, then only exposures in that filter are used for the verification. The number of exposures to use in the verification process is set by MAX_EXP.

Faults

Manual Operation Modes