Nightly Science: pantasks utilities

nightly.stacks.on	Turn on the entire automation code.
nightly.stacks.off	Turn off the entire automation code.
ns.add.date YYYY-MM-DD	Add a date to be processed.
ns.show.dates	List the dates currently in the book, along with their associated states.
ns.del.date	Remove a date from processing (dates that make it to stack are automatically removed).
ns.set.date YYYY-MM-DD STATE	Set the state for a given date.

State values:

State:	Criterion
NEW	Registration has not completed yet (or registration for this date has not been checked).
REGISTERED	Registration complete.
NEEDSBURNING	No burntool tables found for some portion of the data.
QUEUEBURNING	Burntool jobs are being sent to hosts for processing.
BURNING	The first pass of sending out jobs has completed (signals ns.chips.load to check for completion).
QUEUECHIPS	Burntool has finished, and chips can be queued.
TOWARP	Chips have been queued, and ns.stacks.load is waiting for all stackable warps to finish.
QUEUESTACKS	Stackable warps have finished, and stacks will be queued.
STACKING	stacktool commands have run to queue all stackable data.
DROP	No exposures were found for the date, or the date string was malformed.

The following states are internal, and should not appear in standard use:

State:	Criterion
CHECKDQSTATS	Check to see if a dqstats bundle can be created (unused in pantasks module).
QUEUEDQSTATS	Queue a dqstats bundle (unused in pantasks module).
CHECKDETRENDS	Check to see if a detrend verify run can be created (unused in pantasks module).
QUEUEDETRENDS	Queue detrend verify runs (unused in pantasks module).
FORCETOWARP	Internal state to ensure stacks are not prematurely queued.
STACKING_POSSIBLE_ERROR	Internal state to note when new stacks queues have been attempted when such stacks already exist.
ERROR	Burntool definition failed.
DETREND_QUEUED	Successfully queued detrend verify runs (unused in pantasks module).
DETREND_DROP	No detrend exposures were found for the data (unused in pantasks module).

automate_stacks.pl script

Useful modes for manually checking and queueing data:

automate_stacks.pl --date YYYY-MM-DD --check_registration

Confirm that registration is complete, and all needed exposures are available. If exposures are missing (due to an incomplete download or registration error), they are listed along with a note stating whether they are important to the automation or not.

automate_stacks.pl --date YYYY-MM-DD --check_chips

automate_stacks.pl --date YYYY-MM-DD --queue_chips

The first checks the chips to see if they're ready to be queued (confirms that burntool has finished for all exposures), and the second issues the appropriate chiptool commands.

automate_stacks.pl --date YYYY-MM-DD --check_stacks

automate_stacks.pl --date YYYY-MM-DD --queue_stacks

automate_stacks.pl --date YYYY-MM-DD --queue_stacks --force_stack_count

Similar to the chip mode options, check the number of warps against the input exposures to determine if all the warps for a stack are finished, and queue them if so. The --force_stack_count allows this counting to be skipped if an exposure is known to have a uncorrectable error.

Both chip and stack mode options support the --this_target_only and --this_filter_only which allow a specific target or filter to be queued without checking the rest. This is safe because previously queued runs will not be duplicated.

Other modes:

automate_stacks.pl --date YYYY-MM-DD --check_detrends

automate_stacks.pl --date YYYY-MM-DD --queue_detrends

Check and queue detrend verify runs based on the types of detrends defined in the config file.

automate_stacks.pl --date YYYY-MM-DD --check_dqstats

automate_stacks.pl --date YYYY-MM-DD --queue_dqstats

Check and queue dqstats runs to bundle image statistics for the IPP-OTIS interface.

automate_stacks.pl --date YYYY-MM-DD --clean_old --debug

automate_stacks.pl --date YYYY-MM-DD --clean_old

Check and queue old data cleanup based on the rules defined in the configuration file.

automate_stacks.pl config file

The configuration file defines the parameters for four different components:

FILTERS

This is a simple MULTI metadata object containing the strings of all filters defined.

CLEAN_MODES

These define control how data from different stages is cleaned. The parameters are:

MODE	STR	Mode name being cleaned. Used to index the methods, and to allow overrides (see below).
COMMAND	STR	Command used to queue the data for cleanup.
RETENTION_TIME	S16	Number of days to retain data before cleanup. Values less than one day disable cleaning for that target.
ALTERNATE_CMD	STR	Used to mark the format of the options for the cleanup command.
	A	-dbname $dbname -updaterun -set_state goto_cleaned -full -set_label goto_cleaned -time_stamp_end $cleaning_date
	B	-dbname $dbname -updaterun -set_state goto_cleaned -state full -set_label goto_cleaned -label $label
	NULL	-dbname $dbname -updaterun -set_state goto_cleaned -state full -set_label goto_cleaned -label $label -data_group $data_group

TARGETS

Each target in ippconfig/recipes/nightly_science.config has the following parameters:

NAME	STR	Target name. Used as the base of the data_group variable.
TESS	STR	Tessellation ID.
OBSMODE	STR	obs_mode value from rawExp. This is a LIKE search, so % wildcards are supported.
OBJECT	STR	object from rawExp. This was not populated, so although it is supported, none of the current target uses it.
COMMENT	STR	comment from rawExp. Also a LIKE search.
STACKABLE	BOOL	This flag determines whether or not nightly stacks can be made of this target.
CLEAN_MODE	S16	Using the clean mode names defined in the CLEAN_MODE section, an alternate retention time can be specified for this target/mode combination.

DETRENDS

Nightly detrend verification runs are controlled by the following parameters:

NAME	STR	Detrend name.
DETTYPE	STR	Value of the det_type column in the rawExp table to select on.
EXPTYPE	STR	Value of the exp_type column in the rawExp table to select on.
REF_ID	S64	det_id of the reference detrend to verify against.
REF_ITER	S32	iteration of the reference detrend to verify against.
FILTER	STR	Optional filter to select on.
MAX	S32	Optional maximum number of detrend frames to use in this verify run.