Nightly Science: pantasks utilities

nightly.stacks.on	Turn on the entire automation code.
nightly.stacks.off	Turn off the entire automation code.
ns.add.date YYYY-MM-DD	Add a date to be processed.
ns.show.dates	List the dates currently in the book, along with their associated states.
ns.del.date	Remove a date from processing (dates that make it to stack are automatically removed).
ns.set.date YYYY-MM-DD STATE	Set the state for a given date.

State values:

State:	Criterion
NEW	Registration has not completed yet (or registration for this date has not been checked).
REGISTERED	Registration complete.
NEEDSBURNING	No burntool tables found for some portion of the data.
QUEUEBURNING	Burntool jobs are being sent to hosts for processing.
BURNING	The first pass of sending out jobs has completed (signals ns.chips.load to check for completion).
QUEUECHIPS	Burntool has finished, and chips can be queued.
TOWARP	Chips have been queued, and ns.stacks.load is waiting for all stackable warps to finish.
QUEUESTACKS	Stackable warps have finished, and stacks will be queued.
STACKING	stacktool commands have run to queue all stackable data.
DROP	No exposures were found for the date, or the date string was malformed.

automate_stacks.pl script

Useful modes for manually checking and queueing data:

automate_stacks.pl --date YYYY-MM-DD --check_registration

Confirm that registration is complete, and all needed exposures are available. If exposures are missing (due to an incomplete download or registration error), they are listed along with a note stating whether they are important to the automation or not.

automate_stacks.pl --date YYYY-MM-DD --check_chips

automate_stacks.pl --date YYYY-MM-DD --queue_chips

The first checks the chips to see if they're ready to be queued (confirms that burntool has finished for all exposures), and the second issues the appropriate chiptool commands.

automate_stacks.pl --date YYYY-MM-DD --check_stacks

automate_stacks.pl --date YYYY-MM-DD --queue_stacks

automate_stacks.pl --date YYYY-MM-DD --queue_stacks --force_stack_count

Similar to the chip mode options, check the number of warps against the input exposures to determine if all the warps for a stack are finished, and queue them if so. The --force_stack_count allows this counting to be skipped if an exposure is known to have a uncorrectable error.

Both chip and stack mode options support the --this_target_only and --this_filter_only which allow a specific target or filter to be queued without checking the rest. This is safe because previously queued runs will not be duplicated.

automate_stacks.pl config file

Each target in ippconfig/recipes/nightly_science.config has the following parameters:

NAME	STR	Target name. Used as the base of the data_group variable.
TESS	STR	Tessellation ID.
OBSMODE	STR	obs_mode value from rawExp. This is a LIKE search, so % wildcards are supported.
OBJECT	STR	object from rawExp. This was not populated, so although it is supported, none of the current target uses it.
COMMENT	STR	comment from rawExp. Also a LIKE search.
STACKABLE	BOOL	This flag determines whether or not nightly stacks can be made of this target.

The known filters are also stored in this file, although this is only used while queueing stacks.

The RETENTION_TIME value stores the number of days data should be stored before being marked for cleanup. This is commented out (it is part of ns.initday.load), but is fully implemented.