Changes between Initial Version and Version 1 of 2016_Nightly_Science_Update

Timestamp:

Nov 17, 2016, 4:43:33 PM (10 years ago)

Author:

watersc1

Comment:

--

2016_Nightly_Science_Update

@@ +1 @@
+== 2016 Nightly Science Update ==
+
+=== How to run the script to generate the commands for missing diffs. ===
+
+This requires a build of the branches/czw_branch/20160809/ code, and is based on r39816.  At minimum, ippScripts and ippconfig are needed.
+
+{{{
+ nightly_science.pl --dbname gpc1 --camera GPC1 --verbose --debug --check_diffs --force_registration --date `date +%Y-%m-%d`
+}}}
+
+In addition to the regular options, the effects of the others are:
+ * {{{--debug}}} Ensures all commands have -pretends appended for safety.
+ * {{{--check_diffs}}} Does not execute the commands automatically, just prints them out.
+ * {{{--force_registration}}} Should not be needed in general, but in case morning darks were not taken, this tells the script to assume it is the end of the night.
+
+This command is very verbose, but the missing diffs will be towards the end, in the section with lines prefaced by "desp_diff_queue".
+
+=== Details from my email on the change. ===
+
+The goal of this change is to
+ensure that all diffs that are desired by MOPS are automatically
+generated, removing the need for additional manual work.  This should
+speed MOPS processing.
+
+I have attached the output of the current nightly_science.pl script in
+the "check_diffs" mode, along with the output from the proposed change
+to nightly_science.pl for yesterday (2016-11-14).  The check_diffs
+mode runs the same logic as is used for queuing diffs, but does not
+issue queuing commands, and as such represents a way to demonstrate
+what the effect of this proposed change will be.
+
+The output starts with a summary of the active configuration.  A diff
+illustrates that the current IPP production configuration does not
+match the trunk exactly, with default retention times for warps being
+long, and with differences to the target definitions that will likely
+need to be resolved on the IPP side.
+
+Following the configuration is a set of checks of the database to
+attempt to determine if observing has reached the end of night, by
+looking for the standard dark frame observations in the database.
+This is used to know if it is safe to generate stacks, and also to
+queue the new diffs automatically.
+
+Next is a scan over the observing targets and filters.  A note is
+generated for each possible diff that should be possible, in this case
+noting that the diffs are complete.  The format for this includes the
+date, target, input exp_id, template exp_id, input warp_id, template
+warp_id, the observation object, and the input comment string.  The
+proposed change additionally adds the template comment string, which
+was left out of the original version.
+
+This scan appears to me to show that all diffs generated in the
+current code are generated by the proposed code as well.  Yesterday
+was a good example to run, as object ps1_32_1505 had a bad camera
+stage quality for the third visit.  This is handled in both versions
+of the code, but the proposed change is more explicit about the
+rejection of this exposure (exp_id = 1165338) and the exposure
+rejected to ensure that image/template pairs are fully populated
+(exp_id = 1165358).
+
+Following the standard diff scan, based on the information that the
+end of night has been reached and no new exposures are coming, the
+original code continues to a multi-date diff operation that tries to
+find unpaired exposures that exist on disk and construct differences
+from those.  My understanding is that these products are not desired,
+and as such this has been disabled in the proposed change.
+
+Replacing this is an attempt to construct "desperate diffs" (named
+that way simply because I couldn't think of a better word).  This runs
+the pairing operation again, but with the exposures sorted in a
+reverse order.  Diffs that have already been constructed are
+identified as already queued, but this sorting order allows for the
+rejection rules to generate the next-best diffs for cases where an
+exposure has been lost.  For the case of ps1_32_1505, visit three is
+again rejected for camera quality issues, but this time visit one is
+rejected, allowing for the visit 2 - visit 4 diff to be constructed
+(exp_ids 1165320, 1165358).  A technical note for IPP, these diffs
+have the -rerun option applied, which will allow one exposure to be in
+two diffs with the same labels.
+
+Finally, after all this, an IPP metadata block is written out that is
+used by the pantasks processing to set state information, such that
+all possible diffs have been constructed, and that no further checks
+need to be run.
+
+