Changes between Version 15 and Version 16 of IPP_for_PS2
- Timestamp:
- May 11, 2018, 3:09:21 PM (8 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
IPP_for_PS2
v15 v16 36 36 * ippconfig/gpc2/psastro.config be extremely permissive and allow the max FWHM to be 100 pixels. Use 2017 refcat. 37 37 * ippconfig/gpc2/ppImage.config add PPIMAGE_BKGCONT recipe with BACKGROUND.CONTINUITY disabled. 38 * ippconfig/gpc2/psphot.config add SSTF_WS_DIFF recipe (currently identical to SSTF_DIFF recipe). 39 * ippTasks/survey.pro set default warp-stack diff reduction to SWEETSPOT_WS, which calls the SSTF_WS_DIFF recipe. 40 * ippconfig/gpc2/format_mef.config remove INSTRUME rule, and add DETECTOR rule. It is unclear why this worked prior to 2018-05-09. 41 * ippconfig/system.confg remove REMOTE recipe file location. Again, unclear why this became an issue around 2018-05-09. 42 38 43 39 44 === Known issues === 40 45 41 * The filter is not being interpreted from the header correctly. The configuration lists the FILTERID keyword being used for both FPA.FILTERID and FPA.FILTER concepts. The header lists FILTERID='i.00002 ' AND FILTER='i ', but the tgpc2 database lists the filter as 'i'. The header keywords and configuration both match what is being done with GPC1, so this mis-interpretation is not understood.42 * A closer look shows that GPC1 uses FILTERID for both concepts. ppStats uses the FPA.FILTER concept to populate the filter, so this is the source of the problem. I'm hesitant to fix this right away, as it will require changing all the entries in the database already. It might be best to wait until things are more stable and we're in a true commissioning phase.46 * ~~The filter is not being interpreted from the header correctly. The configuration lists the FILTERID keyword being used for both FPA.FILTERID and FPA.FILTER concepts. The header lists FILTERID='i.00002 ' AND FILTER='i ', but the tgpc2 database lists the filter as 'i'. The header keywords and configuration both match what is being done with GPC1, so this mis-interpretation is not understood.~~ Resolved as of 2018-05-01. 47 * ~~A closer look shows that GPC1 uses FILTERID for both concepts. ppStats uses the FPA.FILTER concept to populate the filter, so this is the source of the problem. I'm hesitant to fix this right away, as it will require changing all the entries in the database already. It might be best to wait until things are more stable and we're in a true commissioning phase.~~ Resolved as of 2018-05-01. 43 48 * The nightly_science desperate diff routine depends on the existence of a morning dark to signal that all observations have been taken. These are not currently being taken, which prevented one diff from being constructed automatically. I launched the diff with the command 44 49 {{{difftool -definewarpwarp -dbname tgpc2 -set_workdir neb://@HOST@.0/tgpc2/OSS.nt/2018/03/30 -set_label czw.up -set_data_group OSS.20180330 -set_reduction SWEETSPOT -set_dist_group SweetSpot -exp_id 5729 -template_exp_id 5737 -input_label czw.up -template_label czw.up -good_frac 0.1 -backwards -trace db 5 -rerun}}}. 45 50 * These darks are now being taken, but as of 2018-04-17, the obs_mode does not match the expected value. If this continues, I will remove this restriction from the configuration file (a comment of 'darks_morn%' is probably sufficiently restrictive). 46 51 * There is no cleanup pantasks, so nightly science exposures go to label="goto_cleaned" and state="goto_cleaned". For the one night of science data, I moved them back using {{{STAGEtool -dbname tgpc2 -updaterun -label goto_cleaned -set_label czw.up -set_state full}}}. 47 * The current MASK is a DARKMASK+FLATMASK that does not exclude static bad pixels/columns that are not caught by those processes. 52 * The current MASK is a DARKMASK+FLATMASK that does not exclude static bad pixels/columns that are not caught by those processes. The mask has been updated (2018-05-10) to mask full cells that are bad or poorly illuminated. 48 53 * Occasionally the OBJECT field is not cleared before engineering data is taken. This has not resulted in non-science exposures being processed, but it is a possibility. 49 54 * The i-band GPC1 stacks with label="LAP.PV3.20140730.final" are being ingested into the tgpc2 database to allow for warp (GPC2) - stack (GPC1) difference image processing. The stackRun.filter column will need to be modified, as difftool blocks diffs that do not have the same filter. The ingest process uses the stack_id value from the gpc1 database, so the tgpc2.stackRun and tgpc2.stackSumSkyfile tables will have a much higher auto-increment value. The stackInputSkyfile entries are lost, as they reference warpSkyfile entries that do not exist. 50 * Warp (GPC2) - stack (GPC1) difference images do not currently work, because there is a database JOIN from the stackInputSkyfile table to the warpRun table, I assume to prevent a stack from being subtracted from an input warp. However, as the stack entries are fakes from the GPC1 database, this JOIN fails, and no stacks are found when the diff is attempted. This is not a queuing issue, as the diffInputSkyfile entries were constructed with appropriate warp1/stack2 values. It is only when processing the diff does the issue come up. I think the solution is to create a sacrificial warp that will be the "input" to all ingested stacks. 55 * Warp (GPC2) - stack (GPC1) difference images do not currently work, because there is a database JOIN from the stackInputSkyfile table to the warpRun table, I assume to prevent a stack from being subtracted from an input warp. However, as the stack entries are fakes from the GPC1 database, this JOIN fails, and no stacks are found when the diff is attempted. This is not a queuing issue, as the diffInputSkyfile entries were constructed with appropriate warp1/stack2 values. It is only when processing the diff does the issue come up. I think the solution is to create a sacrificial warp that will be the "input" to all ingested stacks. This is only the case for w-band stacks, which are ingested, but do not have a processed warp to allow the linkage to complete. 51 56 * I have added stackInputSkyfile entries for all i-band stacks that point at warp_id 60. For all of them. This lie should allow WS diffs to complete, as the camera will match. 52 57 * Due to issues with ppSub and the way files are read, I have had to make a very hacky edit to ppSubCamera.c to allow these diffs to be generated. After loading the warp images (from GPC2), ppSub throws an error when it attempts to read the stack (from GPC1) because the camera in the pmConfig object does not match the camera determined from the stack header. This appears to be a fundamental limitation of the pmFPAfileDefine codebase, and I could not find an easy solution. Instead, upon failing to read the stack, I have the code (near ppSubCamera.c:215) substitute the string GPC1 for GPC2 in the config->cameraName string. This allows a second attempt to read the stack (and all subsequent stack products) to succeed, as the header matches the config expectation. Once all stack products have been loaded, the config->cameraName string is set back to the original GPC2 value, and the difference processing continues as normal. This generates images that appear reasonable, but this edit is not a good long-term solution. The substitution is hardcoded, and ignores the fundamental issue that skycell products should be treated equally by the code, regardless of camera. 58 59 === Publish Client Documentation === 60 61 The warp-warp diff data was originally published to the IPP-MOPS-PS2-TEST product using output format 3. The warp-stack diff data was originally published to the IPP-MOPS-PS2-WS-TEST product using output format 2. At the request of MOPS, I have created new products IPP-MOPS-PS2-WW and IPP-MOPS-PS2-WS: 62 63 {{{ 64 pubtool -dbname gpc2 -defineclient -stage diff -product IPP-MOPS-PS2-WW -workdir neb://any/gpc1/IPP-MOPS-PS2-WW -output_format 3 -unmagicked 65 pubtool -dbname gpc2 -defineclient -stage diff -product IPP-MOPS-PS2-WS -workdir neb://any/gpc1/IPP-MOPS-PS2-WS -output_format 2 -unmagicked 66 pubtool -dbname gpc2 -updateclient -client_id 3 -active 67 pubtool -dbname gpc2 -updateclient -client_id 4 -active 68 }}} 69 70 There currently does not appear to be a way to define a client that is active by default.
