| | 7 | |
| | 8 | I investigated and solved the problem we were having in generating dark models which we believed. Earlier in the summer, when we attempted to update the dark model to better follow the temperature variations, and to use the "new" camera |
| | 9 | temperature keywords, we found that the residuals produced by the analysis were substantially worse than expected: lots of visible structures in the jpegs and average residual flux which was not close to zero. Disturbingly, the mean of the residuals was not roughly zero, which seemed to suggest something was going quite wrong with either the model or the application. I examined the various steps of the process, and discovered that the code was working as we expected and intended, but it was the reported residual statistics and jpegs which were in error. It turns out that the code which was measuring the statistics was also creating an internal image mask, but because of the way that mask was generated, negative pixel values were considered 'BAD' and masked. As a result, even though the residual images were quite clean and had ~zero mean flux, the statistics (and jpegs) measured for these images were always biased high. This particular effect only shows up in this case because other stages of the processing either have non-zero bias (raw images) or have an input mask from an earlier stage. Fixing this problem, I was able to generate darks (using inputs from nearly the full semester) which leave very nice looking residuals. |
| | 10 | |
| | 11 | I also fixed some code issues in the non-linearity correction code. There was a segfault due to ancient code triggered by turning on the choice to correct non-linearity. The code also ran much too slowly, so I re-optimized it to make a single scan of the correction table. The non-linearity correction now can be used, but some further tuning of the correction measurement is needed to generate a reliable correction for the cell sag. |
| | 12 | |
| | 13 | Finally, I have started to tackle the repair of the big ThreePi DVO database which was corrupted when the relevant disk filled up. This revealed placed in the dvomerge code which were not correctly catching and handing the data I/O failure. As for the database, I have created to tool to examine the existing images and measurements, and to identify images for which measurements were dropped. These images and the remaining measurements can then be removed from the database to make the tables consistent again. The dropped images can then be re-added to the database and dvomerge can continue as before. |
| 26 | | |
| 27 | | |
| 28 | | |
| 29 | | |
| 30 | | |
| 31 | | |
| 32 | | |
| 33 | | |
| 34 | | |
| 35 | | |
| 36 | | |
| 37 | | |
| 38 | | |
| 39 | | |
| 40 | | |
| 41 | | |
| 42 | | |
| 43 | | |
| | 33 | * PSPS |
| | 34 | * loading: |
| | 35 | * stopped 3PI, and loaded remainder of MD04 |
| | 36 | * troubles starting 3PI loading again with primary DVO Db: corrupted. Spent some time investigating this |
| | 37 | * loading is now stopped pending a new 3PI DVO Db |
| | 38 | * investigated current state of DVO MD04 Db with Heather. Verdict: needs more exposures |
| | 39 | * added extra filtering to ODM-checking script: can check those only on the datastore, '!MergeWorthy', 'Merged' etc |
| | 40 | * communications with TOPCAT developer about a plan for the PSPS Toolbox |
| | 41 | |
| | 42 | * IPP |
| | 43 | * Czar on Monday and Tuesday |
| | 44 | * the strange case of the missing OSS exposures |
| | 45 | * IPP Metrics in czartool: |
| | 46 | * new !StageMetrics class to encapsulate metrics for a stage for a given time/label |
| | 47 | * new method in Gpc1Db class to get a reliable count of summit exposures for a given date |
| | 48 | * new method in !CzarDb to get a full list of labels in use for a given time period |
| | 49 | * script to use above classes to generate HTML page of metrics for a given day, with plots |
| | 50 | * new stacked histogram showing rate of processing over time for each stage |
