| | 8 | I spent much of the week preparing for the McLaren review committee, and recovering. We spent some time this week addressing the storage issues - We have started to compress and delete a chunk of about 13k exposures (some darks, biases, and donuts from before April 2009). Towards the end of the week, I spent some time finishing up the outstanding CR masking and size classification issues in psphot. Over the weekend, we had some trouble with the DVO ingest of 3pi data. This was caused partly by a repeating failure of a single image and exacerbated by DVO lack of support for 64bit file sizes. I started to address the DVO large file size issue, which will probably take until Wednesday. Bill and I are also working out some modifications to the way DVO ingests work to avoid these issues: we will test the ingest of a night's worth of data into a local dvo database, followed by a bulk merge of the dvo databases. There is already a program for doing this bulk merge, though it needs to be tested more carefully. This modification of the ingest process may resolve the long-standing (unaddressed) concern of how to ship DVO databases to the consortium: we may be able to ship the nightly updates and have these merged by the consortium members. Some process control would be needed to guarantee the object IDs are identical (the order matters), but this would be much easier than shipping the full DVO databases. |
