| | 10 | I spent a chunk of last week trying to understand our storage usage. We have been very tight on storage over the past month despite moving lots of raw data to the ATRC machines. Looking back on the past 6 months, it seemed that the rate of storage usage was higher than we expected. Looking at the output products from specific exposures in the full and cleaned states (with and without the pixel products), I realized that we were leaving more bytes behind in the 'cleaned' stages than expected. The current total per processed and cleaned exposure is between 30-50% of the raw data size. There are a couple of products which can be removed on cleanup : the 'pattern' data is generated on update and can be removed. We also leave behind the uncensored camera mask because we do not have a camera 'cleanup' stage. Finally, the output jpegs for the diff stage are very large : they are apparently binned 4x4 instead of 16x16 like the other stages. Addressing these issues can bring the size of a cleaned processed exposure to ~15-20% of a raw exposure. We may also be able to make a more compact storage of the censored and uncensored camera stage smf files, but that will take a bit more work. The other reason storage has been very tight is that the LAP processing has been queuing too many exposures without pushing them all to completion and cleanup. These fixes, and cleaning up the past LAP data got us back into a comfortable place with 10% of the cluster free at MHPCC. |
