| | 8 | We are starting to re-run the Run 3 & Run 4 data with recent software and detrend improvements. We had been aiming to restart as soon as possible, but with the news last weekend that the telescope will be delayed coming back on line, we decided to limit the scope of re-processing this past week while a few more issues were addressed. Specifically, we wanted to include Paul's recent work on the 'detection' image in the output from the stack and the 'dual-convolution' method for the difference images. We would also like to include improvements to the calculation of the zero-point discussed since Harvard but not yet implemented (single measurement for the exposure; detection of the 'blue edge' rather than just a mean). Finally, we were having some problems with the y-band fringe analysis (the fringe fitting had an error which Chris fixed this past week). So, instead of running large amounts of data, we concentrated on MD08 as a demonstration set. |
| | 9 | |
| | 10 | Unfortunately, as this analysis was begun, one of our machines (ipp014) crashed with a motherboard problem. This machine has long been flaky, so it is good to have a chance to fix whatever ails the motherboard (we had been suspecting memory problems). However, having this machine down highlighted some sensitivity to single point failures: the burntool results files were not replicated, and some of the detrend images were not replicated. With ipp014 unavailable, the processing was unable to go past the chip stage analysis. We've taken the time to turn on replication for the burntool result files and we've replicated the detrend images. Although it would have been possible to re-create the data on ipp014, it would have taken a fair amount of effort. Instead, we waited until Steph and Gavin could get the machine back up and running on Friday (in fact, they swapped the disks and raid controller to ipp037, which has been offline, and will work on the ipp014 motherboard at their leisure). |
| | 11 | |
| | 12 | I spent most of the week doing my duty on the TAC, so I got almost no development work done. I did manage to update pcontrol to shutdown and restart pclients which have been running for more than 10hours (this should reduce the runaway and large memory pclients which build up over time). Will, Nick and I also had a very helpful telecon with Paul Sydney, Paul Kervin and Mark Bolden in which we discussed the prognosis for magic, especially the outlook for automatic acceptance of magic results. Paul, Paul, & Mark will provide further guidance on this point this coming week. |
| | 13 | |
| | 15 | |
| | 16 | * fed more md5sums into the gpc1 database. 236540 are still NULL. |
| | 17 | * continued checking md5sums on raw images, and fixing pairs of raw images with mismatched md5sums. The count is now up to 460 raw images fixed. |
| | 18 | * started processing MD08 data. lots of problems: 0 byte config files (discovered by bill, fixed by paul), and ipp014, which is down, has the burntool tables for various chips. |
| | 19 | * told nebulous we want 2 copies of the burntool tables. (that's still in progress) |
| | 20 | * destreaked MD01 and MD02 |
| | 21 | * investigated magic streaks in %0925 labels. |
| | 22 | * worked with ken to create various plots of fwhm and iq for svs data. |
| | 23 | * sick on tuesday. |
| | 51 | * y-band fringe: fought with clipping reasons, fake signals in the |
| | 52 | fringe samples due to CTE effects, and various fitting algorithms, |
| | 53 | finally discovering that the chips with bad fringe scales also have |
| | 54 | video cells. This causes an offset between science and reference |
| | 55 | samples, producing a bad fit. Fixing this provides reasonable |
| | 56 | fringe scales for all chips in the test set. Currently running the |
| | 57 | fringe verify. |
| | 58 | |
| | 59 | * Fixed bugs caused by the ppImage/burntool masking that prevented |
| | 60 | the processing of Megacam data with IPP. Inadvertantly also fixed |
| | 61 | a bug that was preventing ppImage from properly masking burntool |
| | 62 | trails. Looked at the burntool streaks in M31 data, noting that |
| | 63 | burntool has some difficulty when the dither between images is |
| | 64 | small. Suggest a minimum dither of ~20 pixels to ensure burntool |
| | 65 | does not have problems. |
| | 66 | |
| | 67 | * Scanned disks that are part of nebulous to determine what is |
| | 68 | consuming the most diskspace. The breakdown of raw images / |
| | 69 | processed data / not-in-nebulous data is 51% / 36% / 11%. Full |
| | 70 | details on the wiki under [wiki:Cluster_Storage_Notes Cluster Storage Notes]. |
