| | 7 | |
| | 8 | I worked on 3 issues related to the footprint / peak culling process. The peak culling process attempts to remove insignificant peaks within a footprint by requiring a significant downward deviation in going from any possible peak to the brighter significant peaks in the same footprint. First, I finalized the modification of the culling process to use the smoothed image rather than the unsmoothed image. This reduces the rate of false positives coming from background deviations. Next, I modified the culling process for saturated stars so that it requires a more substantial downward deviation than for fainter sources. This helps to group detections in the saturated cores which otherwise remain as many distinct detections. Finally, I modified the culling process to be faster for large footprints with many peaks. The original algorithm drew a unique threshold footprint for each peak of interest, leading to very long processing times in images with excess backgrounds. Chris discovered that these images, especially coming from a few specific chips where the background variations are worst, take up to half of our processing time (for less than 10% of our data). My modification to the code uses slightly more coarse threshold bins, and only draws a single footprint for each threshold bin. This speeds up processing by factors of 30 - 100. The sample of slow images, each of which took 10k - 30k seconds in the original analysis, now took an average of about 200 seconds per chip. |
| | 9 | |
| | 10 | We found 2 other throughput issues this week which will help us a lot. The first was in the area of nebulous. We identified 2 different issues with easy fixes. First, we have long had a problem where the apache server can get into a states in which in generates segmentation faults. We discovered that these are caused by some internal apache bug when there are too many apache threads. The segfaults can be avoided if we limit the number of threads to < 128. The segfaults are probably introducing a small amount of additional slowness. We also realized that the apache server used for the nebulous interface has a tendency to take a large amount of memory on that machine, especially if the nebulous mysql starts to get a little slow. Both of these issues acted as a runaway problem: as the nebulous load would get high, apache would spawn more threads, either slowing things down by segfaulting or by taking more memory. The slowness on apache meant more queries would be pending, driving up the apache load, etc. As a result, we see periods where the apache error rates goes to very high values and the mysql rates plummets, with everything getting extra slow. |
| | 11 | |
| | 12 | The configuration change fixed the segfaults. To address the memory load, we moved the apache server from ippdb00 (where mysql lives) to another machine. This was a big success: the load on ippdb00 has gone from a typical value of 20-25 during heavy processing to 4-5, and the rate of nebulous operations has increased from a maximum of roughly 150 per second to roughly 600 per second. We also suspect that we could push nebulous harder if we set up web redirects and use 2 or 3 machines for the apache server. There are also some optimizations to the SOAP interface code run on the server which Serge has identified. |
| | 13 | |
| | 14 | The other throughput improvement is in nighttime processing / registration. We had some occasional problems in the past couple of weeks in which the registration process, including the burntool analysis, was getting sluggish and causing high load on the gpc1 database. This problem has only had modest impact in the past, but last week it caused processing to come to a halt one day until we stopped everything and restarted the mysql server. We discovered that slowness comes from the query used to identify images for burntool processing. This particular query needs to join 2 tables (summitExp and summitImfile), but the only joining key is a string. Mysql is not smart enough to limit the query to only interesting rows; it ends up checking millions of database rows. We have made an additional restriction to the query which acts as a short term hack while we work out the code to add a veritable auto-increment index. |
| | 15 | |
| | 16 | Finally, I launched some MD04 test data sets for PSPS to demonstrate the current state of the code. |
