| 19 | 19 | * 12:20 CZW: I just noticed that we have a very large number of chips/warps in the processing queue. There are now 274 lapRuns active, up significantly from the ~75 a few days ago. It appears that the cleanup task is timing out for the one run still in 'full' state, and when it times out, it doesn't correctly clear out the full run, leading to many more lapRuns being launched that is expected. I've pointed ~ipp/lap/current.queue to the empty off.queue, which will prevent new runs from being launched (so we can taper back down to ~75). I'll manually clear the full run, and figure out how long to set the timeout limit to. |
