Context Navigation

Changes between Version 19 and Version 20 of PS1_IPP_Czarlog_20140210

v19	v20
19	19	* 12:20 CZW: I just noticed that we have a very large number of chips/warps in the processing queue. There are now 274 lapRuns active, up significantly from the ~75 a few days ago. It appears that the cleanup task is timing out for the one run still in 'full' state, and when it times out, it doesn't correctly clear out the full run, leading to many more lapRuns being launched that is expected. I've pointed ~ipp/lap/current.queue to the empty off.queue, which will prevent new runs from being launched (so we can taper back down to ~75). I'll manually clear the full run, and figure out how long to set the timeout limit to.
20	20	* 14:40 CZW: A number of pantasks servers are not running. I'm stopping the remaining ones for a fresh restart.
	21	* 15:20 CZW: Back up and running. The job rate was remarkably stable over the past day or so. The current hypothesis is that the slow downs we've been seeing in the past after 100k jobs may have been due to crash/timeout pages filling up the book (that only requires a 1/1000 rate). With the fix to that a few days ago, we may not get slowdowns anymore. This is something to keep an eye on.
21	22	=== Thursday : YYYY.MM.DD ===
22	23