| Version 1 (modified by , 16 years ago) ( diff ) |
|---|
PS1 IPP Czar Logs for the weekend 2010.12.03 - 2010.10.05
(Up to PS1 IPP Czar Logs)
Monday : YYYY.MM.DD
Tuesday : YYYY.MM.DD
Wednesday : YYYY.MM.DD
Thursday : YYYY.MM.DD
Friday : YYYY.MM.DD
Bill is acting czar today. We are running STS.20101202 chip - warp) and MD03.refstack.20101202 stacks.
Apparently something has changed in ppStack and it is a memory hog. Certain skycells fail with memory allocation error and in some cases crash the node running the process.
3 hosts died with this problem a bit after 3 pm. ipp015, c27 and c29. Cindy and Gavin rebooted them. Then Gavin needed to clean up some nfs problems.
As a workaround we're going to stop processing of these stacks in stdscience with a selection of enabled hosts with large memory. This will run out of ~ipp/stackonly with pantasks_server on ipp042.
Around 16:00 i disabled the two labels (using labeltool) and I am allowing the stacks that are running to finish.
