Changes between Initial Version and Version 1 of PS1_IPP_CzarLog_20101203

Timestamp:

Dec 3, 2010, 4:18:36 PM (16 years ago)

Author:

bills

Comment:

--

PS1_IPP_CzarLog_20101203

@@ +1 @@
+== PS1 IPP Czar Logs for the weekend 2010.12.03 - 2010.10.05 == 
+[[PageOutline]]
+
+(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])
+
+=== Monday : YYYY.MM.DD ===
+
+=== Tuesday : YYYY.MM.DD ===
+
+=== Wednesday : YYYY.MM.DD ===
+
+=== Thursday : YYYY.MM.DD ===
+
+=== Friday : YYYY.MM.DD ===
+Bill is acting czar today. We are running STS.20101202 chip - warp) and MD03.refstack.20101202 stacks.
+
+Apparently something has changed in ppStack and it is a memory hog. Certain skycells fail with memory allocation error and in some cases crash the node running the process.
+
+3 hosts died with this problem a bit after 3 pm. ipp015, c27 and c29. Cindy and Gavin rebooted them. Then Gavin needed to clean up some nfs problems.
+
+As a workaround we're going to stop processing of these stacks in stdscience with a selection of enabled hosts with large memory. This will run out of ~ipp/stackonly with pantasks_server on ipp042.
+
+Around 16:00 i disabled the two labels (using labeltool) and I am allowing the stacks that are running to finish.
+
+=== Saturday : YYYY.MM.DD ===
+
+=== Sunday : YYYY.MM.DD ===
+