Changes between Version 138 and Version 139 of PS1_IPP_Czarlog_20141208

Timestamp:

Dec 13, 2014, 5:31:01 AM (12 years ago)

Author:

eugene

Comment:

--

PS1_IPP_Czarlog_20141208

@@ -259 +259 @@
   * cleanup poll 30, resid cleanup stop, ippmd 50% down, stdlocal less
 
-=== Saturday : YYYY.MM.DD ===
+=== Saturday : 2014.11.13 ===
+ * 05:00 : ippdb00 connection rates as reported by Chris' tool (~watersc1/monitor_connections.20141210/mon_con.dat) have been low (100-300) for the second half of the night.  the free slot failures came in only a handful of burst during the night:
+
+{{{
+grep "cannot find a free slot" /var/log/mysql/mysqld.err | sort | uniq -c 
+[snip]
+      2 141212 16:54:57InnoDB: Warning: cannot find a free slot for an undo log. Do you have too
+      8 141212 16:54:58InnoDB: Warning: cannot find a free slot for an undo log. Do you have too
+     26 141212 16:54:59InnoDB: Warning: cannot find a free slot for an undo log. Do you have too
+     31 141212 16:55:00InnoDB: Warning: cannot find a free slot for an undo log. Do you have too
+     10 141212 16:55:01InnoDB: Warning: cannot find a free slot for an undo log. Do you have too
+      2 141212 16:55:02InnoDB: Warning: cannot find a free slot for an undo log. Do you have too
+     23 141212 16:55:03InnoDB: Warning: cannot find a free slot for an undo log. Do you have too
+     28 141212 16:55:04InnoDB: Warning: cannot find a free slot for an undo log. Do you have too
+     57 141212 16:55:05InnoDB: Warning: cannot find a free slot for an undo log. Do you have too
+     51 141212 16:55:06InnoDB: Warning: cannot find a free slot for an undo log. Do you have too
+     41 141212 16:55:07InnoDB: Warning: cannot find a free slot for an undo log. Do you have too
+     55 141212 16:55:08InnoDB: Warning: cannot find a free slot for an undo log. Do you have too
+     77 141212 20:57:03InnoDB: Warning: cannot find a free slot for an undo log. Do you have too
+     45 141212 21:13:48InnoDB: Warning: cannot find a free slot for an undo log. Do you have too
+     17 141212 21:13:50InnoDB: Warning: cannot find a free slot for an undo log. Do you have too
+     23 141212 21:13:51InnoDB: Warning: cannot find a free slot for an undo log. Do you have too
+     88 141212 21:13:52InnoDB: Warning: cannot find a free slot for an undo log. Do you have too
+     12 141212 21:13:54InnoDB: Warning: cannot find a free slot for an undo log. Do you have too
+     12 141212 21:13:55InnoDB: Warning: cannot find a free slot for an undo log. Do you have too
+     80 141212 21:21:15InnoDB: Warning: cannot find a free slot for an undo log. Do you have too
+     42 141212 21:21:16InnoDB: Warning: cannot find a free slot for an undo log. Do you have too
+     73 141212 22:44:32InnoDB: Warning: cannot find a free slot for an undo log. Do you have too
+    100 141212 22:44:33InnoDB: Warning: cannot find a free slot for an undo log. Do you have too
+     17 141212 22:44:34InnoDB: Warning: cannot find a free slot for an undo log. Do you have too
+     78 141212 22:44:35InnoDB: Warning: cannot find a free slot for an undo log. Do you have too
+}}}
+
+We still have no good clue about this, but I suspect cleanup.  I'm going to recommend the following:  let keep cleanup off at night and weekends (I've shut it down now), and let's plan for campaigns to run it hard during the work day when folks can monitor the behavior in real time.  Meanwhile, we need to push stdlocal processing hard to get the rate back up.
+
 
 === Sunday : YYYY.MM.DD ===