Changes between Version 13 and Version 14 of PS1_IPP_Czarlog_20190510

Timestamp:

May 13, 2019, 6:32:17 PM (7 years ago)

Author:

tdeboer

Comment:

--

PS1_IPP_Czarlog_20190510

@@ -222 +222 @@
  The vast majority of them are in fact exps which have a label of goto_clean% which are likely previously cleaned chip exps. However, we cannot simply set their state to cleaned, since we have no way of telling whether the cleanup operation went succesfully or failed. Therefore, we need to send the whole lot back to cleanup. Nonetheless, by parsin the xps, we can preferentially send exps that we still need to use in the update processing before doing things previosuly cleaned. Focus on labels like %LAP.PV3% or %OSS% at first and slowly end them to cleanup in between other cleanup jobs.
 
+ * TdB: at around 17:30, the IPPmonitor stalled. Turns out, mysql crashed on ippdb05. Let's check things:
+
+ Look at the error log under /var/log/mysql (log into ippdb05 as own user)
+{{{
+190513 17:16:46 - mysqld got signal 6 ;
+This could be because you hit a bug. It is also possible that this binary
+or one of the libraries it was linked against is corrupt, improperly built,
+or misconfigured. This error can also be caused by malfunctioning hardware.
+We will try our best to scrape up some info that will hopefully help diagnose
+the problem, but since we have already crashed, something is definitely wrong
+and this may fail.
+
+key_buffer_size=33554432
+read_buffer_size=262144
+max_used_connections=15
+max_connections=2048
+threads_connected=2
+It is possible that mysqld could use up to 
+key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 1605632 K
+bytes of memory
+Hope that's ok; if not, decrease some variables in the equation.
+
+thd=0x7f664c587e30
+Attempting backtrace. You can use the following information to find out
+where mysqld died. If you see no messages after this, something went
+terribly wrong...
+Cannot determine thread, fp=0x7f664c587e30, backtrace may not be correct.
+Bogus stack limit or frame pointer, fp=0x7f664c587e30, stack_bottom=0x44c90000, thread_stack=262144, aborting backtrace.
+Trying to get some variables.
+Some pointers may be invalid and cause the dump to abort...
+thd->query at (nil)  is invalid pointer
+thd->thread_id=4
+The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
+information that should help you find out what is causing the crash.
+pure virtual method called
+terminate called without an active exception
+}}}
+
+ So, let's try to restart mysql on ippdb05 and see if ti comes back up. As root, do:
+{{{
+sudo /etc/init.d/mysql start
+ * Starting mysql ...
+ * Strange, the socket file already exist in "/var/run/mysqld/mysqld.sock"
+ * it will be removed now and re-created by the MySQL server
+ * BUT please make your checks.
+ * Starting mysql (/etc/mysql/my.cnf)
+ * MySQL NOT started (0)     
+}}} 
+ But, it probably did start. Check, using:                                                                                                                            [ !! ]
+{{{
+ps aux | grep mysql
+mysql    27881 93.3  7.5 146558796 14950952 ?  Ssl  18:25   0:33 /usr/sbin/mysqld --defaults-file=/etc/mysql/my.cnf --basedir=/usr --datadir=/var/lib/mysql --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock
+}}}
+ Indeed, it started just fine.
+
+ Checking the error log file again shows:
+{{{
+190513 18:26:29  InnoDB: Started; log sequence number 5173 4202608365
+190513 18:26:29 [Note] Recovering after a crash using /var/lib/mysql/mysqld-bin
+190513 18:26:29 [Note] Starting crash recovery...
+190513 18:26:29 [Note] Crash recovery finished.
+190513 18:26:29 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts as a slave and has his hostname changed!! Please use '--relay-log=mysqld-relay-bin' to avoid this problem.
+
+InnoDB: Rolling back of trx id 2 1776754758 completed
+190513 18:26:29  InnoDB: Rollback of non-prepared transactions completed
+190513 18:26:29 [Note] /usr/sbin/mysqld: ready for connections.
+Version: '5.0.77-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  Gentoo Linux mysql-community-5.0.77-r1
+}}}
 
 === Tuesday : 2019.05.14 ===