Changes between Version 16 and Version 17 of PS1_IPP_Czarlog_20190510

Timestamp:

May 14, 2019, 11:08:05 AM (7 years ago)

Author:

tdeboer

Comment:

--

PS1_IPP_Czarlog_20190510

@@ -291 +291 @@
 }}}
 
+ ------------------------------------------------------
+
+ Then we noticed the replication server is also down. the sql slave is not quite working right.
+ On ippdb05, log into mysql as root:
+{{{
+mysql -ulocalhost -uroot -pX
+show slave status\G
+}}}
+ then, inside mysql do:
+{{{
+STOP SLAVE; SET GLOBAL SQL_SLAVE_SKIP_COUNTER=1; START SLAVE;
+}}}
+ That set the replication back up, and seconds behind master started to tick down.
+
+ ------------------------
+
+ On ippdb08 (the primary) go into mysql as root as well, and do:
+{{{
+show processlist;
+}}}
+ which shows:
+{{{
+| 1636903827 | repl_gpc1 | ippdb05.ifa.hawaii.edu:50012 | NULL   | Binlog Dump |  197 | Has sent all binlog to slave; waiting for binlog to be updated | NULL  
+}}}
+ So, it is probably back up to date
+
+ -------------------
+
+ It seems there is also a problem on ipp116. The replication slave there seems stuck, and does not respond to a stop slave; inside mysql.
+ shut it down externally, using:
+{{{
+mysqladmin -uroot -pxxx shutdown now
+}}}
+ The mysql.err log shows a normal shutdown, which is good. Checking ps aux | grep mysql shows that the mysql process is still running. IPPmonitor is now unstuck, but shows that ipp116 has a DB connection error.  Trying to see how to shut down the process without killing it hard.
+{{{
+sudo /etc/init.d/mysql stop
+ * WARNING:  mysql has not yet been started.
+sudo /etc/init.d/mysql zap
+ * Manually resetting mysql to stopped state.
+}}}
+ It is still there!
+
+ Try this:
+{{{
+sudo mysqladmin -uroot -pXXX shutdown now
+mysqladmin: connect to server at 'localhost' failed
+error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)'
+Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
+}}}
+ It has not updated any files in two hours, since the crash. So, it is likely now a zombie process. Mark tried to kill it nicely and then killed it hard after. Then, we started it anew, using:
+{{{
+sudo /etc/init.d/mysql start
+}}}
+ That worked, and the error log looks good. Now we need to restart the replication on this machine.
+{{{
+mysql -ulocalhost -uroot -pXXX
+show slave status\G
+
+STOP SLAVE; SET GLOBAL SQL_SLAVE_SKIP_COUNTER=1; START SLAVE;
+}}}
+
+ Keep an eye on the seconds behind master, using: show slave status\G
+ After a good 20 minutes, both of them had caught up to the master and things were working well again. 
+
 
  * MEH: starting some older diff cleanup now to help free up space on lower data nodes
 
 === Tuesday : 2019.05.14 ===
+ * TdB: Haydn wants to redistribute the x nodes ippx001-044 to get better temperature behaviour inside the cabinet (they are all adjacent at the top of a half-filled cabinet). remove them from processing:
+ go to ~ippps2/stdscience and in to pantasks_client
+{{{
+controller status
+
+There are 6 sets of them
+controller host off ippx001
+controller host off ippx002
+controller host off ippx003
+controller host off ippx004
+controller host off ippx005
+controller host off ippx006
+controller host off ippx007
+controller host off ippx008
+controller host off ippx009
+controller host off ippx010
+controller host off ippx011
+controller host off ippx012
+controller host off ippx013
+controller host off ippx014
+controller host off ippx015
+controller host off ippx016
+controller host off ippx017
+controller host off ippx018
+controller host off ippx019
+controller host off ippx020
+controller host off ippx021
+controller host off ippx022
+controller host off ippx023
+controller host off ippx024
+controller host off ippx025
+controller host off ippx026
+controller host off ippx027
+controller host off ippx028
+controller host off ippx029
+controller host off ippx030
+controller host off ippx031
+controller host off ippx032
+controller host off ippx033
+controller host off ippx034
+controller host off ippx035
+controller host off ippx036
+controller host off ippx037
+controller host off ippx038
+controller host off ippx039
+controller host off ippx040
+controller host off ippx041
+controller host off ippx042
+controller host off ippx043
+controller host off ippx044
+}}}
 
 === Wednesday : 2019.05.15 ===