The purpose of this page is to show how databases replication problems were solved. Please contribute when you fix a MySQL replication problem (add a header <date>/<time>: <Problem>). If you fix something which is similar to an existing entry, just add an entry something like 'Fixed problem as in <entry reference>'. When we will have a lot of problems, I (SC) will try to reorganize them by categories.

20110429/0850: gpc1 Replication Error

From mysql slave on ippc02 (SHOW SLAVE STATUS) Query partially completed on the master (error on master: 1317) and was aborted. There is a chance that your master is inconsistent at this point. If you are sure that your master is ok, run this query manually on the slave and then restart the slave with SET GLOBAL SQL_SLAVE_SKIP_COUNTER=1; START SLAVE; . Query: 'drop table lapExp'.

I checked with Chris. I manually dropped the table in gpc1 and did a 'STOP SLAVE; SET GLOBAL SQL_SLAVE_SKIP_COUNTER = 1; START SLAVE;'. Hopefully we will not have to ingest a new dump once again...

20110413/1450: ippadmin Replication Error

• Logged onto ipp001
• Started mysql client as root
• 'SHOW SLAVE STATUS\G' shows the following error:

  Last_Error: Error 'Duplicate entry '26' for key 1' on query. Default database: 'ippadmin'. Query: 'insert into users (username, password) values ('denver', '********')'
  

• I did the usual 'STOP SLAVE; SET GLOBAL SQL_SLAVE_SKIP_COUNTER = 1; START SLAVE;'
• The slave is 172833 seconds (i.e. 2 days) behind its master. It clearly shows that no one looks at the replication server statuses.

20110307/0830: Nebulous Replication Error

• Logged onto ippdb02
• Started mysql client as root
• 'SHOW SLAVE STATUS\G' shows the following error:

  Error 'Table 'nebulous.mountedvol' doesn't exist' on query. 
  Default database: 'nebulous'. 
  Query: 'REPLACE INTO mountedvol SELECT vol_id,name,host,path,allocate,available,xattr,mountpoint, 
         '10651598144', '8973230080',note FROM volume WHERE mountpoint = '/data/ipp004.0''
  

• But, if I check the contents of the database, I can see nebulous.mountedvol exists! It seems that the replicated nebulous didn't pass the optimization of last week.
• I'm setting up a new replicated server.

20101116/1659: gpc1 Replication Error

• Logged onto ipp001
• Started mysql client as root
• 'SHOW SLAVE STATUS\G' shows error similar to 20101104/1429: gpc1 Replication Error
• Same solution was applied (Statement skipped)

20101104/1429: gpc1 Replication Error

Problem detected through czartool

• Logged onto ipp001
• Started mysql client as root
• 'SHOW SLAVE STATUS\G' showed (shortened)

            Slave_SQL_Running: No
                   Last_Errno: 1062
                   Last_Error: Error 'Duplicate entry '23' for key 1' on query. Default database: 'ippadmin'. 
                               Query: 'INSERT into cookies (username, cookie) values ('psobs', '99497')'
  

• Local non-replicated database involved... Statement skipped (in MySQL client):

    STOP SLAVE;
    SET GLOBAL SQL_SLAVE_SKIP_COUNTER = 1; 
    START SLAVE;
  

  on one line:

    STOP SLAVE; SET GLOBAL SQL_SLAVE_SKIP_COUNTER = 1; START SLAVE;
  

• Then 'SHOW SLAVE STATUS\G' showed (shortened)

            Slave_SQL_Running: Yes
                   Last_Errno: 0
                   Last_Error: 
  

• Czartool didn't show any problem after the fix