三叶运维大清早收到一个 MySQL 的自定义语言告警 ：replication interrupt，看来是主从同步报错了。
登陆 MySQL，执行 show slave status \G 发现 salve 已经停止了，于是使用 start slave 启动，结果有如下报错：
|
1
|
ERROR 1201 (HY000): Could not initialize master info structure; more error messages can be found in the MySQL error log
|
搜索了下，网上给出的解决办法基本都是：
①、执行 reset slave；
②、重新执行 change master：
|
1
2
3
4
5
6
7
|
mysql> change master to
-> master_host='192.168.1.100',
-> master_port=3306,
-> master_user='repl',
-> master_password='repl',
-> master_log_file='mysql-bin.000051',
-> master_log_pos=254105;
|
③、执行 start slave
于是试了下，发现还是一样的报错，顺着报错信息，看了下 error 日志：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
160324 6:40:10 [ERROR] log *** glibc detected *** /data/mysql/bin/mysqld: corrupted double-linked list: 0x00002ab038100ab0 *** listed in the index, but failed to stat
160324 6:40:10 [ERROR] Error counting relay log space
160324 6:40:10 [ERROR] Failed to initialize the master info structure
160324 6:40:10 [Note] Event Scheduler: Loaded 0 events
160324 6:40:10 [Note] /data/mysql/bin/mysqld: ready for connections.
Version: '5.5.13-log' socket: '/data/mysql/data/mysqld.sock' port: 3306 Source distribution
160324 9:33:41 [ERROR] log *** glibc detected *** /data/mysql/bin/mysqld: corrupted double-linked list: 0x00002ab038100ab0 *** listed in the index, but failed to stat
160324 9:33:41 [ERROR] Error counting relay log space
160324 9:33:54 [ERROR] log *** glibc detected *** /data/mysql/bin/mysqld: corrupted double-linked list: 0x00002ab038100ab0 *** listed in the index, but failed to stat
160324 9:33:54 [ERROR] Error counting relay log space
160324 9:35:29 [ERROR] log *** glibc detected *** /data/mysql/bin/mysqld: corrupted double-linked list: 0x00002ab038100ab0 *** listed in the index, but failed to stat
160324 9:35:29 [ERROR] Error counting relay log space
160324 9:37:47 [ERROR] log *** glibc detected *** /data/mysql/bin/mysqld: corrupted double-linked list: 0x00002ab038100ab0 *** listed in the index, but failed to stat
160324 9:37:47 [ERROR] Error counting relay log space
|
发现其中频繁出现如下错误：
|
1
|
ist: 0x00002ab038100ab0 *** listed in the index, but failed to stat
|
于是，把 MySQL 数据文件目录下的 index info 文件都看了一遍，结果发现 Centos64-relay-bin.index 文件中出现异常内容：
这个文件正常情况应该是记录了 bin-log 文件名称才对，比如：
|
1
2
3
4
5
6
7
|
./Centos64-relay-bin.002064
./Centos64-relay-bin.002065
./Centos64-relay-bin.002066
./Centos64-relay-bin.002067
./Centos64-relay-bin.002068
./Centos64-relay-bin.002069
./Centos64-relay-bin.002070
|
因此，将 Centos64-relay-bin.index 这个文件 mv 重命名，然后重启 MySQL，重新按照上文从网上照来的的步骤：reset、change master、start 成功搞定！
如果你也遇到类似问题，可以先按照网上提供的三个步骤试试，如果还是和本文类似的报错，请特别看下 MySQL 数据目录下的这几种文件：
mysql-bin.index
relay-log.info
主机名-relay-bin.index
可以尝试将他们重命名或移走，重启 MySQL 再走一遍上面的三个步骤应该可以搞定了！
