MySQL 5.6 fails to start on the incremental backup prepared with 2.4 and non-default innodb_data_file_path

Tested with MySQL 5.6 with PXB 2.4:

Result after copy-back+start:

2017-03-06 06:43:10 27284 [ERROR] InnoDB: Cannot create log files because data files are corrupt or not in sync with each other
2017-03-06 06:43:10 27284 [ERROR] Plugin 'InnoDB' init function returned error.
2017-03-06 06:43:10 27284 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2017-03-06 06:43:10 27284 [ERROR] Unknown/unsupported storage engine: InnoDB
2017-03-06 06:43:10 27284 [ERROR] Aborting

Took 1 full + 2 incrementals with following my.cnf:

$ cat my.sandbox.cnf
[client]
user=root
password=
port=5635
socket=/tmp/mysql_sandbox5635.sock

[mysqld]
user = shahriyar.rzaev
port = 5635
socket = /tmp/mysql_sandbox5635.sock
basedir = /home/shahriyar.rzaev/Percona_Servers/MySQLs/5.6.35
datadir = /home/shahriyar.rzaev/sandboxes/msb_5_6_35/data
tmpdir = /home/shahriyar.rzaev/sandboxes/msb_5_6_35/tmp
lower_case_table_names = 0
pid-file = /home/shahriyar.rzaev/sandboxes/msb_5_6_35/data/mysql_sandbox5635.pid
bind-address = 127.0.0.1
log-error=msandbox.err
innodb_data_file_path = ibdata1:100M;ibdata2:10M:autoextend

There is a workaround. When instead of

xtrabackup --prepare --incremental-dir=... --target-dir..

I use

xtrabackup --prepare --apply-log-only --incremental-dir=... --target-dir..
xtrabackup --prepare --target-dir..

it works for me

I used following with 2.4 but failed:

/home/shahriyar.rzaev/Percona_Xtrabackups/xb_2.4/usr/local/xtrabackup/bin/xtrabackup --prepare --apply-log-only --target-dir=/home/shahriyar.rzaev/backup_dirs/msb_5_6_35//full/2017-03-06_06-33-29

/home/shahriyar.rzaev/Percona_Xtrabackups/xb_2.4/usr/local/xtrabackup/bin/xtrabackup --prepare --apply-log-only --target-dir=/home/shahriyar.rzaev/backup_dirs/msb_5_6_35//full/2017-03-06_06-33-29 --incremental-dir=/home/shahriyar.rzaev/backup_dirs/msb_5_6_35//inc/2017-03-06_06-33-47

/home/shahriyar.rzaev/Percona_Xtrabackups/xb_2.4/usr/local/xtrabackup/bin/xtrabackup --prepare --target-dir=/home/shahriyar.rzaev/backup_dirs/msb_5_6_35//full/2017-03-06_06-33-29 --incremental-dir=/home/shahriyar.rzaev/backup_dirs/msb_5_6_35//inc/2017-03-06_06-35-38

Shako,

The last prepare has to be without --incremental-dir like I posted in my comment. Problem turned out to be that both 2.3 and 2.4 create ib_logfile's in the incremental_basedir when it is specified.

When you prepare with
xtrabackup --prepare --target-dir=...

logs created inside of the target-dir

But when you prepare with
xtrabackup --prepare --inremental-dir=... --target-dir=...

logs created inside of the incremental-dir

Yet to find out why the combination of 5.6 + 2.4 doesn't work, but 5.6 + 2.3 and 5.7 + 2.4 work fine.

It is all clear now.

1. xtrabackup --prepare --incremental-dir does not create ib_logfile's, it is a bug
2. If log files aren't found MySQL 5.6 is trying to check that both ibdata1 and ibdata2 have the same flushed_lsn number stored in their first pages. MySQL 5.7 does not perform that check, it trusts flushed_lsn from the first file. It doesn't store this number in second file either, and xtrabackup 2.4 being based on 5.7 has the same behaviour.

The fix will come in two variants:
1. In 2.3 and 2.4 we will fix ib_logfile's creation
2. In 2.4 we will make sure to store flushed_lsn to the second datafile as well to make MySQL 5.6 happy.

https://github.com/percona/percona-xtrabackup/pull/327
https://github.com/percona/percona-xtrabackup/pull/328

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-498