streaming backups of large databases fail when receiving side runs pxb 2.3.8

Bug #1690823 reported by Rene H. Larsen
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Percona XtraBackup moved to https://jira.percona.com/projects/PXB
Status tracked in 2.4
2.3
Fix Released
High
Sergei Glushchenko
2.4
Fix Released
High
Sergei Glushchenko

Bug Description

Streaming backups from a database containing many tables via xbstream to a host that has the xbstream from Percona XtraBackup 2.3.8 installed fails. This is due to the receiving xbstream process running out of file descriptors:

xbstream: Out of resources when opening file './magento/sales_invoice.ibd.qp' (Errcode: 24 - Too many open files)
xbstream: failed to create file.

Meanwhile, this is logged to the innobackupex log file:

innobackupex: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
[01] xtrabackup: Error: failed to copy datafile.

Instrumenting the xbstream binary with strace reveals the cause of the problem:

[...]
6578 read(0, "XBSTCK01\0E \0\0\0", 14) = 14
6578 read(0, "magento/directory_country.ibd.qp", 32) = 32
6578 read(0, "XBSTCK01\0P\"\0\0\0", 14) = 14
6578 read(0, "magento/eav_attribute_group.ibd."..., 34) = 34
6578 read(0, "\n\20\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 16) = 16
6578 read(0, "\223L\271\25", 4) = 4
6578 read(0, "qpress10\0\0\1\0\0\0\0\0F\27\0\0\0eav_attribu"..., 4106) = 4106
6578 mkdir("./magento/", 0700) = -1 EEXIST (File exists)
6578 open("./magento/eav_attribute_group.ibd.qp", O_WRONLY|O_CREAT|O_EXCL|O_NOFOLLOW, 0660) = 978
6578 write(978, "qpress10\0\0\1\0\0\0\0\0F\27\0\0\0eav_attribu"..., 4106) = 4106
6578 fadvise64(978, 0, 0, POSIX_FADV_DONTNEED) = 0
6578 read(0, "XBSTCK01\0E\"\0\0\0", 14) = 14
6578 read(0, "magento/eav_attribute_group.ibd."..., 34) = 34
6578 read(0, "XBSTCK01\0P(\0\0\0", 14) = 14
6578 read(0, "magento/sales_refunded_aggregate"..., 40) = 40
6578 read(0, "\307\t\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 16) = 16
6578 read(0, "i\210\v\313", 4) = 4
6578 read(0, "qpress10\0\0\1\0\0\0\0\0F\35\0\0\0sales_refun"..., 2503) = 2503
6578 mkdir("./magento/", 0700) = -1 EEXIST (File exists)
6578 open("./magento/sales_refunded_aggregated.ibd.qp", O_WRONLY|O_CREAT|O_EXCL|O_NOFOLLOW, 0660) = 979
6578 write(979, "qpress10\0\0\1\0\0\0\0\0F\35\0\0\0sales_refun"..., 2503) = 2503
6578 fadvise64(979, 0, 0, POSIX_FADV_DONTNEED) = 0
[...]

Note that the created files are never closed. With the xbstream binary from 2.3.7 it looks like this, and it works perfectly:

[...]
22980 read(0, "XBSTCK01\0P\"\0\0\0", 14) = 14
22980 read(0, "magento/eav_attribute_group.ibd."..., 34) = 34
22980 read(0, "\n\20\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 16) = 16
22980 read(0, "\223L\271\25", 4) = 4
22980 read(0, "qpress10\0\0\1\0\0\0\0\0F\27\0\0\0eav_attribu"..., 4106) = 4106
22980 mkdir("./magento/", 0700) = -1 EEXIST (File exists)
22980 open("./magento/eav_attribute_group.ibd.qp", O_WRONLY|O_CREAT|O_EXCL|O_NOFOLLOW, 0660) = 3
22980 write(3, "qpress10\0\0\1\0\0\0\0\0F\27\0\0\0eav_attribu"..., 4106) = 4106
22980 fadvise64(3, 0, 0, POSIX_FADV_DONTNEED) = 0
22980 read(0, "XBSTCK01\0E\"\0\0\0", 14) = 14
22980 read(0, "magento/eav_attribute_group.ibd."..., 34) = 34
22980 fsync(3) = 0
22980 close(3) = 0
22980 read(0, "XBSTCK01\0P(\0\0\0", 14) = 14
22980 read(0, "magento/sales_refunded_aggregate"..., 40) = 40
22980 read(0, "\307\t\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 16) = 16
22980 read(0, "i\210\v\313", 4) = 4
22980 read(0, "qpress10\0\0\1\0\0\0\0\0F\35\0\0\0sales_refun"..., 2503) = 2503
22980 mkdir("./magento/", 0700) = -1 EEXIST (File exists)
22980 open("./magento/sales_refunded_aggregated.ibd.qp", O_WRONLY|O_CREAT|O_EXCL|O_NOFOLLOW, 0660) = 3
22980 write(3, "qpress10\0\0\1\0\0\0\0\0F\35\0\0\0sales_refun"..., 2503) = 2503
22980 fadvise64(3, 0, 0, POSIX_FADV_DONTNEED) = 0
22980 read(0, "XBSTCK01\0E(\0\0\0", 14) = 14
22980 read(0, "magento/sales_refunded_aggregate"..., 40) = 40
22980 fsync(3) = 0
22980 close(3) = 0
[...]

The exact package versions I'm testing with (on Debian wheezy):
2.3.7-2.wheezy
2.3.8-1.wheezy

For now, I've reverted to version 2.3.7 on the receiving side.

Revision history for this message
Sergei Glushchenko (sergei.glushchenko) wrote :

Thank you for the bug report. Indeed, looks like xbstream doesn't close file descriptors. It's a regression introduced by commit 6cc9b954ac3.

Possible workaround is to increase file limit for xbstream.

Revision history for this message
Sergei Glushchenko (sergei.glushchenko) wrote :
Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-503

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.