streaming backups of large databases fail when receiving side runs pxb 2.3.8
Affects | Status | Importance | Assigned to | Milestone | ||
---|---|---|---|---|---|---|
Percona XtraBackup moved to https://jira.percona.com/projects/PXB | Status tracked in 2.4 | |||||
2.3 |
Fix Released
|
High
|
Sergei Glushchenko | |||
2.4 |
Fix Released
|
High
|
Sergei Glushchenko |
Bug Description
Streaming backups from a database containing many tables via xbstream to a host that has the xbstream from Percona XtraBackup 2.3.8 installed fails. This is due to the receiving xbstream process running out of file descriptors:
xbstream: Out of resources when opening file './magento/
xbstream: failed to create file.
Meanwhile, this is logged to the innobackupex log file:
innobackupex: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
[01] xtrabackup: Error: failed to copy datafile.
Instrumenting the xbstream binary with strace reveals the cause of the problem:
[...]
6578 read(0, "XBSTCK01\0E \0\0\0", 14) = 14
6578 read(0, "magento/
6578 read(0, "XBSTCK01\
6578 read(0, "magento/
6578 read(0, "\n\20\
6578 read(0, "\223L\271\25", 4) = 4
6578 read(0, "qpress10\
6578 mkdir("./magento/", 0700) = -1 EEXIST (File exists)
6578 open(".
6578 write(978, "qpress10\
6578 fadvise64(978, 0, 0, POSIX_FADV_
6578 read(0, "XBSTCK01\
6578 read(0, "magento/
6578 read(0, "XBSTCK01\
6578 read(0, "magento/
6578 read(0, "\307\t\
6578 read(0, "i\210\v\313", 4) = 4
6578 read(0, "qpress10\
6578 mkdir("./magento/", 0700) = -1 EEXIST (File exists)
6578 open(".
6578 write(979, "qpress10\
6578 fadvise64(979, 0, 0, POSIX_FADV_
[...]
Note that the created files are never closed. With the xbstream binary from 2.3.7 it looks like this, and it works perfectly:
[...]
22980 read(0, "XBSTCK01\
22980 read(0, "magento/
22980 read(0, "\n\20\
22980 read(0, "\223L\271\25", 4) = 4
22980 read(0, "qpress10\
22980 mkdir("./magento/", 0700) = -1 EEXIST (File exists)
22980 open(".
22980 write(3, "qpress10\
22980 fadvise64(3, 0, 0, POSIX_FADV_
22980 read(0, "XBSTCK01\
22980 read(0, "magento/
22980 fsync(3) = 0
22980 close(3) = 0
22980 read(0, "XBSTCK01\
22980 read(0, "magento/
22980 read(0, "\307\t\
22980 read(0, "i\210\v\313", 4) = 4
22980 read(0, "qpress10\
22980 mkdir("./magento/", 0700) = -1 EEXIST (File exists)
22980 open(".
22980 write(3, "qpress10\
22980 fadvise64(3, 0, 0, POSIX_FADV_
22980 read(0, "XBSTCK01\
22980 read(0, "magento/
22980 fsync(3) = 0
22980 close(3) = 0
[...]
The exact package versions I'm testing with (on Debian wheezy):
2.3.7-2.wheezy
2.3.8-1.wheezy
For now, I've reverted to version 2.3.7 on the receiving side.
Thank you for the bug report. Indeed, looks like xbstream doesn't close file descriptors. It's a regression introduced by commit 6cc9b954ac3.
Possible workaround is to increase file limit for xbstream.