cluster SST fail - /usr/bin/du: Argument list too long

Bug #1294431 reported by pawouk
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MySQL patches by Codership
Status tracked in 5.6
5.5
Fix Committed
Medium
Alex Yurchenko
5.6
Fix Released
Medium
Alex Yurchenko
Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC
Status tracked in 5.6
5.5
Fix Released
Undecided
Unassigned
5.6
Fix Released
Undecided
Unassigned

Bug Description

Hi Guys,

we have quite a lot of tables and databases (1000DB, each with 60tables).

When I try to add a new node to the cluster I receive an error:

WSREP_SST: [INFO] Evaluating xbstream -c ${INFO_FILE} | socat -u stdio TCP:10.100.2.3:4444; RC=( ${PIPESTATUS[@]} ) (20140319 02:22:23.431)

/usr//bin/wsrep_sst_xtrabackup-v2: line 234: /usr//bin/du: Argument list too long

WSREP_SST: [INFO] Sleeping before data transfer for SST (20140319 02:22:25.456)
WSREP_SST: [INFO] Streaming the backup to joiner at 10.100.2.3 4444 (20140319 02:22:35.463)
WSREP_SST: [INFO] Evaluating innobackupex --defaults-file=/etc/mysql/my.cnf $INNOEXTRA --galera-info --stream=$sfmt ${TMPDIR} 2>${DATA}/innobackup.backup.log | pv -f -i 10 -N donor -s 2>0 | socat -u stdio TCP:10.100.2.3:4444; RC=( ${PIPESTATUS[@]} ) (20140319 02:22:35.468)
WSREP_SST: [ERROR] innobackupex finished with error: 1. Check /var/lib/mysql/data//innobackup.backup.log (20140319 02:22:35.697)
WSREP_SST: [ERROR] Cleanup after exit with status:22 (20140319 02:22:35.702)
140319 2:22:35 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.100.2.3:4444/xtrabackup_sst' --auth 'xtrabackuper:h@wagaV8$uyE' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/data/' --defaults-file '/etc/mysql/my.cnf' --gtid '7002b138-9d04-11e2-0800-0dbb30030abe:1402150009'
140319 2:22:35 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.100.2.3:4444/xtrabackup_sst' --auth 'xtrabackuper:h@wagaV8$uyE' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/data/' --defaults-file '/etc/mysql/my.cnf' --gtid '7002b138-9d04-11e2-0800-0dbb30030abe:1402150009': 22 (Invalid argument)
140319 2:22:35 [Warning] WSREP: 0 (sql1002): State transfer to 1 (sql1003) failed: -1 (Operation not permitted)
140319 2:22:35 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 1402150009)

The problematic code line is:

payload=$(du --block-size=1 -c **/*.ibd **/*.MYI **/*.MYI ibdata1 | awk 'END { print $1 }')

When I try to run it manualy at mysql data directory I receive the same error.

root@sql1002:/var/lib/mysql/data# du --block-size=1 -c **/*.ibd **/*.MYI **/*.MYI ibdata1
-bash: /usr/bin/du: Argument list too long
root@sql1002:/var/lib/mysql/data#

I believe it can be fixed by replacing the line with this command:

payload=$(find . -name "*.ibd" -name "*.MYI" -name "ibdata1" -type f -print0 | xargs -0 du --block-size=1 -c | awk 'END { print $1 }')

Tags: sst xtrabackup
tags: added: sst xtrabackup
Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

Yes, it can hit this bug when the number of files matched by " **/*.ibd **/*.MYI **/*.MYI ibdata1 " is really high.

Also, this bug is hit only when 'progress' sst option is used.

Using find should help I guess.

Revision history for this message
Alex Yurchenko (ayurchen) wrote :
Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1653

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.