Infra node was down for about 2 days due to hardware failure. Upon bringing the node back up it fails to rejoin the cluster with errors[1]. This appears to be a bug in percona-xtrabackup-2.3 with MariaDB10. [0] This is a critical failure as the only options to rejoin the cluster are to change the "wsrep_sst_method = xtrabackup-v2" to either "rsync" or "mysqldump". Neither of which are very good options due to speed or DB locking.
[0]
https://bugs.launchpad.net/percona-xtrabackup/+bug/1570560
[1]
InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 2016-06-07 16:47:12 7efea54c1700 InnoDB: Assertion failure in thread 139632160020224 in file page0cur.cc line 931
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
20:47:12 UTC - xtrabackup got signal 6 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x10000
innobackupex(my_print_stacktrace+0x2e) [0x8f139e]
innobackupex(handle_fatal_signal+0x262) [0x7d1c82]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7efea87a3340]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39) [0x7efea6f07cc9]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7efea6f0b0d8]
innobackupex(page_cur_parse_insert_rec(unsigned long, unsigned char*, unsigned char*, buf_block_t*, dict_index_t*, mtr_t*)+0x8d5) [0x6db465]
innobackupex() [0x68249c]
innobackupex(recv_recover_page_func(unsigned long, buf_block_t*)+0x509) [0x684009]
innobackupex(buf_page_io_complete(buf_page_t*)+0x410) [0x6c1560]
innobackupex(fil_aio_wait(unsigned long)+0x138) [0x6f45f8]
innobackupex(io_handler_thread+0x28) [0x7215b8]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7efea879b182]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7efea6fcb47d]
Nobody got the opportunity to test this bug and was active during the bug triage of this week. Therefore, all the classification and work will be delayed to next week.
Sorry for the inconvenience.