Comment 0 for bug 1366997

We (PagerDuty) have experienced on multiple occasions duplicate entry errors on the primary key for some tables after our cluster's size changes. In the cases we have experienced so far, this is when gracefully adding or removing a node to vertically scale the nodes in the cluster. We don't experience a total failure of all transactions. Instead, a small subset error out. In some cases, the problem corrects itself, but several times now we have had the errors persist for several hours. When the problem persists, we have found that restarting one of the cluster members will sometimes fix the issue.

Restarting all transactions by rebooting the application does not fix the issue. Only a restart of a cluster member can eventually solve the problem.

Here are the packages we are running

ii percona-toolkit 2.2.7 Advanced MySQL and system command-line tools
ii percona-xtrabackup 2.1.9-744-1.lucid Open source backup tool for InnoDB and XtraDB
ii percona-xtradb-cluster-client-5.5 5.5.37-25.10-756.lucid Percona XtraDB Cluster database client binaries
ii percona-xtradb-cluster-common-5.5 5.5.37-25.10-756.lucid Percona XtraDB Cluster database common files (e.g. /e
ii percona-xtradb-cluster-galera-2.x 188.lucid Galera components of Percona XtraDB Cluster
ii percona-xtradb-cluster-server-5.5 5.5.37-25.10-756.lucid Percona XtraDB Cluster database server binaries

Here is an example error that we see.

Duplicate entry '623287' for key 'PRIMARY'

That error will be seen for multiple tables, and sometimes again for the same table.

Please let me know what additional information can help. The MySQL error log doesn't appear to have any interesting details in it (just node join and leave events) but I would be glad to pass it along if you think it would be helpful.