MySQL InnoDB Cluster Charm

Scale out node is leader

Bug #1877616 reported by Andrew McLeod on 2020-05-08

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Charm Helpers	Invalid	High	David Ames
	MySQL InnoDB Cluster Charm	Fix Released	Medium	David Ames	MySQL InnoDB Cluster Charm 20.05

Bug Description

focal ussuri s390x

unit-mysql-innodb-cluster-2: 15:41:24 DEBUG unit.mysql-innodb-cluster/2.leader-elected MySQLdb._exceptions.OperationalError: (1290, 'The MySQL server is running with the --super-read-only option so it cannot execute this statement')

partial log:

https://pastebin.canonical.com/p/dgrg9fC9tB/

full juju logs and mysql error log attached in tar

Revision history for this message

Andrew McLeod (admcleod) wrote on 2020-05-08:

logs.tar.gz Edit (42.3 KiB, application/x-tar)

Revision history for this message

David Ames (thedac) wrote on 2020-05-08:

TRIAGE:

The code correctly connects to the RW node 10.13.3.225 and then immediately connects locally which turns out not to be a RO node.

unit-mysql-innodb-cluster-2: 15:42:33 DEBUG unit.mysql-innodb-cluster/2.juju-log Opening db connection for clusteruser@10.13.3.225
unit-mysql-innodb-cluster-2: 15:42:34 DEBUG unit.mysql-innodb-cluster/2.juju-log Opening db connection for root@None

Check charmhelpers, if we are already connected don't connect again.

Changed in charm-mysql-innodb-cluster:
status:	New → Triaged
importance:	Undecided → High
assignee:	nobody → David Ames (thedac)
Changed in charm-helpers:
status:	New → Confirmed
importance:	Undecided → High
assignee:	nobody → David Ames (thedac)

Revision history for this message

David Ames (thedac) wrote on 2020-05-15:

This turns out to be an edge case.

If one has *just* removed a unit and adds a new unit immediately, the new unit often wins the leader election. This puts the charm into a state that it cannot recover from automatically, as the new unit is not yet a member of the cluster and therefore cannot add itself to the cluster.

TRIAGE:

Catch and log this edge case and provide an "add_instance" action to solve it.

Changed in charm-helpers:
status:	Confirmed → Invalid
summary:	- hook failed: "leader-elected" + Scale out node is leader
Changed in charm-mysql-innodb-cluster:
importance:	High → Medium

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-05-15: Fix proposed to charm-mysql-innodb-cluster (master)

Fix proposed to branch: master
Review: https://review.opendev.org/728587

Changed in charm-mysql-innodb-cluster:
status:	Triaged → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-05-20: Fix merged to charm-mysql-innodb-cluster (master)

Reviewed: https://review.opendev.org/728587
Committed: https://git.openstack.org/cgit/openstack/charm-mysql-innodb-cluster/commit/?id=7cbf9a88b001d73d290bb123f5bba010f51319b3
Submitter: Zuul
Branch: master

commit 7cbf9a88b001d73d290bb123f5bba010f51319b3
Author: David Ames <email address hidden>
Date: Wed May 13 11:37:31 2020 -0700

Scale in and scale out

Handle adding and removing units of the mysql-innodb-cluster gracefully.

    Review and land:
    https://github.com/openstack-charmers/zaza/pull/353
    https://github.com/openstack-charmers/zaza-openstack-tests/pull/275

    Closes-Bug: #1874479
    Closes-Bug: #1877546
    Closes-Bug: #1877616
    Depends-On: Ie658c42b095bcff822cdfb0b771d41704ddc85ea
    Change-Id: I39210c760e7204f84365d5dcb4eeedbc0889041f