MySQL InnoDB Cluster Charm

Scale out: Adding new node fails

Bug #1877546 reported by Andrew McLeod
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MySQL InnoDB Cluster Charm
Fix Released
High
David Ames
MySQL InnoDB Cluster Charm 20.05
charm-interface-mysql-innodb-cluster
Fix Released
High
David Ames

Bug Description

focal ussuri s390x

Install hook intermittently failing (e.g. on one of 3 units) with the following error:

subprocess.CalledProcessError: Command '['/snap/bin/mysql-shell.mysqlsh', '--no-wizard', '--python', '-f', '/root/snap/mysql-shell/common/tmpl8_2dg_k.py']' returned non-zero exit status 1

reason: the tmp file does not exist in the directory (the directory is there and contains nothing)

https://pastebin.canonical.com/p/YmV77wWWpD/

Revision history for this message
David Ames (thedac) wrote :

Triage:

Per the Traceback

One of the non-leaders is executing config_changed during the install hook. config_changed calls wait_until_cluster_available but the cluster is not ready yet.

More precise flags are required on the config_changed reactive handler.

summary: - hook failed: "install" - executing /snap/bin/mysql-shell.mysqlsh
+ wait_until_cluster_available called before the cluster is ready
Revision history for this message
David Ames (thedac) wrote : Re: wait_until_cluster_available called before the cluster is ready

Correction, it is the leader that is running config_changed too early.

David Ames (thedac)
Changed in charm-mysql-innodb-cluster:
status: New → Confirmed
status: Confirmed → Triaged
importance: Undecided → High
assignee: nobody → David Ames (thedac)
Revision history for this message
David Ames (thedac) wrote :

This is a scale out failure when adding a node to the cluster

summary: - wait_until_cluster_available called before the cluster is ready
+ Scale out: Adding new node fails
David Ames (thedac)
Changed in charm-interface-mysql-innodb-cluster:
status: New → Triaged
importance: Undecided → High
assignee: nobody → David Ames (thedac)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-interface-mysql-innodb-cluster (master)

Fix proposed to branch: master
Review: https://review.opendev.org/728585

Changed in charm-interface-mysql-innodb-cluster:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-mysql-innodb-cluster (master)

Fix proposed to branch: master
Review: https://review.opendev.org/728587

Changed in charm-mysql-innodb-cluster:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-interface-mysql-innodb-cluster (master)

Reviewed: https://review.opendev.org/728585
Committed: https://git.openstack.org/cgit/openstack/charm-interface-mysql-innodb-cluster/commit/?id=5f87bb1ddec3868864d2a8d1bf9b66ae5dca4e48
Submitter: Zuul
Branch: master

commit 5f87bb1ddec3868864d2a8d1bf9b66ae5dca4e48
Author: David Ames <email address hidden>
Date: Fri May 15 15:17:59 2020 -0700

    Scale-in and Scale-out of mysql-innodb-cluster

    Handle adding and removing units to the mysql-innodb-cluster.

    Change-Id: Ie658c42b095bcff822cdfb0b771d41704ddc85ea
    Closes-Bug: #1877546
    Closes-Bug: #1874479

Changed in charm-interface-mysql-innodb-cluster:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-mysql-innodb-cluster (master)

Reviewed: https://review.opendev.org/728587
Committed: https://git.openstack.org/cgit/openstack/charm-mysql-innodb-cluster/commit/?id=7cbf9a88b001d73d290bb123f5bba010f51319b3
Submitter: Zuul
Branch: master

commit 7cbf9a88b001d73d290bb123f5bba010f51319b3
Author: David Ames <email address hidden>
Date: Wed May 13 11:37:31 2020 -0700

    Scale in and scale out

    Handle adding and removing units of the mysql-innodb-cluster gracefully.

    Review and land:
    https://github.com/openstack-charmers/zaza/pull/353
    https://github.com/openstack-charmers/zaza-openstack-tests/pull/275

    Closes-Bug: #1874479
    Closes-Bug: #1877546
    Closes-Bug: #1877616
    Depends-On: Ie658c42b095bcff822cdfb0b771d41704ddc85ea
    Change-Id: I39210c760e7204f84365d5dcb4eeedbc0889041f

Changed in charm-mysql-innodb-cluster:
status: In Progress → Fix Committed
David Ames (thedac)
Changed in charm-mysql-innodb-cluster:
milestone: none → 20.05
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.