Scale out handler does not have enough guarding flags

Bug #1915062 reported by David Ames
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MySQL InnoDB Cluster Charm
Fix Released
High
David Ames

Bug Description

Related to LP Bug #1914299 and the attempt to handle departed hooks [1] that *may* be reverted.

The reactive.mysql_innodb_cluster_handlers.scale_out function may be called when the functions it calls do not have all of their required flags set:
    create_remote_cluster_user()
    configure_instances_for_clustering()
    add_instances_to_cluster()

Leading to:

2021-02-06 22:29:08 INFO juju-log Invoking reactive handler: reactive/mysql_innodb_cluster_handlers.py:291:scale_out
2021-02-06 22:29:08 ERROR juju-log Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-2/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-2/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-2/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-2/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-2/charm/reactive/mysql_innodb_cluster_handlers.py", line 310, in scale_out
    create_remote_cluster_user()
  File "/var/lib/juju/agents/unit-mysql-innodb-cluster-2/charm/reactive/mysql_innodb_cluster_handlers.py", line 105, in create_remote_cluster_user
    for unit in cluster.all_joined_units:
AttributeError: 'NoneType' object has no attribute 'all_joined_units'

Also related is the discussion on issue/231 [2]. That endpoint_from_flag may return None.

We should audit the charm for these edge cases and guard against None from endpoint_from_flag.

A comprehensive solution for departed/broken relationships is needed.

[0] https://bugs.launchpad.net/charm-mysql-router/+bug/1914299
[1] https://review.opendev.org/c/openstack/charm-mysql-innodb-cluster/+/774033
[2] https://github.com/juju-solutions/charms.reactive/issues/231

David Ames (thedac)
Changed in charm-mysql-innodb-cluster:
status: New → Triaged
importance: Undecided → High
Revision history for this message
David Ames (thedac) wrote :

Added the 'cluster.available' flag to the scale_out handler [0]

[0] https://review.opendev.org/c/openstack/charm-mysql-innodb-cluster/+/774356

Changed in charm-mysql-innodb-cluster:
assignee: nobody → David Ames (thedac)
milestone: none → 21.04
status: Triaged → Fix Committed
Changed in charm-mysql-innodb-cluster:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.