MySQL InnoDB Cluster Charm

AttributeError: 'NoneType' object has no attribute 'all_joined_units'

Bug #1896809 reported by Felipe Reyes
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MySQL InnoDB Cluster Charm
Fix Released
High
David Ames
MySQL InnoDB Cluster Charm 21.04

Bug Description

This happens when destroying a model and the peers already went away making reactive.endpoint_from_flag("cluster.available") return None.

unit-mysql-1: 16:54:20 ERROR unit.mysql/1.juju-log shared-db:13: Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mysql-1/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-mysql-1/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-mysql-1/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-mysql-1/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-mysql-1/charm/reactive/mysql_innodb_cluster_handlers.py", line 305, in scale_out
    create_remote_cluster_user()
  File "/var/lib/juju/agents/unit-mysql-1/charm/reactive/mysql_innodb_cluster_handlers.py", line 102, in create_remote_cluster_user
    for unit in cluster.all_joined_units:
AttributeError: 'NoneType' object has no attribute 'all_joined_units'

unit-mysql-1: 16:54:20 ERROR juju.worker.uniter.operation hook "shared-db-relation-departed" (via explicit, bespoke hook script) failed: exit status 1

 juju status
Model Controller Cloud/Region Version SLA Timestamp Notes
octavia freyes stsstack/stsstack 2.8.1 unsupported 17:00:35Z attempt 20 to destroy model failed (will retry): model not empty, found 1 machine, 7 applications (model not empty)

App Version Status Scale Charm Store Rev OS Notes
glance unknown 0 glance jujucharms 429 ubuntu
keystone waiting 0 keystone jujucharms 513 ubuntu
mysql 8.0.21 error 1 mysql-innodb-cluster jujucharms 48 ubuntu
neutron-api unknown 0 neutron-api jujucharms 455 ubuntu
nova-cloud-controller unknown 0 nova-cloud-controller jujucharms 499 ubuntu
octavia unknown 0 octavia jujucharms 84 ubuntu
placement unknown 0 placement jujucharms 28 ubuntu

Unit Workload Agent Machine Public address Ports Message
mysql/1* error idle 4 10.5.0.140 hook failed: "shared-db-relation-departed"

Machine State DNS Inst id Series AZ Message
4 stopped 10.5.0.140 ec618e8b-7b73-4cc7-9132-e48c4ff1ae1f focal nova ACTIVE

Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

I think this is a bug in charms.reactive: https://github.com/juju-solutions/charms.reactive/issues/231

I'm going to work around it in the charm to ensure that it doesn't crash if the return value from endpoint_from_flag() is None.

Changed in charm-mysql-innodb-cluster:
assignee: nobody → Alex Kavanagh (ajkavanagh)
status: New → In Progress
Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

This review is a workaround / fix for the crashing issue (endpoint_from_flag returns None in handler where the relation 'should' exist).

https://review.opendev.org/c/openstack/charm-mysql-innodb-cluster/+/774376

Revision history for this message
Cory Johns (johnsca) wrote :

I'd recommend changing the interface layer [1] to use Endpoint.manage_flags() [2] to avoid this race condition (the issue being that the interface layer's @when handlers which manage the cluster.available flag aren't guaranteed to be run before the charm's handlers which use it, while manage_flags is).

[1]: https://github.com/openstack/charm-interface-mysql-innodb-cluster/blob/master/peers.py
[2]: https://charmsreactive.readthedocs.io/en/latest/charms.reactive.relations.html#charms.reactive.endpoints.Endpoint.manage_flags

Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

Assigned David as he is working on a comprehensive solution to the issue.

Changed in charm-mysql-innodb-cluster:
assignee: Alex Kavanagh (ajkavanagh) → nobody
status: In Progress → Triaged
importance: Undecided → High
assignee: nobody → David Ames (thedac)
Revision history for this message
David Ames (thedac) wrote :

This is in progress [0]. The issue was create_remote_cluster_user being called in scale_out when we did not have cluster.available.

[0] https://review.opendev.org/c/openstack/charm-mysql-innodb-cluster/+/774356

Changed in charm-mysql-innodb-cluster:
status: Triaged → In Progress
Revision history for this message
David Ames (thedac) wrote :

The PR adding the correct flag to the scale out handler has landed:

https://review.opendev.org/c/openstack/charm-mysql-innodb-cluster/+/774356

Changed in charm-mysql-innodb-cluster:
status: In Progress → Fix Committed
milestone: none → 21.04
Alex Kavanagh (ajkavanagh)
Changed in charm-mysql-innodb-cluster:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on charm-mysql-innodb-cluster (master)

Change abandoned by "Alex Kavanagh <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/charm-mysql-innodb-cluster/+/774376
Reason: Overtaken by events (and other patches)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.