Point operators to logs when instance is not clustered

Bug #1917337 reported by David Ames
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MySQL InnoDB Cluster Charm
Fix Released
High
David Ames

Bug Description

Never report "None" as the cluster status. Rather, acknowledge the instance is not clustered and point the operator to the logs for further investigation.

There are a number of circumstances that can lead to an instance not successfully being joined to the cluster.

Example [0]

mysql-innodb-cluster/1 active executing 2 172.17.110.3 Unit is ready: Mode: R/W
mysql-innodb-cluster/2* active executing 3 172.17.110.33 Unit is ready: Mode: R/O
mysql-innodb-cluster/4 blocked idle 6 172.17.110.30 MySQL InnoDB Cluster not healthy: None

Example [1]

Unit Workload Agent Machine Public address Ports Message
mysql-innodb-cluster/0* blocked idle 0 10.5.0.7 MySQL InnoDB Cluster not healthy: Cluster has no quorum as visible from '10.5.0.7:3306' and cannot process write transactions. 2 members are not active
mysql-innodb-cluster/1 blocked idle 1 10.5.0.18 MySQL InnoDB Cluster not healthy: None
mysql-innodb-cluster/2 blocked idle 2 10.5.0.9 MySQL InnoDB Cluster not healthy: None

Currently, the workload status shows "MySQL InnoDB Cluster not healthy: None" which is not helpful. This also leads to bugs being filed on the work load status rather than the root cause of the failure to cluster.

The work load status of an unclustered instance should point the operator to investigate the logs or report the root cause error.

Note: The change will need to avoid false positives as much as possible for expected circumstances as in during deploy time or scale in scale out testing.

[0] https://bugs.launchpad.net/charm-mysql-innodb-cluster/+bug/1889792
[1] https://bugs.launchpad.net/charm-mysql-innodb-cluster/+bug/1917332

David Ames (thedac)
Changed in charm-mysql-innodb-cluster:
status: New → Triaged
importance: Undecided → High
Revision history for this message
David Ames (thedac) wrote :

PR to update workload status [0] has landed.

Now workload status points operators to logs if the instance cannot access the cluster. Workload status also shows cluster status text for more information on the state of the cluster.

[0] https://review.opendev.org/c/openstack/charm-mysql-innodb-cluster/+/779384

Changed in charm-mysql-innodb-cluster:
status: Triaged → Fix Committed
assignee: nobody → David Ames (thedac)
milestone: none → 21.04
Changed in charm-mysql-innodb-cluster:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.