incomplete relations too often

See juju status attached for example. The affected apps are usually random, most of the time I get placement and vault affected, sometimes cinder and glance

Added charm-percona-cluster as affected as well because it seems all the affected units are affected in the way they relate to percona-cluster (mysql in my bundle)

to fix the deployment which I pasted the juju status from I had to remove relation between vault and mysql, placement and mysql, keystone and mysql, and re-add all of them again (in the same order, one by one). Those are the only steps I took to fix all issues in that juju status and now the deployment is working fine.

This exact same issue happens when adding new placement units. Tested with charm rev. 32 on focal/xena.

What does "juju status --relations" show? We need to determine whether the relations are "incomplete" because juju isn't making them properly, ie they do not show as "Joined" in status, or is it that charms aren't properly processing the async events which are triggered as part of standing up the model.

very good point, I should have attached a juju status --relations instead :facepalm:

since I fixed my deployment for the purpose of testing something else, I will be doing that today and then I can tear it down and redeploy to hit the same issue, and then attach a new juju status with --relations.

But in the meantime what I can say is that I had checked the juju status --relations and none of them (at that time) were stuck in "joining", I had also checked the relation data using juju show-unit for the affected units.

Re-deployed env and captured juju status --relations (attached).

This time keystone was not affected by the issue. I also don't see the "joined"/"joining" values in the output.

we've been flagging test runs in the SQA lab with this bug lately as we've seen a large number of cases where all but one unit in a service is up and running, and the one unit that isn't running says its missing a relation.

Most often these services are mysql-router's which are not getting the shared-db relation to finish or trigger a hook.

Here are some logs from a recent example, where neutron-api-mysql-router/0 (the leader unit) is waiting on "'shared-db' incomplete, Waiting for proxied DB creation from cluster" for a little over 3 hours.

We've recently added more logs to the juju-crashdump as well. Each units folder now has logs for juju show-machine and juju show-unit as well as the juju show-status-log.

Crashdump: https://oil-jenkins.canonical.com/artifacts/6834065e-9050-48a1-9da2-660752d77bba/generated/generated/openstack/juju-crashdump-openstack-2022-05-03-15.01.53.tar.gz

Controller Crashdump:https://oil-jenkins.canonical.com/artifacts/6834065e-9050-48a1-9da2-660752d77bba/generated/generated/juju_maas_controller/juju-crashdump-controller-2022-05-03-15.12.42.tar.gz

Testrun:https://solutions.qa.canonical.com/testruns/testRun/6834065e-9050-48a1-9da2-660752d77bba
All the logs for the run can be found at the "view artifacts for this run" at the bottom of the page.

We can see that the ovn-chassis side had run the relation hooks, but not the ovn-central side.

We need some additional information to diagnose this further - a database dump and extra tracing.

If possible, "juju dump-db -m <model>" would be good (you need JUJU_DEV_FEATURE_FLAGS set to "developer-mode"). But at the least, a json dump of the collections is needed.

Also required is trace level logging for these loggers:

juju.state.relationunits
juju.worker.uniter.operation
juju.worker.uniter.resolved
juju.worker.uniter.relation

The above will help us diagnose why units on one side of the relation do not get to run the relation hooks.

Folling up with some logs
there are 2 major places we hit this, 1 is with ha cluster
and the other is with mysql-innodb-router

in these logs (with the extra logging above)vault-mysql-router is stuck waiting for the shared-db relation, but everything else reached a stable state:

crashdump: https://oil-jenkins.canonical.com/artifacts/09759a4c-0088-4bf4-9c97-82288f85f940/generated/generated/openstack/juju-crashdump-openstack-2022-09-06-19.30.38.tar.gz
db-dump: https://oil-jenkins.canonical.com/artifacts/09759a4c-0088-4bf4-9c97-82288f85f940/generated/generated/openstack/juju-dump-db-openstack-2022-09-06-19.30.38.tar.gz

testrun: https://solutions.qa.canonical.com/testruns/testRun/09759a4c-0088-4bf4-9c97-82288f85f940

Was there a controller crash-dump for the run above?

Sorry for late reply, just listed a few runs with this bug, and crashdumps could be found there.

https://oil-jenkins.canonical.com/artifacts/90901686-39b0-472b-9bb1-59cd55f09ac0/index.html
https://oil-jenkins.canonical.com/artifacts/a46236b6-d7b9-423c-a453-ee63d384160f/index.html
https://oil-jenkins.canonical.com/artifacts/b63655d7-a61e-4322-bbcc-aa9f4be0be64/index.html
https://oil-jenkins.canonical.com/artifacts/1ab7e88f-d286-4efd-89a3-56551fd7438d/index.html

I don't think this is a charm bug as it's also reported as affecting other charms 'randomly'. Please feel free to re-open if you have more information showing that it's more likely to be a charm bug.

Checked a handful recent occurrences with juju 2.9.43/44 on openstack yoga.
The applications has "incomplete relations" are the same for all of them.

ceilometer 18.0.0 blocked 3 ceilometer yoga/stable 527 no Incomplete relations: database, Run the ceilometer-upgrade action on the leader to initialize ceilometer and gnocchi
ceph-radosgw 17.2.5 waiting 3 ceph-radosgw quincy/stable 548 no Incomplete relations: mon
glance 24.2.0 waiting 3 glance yoga/stable 562 no Incomplete relations: storage-backend
nova-compute 25.1.1 waiting 6 nova-compute yoga/stable 664 no Incomplete relations: storage-backend, vault

The next 2.9.46 candidate release will not include a fix for this bug and we don't plan on any more 2.9 releases. As such it is being removed from its 2.9 milestone.

If the bug is still important to you, let us know and we can consider it for inclusion on a 3.x milestone.

@wallyworld
SolQA still see this in many test runs with Juju 3.x, like 2~3 times a week.
Please reconsider this for 3.x.

You might find all runs from https://solutions.qa.canonical.com/bugs/1967177

Thanks for the input. We'll look to see how future test runs behave and once we can gather enough evidence to understand the root cause, can assign the bug to the next release milestone.

This is affecting us as well... I can see clearly in the juju logs on the unit that it fails to connect:

2024-02-29 00:27:01 ERROR unit.placement-mysql-router/5.juju-log server.go:316 db-router:70: Failed to bootstrap mysqlrouter: Error: Unable to connect to the metadata server: Error connecting to MySQL server at 10.100.33.27:0: Access denied for user 'mysqlrouteruser'@'10.100.33.2' (using password: YES) (1045)

But looking in the database... the grants look right. At the end of all of this we even see an established connection to the RW member of the DB cluster.

But... it's clearly not progressing beyond the bootstrap:
[metadata_cache:bootstrap]
cluster_type = gr
router_id = 35
user = mysql_router35_jboetej22v83
metadata_cluster = jujuCluster
ttl = 0.5
auth_cache_ttl = -1
auth_cache_refresh_interval = 2
use_gr_notifications = 0

[routing:bootstrap_rw]
bind_address = 127.0.0.1
bind_port = 3306
socket = /var/lib/mysql/placement-mysql-router/mysql.sock
destinations = metadata-cache://jujuCluster/?role=PRIMARY
routing_strategy = first-available
protocol = classic

[routing:bootstrap_ro]
bind_address = 127.0.0.1
bind_port = 3307
socket = /var/lib/mysql/placement-mysql-router/mysqlro.sock
destinations = metadata-cache://jujuCluster/?role=SECONDARY
routing_strategy = round-robin-with-fallback
protocol = classic

[routing:bootstrap_x_rw]
bind_address = 127.0.0.1
bind_port = 3308
socket = /var/lib/mysql/placement-mysql-router/mysqlx.sock
destinations = metadata-cache://jujuCluster/?role=PRIMARY
routing_strategy = first-available
protocol = x

[routing:bootstrap_x_ro]
bind_address = 127.0.0.1
bind_port = 3309
socket = /var/lib/mysql/placement-mysql-router/mysqlxro.sock
destinations = metadata-cache://jujuCluster/?role=SECONDARY
routing_strategy = round-robin-with-fallback
protocol = x