secrets-storage relation stuck waiting

Bug #1870910 reported by Edward Hope-Morley
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Nova Compute Charm
Invalid
Undecided
Unassigned
vault-charm
Triaged
Medium
Unassigned

Bug Description

A freshly deployed nova-compute+vault is remaining stuck in waiting status saying there are no remote units on the secrets-storage relation despite that not being the case:

$ juju status nova-compute --relations
Model Controller Cloud/Region Version SLA Timestamp
octavia-ovn-testing hopem stsstack/stsstack 2.7.0 unsupported 12:15:37Z

App Version Status Scale Charm Store Rev OS Notes
nova-compute 20.0.1 waiting 1 nova-compute jujucharms 489 ubuntu
ovn-chassis 2.12.0 active 1 ovn-chassis jujucharms 23 ubuntu

Unit Workload Agent Machine Public address Ports Message
nova-compute/0* waiting idle 6 10.5.0.86 Incomplete relations: vault
  ovn-chassis/0* active idle 10.5.0.86 Unit is ready

Machine State DNS Inst id Series AZ Message
6 started 10.5.0.86 395c174e-ca73-41f7-99bd-f9ef15a6c1fd bionic nova ACTIVE

Relation provider Requirer Interface Type Message
glance:image-service nova-compute:image-service glance regular
nova-compute:cloud-compute nova-cloud-controller:cloud-compute nova-compute regular
nova-compute:compute-peer nova-compute:compute-peer nova peer
ovn-central:ovsdb ovn-chassis:ovsdb ovsdb regular
ovn-chassis:nova-compute nova-compute:neutron-plugin neutron-plugin subordinate
ovn-chassis:ovsdb-subordinate octavia:ovsdb-subordinate ovsdb-subordinate subordinate
rabbitmq-server:amqp nova-compute:amqp rabbitmq regular
vault:certificates ovn-chassis:certificates tls-certificates regular
vault:secrets nova-compute:secrets-storage vault-kv regular
$ juju run -a nova-compute -- relation-ids secrets-storage
secrets-storage:51
$ juju run -a nova-compute -- relation-list -r secrets-storage:51
vault/0
$ juju run -a nova-compute -- relation-get -r secrets-storage:51 - vault/0
egress-subnets: 10.5.0.8/32
ingress-address: 10.5.0.8
private-address: 10.5.0.8
vault_url: '"http://10.5.0.8:8200"'

And in the logs I have:

# tail /var/log/juju/unit-nova-compute-0.log
2020-04-05 12:11:52 INFO juju-log Updating status.
2020-04-05 12:11:54 INFO juju-log Vault dependencies met so including in status check
2020-04-05 12:11:55 DEBUG juju-log Generating template context for amqp
2020-04-05 12:11:55 DEBUG update-status ERROR no relation id specified
2020-04-05 12:11:55 DEBUG juju-log Generating template context for image-service.
2020-04-05 12:11:57 INFO juju-log Generated config context for neutron network manager.
2020-04-05 12:11:57 DEBUG juju-log 0 section(s) found
2020-04-05 12:11:58 DEBUG update-status none
2020-04-05 12:11:58 DEBUG juju-log Generating template context for cloud-credentials
2020-04-05 12:11:58 INFO juju-log vault relations's interface, secrets-storage, is related but has no units in the relation.

Revision history for this message
Edward Hope-Morley (hopem) wrote :

Further analsyis shows that the message "vault relations's interface, secrets-storage, is related but has no units in the relation." is misleading and is being show as a result of charm-helpers.contrib.openstack.vaultlocker.VaultKVContext().complete being false due to the fact that the related vault charm has not provided a token. I assume this a race issue on the vault charm side since a token should have been provided when vault was unsealed.

Revision history for this message
Edward Hope-Morley (hopem) wrote :

removing then adding the relation again fixed the problem i.e.

$ juju run -a nova-compute -- relation-get -r secrets-storage:59 - vault/0
egress-subnets: 10.5.0.8/32
ingress-address: 10.5.0.8
nova-compute/0_role_id: '"e1d47084-579a-afe1-326e-ec643b5a1b87"'
nova-compute/0_token: '"s.B3Ynbrb2hz1omo5BPQUH2oAF"'
private-address: 10.5.0.8
vault_url: '"http://10.5.0.8:8200"'

Changed in vault-charm:
milestone: none → 20.05
Changed in charm-nova-compute:
status: New → Invalid
Changed in vault-charm:
importance: Undecided → Medium
status: New → Triaged
David Ames (thedac)
Changed in vault-charm:
milestone: 20.05 → 20.08
James Page (james-page)
Changed in vault-charm:
milestone: 20.08 → none
Revision history for this message
Cory Johns (johnsca) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.