OpenStack Placement Charm

Juju lease acquired by non-existent unit

Bug #2043621 reported by Alan Baghumian
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Canonical Juju
Triaged
Medium
Joseph Phillips
OpenStack Placement Charm
Invalid
Undecided
Unassigned

Bug Description

Hello OpenStack Team,

While shuffling placement units in my focal/yoga cloud today, I noticed the newly deployed Placement units are missing TLS certificates despite the relation to Vault being present.

I am running Juju 2.9.37, Placement charm rev 94, in yoga/stable channel.

Looking at the tls-certificates in the --relations is showing "joining" (See attachment)

Removing and adding the Vault relation does not seem to do anything:

2023-11-15 21:42:03 INFO unit.placement/11.juju-log server.go:316 certificates:444: Invoking reactive handler: reactive/placement_handlers.py:82:cluster_connected
2023-11-15 21:42:03 INFO unit.placement/11.juju-log server.go:316 certificates:444: Invoking reactive handler: reactive/layer_openstack_api.py:20:default_setup_database
2023-11-15 21:42:03 INFO unit.placement/11.juju-log server.go:316 certificates:444: Invoking reactive handler: reactive/layer_openstack_api.py:37:default_setup_endpoint_connection
2023-11-15 21:42:03 INFO unit.placement/11.juju-log server.go:316 certificates:444: Invoking reactive handler: reactive/layer_openstack_api.py:54:default_update_peers
2023-11-15 21:42:03 INFO unit.placement/11.juju-log server.go:316 certificates:444: Invoking reactive handler: hooks/relations/placement/provides.py:26:joined:placement
2023-11-15 21:42:03 INFO unit.placement/11.juju-log server.go:316 certificates:444: Invoking reactive handler: hooks/relations/tls-certificates/requires.py:79:joined:certificates
2023-11-15 21:42:04 INFO unit.placement/11.juju-log server.go:316 certificates:444: DEPRECATION WARNING: Function _ows_check_services_running is being removed on/around 2022-05 : use ows_check_services_running() instead
2023-11-15 21:42:04 INFO juju.worker.uniter.operation runhook.go:146 ran "certificates-relation-changed" hook (via explicit, bespoke hook script)
2023-11-15 21:44:58 INFO unit.placement/11.juju-log server.go:316 Reactive main running for hook update-status
2023-11-15 21:44:58 INFO unit.placement/11.juju-log server.go:316 Initializing Leadership Layer (is leader)
2023-11-15 21:44:58 INFO unit.placement/11.juju-log server.go:316 Invoking reactive handler: reactive/layer_openstack.py:64:default_update_status
2023-11-15 21:44:58 INFO unit.placement/11.juju-log server.go:316 Invoking reactive handler: reactive/layer_openstack.py:82:check_really_is_update_status
2023-11-15 21:44:58 INFO unit.placement/11.juju-log server.go:316 Invoking reactive handler: reactive/layer_openstack.py:93:run_default_update_status
2023-11-15 21:44:58 INFO unit.placement/11.juju-log server.go:316 Invoking reactive handler: reactive/layer_openstack_api.py:20:default_setup_database
2023-11-15 21:44:58 INFO unit.placement/11.juju-log server.go:316 Invoking reactive handler: reactive/layer_openstack_api.py:37:default_setup_endpoint_connection
2023-11-15 21:44:58 INFO unit.placement/11.juju-log server.go:316 Invoking reactive handler: reactive/layer_openstack_api.py:54:default_update_peers
2023-11-15 21:44:58 INFO unit.placement/11.juju-log server.go:316 Invoking reactive handler: hooks/relations/placement/provides.py:26:joined:placement
2023-11-15 21:44:58 INFO unit.placement/11.juju-log server.go:316 Invoking reactive handler: hooks/relations/tls-certificates/requires.py:79:joined:certificates

Vault units are unsealed and healthy (See attachment).

Please let me know if you'd like me to provide logs etc.

Thank you,
Alan

Revision history for this message
Alan Baghumian (alanbach) wrote :
Revision history for this message
Felipe Reyes (freyes) wrote : Re: [Bug 2043621] [NEW] New placement units fail to fetch TLS certificates from Vault

On Wed, 2023-11-15 at 21:58 +0000, Alan Baghumian wrote:
> Public bug reported:
>
> Hello OpenStack Team,
>
> While shuffling placement units in my focal/yoga cloud today, I noticed
> the newly deployed Placement units are missing TLS certificates despite
> the relation to Vault being present.
>
> I am running Juju 2.9.37, Placement charm rev 94, in yoga/stable
> channel.
>
> Looking at the tls-certificates in the --relations is showing "joining"
> (See attachment)

is a there a hook running in the unit?, what's the output of "juju show-status-log placement/11"?

Revision history for this message
Alan Baghumian (alanbach) wrote : Re: New placement units fail to fetch TLS certificates from Vault

This issue turned out to be related to https://bugs.launchpad.net/juju/+bug/1942354

Please see comment #18.

After fixing the Vault leadership issue, and removing and adding the vault:certificates relationship, the new units properly received their SSL certificates.

Changed in charm-placement:
status: New → Invalid
Felipe Reyes (freyes)
summary: - New placement units fail to fetch TLS certificates from Vault
+ Juju lease acquired by non-existent unit
Revision history for this message
Joseph Phillips (manadart) wrote :

To be clear, this issue is caused by OS upgrades that get stuck.

When the prepare stage runs, leases for applications with units on the machine are "pinned" so that they do not expire.

If the upgrade can not be completed and forced removal is used, we've seen the pin persisting (for a unit now gone), though this should have since been addressed.

We know that OS upgrades need rework following the Charmhub switch, but that work has not made it onto cycle priorities so far.

This will likely get addressed at that time, rather than as an exercise specific to this bug.

Changed in juju:
status: New → Triaged
importance: Undecided → Medium
assignee: nobody → Joseph Phillips (manadart)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.