Comment 3 for bug 1827668

Revision history for this message
Dmitrii Shcherbakov (dmitriis) wrote : Re: [19.04] keystone leader unit sometimes fails to add endpoints when "certificates" relation is present

The difference between the deployments where it goes well and where it does not is the time when a database migration is triggered (shared-db-relation-changed).

1) there is no error if a database migration is done after the certificates relation is handled and TLS is set up for Keystone;
2) the error occurs when a database migration is done before TLS is set up (this is equivalent to setting up keystone without TLS and then adding it).

The issue can be fixed by (1) introducing the usage of the goal-state hook tool for the new deployment case and (2) properly handling the conversion from running without TLS to running with TLS.

show-status-log for the lead keystone unit with a reproducer:
https://paste.ubuntu.com/p/36fJJpXJ4Q/ (full log)

10 May 2019 10:13:48Z juju-unit executing running start hook
10 May 2019 10:13:54Z workload blocked Missing relations: database <-----------------
# ...
10 May 2019 10:17:27Z juju-unit executing running shared-db-relation-changed hook
# ...
10 May 2019 10:19:45Z juju-unit executing running shared-db-relation-joined hook <-----------------
# ...
10 May 2019 10:25:52Z juju-unit executing running shared-db-relation-changed hook
10 May 2019 10:26:05Z workload maintenance Migrating the keystone database <-----------------
10 May 2019 10:27:06Z juju-unit executing running identity-service-relation-changed hook
10 May 2019 10:27:22Z juju-unit executing running shared-db-relation-changed hook
# ...

10 May 2019 10:38:52Z juju-unit executing running certificates-relation-joined hook <-----------------
10 May 2019 10:39:08Z juju-unit executing running identity-service-relation-joined hook
10 May 2019 10:39:21Z workload active Unit is ready
10 May 2019 10:39:23Z juju-unit executing running identity-service-relation-changed hook
10 May 2019 10:40:19Z juju-unit error hook failed: "identity-service-relation-changed" <-----------------

without a reproducer (different deployment):

https://paste.ubuntu.com/p/qX8csvqRWt/ (full log)

10 May 2019 21:32:23Z juju-unit executing running identity-service-relation-joined hook
10 May 2019 21:34:44Z juju-unit executing running certificates-relation-joined hook <-----------------
10 May 2019 21:35:03Z juju-unit executing running identity-service-relation-changed hook
10 May 2019 21:36:09Z juju-unit executing running certificates-relation-changed hook <-----------------
10 May 2019 21:36:33Z juju-unit executing running identity-service-relation-joined hook
10 May 2019 21:37:09Z juju-unit executing running keystone-fid-service-provider-relation-joined hook
10 May 2019 21:37:40Z juju-unit executing running identity-service-relation-joined hook
10 May 2019 21:38:35Z juju-unit executing running certificates-relation-joined hook <-----------------

# ...

10 May 2019 21:42:57Z juju-unit executing running certificates-relation-changed hook <-----------------

# ...

10 May 2019 21:51:07Z workload waiting Incomplete relations: database
10 May 2019 21:51:09Z juju-unit executing running shared-db-relation-joined hook <-----------------

# ...
10 May 2019 21:52:54Z juju-unit executing running shared-db-relation-changed hook
10 May 2019 21:53:08Z workload maintenance Migrating the keystone database