Comment 17 for bug 1827668

I tested unsealing vault but _not_ starting the CA. Keystone started updating endpoints and then went into error state, this time a "cluster-relation-changed". No SSL certificates had been emited yet.

keystone/0* error idle 4/lxd/13 100.126.0.181 5000/tcp hook failed: "cluster-relation-changed"

Running the hook by hand inside a debug-hook session, the result was very similar to the error descibed in comment #13.

root@juju-f8838a-4-lxd-13:/var/lib/juju/agents/unit-keystone-0/charm# ./hooks/cluster-relation-changed
Internal Server Error (HTTP 500)
Internal Server Error (HTTP 500)
Internal Server Error (HTTP 500)
Internal Server Error (HTTP 500)
Internal Server Error (HTTP 500)
Internal Server Error (HTTP 500)
Traceback (most recent call last):
  File "./hooks/cluster-relation-changed", line 937, in <module>
    main()
  File "./hooks/cluster-relation-changed", line 930, in main
    hooks.execute(sys.argv)
  File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/core/hookenv.py", line 956, in execute
    self._hooks[hook_name]()
  File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/contrib/openstack/utils.py", line 1903, in wrapped_f
    return restart_on_change_helper(
  File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/core/host.py", line 837, in restart_on_change_helper
    r = lambda_f()
  File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/contrib/openstack/utils.py", line 1904, in <lambda>
    (lambda: f(*args, **kwargs)),
  File "./hooks/cluster-relation-changed", line 547, in cluster_changed
    update_all_identity_relation_units()
  File "./hooks/cluster-relation-changed", line 340, in update_all_identity_relation_units
    identity_changed(relation_id=rid, remote_unit=unit)
  File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/contrib/openstack/utils.py", line 1903, in wrapped_f
    return restart_on_change_helper(
  File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/core/host.py", line 837, in restart_on_change_helper
    r = lambda_f()
  File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/contrib/openstack/utils.py", line 1904, in <lambda>
    (lambda: f(*args, **kwargs)),
  File "./hooks/cluster-relation-changed", line 445, in identity_changed
    add_service_to_keystone(relation_id, remote_unit)
  File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1895, in add_service_to_keystone
    add_endpoint(region=settings['region'],
  File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 2116, in add_endpoint
    create_service_entry(service, service_type, desc,
  File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1001, in create_service_entry
    list_services = manager.list_services()
  File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1226, in __call__
    return _proxy_manager_call(self._path, self.api_version, args, kwargs)
  File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/core/decorators.py", line 40, in _retry_on_exception_inner_2
    return f(*args, **kwargs)
  File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1263, in _proxy_manager_call
    raise RetryProxyManagerCall()
keystone_utils.RetryProxyManagerCall

And the log file at /var/log/keystone/keystone-manage.log has:

2021-10-13 00:57:30.951 31813 ERROR keystone sqlite3.OperationalError: no such table: credential
[...]
2021-10-13 00:57:30.951 31813 ERROR keystone [SQL: SELECT credential.encrypted_blob AS credential_encrypted_blob, credential.id AS credential_id, credential.user_id AS credential_user_id, credential.project_id AS credential_project_id, credential.type AS credential_type, credential.key_hash AS credential_key_hash, credential.extra AS credential_extra
2021-10-13 00:57:30.951 31813 ERROR keystone FROM credential]

Debug-log for keystone unit shows (apart from same output as above):

unit-keystone-0: 01:26:12 ERROR unit.keystone/0.juju-log cluster:38: The call within manager.py failed with the error: 'Internal Server Error (HTTP 500)'. The call was: path=['list_services'], args=(), kwargs={}, api_version=None