This bug is about test failures for jammy-caracal, jammy-bobcat, and jammy-antelope in cherry-pick commits from this change:
https://review.opendev.org/c/openstack/charm-keystone-openidc/+/922049
That change fixed some bugs in the Keystone OpenIDC charm and added some additional configuration options to help with proxies.
The tests all fail with a JSONDecodeError during the Zaza tests for the Keystone OpenIDC charm. Here is an example of the error:
Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
File "/home/jadon/py3-venv/lib/python3.10/site-packages/requests/models.py", line 974, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/lib/python3.10/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/jadon/py3-venv/lib/python3.10/site-packages/cliff/app.py", line 414, in run_subcommand
self.prepare_to_run_command(cmd)
File "/home/jadon/py3-venv/lib/python3.10/site-packages/osc_lib/shell.py", line 516, in prepare_to_run_command
self.client_manager.auth_ref
File "/home/jadon/py3-venv/lib/python3.10/site-packages/osc_lib/clientmanager.py", line 208, in auth_ref
self._auth_ref = self.auth.get_auth_ref(self.session)
File "/home/jadon/py3-venv/lib/python3.10/site-packages/keystoneauth1/identity/v3/federation.py", line 62, in get_auth_ref
auth_ref = self.get_unscoped_auth_ref(session)
File "/home/jadon/py3-venv/lib/python3.10/site-packages/keystoneauth1/identity/v3/oidc.py", line 293, in get_unscoped_auth_ref
return access.create(resp=response)
File "/home/jadon/py3-venv/lib/python3.10/site-packages/keystoneauth1/access/access.py", line 36, in create
body = resp.json()
File "/home/jadon/py3-venv/lib/python3.10/site-packages/requests/models.py", line 978, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
clean_up ListServer: Expecting value: line 1 column 1 (char 0)
END return value: 1
According to debug output, the failure happens during the OIDC authentication flow. Testing using the OpenStack CLI shows the failure happen right after this request:
REQ: curl -g -i --insecure -X POST https://10.70.143.111:5000/v3/OS-FEDERATION/identity_providers/keycloak/protocols/openid/auth -H "Authorization: {SHA256}45dbb29ea555e0bd24995cbb1481c8ac66c2d03383bc0c335be977d0daaf6959" -H "User-Agent: openstacksdk/3.3.0 keystoneauth1/5.7.0 python-requests/2.32.3 CPython/3.10.12"
Starting new HTTPS connection (1): 10.70.143.111:5000
RESP: [200] Connection: Keep-Alive Content-Length: 0 Date: Tue, 30 Jul 2024 19:28:17 GMT Keep-Alive: timeout=75, max=1000 Server: Apache/2.4.52 (Ubuntu)
RESP BODY: Omitted, Content-Type is set to None. Only text/plain, application/json responses have their bodies logged.
This request is unusual in that the request is a POST request with no request body, and the response is an empty response. The empty response causes the JSONDecodeError because the keystoneauth package expects a JSON document to return from the request for a Keystone token. The empty response causes the JSONDecodeError because an empty string is not a valid document.
This strange behavior happens due to a misconfiguration in the mod_auth_openidc Apache configuration. I looked up how Kolla-Ansible configures OpenIDC in Keystone, and I noticed that they used a different value for the OIDCRedirectURI in the mod_auth_openidc Apache configuration than the Keystone OpenIDC charm. The value of OIDCRedirectURI is supposed to be a fake URI that does not map to any real URI in the protected service. The fake URI should be protected by mod_auth_openidc in Apache's configuration. When someone accesses the path in OIDCRedirectURI mod_auth_openidc handles the request and returns an empty response.
In our configuration, the OIDCRedirectURI is the same URI as the Keystone federation protocol authentication endpoint. Keystone generates the federation protocol authentication endpoint dynamically based on the identity provider and federation protocol configured in Keystone. Here is an example of a federation protocol endpoint for a Keystone identity provider named "keycloak" using a Keystone federation protocol called "openid":
/v3/OS-FEDERATION/identity_providers/keycloak/protocols/openid/auth
When Keystone is attempting to access the Keystone federation protocol authentication endpoint, mod_auth_openidc is intercepting the request and returning an empty response. This is blocking Keystone from properly handling the OIDC authentication flow.
If you change OIDCRedirectURI to something other than your federation protocol authentication endpoint, Keystone can complete the OIDC authentication flow successfully.
This contradicts the configuration outlined in the official Keystone federation configuration documentation at https://docs.openstack.org/keystone/latest/admin/federation/configure_federation.html#configure-mod-auth-openidc.
Here is a corrected Apache configuration for mod_auth_openidc that should illustrate the problem and the solution.