I've been trying to figure out a solution here. We need to somehow get `project_domain_id` or `project_domain` into the context here: 2022-04-12 23:43:07.956 151261 WARNING cinder.compute.nova [req-a23f3da3-3004-4798-b248-3348fe1309ef ff32a8ed05614f7a8f615be6da92fa18 91468a8dcfa94fa98db004ff4c175a9b - - -] XXX context = {'user': 'ff32a8ed05614f7a8f615be6da92fa18', 'tenant': '91468a8dcfa94fa98db004ff4c175a9b', 'system_scope': None, 'project': '91468a8dcfa94fa98db004ff4c175a9b', 'domain': None, 'user_domain': None, 'project_domain': None, 'is_admin': True, 'read_only': False, 'show_deleted': False, 'auth_token': 'gAAAAABiVg6LQxoGaMiTVufoGBn1nA7drrEM9k4cf4CGNtZyM-iw68RLEO2f9OPZ_0_6WzD7BLwPYSO9XpKUBoVVDchWqcefKctRcsb_wStkK4B4eJd64pTxEC6lSH5iExWwUpH712j_XNdjo8lwMY_Cr2xjwAWBtqZo9fb7WiqLp7P1vSxN27E', 'request_id': 'req-a23f3da3-3004-4798-b248-3348fe1309ef', 'global_request_id': 'req-a718a415-0414-4a86-b4bd-87c8f5c832e4', 'resource_uuid': None, 'roles': ['Admin', 'reader', 'member', 'load-balancer_admin', 'admin'], 'user_identity': 'ff32a8ed05614f7a8f615be6da92fa18 91468a8dcfa94fa98db004ff4c175a9b - - -', 'is_admin_project': True, 'user_id': 'ff32a8ed05614f7a8f615be6da92fa18', 'project_id': '91468a8dcfa94fa98db004ff4c175a9b', 'project_name': 'admin', 'domain_id': None, 'read_deleted': 'no', 'remote_address': '192.168.151.129', 'timestamp': '2022-04-12T23:43:07.378961+00:00', 'quota_class': None, 'service_catalog': [{'type': 'identity', 'name': 'keystone', 'endpoints': [{'region': 'RegionOne', 'adminURL': 'https://192.168.151.156:35357/v3', 'publicURL': 'https://192.168.151.156:5000/v3', 'internalURL': 'https://192.168.151.156:5000/v3'}]}, {'type': 'compute', 'name': 'nova', 'endpoints': [{'region': 'RegionOne', 'publicURL': 'https://192.168.151.108:8774/v2.1', 'internalURL': 'https://192.168.151.108:8774/v2.1', 'adminURL': 'https://192.168.151.108:8774/v2.1'}]}, {'type': 'object-store', 'name': 'swift', 'endpoints': [{'region': 'RegionOne', 'adminURL': 'https://192.168.151.138:443/swift', 'internalURL': 'https://192.168.151.138:443/swift/v1', 'publicURL': 'https://192.168.151.138:443/swift/v1'}]}, {'type': 'key-manager', 'name': 'barbican', 'endpoints': [{'region': 'RegionOne', 'internalURL': 'https://192.168.151.109:9311', 'publicURL': 'https://192.168.151.109:9311', 'adminURL': 'https://192.168.151.109:9312'}]}, {'type': 'image', 'name': 'glance', 'endpoints': [{'region': 'RegionOne', 'internalURL': 'https://192.168.151.110:9292', 'publicURL': 'https://192.168.151.110:9292', 'adminURL': 'https://192.168.151.110:9292'}]}]} I could trace the context as far back as in oslo messaging rpc message received: 2022-04-12 23:43:07.506 151261 WARNING oslo_messaging.rpc.server [-] ZZZ message.ctxt = {'user': 'ff32a8ed05614f7a8f615be6da92fa18', 'tenant': '91468a8dcfa94fa98db004ff4c175a9b', 'system_scope': None, 'project': '91468a8dcfa94fa98db004ff4c175a9b', 'domain': None, 'user_domain': None, 'project_domain': None, 'is_admin': True, 'read_only': False, 'show_deleted': False, 'auth_token': 'gAAAAABiVg6LQxoGaMiTVufoGBn1nA7drrEM9k4cf4CGNtZyM-iw68RLEO2f9OPZ_0_6WzD7BLwPYSO9XpKUBoVVDchWqcefKctRcsb_wStkK4B4eJd64pTxEC6lSH5iExWwUpH712j_XNdjo8lwMY_Cr2xjwAWBtqZo9fb7WiqLp7P1vSxN27E', 'request_id': 'req-a23f3da3-3004-4798-b248-3348fe1309ef', 'global_request_id': 'req-a718a415-0414-4a86-b4bd-87c8f5c832e4', 'resource_uuid': None, 'roles': ['Admin', 'reader', 'member', 'load-balancer_admin', 'admin'], 'user_identity': 'ff32a8ed05614f7a8f615be6da92fa18 91468a8dcfa94fa98db004ff4c175a9b - - -', 'is_admin_project': True, 'user_id': 'ff32a8ed05614f7a8f615be6da92fa18', 'project_id': '91468a8dcfa94fa98db004ff4c175a9b', 'project_name': 'admin', 'domain_id': None, 'read_deleted': 'no', 'remote_address': '192.168.151.129', 'timestamp': '2022-04-12T23:43:07.378961+00:00', 'quota_class': None, 'service_catalog': [{'type': 'identity', 'name': 'keystone', 'endpoints': [{'region': 'RegionOne', 'adminURL': 'https://192.168.151.156:35357/v3', 'publicURL': 'https://192.168.151.156:5000/v3', 'internalURL': 'https://192.168.151.156:5000/v3'}]}, {'type': 'compute', 'name': 'nova', 'endpoints': [{'region': 'RegionOne', 'publicURL': 'https://192.168.151.108:8774/v2.1', 'internalURL': 'https://192.168.151.108:8774/v2.1', 'adminURL': 'https://192.168.151.108:8774/v2.1'}]}, {'type': 'object-store', 'name': 'swift', 'endpoints': [{'region': 'RegionOne', 'adminURL': 'https://192.168.151.138:443/swift', 'internalURL': 'https://192.168.151.138:443/swift/v1', 'publicURL': 'https://192.168.151.138:443/swift/v1'}]}, {'type': 'key-manager', 'name': 'barbican', 'endpoints': [{'region': 'RegionOne', 'internalURL': 'https://192.168.151.109:9311', 'publicURL': 'https://192.168.151.109:9311', 'adminURL': 'https://192.168.151.109:9312'}]}, {'type': 'image', 'name': 'glance', 'endpoints': [{'region': 'RegionOne', 'internalURL': 'https://192.168.151.110:9292', 'publicURL': 'https://192.168.151.110:9292', 'adminURL': 'https://192.168.151.110:9292'}]}], 'client_timeout': None} Context is built somewhere, external to cinder, and sent via amqp message? Need to figure out how this context is built. Sometimes in context there is `'project_domain': None`, and sometimes it's `'project_domain': 'c2a7d9d1e315405ca054dda569aefa93'` (where the value is the correct domain id). eg.: 2022-04-12 23:43:00.430 151261 WARNING oslo_messaging.rpc.server [-] ZZZ message.ctxt = {'user': 'ff32a8ed05614f7a8f615be6da92fa18', 'tenant': '91468a8dcfa94fa98db004ff4c175a9b', 'system_scope': None, 'project': '91468a8dcfa94fa98db004ff4c175a9b', 'domain': None, 'user_domain': 'c2a7d9d1e315405ca054dda569aefa93', 'project_domain': 'c2a7d9d1e315405ca054dda569aefa93', 'is_admin': True, 'read_only': False, 'show_deleted': False, 'auth_token': 'gAAAAABiVg6E4AmOEgJHlkHFibNFAlt3TeIbfBdLIO-jiyDVQVfjCchNoGVsb1LpOchEaTWPqGcNE1HlGkGakNkgfACpHPoWfyivaemK9Ke7JNTQDDJVSHwciLtcNN_WgWeeKU9E1BOjNzVaX6OdVwJpnnZ0PXjwg_G4OESery2cdCQ1HgRKt5c', 'request_id': 'req-f9c43b35-5153-42b0-bb30-ed878428b5c5', 'global_request_id': 'req-f79b421a-998b-464c-929a-3d93177e06b4', 'resource_uuid': None, 'roles': ['Admin', 'reader', 'member', 'load-balancer_admin'], 'user_identity': 'ff32a8ed05614f7a8f615be6da92fa18 91468a8dcfa94fa98db004ff4c175a9b - c2a7d9d1e315405ca054dda569aefa93 c2a7d9d1e315405ca054dda569aefa93', 'is_admin_project': True, 'user_id': 'ff32a8ed05614f7a8f615be6da92fa18', 'project_id': '91468a8dcfa94fa98db004ff4c175a9b', 'project_name': 'admin', 'domain_id': None, 'read_deleted': 'no', 'remote_address': '192.168.151.129', 'timestamp': '2022-04-12T23:43:00.333440', 'quota_class': None, 'service_catalog': [{'type': 'identity', 'name': 'keystone', 'endpoints': [{'region': 'RegionOne', 'adminURL': 'https://192.168.151.156:35357/v3', 'publicURL': 'https://192.168.151.156:5000/v3', 'internalURL': 'https://192.168.151.156:5000/v3'}]}, {'type': 'compute', 'name': 'nova', 'endpoints': [{'region': 'RegionOne', 'publicURL': 'https://192.168.151.108:8774/v2.1', 'internalURL': 'https://192.168.151.108:8774/v2.1', 'adminURL': 'https://192.168.151.108:8774/v2.1'}]}, {'type': 'object-store', 'name': 'swift', 'endpoints': [{'region': 'RegionOne', 'adminURL': 'https://192.168.151.138:443/swift', 'internalURL': 'https://192.168.151.138:443/swift/v1', 'publicURL': 'https://192.168.151.138:443/swift/v1'}]}, {'type': 'key-manager', 'name': 'barbican', 'endpoints': [{'region': 'RegionOne', 'internalURL': 'https://192.168.151.109:9311', 'publicURL': 'https://192.168.151.109:9311', 'adminURL': 'https://192.168.151.109:9312'}]}, {'type': 'image', 'name': 'glance', 'endpoints': [{'region': 'RegionOne', 'internalURL': 'https://192.168.151.110:9292', 'publicURL': 'https://192.168.151.110:9292', 'adminURL': 'https://192.168.151.110:9292'}]}], 'client_timeout': None} So what's the difference between the two? Why does the context sometimes get populated correctly with the domain information (project_domain, user_domain), and other times not? It seems to be roughly 50% of the time, which may suggest some kind of data race / multithreading error, but then again it's not populating the project_domain 100% of the time for resizing the lvm volume. On the oslo_messaging side, the `ctxt` appears to be a standard python dictionary. By the time it gets to cinder/compute/nova.py, it's a cinder.context.RequestContext, which is a subclass of the oslo RequestContext ( https://github.com/openstack/oslo.context/blob/master/oslo_context/context.py ). Manually adding a "project_domain_id" key with the correct value to the ctxt in oslo_messaging in /usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py fixes the issue. So the question is: where should project_domain_id be coming from? The nova section in cinder.conf appears to be a red herring; that doesn't officially have a project_domain_id section. I'm not sure about the section that Gabriel added to the file. Other similar keystone-authtoken part files begin with the [keystone_authtoken] section header. https://opendev.org/openstack/cinder/src/commit/1eb7384cb6542784a904891666cca4abbe42841d/cinder/api/middleware/auth.py#L112-L113 looks promising, but I added logging there and it was never called. Potentially interesting sections of cinder.conf: ``` [keystone_authtoken] auth_type = password auth_uri = https://192.168.151.156:5000/v3 auth_url = https://192.168.151.156:35357/v3 # should this be admin_domain? I tried changing this but it didn't appear to help project_domain_name = service_domain user_domain_name = service_domain project_name = services ``` According to https://docs.openstack.org/cinder/ussuri/_static/cinder.conf.sample , there is support for a [service_user].project_domain_id option, but setting that also didn't help. Example traceback extended back further to oslo messaging, to figure out where the context comes from. File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming res = self.dispatcher.dispatch(message) File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch return self._do_dispatch(endpoint, method, ctxt, args) File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch result = func(ctxt, **new_args) File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 2941, in extend_volume nova_api.extend_volume(context, instance_uuids, volume.id) File "/usr/lib/python3/dist-packages/cinder/compute/nova.py", line 216, in extend_volume result = self._send_events(context, events, api_version=api_version) File "/usr/lib/python3/dist-packages/cinder/compute/nova.py", line 151, in _send_events response = nova.server_external_events.create(events) File "/usr/lib/python3/dist-packages/novaclient/v2/server_external_events.py", line 38, in create return self._create('/os-server-external-events', body, 'events', File "/usr/lib/python3/dist-packages/novaclient/base.py", line 363, in _create resp, body = self.api.client.post(url, body=body) File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 401, in post return self.request(url, 'POST', **kwargs) File "/usr/lib/python3/dist-packages/novaclient/client.py", line 69, in request resp, body = super(SessionClient, self).request(url, File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 554, in request resp = super(LegacyJsonAdapter, self).request(*args, **kwargs) File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 257, in request return self.session.request(url, method, **kwargs) File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 790, in request auth_headers = self.get_auth_headers(auth) File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1201, in get_auth_headers return auth.get_headers(self, **kwargs) File "/usr/lib/python3/dist-packages/keystoneauth1/plugin.py", line 106, in get_headers token = self.get_token(session) File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 88, in get_token return self.get_access(session).auth_token File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access self.auth_ref = self.get_auth_ref(session) File "/usr/lib/python3/dist-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref return self._plugin.get_auth_ref(session, **kwargs) File "/usr/lib/python3/dist-packages/keystoneauth1/identity/v3/base.py", line 187, in get_auth_ref resp = session.post(token_url, json=body, headers=headers, File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1149, in post return self.request(url, 'POST', **kwargs) File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 986, in request raise exceptions.from_response(resp, method, url) keystoneauth1.exceptions.http.BadRequest: Expecting to find domain in project. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-de6f6d63-40a6-4418-bea3-f924622b1595)