When using memcached with encryption (eg. memcache_security_strategy = MAC), its possible that data in memcached becomes un-decryptable for some reason.
The cache implementation returns null value (None) in case decryption fails but it seems that case is not properly handled, and the process immediately fails with the following traceback.
The problem repeats until the broken cache data in memcached expires.
2023-05-31 16:52:59.839 45 ERROR keystonemiddleware.auth_token [-] Failed to decrypt/verify cache data: keystonemiddleware.auth_token._memcache_crypt.InvalidMacError: Invalid MAC; data appears to be corrupted.
2023-05-31 16:52:59.839 45 ERROR keystonemiddleware.auth_token Traceback (most recent call last):
2023-05-31 16:52:59.839 45 ERROR keystonemiddleware.auth_token File "/var/lib/kolla/venv/lib/python3.6/site-packages/keystonemiddleware/auth_token/_cache.py", line 287, in _deserialize
2023-05-31 16:52:59.839 45 ERROR keystonemiddleware.auth_token return memcache_crypt.unprotect_data(context, data)
2023-05-31 16:52:59.839 45 ERROR keystonemiddleware.auth_token File "/var/lib/kolla/venv/lib/python3.6/site-packages/keystonemiddleware/auth_token/_memcache_crypt.py", line 208, in unprotect_data
2023-05-31 16:52:59.839 45 ERROR keystonemiddleware.auth_token raise InvalidMacError(_('Invalid MAC; data appears to be corrupted.'))
2023-05-31 16:52:59.839 45 ERROR keystonemiddleware.auth_token keystonemiddleware.auth_token._memcache_crypt.InvalidMacError: Invalid MAC; data appears to be corrupted.
2023-05-31 16:52:59.839 45 ERROR keystonemiddleware.auth_token
2023-05-31 16:52:59.840 45 ERROR oslo_middleware.catch_errors [-] An error occurred during processing the request: GET /v2.0/security-groups?limit=20&marker=dea78d77-4054-4e97-8f5a-10c38184fa77 HTTP/1.0
2023-05-31 16:52:59.840 45 ERROR oslo_middleware.catch_errors Traceback (most recent call last):
2023-05-31 16:52:59.840 45 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_middleware/catch_errors.py", line 40, in __call__
2023-05-31 16:52:59.840 45 ERROR oslo_middleware.catch_errors response = req.get_response(self.application)
2023-05-31 16:52:59.840 45 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python3.6/site-packages/webob/request.py", line 1314, in send
2023-05-31 16:52:59.840 45 ERROR oslo_middleware.catch_errors application, catch_exc_info=False)
2023-05-31 16:52:59.840 45 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application
2023-05-31 16:52:59.840 45 ERROR oslo_middleware.catch_errors app_iter = application(self.environ, start_response)
2023-05-31 16:52:59.840 45 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
2023-05-31 16:52:59.840 45 ERROR oslo_middleware.catch_errors resp = self.call_func(req, *args, **kw)
2023-05-31 16:52:59.840 45 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
2023-05-31 16:52:59.840 45 ERROR oslo_middleware.catch_errors return self.func(req, *args, **kwargs)
2023-05-31 16:52:59.840 45 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python3.6/site-packages/osprofiler/web.py", line 112, in __call__
2023-05-31 16:52:59.840 45 ERROR oslo_middleware.catch_errors return request.get_response(self.application)
2023-05-31 16:52:59.840 45 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python3.6/site-packages/webob/request.py", line 1314, in send
2023-05-31 16:52:59.840 45 ERROR oslo_middleware.catch_errors application, catch_exc_info=False)
2023-05-31 16:52:59.840 45 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application
2023-05-31 16:52:59.840 45 ERROR oslo_middleware.catch_errors app_iter = application(self.environ, start_response)
2023-05-31 16:52:59.840 45 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
2023-05-31 16:52:59.840 45 ERROR oslo_middleware.catch_errors resp = self.call_func(req, *args, **kw)
2023-05-31 16:52:59.840 45 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
2023-05-31 16:52:59.840 45 ERROR oslo_middleware.catch_errors return self.func(req, *args, **kwargs)
2023-05-31 16:52:59.840 45 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 338, in __call__
2023-05-31 16:52:59.840 45 ERROR oslo_middleware.catch_errors response = self.process_request(req)
2023-05-31 16:52:59.840 45 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 659, in process_request
2023-05-31 16:52:59.840 45 ERROR oslo_middleware.catch_errors resp = super(AuthProtocol, self).process_request(request)
2023-05-31 16:52:59.840 45 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 411, in process_request
2023-05-31 16:52:59.840 45 ERROR oslo_middleware.catch_errors allow_expired=allow_expired)
2023-05-31 16:52:59.840 45 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 445, in _do_fetch_token
2023-05-31 16:52:59.840 45 ERROR oslo_middleware.catch_errors data = self.fetch_token(token, **kwargs)
2023-05-31 16:52:59.840 45 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 736, in fetch_token
2023-05-31 16:52:59.840 45 ERROR oslo_middleware.catch_errors cached = self._token_cache.get(token)
2023-05-31 16:52:59.840 45 ERROR oslo_middleware.catch_errors File "/var/lib/kolla/venv/lib/python3.6/site-packages/keystonemiddleware/auth_token/_cache.py", line 237, in get
2023-05-31 16:52:59.840 45 ERROR oslo_middleware.catch_errors data = data.decode('utf-8')
2023-05-31 16:52:59.840 45 ERROR oslo_middleware.catch_errors AttributeError: 'NoneType' object has no attribute 'decode'
Fix proposed to branch: master /review. opendev. org/c/openstack /keystonemiddle ware/+/ 885351
Review: https:/