Environment:
Reproduced on RackSpace lab, 3 controllers, 197 computes, VxLAN+DVR, MOS 9.0 ISO 188
Detailed description:
Keystone caches the whole revoke tree, which can exceed the 1M memcached object size limit if huge number of tokens get revoked at the same time (details: https://github.com/lericson/pylibmc/issues/184) .
from keystone adimn log file http://paste.openstack.org/show/494384/
After that keystone breaks its operation and cluster in not usable.
Keystone error:
2016-04-18 09:44:57.484 33105 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/revoke/core.py", line 225, in check_token
2016-04-18 09:44:57.484 33105 ERROR keystone.common.wsgi if self._get_revoke_tree().is_revoked(token_values):
Steps to reproduce:
1. set backend = dogpile.cache.pylibmc' in [cache] section in /etc/keystone/keystone.conf
2. Perform raly tests. All rally tests was failed excluding only three of them (results of the three tests are attached - rally_report.html)
3. found the following bug https://bugs.launchpad.net/mos/+bug/1571626
4. tried http://paste.openstack.org/show/494498/ + [revoke]caching=False in keystone.conf - No error any more, but request takes more then 60 sec and we get 504 even if request "opestack user list"
run rally scenario KeystoneBasic.add_and_remove_user_role, against large cluster. Example scenario:
{
"kw": {
"runner": {
"type": "constant",
"concurrency": 20,
"times": 1970
},
"sla": {
"failure_rate": {
"max": 0
}
},
"context": {
"api_versions": {
"keystone": {
"version": 2
}
}
}
},
"name": "KeystoneBasic.add_and_remove_user_role",
"pos": 0
}
diagnostic snapshot: http://mos-scale-share.mirantis.com/fuel-snapshot-2016-04-17_17-47-57.tar.xz
etc and log folders from controller nodes: http://mos-scale-share.mirantis.com/controller-data.tar.xz
First glance: revocation tree grows larger 1M and gets inacceptable for caching in memcached.
I.e. memcahced doesn't accept such size.
The problem appeared to be well-known in the Community and they suggest just turning revocation caching off.