"Unable to add token to revocation list" warning happened when revoking token in memcache
Bug #1242620 reported by
Eric Zhou
This bug affects 10 people
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Identity (keystone) |
Won't Fix
|
Medium
|
Adam Young |
Bug Description
Memcache backend is used to store the token. When revoking a token, such error reported.
"Unable to add token to revocation list"
As a result, the revoked token could not be added to revocation-list in memcache although the token was actually revoked.
I found this warning always happen when the size of value of the revocation-list key in memcache is about 512K.
Expected result:
No warning exception should be raised when revoking token.
Changed in keystone: | |
status: | New → Triaged |
importance: | Undecided → Medium |
Changed in keystone: | |
assignee: | nobody → Adam Young (ayoung) |
To post a comment you must log in.
Just stumble upon the same problem while running tempest on an openstack installation that use memcached as keystone backend, and there is my analyze of the situation.
As far as i can tell, the problem is the way the revoked token are stored in memcached, basically each revoked token is appended to this item "revocation-list" and because memcached has a default max_item_size set to 1MB (echo 'stats settings' | nc localhost 11211 | grep 'item_size_max') than as soon as this limit is hit keystone will start raising an error when it try to append to this item.
Code taken from keystone/ token/backends/ memcache. py:
def _add_to_ revocation_ list(self, data): dumps(data) append( self.revocation _key, ',%s' % data_json): add(self. revocation_ key, data_json): append( self.revocation _key,
',%s' % data_json):
msg = _('Unable to add token to revocation list.')
raise exception. UnexpectedError (msg)
data_json = jsonutils.
if not self.client.
if not self.client.
if not self.client.
The quick and dirty and temporary fix (which is of course not recommended) is to change the memcached default max_item_size to something bigger than 1MB and thus is possible only with memcached 1.4.2 and above by supplying the -I (capital i) argument.
$ memcached -I 10m ... # max_item_size = 10MB
This will increase heavily memory consumption of memcached, that's one of the reason why it's not recommended.
A permanent fix will be when this https:/ /blueprints. launchpad. net/keystone/ +spec/revocatio n-backend will be implemented i guess.
A lesson to take from this is that memcached is not meant to store big lists.