Comment 8 for bug 1578401

Revision history for this message
Pavlo Shchelokovskyy (pshchelo) wrote :

I'd like to re-start the discussion about this bug.

What this combination currently leads to is effectively false positive alerts on memcached monitoring.

Monitoring memcached for forced evictions is a well established practice to get alerted when you need to increase the cache size or redistribute it between slabs or check in any other way for possible abnormalities of cache usage.
But this of course relies on cache users to not dump stuff into the cache to keep it "forever". Memcached server AFAIU has no periodic garbage collection, so even with TTL set, memcached will only delete the data if either client tries to access that expired key, or to make room for new data - in which case it won't consider it a forced eviction.

Here however keystone keeps storing token creation/validation data, for whatever reason it is never deleted from memcached by dogpile itself (because at some point too nothing asks for that potentially expired info, and it also does not have any periodic GC? anyway basically impossible in a wsgi app) and it has no TTL in memcached as well, so when memcached tries to make room for new data, it deletes them and fires forced eviction alerts.

I'd like oslo.cache maintainers to please restore the patch https://review.opendev.org/c/openstack/oslo.cache/+/350942 and I will take it over.