Comment 0 for bug 1771114

[Problem]

Currently when you make changes to a role for a user, these changes may be inconsistently reflected when you have a HA Keystone configuration.

The reason for this is the use of an individual memcache on each keystone unit, where all memcache servers don't have their cache invalidated when a role is removed.

[Reproduction]

- Deploy a xenial-mitaka through queens environment with 3 keystone units and a VIP

openstack project create test
openstack user create test --password test --project test --domain admin_domain

- Download an OpenStack v3 RC file from openstack dashboard for 'admin' and 'test'

* As 'admin' user
source admin-openrc.sh
openstack network create admin1

* As 'test' user
source test-openrc.sh
openstack network create test1
openstack network list # should show only 'test1'

* As 'admin' user
source admin-openrc.sh
openstack role add --user test --project test Admin

* As 'test' user
source test-openrc.sh
openstack network list # do this a few times, should now show both 'test1' and 'admin1'
openstack network list
openstack network list

* As 'admin' user
openstack role remove --user test --project test Admin

* As 'test' user
source test-openrc.sh
openstack network list # do this a few times, sometimes you will see an inconsistent list showing either test1 or test1 and admin1 - depending on whether the keystone endpoint that 'neutron' hits had it's cache invalidated or not.
openstack network list
openstack network list

* Restart 'memcached' on each of the keystone servers
systemctl restart memcached

* Repeat test, inconsistency goes away.

You can further try delete the test user/project, re-add it and but then re-use the old test-openrc.sh which has the user and project ID hard coded and those IDs will partially work again depending on whether the cache was invalidated on that keystone host or not. Roles are not the only inconsistency.

[Possible Fixes]
 - Disable memcached on HA installations
 - Use a peered memcached solution (memcached itself does not have this built-in but other implementations and forks do)
 - Switch to redis (which supports peered implementations)
 - Set a faster memcached expiry and/or try to send keystone requests to a single server instead of round-robin