| 2018-05-14 12:51:23 |
Trent Lloyd |
bug |
|
|
added bug |
| 2018-05-14 12:53:15 |
Trent Lloyd |
description |
[Problem]
Currently when you make changes to a role for a user, these changes may be inconsistently reflected when you have a HA Keystone configuration.
The reason for this is the use of an individual memcache on each keystone unit, where all memcache servers don't have their cache invalidated when a role is removed.
[Reproduction]
- Deploy a xenial-mitaka through queens environment with 3 keystone units and a VIP
openstack project create test
openstack user create test --password test --project test --domain admin_domain
- Download an OpenStack v3 RC file from openstack dashboard for 'admin' and 'test'
* As 'admin' user
source admin-openrc.sh
openstack network create admin1
* As 'test' user
source test-openrc.sh
openstack network create test1
openstack network list # should show only 'test1'
* As 'admin' user
source admin-openrc.sh
openstack role add --user test --project test Admin
* As 'test' user
source test-openrc.sh
openstack network list # do this a few times, should now show both 'test1' and 'admin1'
openstack network list
openstack network list
* As 'admin' user
openstack role remove --user test --project test Admin
* As 'test' user
source test-openrc.sh
openstack network list # do this a few times, sometimes you will see an inconsistent list showing either test1 or test1 and admin1 - depending on whether the keystone endpoint that 'neutron' hits had it's cache invalidated or not.
openstack network list
openstack network list
* Restart 'memcached' on each of the keystone servers
systemctl restart memcached
* Repeat test, inconsistency goes away.
You can further try delete the test user/project, re-add it and but then re-use the old test-openrc.sh which has the user and project ID hard coded and those IDs will partially work again depending on whether the cache was invalidated on that keystone host or not. Roles are not the only inconsistency.
[Possible Fixes]
- Disable memcached on HA installations
- Use a peered memcached solution (memcached itself does not have this built-in but other implementations and forks do)
- Switch to redis (which supports peered implementations)
- Set a faster memcached expiry and/or try to send keystone requests to a single server instead of round-robin |
[Problem]
Currently when you make changes to a role for a user, these changes may be inconsistently reflected when you have a HA Keystone configuration.
The reason for this is the use of an individual memcache on each keystone unit, where all memcache servers don't have their cache invalidated when a role is removed.
[Reproduction]
- Deploy a xenial-mitaka through queens environment with 3 keystone units and a VIP
openstack project create test
openstack user create test --password test --project test --domain admin_domain
- Download an OpenStack v3 RC file from openstack dashboard for 'admin' and 'test'
* As 'admin' user
source admin-openrc.sh
openstack network create admin1
* As 'test' user
source test-openrc.sh
openstack network create test1
openstack network list # should show only 'test1'
* As 'admin' user
source admin-openrc.sh
openstack role add --user test --project test Admin
* As 'test' user
source test-openrc.sh
openstack network list # do this a few times, should now show both 'test1' and 'admin1'
openstack network list
openstack network list
* As 'admin' user
openstack role remove --user test --project test Admin
* As 'test' user
source test-openrc.sh
openstack network list # do this a few times, sometimes you will see an inconsistent list showing either test1 or test1 and admin1 - depending on whether the keystone endpoint that 'neutron' hits had it's cache invalidated or not.
openstack network list
openstack network list
* Restart 'memcached' on each of the keystone servers
systemctl restart memcached
* Repeat test, inconsistency goes away.
You can further try delete the test user/project, re-add it and but then re-use the old test-openrc.sh which has the user and project ID hard coded and those IDs will partially work again depending on whether the cache was invalidated on that keystone host or not. Roles are not the only inconsistency.
[Possible Fixes]
- Disable memcached on HA installations
- Use a peered memcached solution (memcached itself does not have this built-in but other implementations and forks do)
- Switch to redis (which supports peered implementations)
- Set a faster memcached expiry and/or try to send keystone requests to a single server instead of round-robin |
|
| 2018-05-14 13:05:51 |
Trent Lloyd |
charm-keystone: status |
New |
Confirmed |
|
| 2018-05-14 13:13:52 |
Felipe Reyes |
tags |
|
sts |
|
| 2018-05-23 15:23:14 |
Edward Hope-Morley |
charm-keystone: milestone |
|
18.08 |
|
| 2018-05-28 06:12:56 |
Amad Ali |
bug |
|
|
added subscriber Amad Ali |
| 2018-09-12 20:29:10 |
Edward Hope-Morley |
charm-keystone: status |
Confirmed |
Incomplete |
|
| 2018-09-12 20:38:09 |
James Page |
charm-keystone: milestone |
18.08 |
18.11 |
|
| 2018-11-08 17:01:42 |
Trent Lloyd |
charm-keystone: status |
Incomplete |
New |
|
| 2018-11-08 18:21:56 |
Dmitrii Shcherbakov |
tags |
sts |
cpe-onsite sts |
|
| 2018-11-08 18:22:15 |
Dmitrii Shcherbakov |
bug |
|
|
added subscriber Dmitrii Shcherbakov |
| 2018-11-20 09:24:31 |
James Page |
charm-keystone: milestone |
18.11 |
19.04 |
|
| 2019-04-17 22:07:55 |
David Ames |
charm-keystone: milestone |
19.04 |
19.07 |
|
| 2019-05-13 06:51:28 |
Chris MacNaughton |
charm-keystone: importance |
Undecided |
Medium |
|
| 2019-05-13 06:51:31 |
Chris MacNaughton |
charm-keystone: status |
New |
Triaged |
|
| 2019-08-12 21:31:07 |
David Ames |
charm-keystone: milestone |
19.07 |
19.10 |
|
| 2019-10-24 23:29:10 |
David Ames |
charm-keystone: milestone |
19.10 |
20.01 |
|
| 2020-03-02 15:40:20 |
James Page |
charm-keystone: milestone |
20.01 |
20.05 |
|
| 2020-05-21 20:40:53 |
David Ames |
charm-keystone: milestone |
20.05 |
20.08 |
|
| 2020-08-03 14:02:12 |
James Page |
charm-keystone: milestone |
20.08 |
|
|
| 2021-02-22 21:49:30 |
Nicolas Bock |
charm-keystone: assignee |
|
Nicolas Bock (nicolasbock) |
|
| 2021-03-10 10:57:37 |
Aurelien Lourot |
charm-keystone: status |
Triaged |
In Progress |
|
| 2021-03-18 00:18:12 |
Brett Milford |
bug |
|
|
added subscriber Brett Milford |
| 2021-12-29 18:47:48 |
Chris Johnston |
bug |
|
|
added subscriber Chris Johnston |
| 2023-06-07 13:06:20 |
Edward Hope-Morley |
charm-keystone: status |
In Progress |
New |
|
| 2023-06-07 13:06:23 |
Edward Hope-Morley |
charm-keystone: assignee |
Nicolas Bock (nicolasbock) |
|
|
| 2023-06-07 13:06:30 |
Edward Hope-Morley |
bug task added |
|
charm-helpers |
|
| 2023-06-07 13:10:18 |
Edward Hope-Morley |
charm-helpers: status |
New |
Invalid |
|
| 2023-06-07 13:16:08 |
OpenStack Infra |
charm-keystone: status |
New |
In Progress |
|
| 2023-06-07 13:16:09 |
Edward Hope-Morley |
charm-keystone: assignee |
|
Edward Hope-Morley (hopem) |
|
| 2023-06-09 13:44:58 |
Felipe Reyes |
bug task added |
|
charm-guide |
|
| 2023-06-09 13:45:08 |
Felipe Reyes |
charm-guide: status |
New |
Triaged |
|
| 2023-06-12 10:53:38 |
OpenStack Infra |
charm-keystone: status |
In Progress |
Fix Committed |
|
| 2023-06-16 08:18:13 |
OpenStack Infra |
charm-guide: status |
Triaged |
In Progress |
|
| 2023-06-16 14:03:45 |
Felipe Reyes |
nominated for series |
|
charm-keystone/2023.1 |
|
| 2023-06-16 14:03:45 |
Felipe Reyes |
bug task added |
|
charm-keystone/2023.1 |
|
| 2023-06-16 14:05:23 |
OpenStack Infra |
charm-guide: status |
In Progress |
Fix Released |
|
| 2023-06-16 14:12:56 |
OpenStack Infra |
charm-keystone/2023.1: status |
New |
Fix Committed |
|
| 2023-06-19 14:31:58 |
OpenStack Infra |
tags |
cpe-onsite sts |
cpe-onsite in-stable-zed sts |
|
| 2023-06-21 14:34:09 |
OpenStack Infra |
tags |
cpe-onsite in-stable-zed sts |
cpe-onsite in-stable-yoga in-stable-zed sts |
|
