Keystone become extremely slow after controller shutdown
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Fix Released
|
High
|
MOS Keystone | ||
6.0.x |
Fix Released
|
High
|
Bogdan Dobrelya | ||
6.1.x |
Invalid
|
High
|
MOS Maintenance | ||
7.0.x |
Fix Released
|
High
|
MOS Keystone |
Bug Description
Fuel version( 6.0 #56): http://
The problem described here https:/
1. Create HA environment (Ubuntu+
2. Add 3 controller+mongo and 4 copmute+ceph nodes.
3. Deploy changes. Run OSTF. All tests passed.
4. Shutdown some controller (or just block traffic to memcached that is running on it - `iptables -I INPUT -p tcp --dport 11211 -j DROP`)
Expected result:
- OS services continue to work fine, environment passes health checks
Actual result:
- lots of OSTF smoke tests fails by timeout "AssertionError: Failed to get to expected status. In error state."
When all (3) memcached instances are available keystone needs 0.1-0.2 sec to create new token:
http://
and opening of '/horizon/
After destroying one of controllers keystone spends >3 seconds creating a token:
http://
Loading of '/horizon/
I observed the same behaviour on CentOS clouds.
summary: |
- Kyestone become extremely slow after controller shutdown + Keystone become extremely slow after controller shutdown |
tags: |
added: release-notes-done removed: release-notes |
tags: | added: customer-found |
tags: | added: on-verification |
I guess we should provide some workaround for 6.0 and document it in release notes. If one of controllers in HA environment is destroyed/deleted all OS services work slowly, especially Horizon dashboard, due to delays from keystone. Such issue can be manually fixed in the following way:
1. determine the IP address assigned to management interface of unavailable controller node (should be in /etc/hosts on alive nodes); IP="192. 168.0.5" ; sed -r 's/'$DOWN_ IP':11211, ?//g' -i.bak /etc/keystone/ keystone. conf`)
2. remove this address from list of memcached servers in keystone.conf on each controller (`DOWN_
3. restart keystone daemon on all controllers (`service openstack-keystone restart || restart keystone`)