Fuel for OpenStack

Keystone user creation fails: 'IndexError: deque index out of range'

Bug #1581133 reported by Artem Panchenko on 2016-05-12

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Fuel for OpenStack	Fix Committed	High	Dmitry Burmistrov	Fuel for OpenStack 10.0
	Mitaka	Fix Released	High	Dmitry Burmistrov	Fuel for OpenStack 9.0

Bug Description

Rally benchmark test for nova "NovaServers.boot_and_delete_server" failed on pre-step while trying to create tests users in Keystone:

2016-05-11 23:16:48.291 7 WARNING rally.common.broker [-] Failed to consume a task from the queue: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-0c8dd246-84e2-4c30-8a5c-1bed5d9471ed)
...
2016-05-11 23:16:59.085 7 ERROR rally.task.engine [-] Unable to setup context 'users': 'Failed to create the requested number of users.'

There is a corresponding error in keystone logs on controller (node-3):

http://paste.openstack.org/show/496926/

Here is a part of keystone.conf file related to caching:

root@node-3:~# grep -vE '^\s*(#|$)' /etc/keystone/keystone.conf | fgrep -A 7 '[cache]'
[cache]
backend = keystone.cache.memcache_pool
enabled = True
memcache_servers = 10.109.8.4:11211,10.109.8.5:11211,10.109.8.6:11211
memcache_dead_retry = 60
memcache_socket_timeout = 1
memcache_pool_maxsize = 1000
memcache_pool_unused_timeout = 60

Steps to reproduce:

1. Create some environment with Fuel 9.0
2. Start Rally benchmark test "NovaServers.boot_and_delete_server" with this scenario https://github.com/openstack/fuel-qa/blob/6cae33f50a5d4b3decf7e128687732d813846015/fuelweb_test/rally/scenarios/nova.json

Expected result:

tests is successful

Actual result:

sometimes test fails

I contacted Alex Makarov from mos-keystone team and he pointed that driver used by keystone for caching is incorrect, "memcache_pool" must be replaced by "memcache".

Tags:

Revision history for this message

Artem Panchenko (apanchenko-8) wrote on 2016-05-12:

fail_error_ceph_partitions_repetitive_cold_restart-fuel-snapshot-2016-05-11_23-17-11.tar.xz Edit (63.3 MiB, application/octet-stream)

Revision history for this message

Roman Podoliaka (rpodolyaka) wrote on 2016-05-13:

Not sure it deserves to be of High importance, though.

tags:	added: area-puppet removed: area-mos
Changed in fuel:
status:	New → Confirmed

Dmitry Burmistrov (dmburmistrov) on 2016-05-16

Changed in fuel:
assignee:	MOS Puppet Team (mos-puppet) → Dmitry Burmistrov (dmburmistrov)

Revision history for this message

Dmitry Burmistrov (dmburmistrov) wrote on 2016-05-16:

We have
1) Alexandr Makarov believed MOS uses "fernet" as token backend.
2) I see "uuid" provider (with "sql" driver) in "[token]" section in puppets.
3) Also "uuid|sql" are default values in keystone code (current master): https://github.com/openstack/keystone/blob/3456a9e8a8ecfb74d4bb814a625c19b161306b8f/keystone/common/config.py#L304-L315
4) Current "documentation" in keystone.conf says ("cache" section, "backend" setting): "It is recommended that Memcache with pooling (oslo_cache.memcache_pool) or Redis (dogpile.cache.redis) be used in production deployments".
5) Proposed fix is to change "cache/backend" (and all other occurrences of "memcache_pool") from "keystone.cache.memcache_pool" to "keystone.cache.memcache".
6) For fernet: Alexandr suggested to use "provider = keystone.token.providers.fernet.Provider" to prevent token reading attempts from memcache.
7) from chat conversation: "when they extracted memcache_pool from keystone to oslo they've forgotten the fix" - there may be an error in keystone/oslo refactoring process (as I guess)

I suggest for the first stage
a) find the real problem (I am not sure it is "puppet issue")
b) if it is puppet problem
b.1) fix puppet
b.2) fix docs (because I see conflict here)
c) if the problem is in code, but we can't solve it in the near future - we can temporary hack puppet

After that in stage 2 we can discuss fernet in fuel.

Colleagues, your thoughts?

Revision history for this message

Alexander Makarov (amakarov) wrote on 2016-05-16:

Dmirty,can you please provide a link to the place you see in 2) ?

Revision history for this message

Alexander Makarov (amakarov) wrote on 2016-05-16:

What else configures MOS?

Revision history for this message

Dmitry Burmistrov (dmburmistrov) wrote on 2016-05-16:

2. MOS/fuel.
I did a mistake in "2)" - "uuid"/"sql" is not used in puppets.

- "token_driver" is 'keystone.token.persistence.backends.memcache_pool.Token'
- "token_provider" is read from hiera, the value should be "keystone.token.providers.fernet.Provider"

here they are: https://github.com/openstack/fuel-library/blob/master/deployment/puppet/openstack_tasks/manifests/keystone/keystone.pp#L93-L94

1. About defaults from puppet-keystone (upstream).
token section with defaults:
- 'token/driver', default: 'sql' # https://github.com/openstack/puppet-keystone/blob/4e386acb4ec81c1e812d2cefef457fa2dcc867fe/manifests/init.pp#L593
- 'token/expiration', default: 3600
- 'token/caching', default: <is not set in keystone.conf>
- 'token/provider', default: 'uuid' # https://github.com/openstack/puppet-keystone/blob/4e386acb4ec81c1e812d2cefef457fa2dcc867fe/manifests/init.pp#L592
- 'token/revoke_by_id', default: true

For more details try "token/" in your browser search field for this file: https://github.com/openstack/puppet-keystone/blob/4e386acb4ec81c1e812d2cefef457fa2dcc867fe/manifests/init.pp
All options are set in "keystone_config {}" cunstructions.

Revision history for this message

Dmitry Burmistrov (dmburmistrov) wrote on 2016-05-16:

Anyway, I got this from current keystone master after "tox -e genconfig" in etc/keystone.conf:
[cache]
...
# Dogpile.cache backend module. It is recommended that Memcache with pooling
# (oslo_cache.memcache_pool) or Redis (dogpile.cache.redis) be used in
# production deployments. Small workloads (single process) like devstack can
# use the dogpile.cache.memory backend. (string value)
#backend = dogpile.cache.null

As we use this options, do we cache somethig for keystone?
Is there an error in docs (it recommends "oslo_cache.memcache_pool")?

I'm confused a bit about this topic, please, help me understand the situation.
How configuration should look like?

Revision history for this message

Alexander Makarov (amakarov) wrote on 2016-05-16: Re: [Bug 1581133] Re: Keystone user creation fails: 'IndexError: deque index out of range'

#10

This doc is a bit outdated. memcache_pool was a workaround in it's time of
eventlet and currently there are replacement recommendations: pylibmc,
pymemcached.
For our tasks simple memcache client is the optimal solution as
memcache_pool has some performance issues with our current
multithreading/non-eventlet model.
Another thing is that Fernet tokens do not require storage anymore, so
memcached is used just for caching right now and not for token storage
anymore.

Information about tokens in use you can find in keystone.conf
[token]provider parameter. You can find `driver` nearby that can confuse
you, but if provider is set to fernet, driver is not used.

On Mon, May 16, 2016 at 9:41 PM, Dmitry Burmistrov <
<email address hidden>> wrote:

> Anyway, I got this from current keystone master after "tox -e genconfig"
> in etc/keystone.conf:
> [cache]
> ...
> # Dogpile.cache backend module. It is recommended that Memcache with
> pooling
> # (oslo_cache.memcache_pool) or Redis (dogpile.cache.redis) be used in
> # production deployments. Small workloads (single process) like devstack
> can
> # use the dogpile.cache.memory backend. (string value)
> #backend = dogpile.cache.null
>
> As we use this options, do we cache somethig for keystone?
> Is there an error in docs (it recommends "oslo_cache.memcache_pool")?
>
>
> I'm confused a bit about this topic, please, help me understand the
> situation.
> How configuration should look like?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1581133
>
> Title:
> Keystone user creation fails: 'IndexError: deque index out of range'
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/fuel/+bug/1581133/+subscriptions
>

--
Kind Regards,
Alexander Makarov,
Senior Software Developer,

Mirantis, Inc.
35b/3, Vorontsovskaya St., 109147, Moscow, Russia

Tel.: +7 (495) 640-49-04
Tel.: +7 (926) 204-50-60

Skype: MAKAPOB.AJIEKCAHDP

Information about tokens in use you can find in keystone.conf
[token]provider parameter. You can find `driver` nearby that can confuse
you, but if provider is set to fernet, driver is not used.

On Mon, May 16, 2016 at 9:41 PM, Dmitry Burmistrov <
dmburmistrov@mirantis.com> wrote:

> Anyway, I got this from current keystone master after "tox -e genconfig"
> in etc/keystone.conf:
> [cache]
> ...
> # Dogpile.cache backend module. It is recommended that Memcache with
> pooling
> # (oslo_cache.memcache_pool) or Redis (dogpile.cache.redis) be used in
> # production deployments.  Small workloads (single process) like devstack
> can
> # use the dogpile.cache.memory backend. (string value)
> #backend = dogpile.cache.null
>
> As we use this options, do we cache somethig for keystone?
> Is there an error in docs (it recommends "oslo_cache.memcache_pool")?
>
>
> I'm confused a bit about this topic, please, help me understand the
> situation.
> How configuration should look like?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1581133
>
> Title:
>   Keystone user creation fails: 'IndexError: deque index out of range'
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/fuel/+bug/1581133/+subscriptions
>

-- 
Kind Regards,
Alexander Makarov,
Senior Software Developer,

Mirantis, Inc.
35b/3, Vorontsovskaya St., 109147, Moscow, Russia

Tel.: +7 (495) 640-49-04
Tel.: +7 (926) 204-50-60

Skype: MAKAPOB.AJIEKCAHDP

Revision history for this message

Alexander Makarov (amakarov) wrote on 2016-05-16:

This doc is a bit outdated. memcache_pool was a workaround in it's time of eventlet and currently there are replacement recommendations: pylibmc, pymemcached.
For our tasks simple memcache client is the optimal solution as memcache_pool has some performance issues with our current multithreading/non-eventlet model.
Another thing is that Fernet tokens do not require storage anymore, so memcached is used just for caching right now and not for token storage anymore.

Information about tokens in use you can find in keystone.conf [token]provider parameter. You can find `driver` nearby that can confuse you, but if provider is set to fernet, driver is not used.

Revision history for this message

Alexander Makarov (amakarov) wrote on 2016-05-16:

> How configuration should look like?

[cache]
backend = dogpile.cache.memcache

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-05-17: Fix proposed to fuel-library (master)

#11

Fix proposed to branch: master
Review: https://review.openstack.org/317384

Changed in fuel:
status:	Confirmed → In Progress

Revision history for this message

Ivan Berezovskiy (iberezovskiy) wrote on 2016-05-19:

#12

Patch for master in upstream - https://review.openstack.org/#/c/318215/

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-05-19: Fix proposed to fuel-library (stable/mitaka)

#13

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/318741

OpenStack Infra (hudson-openstack) on 2016-05-27

Changed in fuel:
assignee:	Dmitry Burmistrov (dmburmistrov) → Ivan Berezovskiy (iberezovskiy)

Ivan Berezovskiy (iberezovskiy) on 2016-05-27

Changed in fuel:
assignee:	Ivan Berezovskiy (iberezovskiy) → Dmitry Burmistrov (dmburmistrov)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-06-01: Fix merged to fuel-library (master)

#14

Reviewed: https://review.openstack.org/317384
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=da0f0dd670824eeddd4580ba1f56099d5d0299b8
Submitter: Jenkins
Branch: master

commit da0f0dd670824eeddd4580ba1f56099d5d0299b8
Author: dmburmistrov <email address hidden>
Date: Tue May 17 13:49:32 2016 +0300

Set proper cache backend for keystone.conf

    * "memcache_pool" was a workaround for
      eventlet. As we don't use eventlet we
      should use proper backend for caching
    * cache backend can be configured via hiera
    * package installation provided by upstream
      puppet-oslo module
    * noop tests are updated

Closes-bug: #1581133

Change-Id: If86ce795a8f53d8e93b6e5eeab79f9231621ee0e
Depends-On: I2e6a705971761754c59b4069b5dfdfd1dfd7f00f

Changed in fuel:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-06-01: Fix merged to fuel-library (stable/mitaka)

#15

Reviewed: https://review.openstack.org/318741
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=e65f393e419a07ecd1a3cd1e5ade697a792e6617
Submitter: Jenkins
Branch: stable/mitaka

commit e65f393e419a07ecd1a3cd1e5ade697a792e6617
Author: dmburmistrov <email address hidden>
Date: Tue May 17 13:49:32 2016 +0300

Set proper cache backend for keystone.conf

    * "memcache_pool" was a workaround for
      eventlet. As we don't use eventlet we
      should use proper backend for caching
    * noop tests are updated

Closes-bug: #1581133

Change-Id: If86ce795a8f53d8e93b6e5eeab79f9231621ee0e

Revision history for this message

Alexander Makarov (amakarov) wrote on 2016-06-15:

#16

Fixed with downstream patch: https://review.fuel-infra.org/#/c/21651/
Corresponding upstream patch: https://review.openstack.org/#/c/325242/