OpenStack Identity (keystone)

Upgrade to Ocata: Keystone Intermittent Missing 'options' Key

Bug #1793389 reported by Alex Redinger
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
In Progress
Medium
Unassigned
openstack-ansible
Fix Released
Medium
Alex Redinger

Bug Description

During upgrades of Newton-EOL AIOs to Ocata, Keystone installation fails at the "Ensure service tenant" play of the os-keystone_install.

This occurs using the provided run-upgrade.sh script.

Keystone logs are thus:

INFO keystone.common.wsgi [req-11844ac2-f2d5-46b6-986d-05019432f264 - - - - -] HEAD http://aio1-keystone-container-14a3e1ad:5000/
DEBUG keystone.middleware.auth [req-6523488f-be1a-4ba7-a264-6b6b8ca4c936 - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set. fill_context /openstack/venvs/keystone-15.1.24/lib/python2.7/site-packages/keystone/middleware/auth.py:188
INFO keystone.common.wsgi [req-6523488f-be1a-4ba7-a264-6b6b8ca4c936 - - - - -] POST http://172.29.236.66:35357/v3/auth/tokens
ERROR keystone.common.wsgi [req-6523488f-be1a-4ba7-a264-6b6b8ca4c936 - - - - -] 'options'
ERROR keystone.common.wsgi Traceback (most recent call last):
ERROR keystone.common.wsgi File "/openstack/venvs/keystone-15.1.24/lib/python2.7/site-packages/keystone/common/wsgi.py", line 228, in __call__
ERROR keystone.common.wsgi result = method(req, **params)
ERROR keystone.common.wsgi File "/openstack/venvs/keystone-15.1.24/lib/python2.7/site-packages/keystone/auth/controllers.py", line 132, in authenticate_for_token
ERROR keystone.common.wsgi auth_context['user_id'], method_names_set):
ERROR keystone.common.wsgi File "/openstack/venvs/keystone-15.1.24/lib/python2.7/site-packages/keystone/auth/core.py", line 377, in check_auth_methods_against_rules
ERROR keystone.common.wsgi mfa_rules = user_ref['options'].get(ro.MFA_RULES_OPT.option_name, [])
ERROR keystone.common.wsgi KeyError: 'options'

It appears that the sql identity backend ensures an 'options' key should exist with .../keystone/identity/backends/sql_schema.py:225, but obviously that code's not being hit.

It should be noted that rerunning the install process causes it to be successful.

Revision history for this message
Alex Redinger (rexredinger) wrote :

Noticing that even on successfull upgrades there are usually a few failed attempts at running the "Ensure service tenant" play. Simply adding more retires should greatly mitigate upgrade failures going forward.

Changed in openstack-ansible:
assignee: nobody → Alex Redinger (rexredinger)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-ansible-os_keystone (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/604804

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on openstack-ansible-os_keystone (stable/pike)

Change abandoned by Alex Redinger (<email address hidden>) on branch: stable/pike
Review: https://review.openstack.org/604147
Reason: Abandoning for upstream changed here https://review.openstack.org/#/c/604804/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on openstack-ansible-os_keystone (stable/ocata)

Change abandoned by Alex Redinger (<email address hidden>) on branch: stable/ocata
Review: https://review.openstack.org/604144
Reason: Abandoning for upstream changed here https://review.openstack.org/#/c/604804/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-ansible-os_keystone (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/604845

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-ansible-os_keystone (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/604846

Mohammed Naser (mnaser)
Changed in openstack-ansible:
status: New → Confirmed
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-ansible-os_keystone (master)

Fix proposed to branch: master
Review: https://review.openstack.org/605146

Changed in openstack-ansible:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-ansible-os_keystone (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/605148

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-ansible-os_keystone (master)

Fix proposed to branch: master
Review: https://review.openstack.org/608066

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on openstack-ansible-os_keystone (master)

Change abandoned by Alex Redinger (<email address hidden>) on branch: master
Review: https://review.openstack.org/605146
Reason: Abandoning in favor of https://review.openstack.org/#/c/608066/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on openstack-ansible-os_keystone (stable/queens)

Change abandoned by Alex Redinger (<email address hidden>) on branch: stable/queens
Review: https://review.openstack.org/604804

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to openstack-ansible-os_keystone (master)

Reviewed: https://review.openstack.org/608066
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_keystone/commit/?id=afc0e5b1ce660224593f2f9d1886d45f1c69f174
Submitter: Zuul
Branch: master

commit afc0e5b1ce660224593f2f9d1886d45f1c69f174
Author: rexredinger <email address hidden>
Date: Thu Oct 4 15:38:01 2018 -0400

    Add memcache flushing handler on db migrations

    Flushing cache when schema changes occur to avoid drift between
    what might be cached and what keystone queries expect.

    Change-Id: Ibf8f3dd60d6f3c446a14dc8228fa005f12fcc840
    Closes-Bug: 1793389

Changed in openstack-ansible:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-ansible-os_keystone (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/612482

Revision history for this message
Lance Bragstad (lbragstad) wrote :

I proposed a patch to keystone's stable/ocata branch that should help with this.

https://review.openstack.org/#/c/612686/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to openstack-ansible-os_keystone (master)

Reviewed: https://review.openstack.org/613256
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_keystone/commit/?id=280f3e062a30ebffdc2986d0b3c20e323486e3da
Submitter: Zuul
Branch: master

commit 280f3e062a30ebffdc2986d0b3c20e323486e3da
Author: Jesse Pretorius <email address hidden>
Date: Thu Oct 25 10:34:48 2018 +0100

    Make the memcache flush optional

    The memcache flush implemented in https://review.openstack.org/608066
    is actually a workaround. The implementation should really be
    implemented more surgically in keystone itself. This has begun with
    https://review.openstack.org/612686 but it is not complete. However,
    something that's been made clear by the team is that this was only
    required for the Newton->Ocata upgrade and has not been required since.

    This workaround may be required again in the future, so instead of
    removing it, we make the tool opt-in via a toggle which can then easily
    be set when doing the appropriate major upgrade.

    Related-Bug: 1793389
    Change-Id: Ied0ce1e9877697bb627f784a0590f7c7e924479b

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-ansible-os_keystone (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/613321

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-ansible-os_keystone (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/613323

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-ansible-os_keystone (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/613325

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to openstack-ansible-os_keystone (stable/ocata)

Reviewed: https://review.openstack.org/613325
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_keystone/commit/?id=b8798d89f8b5537c58a7d3636c47b3513f06bc64
Submitter: Zuul
Branch: stable/ocata

commit b8798d89f8b5537c58a7d3636c47b3513f06bc64
Author: rexredinger <email address hidden>
Date: Thu Oct 4 15:38:01 2018 -0400

    Add memcache flushing handler on db migrations

    Flushing cache when schema changes occur to avoid drift between
    what might be cached and what keystone queries expect.

    Given that this is a workaround which is only usually required
    in major upgrades (and should really be handled in process by
    keystone itself), we make the execution optional so that it can
    be set to happen only on major upgrades.

    To prevent test breakage as happened in master, we combine the
    two master patches into one for the backport:

    https://review.openstack.org/608066
    https://review.openstack.org/613256

    Depends-On: https://review.openstack.org/613293
    Change-Id: Ibf8f3dd60d6f3c446a14dc8228fa005f12fcc840
    Closes-Bug: 1793389

tags: added: in-stable-ocata
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to openstack-ansible-os_keystone (stable/pike)

Reviewed: https://review.openstack.org/613323
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_keystone/commit/?id=0b4eb4162a34d2f9947aca2f9e7f42504b5ba010
Submitter: Zuul
Branch: stable/pike

commit 0b4eb4162a34d2f9947aca2f9e7f42504b5ba010
Author: rexredinger <email address hidden>
Date: Thu Oct 4 15:38:01 2018 -0400

    Add memcache flushing handler on db migrations

    Flushing cache when schema changes occur to avoid drift between
    what might be cached and what keystone queries expect.

    Given that this is a workaround which is only usually required
    in major upgrades (and should really be handled in process by
    keystone itself), we make the execution optional so that it can
    be set to happen only on major upgrades.

    To prevent test breakage as happened in master, we combine the
    two master patches into one for the backport:

    https://review.openstack.org/608066
    https://review.openstack.org/613256

    Depends-On: https://review.openstack.org/613292
    Change-Id: Ibf8f3dd60d6f3c446a14dc8228fa005f12fcc840
    Closes-Bug: 1793389

tags: added: in-stable-pike
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to openstack-ansible-os_keystone (stable/queens)

Reviewed: https://review.openstack.org/612482
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_keystone/commit/?id=03e3e83953218ad3a99c7c239037a94b60313ee7
Submitter: Zuul
Branch: stable/queens

commit 03e3e83953218ad3a99c7c239037a94b60313ee7
Author: rexredinger <email address hidden>
Date: Thu Oct 4 15:38:01 2018 -0400

    Add memcache flushing handler on db migrations

    Flushing cache when schema changes occur to avoid drift between
    what might be cached and what keystone queries expect.

    Given that this is a workaround which is only usually required
    in major upgrades (and should really be handled in process by
    keystone itself), we make the execution optional so that it can
    be set to happen only on major upgrades.

    To prevent test breakage as happened in master, we combine the
    two master patches into one for the backport:

    https://review.openstack.org/608066
    https://review.openstack.org/613256

    Depends-On: https://review.openstack.org/613291
    Change-Id: Ibf8f3dd60d6f3c446a14dc8228fa005f12fcc840
    Closes-Bug: 1793389

tags: added: in-stable-queens
Revision history for this message
wangxiyuan (wangxiyuan) wrote :

https://review.openstack.org/#/c/612686/

Changed in keystone:
assignee: nobody → Lance Bragstad (lbragstad)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to openstack-ansible-os_keystone (stable/rocky)

Reviewed: https://review.openstack.org/613321
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_keystone/commit/?id=ce0df5355f837dc406e0a94e6fcc6e4c8f1b37e0
Submitter: Zuul
Branch: stable/rocky

commit ce0df5355f837dc406e0a94e6fcc6e4c8f1b37e0
Author: rexredinger <email address hidden>
Date: Thu Oct 4 15:38:01 2018 -0400

    Add memcache flushing handler on db migrations

    Flushing cache when schema changes occur to avoid drift between
    what might be cached and what keystone queries expect.

    Given that this is a workaround which is only usually required
    in major upgrades (and should really be handled in process by
    keystone itself), we make the execution optional so that it can
    be set to happen only on major upgrades.

    To prevent test breakage as happened in master, we combine the
    two master patches into one for the backport:

    https://review.openstack.org/608066
    https://review.openstack.org/613256

    Depends-On: https://review.openstack.org/613290
    Change-Id: Ibf8f3dd60d6f3c446a14dc8228fa005f12fcc840
    Closes-Bug: 1793389

tags: added: in-stable-rocky
Revision history for this message
Morgan Fainberg (mdrnstm) wrote :

We (Keystone) will add the db schema version ID / Number to the cache key generator so that we do an implicit cache pop on schema upgrades.

Changed in keystone:
status: New → Triaged
importance: Undecided → Medium
Revision history for this message
Lance Bragstad (lbragstad) wrote :

Removing myself in case others have more time to pick this up [0].

[0] https://review.openstack.org/#/c/612686/

Changed in keystone:
status: Triaged → In Progress
assignee: Lance Bragstad (lbragstad) → nobody
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on keystone (stable/ocata)

Change abandoned by Lance Bragstad (<email address hidden>) on branch: stable/ocata
Review: https://review.openstack.org/612686
Reason: Removing from my work queue for now. If someone has the time to pick this up, please feel free to do so.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/openstack-ansible-os_keystone ocata-em

This issue was fixed in the openstack/openstack-ansible-os_keystone ocata-em release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/openstack-ansible-os_keystone 16.0.29

This issue was fixed in the openstack/openstack-ansible-os_keystone 16.0.29 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on openstack-ansible-os_keystone (stable/pike)

Change abandoned by Jesse Pretorius (odyssey4me) (<email address hidden>) on branch: stable/pike
Review: https://review.opendev.org/604845
Reason: This has been replaced by a better solution

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on openstack-ansible-os_keystone (stable/ocata)

Change abandoned by Jesse Pretorius (odyssey4me) (<email address hidden>) on branch: stable/ocata
Review: https://review.opendev.org/604846
Reason: This has been replaced by a better solution

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on openstack-ansible-os_keystone (stable/rocky)

Change abandoned by Jesse Pretorius (odyssey4me) (<email address hidden>) on branch: stable/rocky
Review: https://review.opendev.org/605148
Reason: This has been replaced by a better solution

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/openstack-ansible-os_keystone queens-eol

This issue was fixed in the openstack/openstack-ansible-os_keystone queens-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/openstack-ansible-os_keystone rocky-eol

This issue was fixed in the openstack/openstack-ansible-os_keystone rocky-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/openstack-ansible-os_keystone stein-eol

This issue was fixed in the openstack/openstack-ansible-os_keystone stein-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.