OpenStack Security Advisory

Service tokens seems to be incorrectly implemented in kolla-ansible

Bug #2049762 reported by Franciszek Przewoźny
276
This bug affects 4 people
Affects Status Importance Assigned to Milestone
OpenStack Security Advisory
New
Undecided
Unassigned
OpenStack Security Notes
New
Undecided
Unassigned
kolla-ansible
Fix Released
Undecided
Unassigned
Antelope
Fix Released
Undecided
Unassigned
Bobcat
Fix Released
Undecided
Unassigned
Caracal
Fix Released
Undecided
Unassigned
Yoga
Fix Committed
Undecided
Unassigned
Zed
Fix Released
Undecided
Unassigned

Bug Description

Hi all,

It seems, that configuration done by kolla-ansible regarding service tokens seems to be not compliant with the documentation. If I'm right, this might mean that OSSA-2023-003 isn't fully fixed there.

https://docs.openstack.org/cinder/2023.2/configuration/block-storage/service-token.html#configuration

1. Send service token
  Section [service_user] is not defined at all for Cinder: https://github.com/openstack/kolla-ansible/blob/stable/2023.2/ansible/roles/cinder/templates/cinder.conf.j2, Nova got it defined: https://github.com/openstack/kolla-ansible/blob/stable/2023.2/ansible/roles/nova/templates/nova.conf.j2#L211

2. Service role
  From documentation "A service role is nothing more than a Keystone role that allows a deployment to identify a service without the need to make them admins", and later: "The default service role is service, but we can use a different name or even have multiple service roles", where in kolla-ansible, Nova service jinja does not have parameter to be set for service_token_roles, and it uses default "service", Cinder on the other hand have HARDCODED "admin" value (https://github.com/openstack/kolla-ansible/blob/stable/2023.2/ansible/roles/cinder/templates/cinder.conf.j2#L119), and that's where I'm a bit lost. I thought that service users "nova" and "cinder" should have only f.e. "service" role in "service" project, not some multiple roles including "admin" one.

3. Policies not prepared for "service" role only
  Configuration of other role "service" for services (tested with Neutron and Placement), cannot be done as there are policies restricting some API calls, f.e.:

  Neutron:
  ERROR neutron.notifiers.nova novaclient.exceptions.Forbidden: Policy doesn't allow os_compute_api:os-server-external-events:create to be performed. (HTTP 403)

  Policy restricting that call is:
  "os_compute_api:os-server-external-events:create": "rule:context_is_admin"

  And rule:context_is_admin definition is:
  "context_is_admin": "role:admin"

  Which totally blocks any effort put into getting rid of "admin" service role for other services.

My environment isn't kolla-ansible, but I'm using kolla's jinja templates as a configuration reference sometimes. I'm raising this ticket, as it's a bit suspicious for me that something else is configured in kolla, than is described in manual.

Best regards,
Franciszek

See original description
Franciszek Przewoźny (fprzewozn)
tags: added: cinder nova
Franciszek Przewoźny (fprzewozn)
description: updated
Franciszek Przewoźny (fprzewozn)
information type: Private Security → Public Security
Revision history for this message
sean mooney (sean-k-mooney) wrote :

[service_user] for services that work like not is used for making request to other services
in the context of the cve cinder does not actually need that section.

note ironic does not use a [service_user] section it uses the details form the relevant service section i.e. cinder instead and assumes that it has the admin and service roles.

so cinder does not need the [service_user] section for the cve mitigation in its config.
it may use it for the other uses of service_user which is to test that when calling other services with a user token that may expire it was valid when cinder received it and should be honoured as if it was still valid by the receiving service.

you are correct that policies in services are not prepared for service role only in general which is why we recommend using both the service and admin role for the user configured in [service_user] in the interm period.

the i have not looked at the details of all of kolla config but i proposed the original patch for the service_user support for nova in kolla
https://github.com/openstack/kolla-ansible/commit/ddadaa282e72cc437470859766ac963ac757a26a

since my original version the patch was updated to use the admin role when verifying the service token in cinder

https://github.com/openstack/kolla-ansible/commit/ddadaa282e72cc437470859766ac963ac757a26a#diff-61b022281f112ed49c0169e67dd999658a41397df88acb4c224904d1c11b03cdR110

that is not correct and that could have security implications.
the service role is intentionally referenced in the upstream docs regarding this because its a role a normal admin.human should not have.

it was intentionally used so that admin could not delete cinder volume atachemetn with there normal admin accounts.

by enforceing the token has the admin role instead that protection has been removed but the cve is not reintudced for end users as normal users are still blocked.

kolla should be adding the service role to the service accounts (nova neutron ectra) and they should not be setting service_token_roles = admin in the cinder config.

Revision history for this message
Sven Kieske (s-kieske) wrote :

Thanks for the pointers Sean, I added this topic to our Whiteboard, to discuss it next week.

Maybe I also can take a look at the code a little earlier and propose a new patch:

https://etherpad.opendev.org/p/KollaWhiteBoard#L60

Revision history for this message
Sven Kieske (s-kieske) wrote :

I guess this comment chain is the culprit:

https://review.opendev.org/c/openstack/kolla-ansible/+/882893/comments/bb24d1f0_252d8b74

to quote it:

myself:

> question, mainly for mnasiadka:

> should we add the cinder patch to all other services in this patch also?

> That would mean validating service tokens for every service.

> We could maybe also argue to split this work up in a second/more changesets.

> I have a slight preference to split this up to not let this security bugfix wait for much longer.

> What do you think?

mnasiadika:

> I think we should split the rest, it doesn't seem to be required now.

then in patchset 4:

https://review.opendev.org/c/openstack/kolla-ansible/+/882893/3..4/ansible/roles/cinder/templates/cinder.conf.j2

we have:

```
# TODO(mnasiadka): Work out if that's the desired approach or should we add service role to nova user
service_token_roles = admin

then in patchset 6 the comment got removed:

https://review.opendev.org/c/openstack/kolla-ansible/+/882893/5..6/ansible/roles/cinder/templates/cinder.conf.j2

I didn't comment on that because I thought mnasiadka did indeed had checked that this was necessary.
```

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/908145

Changed in kolla-ansible:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to kolla-ansible (stable/2023.1)

Related fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/909002

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on kolla-ansible (master)

Change abandoned by "Bartosz Bezak <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/815577
Reason: Kolla community decided not to implement service role for all service users in one change. Rather then doing it selectively. As not all openstack projects implemented service role support yet.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to kolla-ansible (stable/2023.1)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/909002
Committed: https://opendev.org/openstack/kolla-ansible/commit/01c359665b53ef51837bd2bde649f969e0223d7f
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 01c359665b53ef51837bd2bde649f969e0223d7f
Author: Bartosz Bezak <email address hidden>
Date: Wed Feb 14 15:59:01 2024 +0100

    Keystone: Add service role

    Keystone is creating service in bootstrap since Bobcat [1].
    Service role is needed for SLURP to work from Antelope.
    This role is also needed in Antelope and Zed for Cinder for
    proper service token support. [2]

    [1] https://review.opendev.org/c/openstack/keystone/+/863420
    [2] https://bugs.launchpad.net/kolla-ansible/+bug/2049762

    Related-Bug: #2049762
    Change-Id: I7685904ef92b60793545dab4b998f88ad48f2e8f

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to kolla-ansible (stable/zed)

Related fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/909084

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to kolla-ansible (unmaintained/yoga)

Related fix proposed to branch: unmaintained/yoga
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/909085

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to kolla-ansible (stable/zed)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/909084
Committed: https://opendev.org/openstack/kolla-ansible/commit/ec9a30ca9c60e329bc69a95ceef3e3496b17f61e
Submitter: "Zuul (22348)"
Branch: stable/zed

commit ec9a30ca9c60e329bc69a95ceef3e3496b17f61e
Author: Bartosz Bezak <email address hidden>
Date: Wed Feb 14 15:59:01 2024 +0100

    Keystone: Add service role

    Keystone is creating service in bootstrap since Bobcat [1].
    Service role is needed for SLURP to work from Antelope.
    This role is also needed in Antelope and Zed for Cinder for
    proper service token support. [2]

    [1] https://review.opendev.org/c/openstack/keystone/+/863420
    [2] https://bugs.launchpad.net/kolla-ansible/+bug/2049762

    Related-Bug: #2049762
    Change-Id: I7685904ef92b60793545dab4b998f88ad48f2e8f
    (cherry picked from commit 01c359665b53ef51837bd2bde649f969e0223d7f)

tags: added: in-stable-zed
tags: added: in-unmaintained-yoga
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to kolla-ansible (unmaintained/yoga)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/909085
Committed: https://opendev.org/openstack/kolla-ansible/commit/2d8a5c588aacb29b4d0c39a45acbe2c604637ca1
Submitter: "Zuul (22348)"
Branch: unmaintained/yoga

commit 2d8a5c588aacb29b4d0c39a45acbe2c604637ca1
Author: Bartosz Bezak <email address hidden>
Date: Wed Feb 14 15:59:01 2024 +0100

    Keystone: Add service role

    Keystone is creating service in bootstrap since Bobcat [1].
    Service role is needed for SLURP to work from Antelope.
    This role is also needed in Antelope and Zed for Cinder for
    proper service token support. [2]

    [1] https://review.opendev.org/c/openstack/keystone/+/863420
    [2] https://bugs.launchpad.net/kolla-ansible/+bug/2049762

    Related-Bug: #2049762
    Change-Id: I7685904ef92b60793545dab4b998f88ad48f2e8f
    (cherry picked from commit 01c359665b53ef51837bd2bde649f969e0223d7f)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (master)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/908405
Committed: https://opendev.org/openstack/kolla-ansible/commit/1ef765f69044abae1b838b261481f420a4007121
Submitter: "Zuul (22348)"
Branch: master

commit 1ef765f69044abae1b838b261481f420a4007121
Author: Michal Nasiadka <email address hidden>
Date: Thu Feb 8 12:01:20 2024 +0100

    cinder: Stop using admin service token

    In order to do this - we need to add service role to Nova and Cinder.

    Closes-Bug: #2049762

    Change-Id: Ic121bf9f90c9865cd4d08890c80247570ef310ae

Changed in kolla-ansible:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/2023.2)

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/909542

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/2023.1)

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/909543

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/zed)

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/909544

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (unmaintained/yoga)

Fix proposed to branch: unmaintained/yoga
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/909545

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/zed)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/909544
Committed: https://opendev.org/openstack/kolla-ansible/commit/47d97bda7e23a94ab8706acfd0887567d91af237
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 47d97bda7e23a94ab8706acfd0887567d91af237
Author: Michal Nasiadka <email address hidden>
Date: Thu Feb 8 12:01:20 2024 +0100

    cinder: Stop using admin service token

    In order to do this - we need to add service role to Nova and Cinder.

    Closes-Bug: #2049762

    Change-Id: Ic121bf9f90c9865cd4d08890c80247570ef310ae
    (cherry picked from commit 1ef765f69044abae1b838b261481f420a4007121)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/2023.2)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/909542
Committed: https://opendev.org/openstack/kolla-ansible/commit/847bcd4364a7765295f73ebc46b81d84f8fdfd6a
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 847bcd4364a7765295f73ebc46b81d84f8fdfd6a
Author: Michal Nasiadka <email address hidden>
Date: Thu Feb 8 12:01:20 2024 +0100

    cinder: Stop using admin service token

    In order to do this - we need to add service role to Nova and Cinder.

    Closes-Bug: #2049762

    Change-Id: Ic121bf9f90c9865cd4d08890c80247570ef310ae
    (cherry picked from commit 1ef765f69044abae1b838b261481f420a4007121)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/2023.1)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/909543
Committed: https://opendev.org/openstack/kolla-ansible/commit/ffcf271f55135e34cf1e7892f48a8bfc617334bf
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit ffcf271f55135e34cf1e7892f48a8bfc617334bf
Author: Michal Nasiadka <email address hidden>
Date: Thu Feb 8 12:01:20 2024 +0100

    cinder: Stop using admin service token

    In order to do this - we need to add service role to Nova and Cinder.

    Closes-Bug: #2049762

    Change-Id: Ic121bf9f90c9865cd4d08890c80247570ef310ae
    (cherry picked from commit 1ef765f69044abae1b838b261481f420a4007121)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (unmaintained/yoga)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/909545
Committed: https://opendev.org/openstack/kolla-ansible/commit/ee1861fe0080c61092b1fdc3e15c36bf77f9429d
Submitter: "Zuul (22348)"
Branch: unmaintained/yoga

commit ee1861fe0080c61092b1fdc3e15c36bf77f9429d
Author: Michal Nasiadka <email address hidden>
Date: Thu Feb 8 12:01:20 2024 +0100

    cinder: Stop using admin service token

    In order to do this - we need to add service role to Nova and Cinder.

    Closes-Bug: #2049762

    Change-Id: Ic121bf9f90c9865cd4d08890c80247570ef310ae
    (cherry picked from commit 1ef765f69044abae1b838b261481f420a4007121)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 16.5.0

This issue was fixed in the openstack/kolla-ansible 16.5.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 17.3.0

This issue was fixed in the openstack/kolla-ansible 17.3.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 15.6.0

This issue was fixed in the openstack/kolla-ansible 15.6.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to kolla-ansible (stable/2023.2)

Related fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/912066

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on kolla-ansible (stable/2023.2)

Change abandoned by "Sven Kieske <email address hidden>" on branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/912066
Reason: superseded by https://review.opendev.org/c/openstack/kolla-ansible/+/912452

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 18.0.0.0rc1

This issue was fixed in the openstack/kolla-ansible 18.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on kolla-ansible (master)

Change abandoned by "Sven Kieske <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/908145
Reason: superseded by https://review.opendev.org/c/openstack/kolla-ansible/+/908405

To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.