Openstack service client caching breaks security and backward compatibility

Bug #1627689 reported by Andras Kovi
258
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mistral
Triaged
Medium
Unassigned

Bug Description

Cached openstack clients are initialized with credentials but only tenant id used as key. This causes subsequent calls going to the same Openstack service endpoint to be dependent on each other.

Example:
1. - invoke stacks_list action with admin credentials
2. - invoke stacks_list action with non-privileged credentials

1. will succeed.
2. will use the admin credentials, thereby breaking security.

The caching heavily relies on the expiration time of the authentication tokens, as well, but does not consider invalidated tokens. The invalidated token will be used for all outgoing calls until the token is considered expired.

Backward compatibility is also broken as the original operation involved always using the credentials provided by the client.

Caching should be disabled until these issues are solved.

Changed in mistral:
assignee: nobody → Jeff Peeler (jpeeler-z)
status: New → Confirmed
importance: Undecided → Critical
milestone: none → newton-rc2
information type: Private Security → Public
information type: Public → Public Security
Dougal Matthews (d0ugal)
Changed in mistral:
assignee: Jeff Peeler (jpeeler-z) → Dougal Matthews (d0ugal)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to mistral (master)

Fix proposed to branch: master
Review: https://review.openstack.org/377420

Changed in mistral:
status: Confirmed → In Progress
Revision history for this message
Dougal Matthews (d0ugal) wrote :

I have proposed a patch that should solve the cache key problem - it adds the user_id to the key. As far as I know a users ID will never change and this should be safe to use.

I do not yet have a solution for the expired token problem. The openstack clients are not consistent with their naming of the token on the client itself, so this could be trickier. Although, I think the same token is used for all clients, so there is probably a way we can check this, I will look into it.

"Backward compatibility is also broken as the original operation involved always using the credentials provided by the client." - Andras, I don't fully understand this issue, can you explain it further?

Revision history for this message
Andras Kovi (akovi) wrote :

The original behavior was that the server side used whatever token the client provided to it. This has changed with caching and the same token will be used for every subsequent call. Implementing caching this way can cause unexpected errors on the client side.

IMHO, the client should explicitly signal to the server side that it is Ok with caching. Otherwise, the caching needs to handle every case to ensure that the original behavior does not change.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on mistral (master)

Change abandoned by Dougal Matthews (<email address hidden>) on branch: master
Review: https://review.openstack.org/377420

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to mistral (master)

Fix proposed to branch: master
Review: https://review.openstack.org/377700

Revision history for this message
Dougal Matthews (d0ugal) wrote :

Thanks for the feedback Andras, we are going to follow your recommendation. See the latest proposed patch.

We will revisit this in Ocata, once that change has landed we can re-target this bug for Ocata-1.

Dougal Matthews (d0ugal)
tags: added: newton-rc-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to mistral (master)

Reviewed: https://review.openstack.org/377700
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=ff4f95f643169dee6a3a863e0bd5bec557c05c08
Submitter: Jenkins
Branch: master

commit ff4f95f643169dee6a3a863e0bd5bec557c05c08
Author: Dougal Matthews <email address hidden>
Date: Tue Sep 27 14:35:53 2016 +0100

    Disable Client Caching

    Caching has caused some regressions and security problems. It will be
    revisited in Ocata.

    Partial-Bug: #1627689
    Change-Id: I86fa58de00d01c89e4bbc21dbe128f1306e2a1bf

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to mistral (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/379012

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to mistral (stable/newton)

Reviewed: https://review.openstack.org/379012
Committed: https://git.openstack.org/cgit/openstack/mistral/commit/?id=2d44cb06e53d09e0c7e8b7ee01499bc035450060
Submitter: Jenkins
Branch: stable/newton

commit 2d44cb06e53d09e0c7e8b7ee01499bc035450060
Author: Dougal Matthews <email address hidden>
Date: Tue Sep 27 14:35:53 2016 +0100

    Disable Client Caching

    Caching has caused some regressions and security problems. It will be
    revisited in Ocata.

    Partial-Bug: #1627689
    Change-Id: I86fa58de00d01c89e4bbc21dbe128f1306e2a1bf
    (cherry picked from commit ff4f95f643169dee6a3a863e0bd5bec557c05c08)

tags: added: in-stable-newton
Revision history for this message
Dougal Matthews (d0ugal) wrote :

Caching has now been disabled in Newton. I have lowered the priority for this and targeted it for Ocata 1 as a result.

Changed in mistral:
importance: Critical → Medium
milestone: newton-rc2 → ocata-1
Changed in mistral:
milestone: ocata-1 → ocata-2
Changed in mistral:
milestone: ocata-2 → ocata-3
Changed in mistral:
milestone: ocata-3 → ocata-rc2
Changed in mistral:
milestone: ocata-rc2 → pike-1
Changed in mistral:
milestone: pike-1 → pike-2
Dougal Matthews (d0ugal)
Changed in mistral:
assignee: Dougal Matthews (d0ugal) → nobody
Changed in mistral:
milestone: pike-2 → pike-3
Changed in mistral:
milestone: pike-3 → queens-1
Ryan Brady (rbrady)
Changed in mistral:
status: In Progress → Triaged
Changed in mistral:
milestone: queens-1 → queens-3
Changed in mistral:
milestone: queens-3 → rocky-2
Dougal Matthews (d0ugal)
Changed in mistral:
milestone: rocky-2 → rocky-3
Dougal Matthews (d0ugal)
Changed in mistral:
milestone: rocky-3 → stein-1
Dougal Matthews (d0ugal)
Changed in mistral:
milestone: stein-1 → stein-2
Changed in mistral:
milestone: stein-2 → none
To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.