Normal user can replace active image data if show_multiple_locations has been set to true

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

I was *just* chatting with flwang about this issue. I'll add Fei Long as he's also working on this.

Thanks for the report, Mike.

Firstly, for the cloud operation perspective, the image publish policy should be enabled to avoid non-admin user publish image. But we still can't avoid user swap image data for his personal image. That's the thing we should discuss. Please let me know if I missed something. Thanks.

For clarity, since I'm regrettably unfamiliar with Glance's internals, how does it actually ensure that the data at every listed location for an image is bit-for-bit identical? Does it retrieve them all, calculate checksums and try to match them? Otherwise, it seems like the idea of multiple locations for a single image is inherently incompatible with the idea of image immutability.

That's a good question. And unfortunately, Glance doesn't comparing the checksums when adding a new location.

Given that, I'm inclined to chalk that up to incomplete design and something that's not going to get properly solved by some short, backward-compatible stable backport patches under embargo. If Glance has no actual ability to guarantee that all locations for an image contain identical data then there is no image immutability guarantee while using multiple locations anyway.

Agreed with the B2 class (according to https://security.openstack.org/vmt-process.html#incident-report-taxonomy ).
If no one objects, let's remove privacy setting of this bug report by the end of the week.

I've subscribed OSSG-coresec to discuss an eventual document about Glance image locations.

I agree with Jeremy - this seems like an incomplete design. If an image has multiple locations the hash for the resource at those locations should be required to match.

I'll add a note task for this.

Agree with an OSSN as there's no clear path to a fix.

Should we keep those glance image location related bug reports private and coordinate their disclosure with the OSSN ?

Yes, I think that sounds like a fine approach.

The glance team is planning to gather at some point during the summit to further discuss general issues with image locations. Until then let's keep these private.

+1, we'll follow our "private" note process.

Private OSSN started in Gitlab: https://gitlab.com/hyakuhei/OSSN-TEMP/commit/6163357da189e5e116fdd93ad06a358db7a2cf82

Thoughts on this?

Users of Glance may be able to replace active image data
---

### Summary ###
When Glance has been configured with the "show_multiple_locations"
parameter enabled it is possible for a non-admin user to replace active
image data.

### Affected Services / Software ###
Glance, Havana, Icehouse, Juno, Kilo, Liberty, Mitaka, Newton

### Discussion ###
Glance has a multiple location feature that allows a single image to be
stored in multiple places. This is intended to offer an extra degree of
resilience by improving the availability of Glance images.

This feature involves a user pushing a new location for an image via the
Glance API. However, this process does not involve a checksum of the
existing or newly created image locations - allowing a malicious user to
push a different or altered image as an alternative location for an
existing one.

An attacker could add a malicious image to a location for an existing
one potentially leading to other users of the cloud unknowingly using
the malicious image.

### Recommended Actions ###
In production clouds the image publish policy should be enabled to
prevent non-admin users publishing images that can be used by other
users of the cloud. This does not mitigate the issue completely but it
does constrain the issue to an individual user. It is still possible
that a workload running under one user could be compromised and in turn
abuse the multiple location feature to compromise other workloads
running under that same user.

The safest course of action is to disable support for multiple locations
by editing glance-api.conf:

---- begin glance-api.conf snippet ----
  [DEFAULT]
  show_multiple_locations = False
---- end glance-api.conf snippet ----

### Contacts / References ###
This OSSN : https://wiki.openstack.org/wiki/OSSN/OSSN-0065
Original LaunchPad Bug : https://bugs.launchpad.net/glance/+bug/1549483
OpenStack Security ML : <email address hidden>
OpenStack Security Group : https://launchpad.net/~openstack-ossg
Multiple Image Location BP : https://blueprints.launchpad.net/glance/+spec/multiple-image-locations

Hi Robert,

Thank you for the write up. I've a few suggestions, let me know your thoughts:

#1

### Summary ###
When Glance API has been configured with the "show_multiple_locations"
parameter enabled along with default policies, it is possible for a non-admin user to replace active
image data by manipulating the location by updating Image record.

#2
### Discussion ###
Glance has a multiple location feature that allows a single image to be
stored in multiple places. This is intended to offer an extra degree of
resilience by improving the availability of Glance images as well as to help operators improve performance and reduce errors during inbound and outbound image data streaming.

This feature can allow a user pushing a new location for an image via the
Glance API. However, this process does not involve a checksum of the
existing or newly created image locations - allowing a malicious user to
push a different or altered image as an alternative location for an
existing one.

An attacker could add a malicious data to a non-public image where they are authorized to a location for an existing
image, potentially leading to other users of the cloud unknowingly using
the malicious image, given that image is shared with other users.

#3
### Recommended Actions ###
In production clouds the image publish policy should be enabled to
prevent non-admin users publishing images that can be used by other
users of the cloud. This does not mitigate the issue completely but it
does constrain the issue to an individual user. It is still possible
that a workload running under one user could be compromised and in turn
abuse the multiple location feature to compromise other workloads
running under that same user or an innocent error for specifying incorrect location may result into potential data loss for that cloud without consistency of checksum.

The safest course of action is to disable support for multiple locations
by editing glance-api.conf:

---- begin glance-api.conf snippet ----
  [DEFAULT]
  show_multiple_locations = False
---- end glance-api.conf snippet ----

---

Thoughts?

OSSN is basically ready (updated version in gitlab). Next steps please?

@Nikhil

"the image publish policy should be enabled to prevent non-admin users publishing images"

There is no 'image publish' policy. Should 'image publish policy' read 'publicize_image policy' or something else?

When you say 'publishing images' do you mean 1. marking images as public or 2. both marking images as public and sharing images or 3. just uploading images? Thanks.

@Robert, the next steps are:
* approve the OSSN
* remove privacy settings of location's related bug
* publish the OSSN
* close OSSA task and points to the Security Note

The proposed note sounds good to me.

Can I '-1' this?

It's not possible to follow the recommended action in the proposed note -- it refers to a policy that does not exist.

Also, standard member to member image sharing has not been considered/addressed, so the proposal only partially addresses the issue.

Stuart, I meant that we first need to approve the OSSN before moving on (in order to coordinate the disclosure with the security note publication).

Well if the first paragraph of the recommended action doesn't work, indeed it needs to be fixed first.

Let's wait on the further action before we've come to consensus on what needs to be done.

Good catch Stuart. Thanks for stopping further action on time!

I guess, I missed updating that information in the description!

Yes, "publishing images" is generic enough term to be confused of the scope of the solution (recommended actions).

We should break this down a bit:

1. Operator defined Public images
If cloud operator disallows image sharing and publicizing images, then they need to worry about only images that are defined by admins. There are no actions needed in this case, the public images belonging to the admins are safe.

2. Publishing own images
This is case when sharing is not allowed and cloud is allowing users to publicize their own images; ie. when publicize_image is not restricted to cloud admins. There are no actions needed in this case, the public images belonging to the owners and admins are safe.

3. Shared Images
This is the most unrestricted deployment scenario, image can be potentially shared with any other user of the cloud without the notice of admins and the owner of the image.

recommended actions should be:
a) "set_image_location" and "delete_image_location" should be restricted to cloud admins and owner & project admin

For the #3 uploading images:

I think we can only communicate that users will making their own images redundant if trying to manipulate image data on active images when this config is enabled. Cloud operators should communicate the users of this issues (possibly in a FAQ section) when they decided to deploy this feature.

The policy "publicize_image" shouldn't have any effect on this bug directly (until what's been figured).

---
Error cases:

Unrestricted CR&D on image location

create: as mentioned in the bug description, users may be able to create a incorrect image behind the scene without the knowledge of the owner (if that image is shared with them and set_image_location policy is allowed) or the owner may create an incorrect one inadvertently.

read: it's safe from this bug's perspective

delete: This has the potential of data loss for the user in case they choose to delete their image location inadvertently or if the shared tenant deletes it without knowledge of the user.
---

Thoughts?

Based on above comment and other related issue (bug 1555590 and bug 1546507), I'd like to handle image-location issues as class B2 type of bug according to VMT taxonomy ( http://security.openstack.org/vmt-process.html#incident-report-taxonomy ).

I'm switching the OSSA task status to opinion until a Security Note (OSSN) is defined.
However if a backportable fix can be implemented, then we can revert that decision and issue an advisory.

(This is limited to enabling the non-default option so, marked as High and not Critical)

I'm adding Nova coresec team to the subscribers' list to help tie the ramifications of this bug with the ongoing Glance v2 work.

For the reason why nova-core sec has been added to this bug, please follow the conversation from comment #46 onward here https://bugs.launchpad.net/glance/+bug/1546507 .

OK this seems stalled. What's required to advance Rob's OSSN draft and get this published?

It seems like the proposed OSSN references an incorrect policy, instead it should mention the "set_image_location" and "delete_image_location" as explained in comment #22

Rob, how are you for making the amendment in comment #22, looks like this will be close to complete then.

Been a while since I looked at this, not sure if it needs more tweaking.

Users of Glance may be able to replace active image data
---
### Summary ###
When Glance has been configured with the "show_multiple_locations"
parameter enabled it is possible for a non-admin user to replace active
image data.

### Affected Services / Software ###
Glance, Havana, Icehouse, Juno, Kilo, Liberty, Mitaka, Newton

### Discussion ###
Glance has a multiple location feature that allows a single image to be
stored in multiple places. This is intended to offer an extra degree of
resilience by improving the availability of Glance images.
This feature involves a user pushing a new location for an image via the
Glance API. However, this process does not involve a checksum of the
existing or newly created image locations - allowing a malicious user to
push a different or altered image as an alternative location for an
existing one.
An attacker could add a malicious image to a location for an existing
one potentially leading to other users of the cloud unknowingly using
the malicious image.

### Recommended Actions ###
In production clouds the image publish policy should be enabled to avoid
non-admin users publishing images that can be used by other users of the
cloud. This does not mitigate the issue completely but it does
constrain the issue an individual user.

Cloud operators should updating their Glance configuration such that “set_image_location” and “delete_image_location” are restricted to cloud admins and owner & project admin.

### Contacts / References ###
This OSSN : <link to OSSN on wiki>
Original LaunchPad Bug : <link to launchpad bug for affected project/service>
OpenStack Security ML : <email address hidden>
OpenStack Security Group : https://launchpad.net/~openstack-ossg
Multiple Image Location BP : https://blueprints.launchpad.net/glance/+spec/multiple-image-locations
CVE: <CVE number if one was filed>

A few typo (I think):
s/multiple location/multiple locations/
s/constrain the issue an/constrain the issue to an/
s/should updating/should update/

glance-coresec, please review the last note (comment #29)

glance-coresec, please confirm if you can this reads ok for you, and then I will send out the note with the above replacements.

Just want to acknowledge that we are looking this over, hopefully will get back to you by end of the day.

I took the liberty of rewriting the note to be more clear about the extent of the attack (i.e., the user cannot manipulate the checksum, just the data) and to be more clear about how images can be shared out in Glance and what policies govern this. But I'm not a security guy, so Tristan and Luke may need to rewrite. Also, I will grab Nikhil from Glance coresec to read over for accuracy.

Users of Glance may be able to replace active image data
---
### Summary ###
When Glance has been configured with the "show_multiple_locations"
option enabled it is possible for a non-admin user to replace active
image data.

### Affected Services / Software ###
Glance, Havana, Icehouse, Juno, Kilo, Liberty, Mitaka, Newton, Ocata

### Discussion ###

As a convenience to operators, Glance has a multiple location feature,
disabled by default, that allows a single image to be stored in multiple
places. This is intended to offer an extra degree of resilience by
improving the availability of Glance images. This feature involves a
user setting a new entry in an image's 'locations' list, not visible to
users by default, via the Glance API. However, this process does not
involve taking a checksum of the data in a newly created image location,
and hence does not involve comparing the 'checksum' field of the image
(which is always visible to users) with the checksum of any added
locations. This design opens the possibility that a malicious user
could create an image in Glance, set an additional location on that
image pointing to an altered image, then delete the original location,
so that consumers of the original image would unwittingly be using the
malicious image. Note, however, that this attack vector cannot change
the original image's checksum, and it is limited to images that are
owned by the attacker.

### Recommended Actions ###

The reach of this attack depends upon how broadly usage of the original
image is spread among consumers who do not checksum images before they
are consumed. Glance enables three ways for an image to be made
available to other users:

1 Making an image "public". This makes an image available to all users
  of a cloud. The ability to do this is governed by the
  'publicize_image' policy, which is restricted to the admin role by
  default since the Juno release.

2 Making an image "community". *This feature is only available since
  Otaca.* This makes an image available to all users of a cloud, but
  unlike a "public" image, it does not appear in the default image-list
  response of any user (other than the owner). It is governed by the
  'communitize_image' policy, which is unrestricted by default.

3 Making an image "shared". Glance allows project-to-project image
  sharing, in which a user in project A shares an image with project B
  by making project B a *member* of the image. The ability to do this
  is governed by the 'add_member' policy, which is unrestricted by
  default.

  * Project-to-project sharing is the default, based on the
    'owner_is_tenant' configuration setting in Glance. In a cloud
    configured so that 'owner_is_tenant' is false, image sharing is
    user-to-user. This is a cloud-wide configuration, users may not
    determine whether sharing is project-to-project or owner-to-owner.

Note that what has been discussed so far is independent of the specific
vulnerablity discussed in this notice. We encourage cloud operators to
review their current settings for the policies mentioned above. In
particular, we recommend that the 'publicize_image' policy be restricted
to admins (as it has...

Really nice note Brian. Pretty elaborate and explains almost all details.

A couple of points to consider in this note:

1. Summary

### Summary ###
When Glance has been configured with the "show_multiple_locations" option enabled with default policy for set and delete locations, it is possible for a non-admin user having write access to the image metadata to replace active image data.

2. Public & Community images

These images do not grant write access to the non-owner users of the cloud. Hence, setting an incorrect or malicious locations or manipulations to the image location is not allowed, thus keeping the image location intact and safe for the user.

I think Nikhil's change (1) to the summary is good.

Unless I'm misreading it, (2) is a little misleading, though I'm having trouble explaining exactly what I don't like about it. I think the key point is that arbitrary normal users can't manipulate locations on images that they don't own, even if set_location is enabled for everybody and show_multiple_locations is true. But the attack vector as I see it is that the *owner* is doing a bait-and-switch on the image data, and then hoping that some suckers use the image without checking the checksum. So maybe your point is that we shouldn't get too hysterical about this as long as the deployer is smart about who has publicize_image permissions, and end-users keep in mind that you never know what's going to be on a community or shared image, even if the owner *hasn't* done a bait-and-switch (that is, someone could just make a malicious 'community' image that checksums fine, but contains malware).

Was looking this over again. Should s/consumed/used/ in the first sentence in the "Recommended Actions" section. (Maybe kind of pedantic, but you can't really checksum the image before you consume it from Glance ... you need to consume it first, then checksum before you use it.)

Hi Brian,

If the change is made in the first sentence for "Recommended Actions", does the note look ready to go for you now?

@Luke: I think so, if you also use Nikhil's rewrite of the "Summary".

I'll ping him for one more lookover, but I think his other concern is already captured by the last sentence currently in the "Discussion" section. But let me ask this way: is it obvious to you, after reading this, that the attack is limited to images that an attacker owns (or pwns by stealing the owner's credentials)? If so, then I think this is ready to go.

So, my emphasis on the points:

1. public images do not come under this attack vector as non-admin user doesn't have rights to manipulate the image location unless someone has enabled publicize_image policy. (As a separate discussion we should consider deprecating and then removing this policy as we have community images now, will save it for later)

2. community images do come, in the case where a user shares an image and intentionally or unintentionally tries to change the image data but the checksum doesn't match. Consumers of the image are affected as the checksum doesn't match and all further references to the image (nova, snapshots, clones, etc) are affected.

3. shared images do come under this umbrella with the policy permissions requires for set & delete location

I stand corrected on the point # 3 where the right answer is shared images "do not" come under this umbrella:

https://github.com/openstack/glance/blob/b55dd079e02a9c78b82d7e341d3cc51e6cbadab1/glance/db/sqlalchemy/api.py#L280-L291

So the case for community and shared images is same (i.e point #2 & #3 can be combined)

Overall, I think we can go with what we have plus the changes Brian suggested 36, 37 and 39.

Luke, I think we're good to go. Please see my question in #39, though. The title of this note is a bit scary, I want to make sure that it's clear that you could replace your *own* active image data by this method, not the data of arbitrary images.

Here is a round up with the edits:

My feedback is that this makes sense to myself. What I may need is some help with how to toggle the various values, well I could like figure this out, but may need to ping one of you for clarification.

Users of Glance may be able to replace active image data
---
### Summary ###
When Glance has been configured with the "show_multiple_locations" option
enabled with default policy for set and delete locations, it is possible for a
non-admin user having write access to the image metadata to replace active
image data.

### Affected Services / Software ###
Glance, Havana, Icehouse, Juno, Kilo, Liberty, Mitaka, Newton, Ocata

### Discussion ###

As a convenience to operators, Glance has a multiple location feature,
disabled by default, that allows a single image to be stored in multiple
places. This is intended to offer an extra degree of resilience by
improving the availability of Glance images. This feature involves a
user setting a new entry in an image's 'locations' list, not visible to
users by default, via the Glance API. However, this process does not
involve taking a checksum of the data in a newly created image location,
and hence does not involve comparing the 'checksum' field of the image
(which is always visible to users) with the checksum of any added
locations. This design opens the possibility that a malicious user
could create an image in Glance, set an additional location on that
image pointing to an altered image, then delete the original location,
so that consumers of the original image would unwittingly be using the
malicious image. Note, however, that this attack vector cannot change
the original image's checksum, and it is limited to images that are
owned by the attacker.

### Recommended Actions ###

The reach of this attack depends upon how broadly usage of the original
image is spread among consumers who do not checksum images before they
are used. Glance enables three ways for an image to be made
available to other users:

1 Making an image "public". This makes an image available to all users
  of a cloud. The ability to do this is governed by the
  'publicize_image' policy, which is restricted to the admin role by
  default since the Juno release.

2 Making an image "community". *This feature is only available since
  Otaca.* This makes an image available to all users of a cloud, but
  unlike a "public" image, it does not appear in the default image-list
  response of any user (other than the owner). It is governed by the
  'communitize_image' policy, which is unrestricted by default.

3 Making an image "shared". Glance allows project-to-project image
  sharing, in which a user in project A shares an image with project B
  by making project B a *member* of the image. The ability to do this
  is governed by the 'add_member' policy, which is unrestricted by
  default.

  * Project-to-project sharing is the default, based on the
    'owner_is_tenant' configuration setting in Glance. In a cloud
    configured so that 'owner_is_tenant' is false, image sharing is
    user-to-user. This is a cloud-wide configuration, users may not
    determine whether sharing is project-to-project or owner-to...

Luke, LGTM. Only thing I noticed is that in "Recommended Actions" item 2, 'Otaca' should be 'Ocata' (and actually, it's my error, not yours!).

As far as "toggling" the options goes, if show_multiple_locations is False, then this attack vector is nullified. If show_multiple_locations is True, then the operator needs to use the three "location" policy targets to minimize the attack surface described in the note. But exactly how to set these for a particular cloud, we can't say, it's too variable.

Great, I will send out..

Update that this will go out to public lists tomorrow