| 2022-09-19 14:33:14 |
Erno Kuvaja |
bug |
|
|
added bug |
| 2022-09-19 14:33:49 |
Erno Kuvaja |
bug |
|
|
added subscriber Pranali Deore |
| 2022-09-19 17:43:39 |
Jeremy Stanley |
description |
The location fix manipulation mentioned in https://security.openstack.org/ossa/OSSA-2016-006.html of disallowing removal of last image location (and thus transition it back to queued) is only partial fix for the issue.
The fix at the time was able to ensure that the hash stayed in the image record causing validations to fail even if the image data itself had changed.
Different efforts to speed up booting and volume creation processes utilizing Copy On Write behaviour of Ceph or various Cinder backends is causing two different scenarios where malicious content can be sneaked into the image.
1) When Nova creates snapshot directly into the Ceph store and creates image through the API adding the location via locations API rather than uploading the data to Glance it omits 2 parts of metadata that would allow identifying alteration of the image data. The image does not have multihash (or checksum) associated with it making validation impossible. The image does not have size metadata either.
2) When image is consumed in COW manner even if it had multihash (checksum) registered in the metadata it does not get validated as the consumer does not read the whole image and calculate checksum of it.
All current implementations of COW handling of the image depends of the direct url and locations API being exposed. As the services are accessing the image with user credentials if Glance is deployed with single API configuration serving both OpenStack services and end users, the malicious end user will have all the tools needed to make this attack valid. Only real mitigation for this issue is to deploy External API endpoint (for user access) and Internal API endpoint (for Openstack services, note that this endpoint needs to be firewalled off from end user access as the credentials are the same). Additional hardening of creating the multihash metadata entries and validating them upon use should be implemented. The dual API deployment should be highlighted clearly in the documentation.
These two behavioural facts causes the image manipulation mentioned in the OSSA-2016-006 (CVE 2016-0757) still possible.
If the image has multihash (checksum) recorded for it, python-glanceclient will reject the image if the data does not match. But this requires manual verification (actually downloading the image) to find out or deep understanding of the technical implementation to match the location URI with the image ID (in Ceph case). The COW consumers will not flag it to anyone and will just happily consume the modified data.
In the case that there is no multihash recorded for the image the only indication for malicious activity would be through comparing the location URI with the image ID (in Ceph case) and there is no other validation channels.
Once the location of the modified image data has been added to the image locations table, Glance will allow deleting the original data as that is not the last location anymore. |
This issue is being treated as a potential security risk under
embargo. Please do not make any public mention of embargoed
(private) security vulnerabilities before their coordinated
publication by the OpenStack Vulnerability Management Team in the
form of an official OpenStack Security Advisory. This includes
discussion of the bug or associated fixes in public forums such as
mailing lists, code review systems and bug trackers. Please also
avoid private disclosure to other individuals not already approved
for access to this information, and provide this same reminder to
those who are made aware of the issue prior to publication. All
discussion should remain confined to this private bug report, and
any proposed fixes should be added to the bug as attachments. This
embargo shall not extend past 2022-12-18 and will be made
public by or on that date even if no fix is identified.
The location fix manipulation mentioned in https://security.openstack.org/ossa/OSSA-2016-006.html of disallowing removal of last image location (and thus transition it back to queued) is only partial fix for the issue.
The fix at the time was able to ensure that the hash stayed in the image record causing validations to fail even if the image data itself had changed.
Different efforts to speed up booting and volume creation processes utilizing Copy On Write behaviour of Ceph or various Cinder backends is causing two different scenarios where malicious content can be sneaked into the image.
1) When Nova creates snapshot directly into the Ceph store and creates image through the API adding the location via locations API rather than uploading the data to Glance it omits 2 parts of metadata that would allow identifying alteration of the image data. The image does not have multihash (or checksum) associated with it making validation impossible. The image does not have size metadata either.
2) When image is consumed in COW manner even if it had multihash (checksum) registered in the metadata it does not get validated as the consumer does not read the whole image and calculate checksum of it.
All current implementations of COW handling of the image depends of the direct url and locations API being exposed. As the services are accessing the image with user credentials if Glance is deployed with single API configuration serving both OpenStack services and end users, the malicious end user will have all the tools needed to make this attack valid. Only real mitigation for this issue is to deploy External API endpoint (for user access) and Internal API endpoint (for Openstack services, note that this endpoint needs to be firewalled off from end user access as the credentials are the same). Additional hardening of creating the multihash metadata entries and validating them upon use should be implemented. The dual API deployment should be highlighted clearly in the documentation.
These two behavioural facts causes the image manipulation mentioned in the OSSA-2016-006 (CVE 2016-0757) still possible.
If the image has multihash (checksum) recorded for it, python-glanceclient will reject the image if the data does not match. But this requires manual verification (actually downloading the image) to find out or deep understanding of the technical implementation to match the location URI with the image ID (in Ceph case). The COW consumers will not flag it to anyone and will just happily consume the modified data.
In the case that there is no multihash recorded for the image the only indication for malicious activity would be through comparing the location URI with the image ID (in Ceph case) and there is no other validation channels.
Once the location of the modified image data has been added to the image locations table, Glance will allow deleting the original data as that is not the last location anymore. |
|
| 2022-09-19 17:43:48 |
Jeremy Stanley |
bug task added |
|
ossa |
|
| 2022-09-19 17:43:55 |
Jeremy Stanley |
ossa: status |
New |
Incomplete |
|
| 2022-09-19 17:44:11 |
Jeremy Stanley |
bug |
|
|
added subscriber Glance Core security contacts |
| 2022-09-19 21:42:45 |
Brian Rosmaita |
cve linked |
|
2016-0757 |
|
| 2022-09-20 10:43:51 |
Erno Kuvaja |
nominated for series |
|
glance/wallaby |
|
| 2022-09-20 10:43:51 |
Erno Kuvaja |
bug task added |
|
glance/wallaby |
|
| 2022-09-20 10:43:51 |
Erno Kuvaja |
nominated for series |
|
glance/yoga |
|
| 2022-09-20 10:43:51 |
Erno Kuvaja |
bug task added |
|
glance/yoga |
|
| 2022-09-20 10:43:51 |
Erno Kuvaja |
nominated for series |
|
glance/xena |
|
| 2022-09-20 10:43:51 |
Erno Kuvaja |
bug task added |
|
glance/xena |
|
| 2022-09-20 10:43:51 |
Erno Kuvaja |
nominated for series |
|
glance/zed |
|
| 2022-09-20 10:43:51 |
Erno Kuvaja |
bug task added |
|
glance/zed |
|
| 2022-09-21 13:26:18 |
Brian Rosmaita |
bug |
|
|
added subscriber Dan Smith |
| 2022-09-22 20:56:44 |
Brian Rosmaita |
affects |
ossa |
ossn |
|
| 2022-10-10 13:59:54 |
Erno Kuvaja |
attachment added |
|
WIP patch for the meeting https://bugs.launchpad.net/glance/+bug/1990157/+attachment/5622726/+files/0001-SEC-WIP-async-hashing-on-location-add.patch |
|
| 2022-10-10 15:10:56 |
Brian Rosmaita |
attachment added |
|
Latest version of the OSSN (6 Oct 2022) https://bugs.launchpad.net/glance/+bug/1990157/+attachment/5622747/+files/0001-Add-OSSN-0090.patch |
|
| 2022-10-12 06:20:44 |
Brian Rosmaita |
attachment added |
|
Rewritten version of the OSSN (12 Oct 2022) https://bugs.launchpad.net/glance/+bug/1990157/+attachment/5623275/+files/0002-Add-OSSN-0090.patch |
|
| 2022-10-12 14:19:01 |
Brian Rosmaita |
bug |
|
|
added subscriber Jay Faulkner |
| 2022-10-12 14:20:42 |
Brian Rosmaita |
bug |
|
|
added subscriber Julia Kreger |
| 2022-10-13 12:49:00 |
Erno Kuvaja |
bug |
|
|
added subscriber unmesh desale |
| 2022-10-13 12:51:10 |
Erno Kuvaja |
removed subscriber unmesh desale |
|
|
|
| 2022-10-13 12:51:29 |
Erno Kuvaja |
bug |
|
|
added subscriber unmesh desale |
| 2022-10-14 14:05:51 |
Jeremy Stanley |
description |
This issue is being treated as a potential security risk under
embargo. Please do not make any public mention of embargoed
(private) security vulnerabilities before their coordinated
publication by the OpenStack Vulnerability Management Team in the
form of an official OpenStack Security Advisory. This includes
discussion of the bug or associated fixes in public forums such as
mailing lists, code review systems and bug trackers. Please also
avoid private disclosure to other individuals not already approved
for access to this information, and provide this same reminder to
those who are made aware of the issue prior to publication. All
discussion should remain confined to this private bug report, and
any proposed fixes should be added to the bug as attachments. This
embargo shall not extend past 2022-12-18 and will be made
public by or on that date even if no fix is identified.
The location fix manipulation mentioned in https://security.openstack.org/ossa/OSSA-2016-006.html of disallowing removal of last image location (and thus transition it back to queued) is only partial fix for the issue.
The fix at the time was able to ensure that the hash stayed in the image record causing validations to fail even if the image data itself had changed.
Different efforts to speed up booting and volume creation processes utilizing Copy On Write behaviour of Ceph or various Cinder backends is causing two different scenarios where malicious content can be sneaked into the image.
1) When Nova creates snapshot directly into the Ceph store and creates image through the API adding the location via locations API rather than uploading the data to Glance it omits 2 parts of metadata that would allow identifying alteration of the image data. The image does not have multihash (or checksum) associated with it making validation impossible. The image does not have size metadata either.
2) When image is consumed in COW manner even if it had multihash (checksum) registered in the metadata it does not get validated as the consumer does not read the whole image and calculate checksum of it.
All current implementations of COW handling of the image depends of the direct url and locations API being exposed. As the services are accessing the image with user credentials if Glance is deployed with single API configuration serving both OpenStack services and end users, the malicious end user will have all the tools needed to make this attack valid. Only real mitigation for this issue is to deploy External API endpoint (for user access) and Internal API endpoint (for Openstack services, note that this endpoint needs to be firewalled off from end user access as the credentials are the same). Additional hardening of creating the multihash metadata entries and validating them upon use should be implemented. The dual API deployment should be highlighted clearly in the documentation.
These two behavioural facts causes the image manipulation mentioned in the OSSA-2016-006 (CVE 2016-0757) still possible.
If the image has multihash (checksum) recorded for it, python-glanceclient will reject the image if the data does not match. But this requires manual verification (actually downloading the image) to find out or deep understanding of the technical implementation to match the location URI with the image ID (in Ceph case). The COW consumers will not flag it to anyone and will just happily consume the modified data.
In the case that there is no multihash recorded for the image the only indication for malicious activity would be through comparing the location URI with the image ID (in Ceph case) and there is no other validation channels.
Once the location of the modified image data has been added to the image locations table, Glance will allow deleting the original data as that is not the last location anymore. |
The location fix manipulation mentioned in https://security.openstack.org/ossa/OSSA-2016-006.html of disallowing removal of last image location (and thus transition it back to queued) is only partial fix for the issue.
The fix at the time was able to ensure that the hash stayed in the image record causing validations to fail even if the image data itself had changed.
Different efforts to speed up booting and volume creation processes utilizing Copy On Write behaviour of Ceph or various Cinder backends is causing two different scenarios where malicious content can be sneaked into the image.
1) When Nova creates snapshot directly into the Ceph store and creates image through the API adding the location via locations API rather than uploading the data to Glance it omits 2 parts of metadata that would allow identifying alteration of the image data. The image does not have multihash (or checksum) associated with it making validation impossible. The image does not have size metadata either.
2) When image is consumed in COW manner even if it had multihash (checksum) registered in the metadata it does not get validated as the consumer does not read the whole image and calculate checksum of it.
All current implementations of COW handling of the image depends of the direct url and locations API being exposed. As the services are accessing the image with user credentials if Glance is deployed with single API configuration serving both OpenStack services and end users, the malicious end user will have all the tools needed to make this attack valid. Only real mitigation for this issue is to deploy External API endpoint (for user access) and Internal API endpoint (for Openstack services, note that this endpoint needs to be firewalled off from end user access as the credentials are the same). Additional hardening of creating the multihash metadata entries and validating them upon use should be implemented. The dual API deployment should be highlighted clearly in the documentation.
These two behavioural facts causes the image manipulation mentioned in the OSSA-2016-006 (CVE 2016-0757) still possible.
If the image has multihash (checksum) recorded for it, python-glanceclient will reject the image if the data does not match. But this requires manual verification (actually downloading the image) to find out or deep understanding of the technical implementation to match the location URI with the image ID (in Ceph case). The COW consumers will not flag it to anyone and will just happily consume the modified data.
In the case that there is no multihash recorded for the image the only indication for malicious activity would be through comparing the location URI with the image ID (in Ceph case) and there is no other validation channels.
Once the location of the modified image data has been added to the image locations table, Glance will allow deleting the original data as that is not the last location anymore. |
|
| 2022-10-14 14:05:59 |
Jeremy Stanley |
ossn: status |
Incomplete |
In Progress |
|
| 2022-10-14 14:06:17 |
Jeremy Stanley |
ossn: assignee |
|
Brian Rosmaita (brian-rosmaita) |
|
| 2022-10-14 14:06:24 |
Jeremy Stanley |
information type |
Private Security |
Public |
|
| 2022-10-14 14:06:33 |
Jeremy Stanley |
tags |
|
security |
|
| 2022-10-14 14:07:00 |
Jeremy Stanley |
summary |
Malicious image data modification can happen when using COW |
OSSN-090: Malicious image data modification can happen when using COW |
|
| 2022-10-14 14:07:16 |
Jeremy Stanley |
summary |
OSSN-090: Malicious image data modification can happen when using COW |
OSSN-0090: Malicious image data modification can happen when using COW |
|
| 2022-10-19 11:18:04 |
Erno Kuvaja |
information type |
Public |
Public Security |
|
| 2022-11-24 00:35:46 |
Nick Tait |
cve linked |
|
2022-4134 |
|
