[SRU] Python3 librados incompatibility

Bug #1816721 reported by Michal Arbet
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu Cloud Archive
Fix Released
Undecided
Unassigned
Rocky
Fix Released
High
Unassigned
Stein
Fix Released
Undecided
Unassigned
glance_store
Fix Released
Undecided
Michal Arbet
Rocky
Fix Committed
Undecided
Unassigned
Stein
New
Undecided
Unassigned
Train
New
Undecided
Unassigned
python-glance-store (Ubuntu)
Invalid
Undecided
Unassigned
Cosmic
Fix Released
High
Unassigned
Disco
Fix Released
Undecided
Unassigned

Bug Description

[Impact]
For >= rocky (i.e. if using py3 packages) librados.cluster.get_fsid() is returning a binary string which results in spurious/invalid invalid characters being stored in the database e.g. for the image_location. This patch ensures that the value is converted to a normal string.

[Test Case]
* deploy openstack rocky (using p3 packages)
* deploy ceph and use for glance backend
* set
/etc/glance/glance-api.conf:show_multiple_locations = True
/etc/glance/glance-api.conf:show_image_direct_url = True
* upload image
* query image_locations table in glance db
mysql -h$host -u${user} -p$passwd glance -e'select * from image_locations;'
* ensure that the stored value is correct i.e. does not have "b%27" at the beginning.

[Regression Potential]
None expected

[Other Info]
Upgrading to this patch won't fix existing images, they will need to be manually updated in the database.

----------------------------------------------------------------------------

Hello, found a bug in glance. Glance is storing bad direct_url='rbd://b%27868c2b5d-12f1-4f3f-aa2a-5701a3bb1041%27/images/7b1f429e-ad2f-40b2-be9e-8552edae8938/snap' in database. This is causing several problems, as in https://bugs.launchpad.net/cinder/+bug/1816468

root@openstack-controller:/tmp# openstack image create --container-format bare --disk-format raw --file cirros-0.3.4-x86_64-disk.img cirros-test-1

+------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| checksum | ee1eca47dc88f4879d8a229cc70a07c6 |
| container_format | bare |
| created_at | 2019-02-20T09:08:45Z |
| disk_format | raw |
| file | /v2/images/7b1f429e-ad2f-40b2-be9e-8552edae8938/file |
| id | 7b1f429e-ad2f-40b2-be9e-8552edae8938 |
| min_disk | 0 |
| min_ram | 0 |
| name | cirros-test-1 |
| owner | ba5ef70fd99642fdb75c9307c88b1164 |
| properties | direct_url='rbd://b%27868c2b5d-12f1-4f3f-aa2a-5701a3bb1041%27/images/7b1f429e-ad2f-40b2-be9e-8552edae8938/snap', os_hash_algo='sha512', os_hash_value='1b03ca1bc3fafe448b90583c12f367949f8b0e665685979d95b004e48574b953316799e23240f4f739d1b5eb4c4ca24d38fdc6f4f9d8247a2bc64db25d6bbdb2', os_hidden='False' |
| protected | False |
| schema | /v2/schemas/image |
| size | 13287936 |
| status | active |
| tags | |
| updated_at | 2019-02-20T09:08:48Z |
| virtual_size | None |
| visibility | shared |
+------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

mysql> select * from image_locations where image_id = '7b1f429e-ad2f-40b2-be9e-8552edae8938'\G;
*************************** 1. row ***************************
        id: 32
  image_id: 7b1f429e-ad2f-40b2-be9e-8552edae8938
     value: rbd://b%27868c2b5d-12f1-4f3f-aa2a-5701a3bb1041%27/images/7b1f429e-ad2f-40b2-be9e-8552edae8938/snap
created_at: 2019-02-20 09:08:48
updated_at: 2019-02-20 09:08:48
deleted_at: NULL
   deleted: 0
 meta_data: {}
    status: active
1 row in set (0.00 sec)

ERROR:
No query specified

get_fsid function from librados returns different ID representation, in py2 it is STR, in py3 it is bytes.

It can be fixed by oslo_utils.encodeutils.safe_decode

I will post the change.

affects: glance → glance-store
Revision history for this message
Michal Arbet (michalarbet) wrote :

Also causing cinder issues, cinder is parsing ID from direct url , when direct url is corrupted in db ... cinder-glance acceleration is not working =>

https://bugs.launchpad.net/cinder/+bug/1816468

Changed in glance-store:
assignee: nobody → Michal Arbet (michalarbet)
Changed in glance-store:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance_store (master)

Reviewed: https://review.openstack.org/635230
Committed: https://git.openstack.org/cgit/openstack/glance_store/commit/?id=9c73370358c02fc7f308d88f4563f22969a42c96
Submitter: Zuul
Branch: master

commit 9c73370358c02fc7f308d88f4563f22969a42c96
Author: Michal Arbet <email address hidden>
Date: Wed Feb 6 18:38:22 2019 +0100

    Fix python3 compatibility of rbd get_fsid

    In python3 conn.get_fsid() is represented as binary.
    Because of this, direct_url is corrupted in DB.
    This patch is fixing this. More informations in
    closing bug.

    Closes-Bug: #1816721

    Change-Id: I6a89042fae63bf5c9096a6bee6774aebf3f5864b

Changed in glance-store:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance_store 0.28.0

This issue was fixed in the openstack/glance_store 0.28.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance_store (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/663857

description: updated
summary: - Python3 librados incompatibility
+ [SRU] Python3 librados incompatibility
tags: added: sts-sru-needed
Changed in python-glance-store (Ubuntu Disco):
status: New → Fix Released
Felipe Reyes (freyes)
tags: added: sts
Changed in python-glance-store (Ubuntu):
status: New → Invalid
Changed in python-glance-store (Ubuntu Cosmic):
status: New → Triaged
importance: Undecided → High
Revision history for this message
Corey Bryant (corey.bryant) wrote :

This has been uploaded to the Cosmic unapproved queue and is awaiting review by the SRU team:
https://launchpad.net/ubuntu/cosmic/+queue?queue_state=1&queue_text=python-glance-store

Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Michal, or anyone else affected,

Accepted python-glance-store into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-glance-store/0.26.1-0ubuntu2.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in python-glance-store (Ubuntu Cosmic):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-cosmic
Revision history for this message
Corey Bryant (corey.bryant) wrote :

Hello Michal, or anyone else affected,

Accepted python-glance-store into rocky-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:rocky-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-rocky-needed to verification-rocky-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-rocky-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-rocky-needed
Revision history for this message
Edward Hope-Morley (hopem) wrote :

Cosmic verified using [Test Case]

tags: added: verification-done-cosmic
removed: verification-needed-cosmic
Revision history for this message
Edward Hope-Morley (hopem) wrote :

Xenial+Rocky verified using [Test Case]

tags: added: verification-done verification-rocky-done
removed: verification-needed verification-rocky-needed
Revision history for this message
Edward Hope-Morley (hopem) wrote :

root@juju-b1c912-lp1816721-cosmic-sru-5:~# dpkg -l| grep glance-api
ii glance-api 2:17.0.0-0ubuntu4.1 all OpenStack Image Registry and Delivery Service - API
root@juju-b1c912-lp1816721-cosmic-sru-5:~# mysql -h$host -u${service} -p$passwd ${service} -e'select * from image_locations;'
+----+--------------------------------------+---------------------------------------------------------------------------------------------+---------------------+---------------------+------------+---------+-----------+--------+
| id | image_id | value | created_at | updated_at | deleted_at | deleted | meta_data | status |
+----+--------------------------------------+---------------------------------------------------------------------------------------------+---------------------+---------------------+------------+---------+-----------+--------+
| 1 | 26e83cd0-1692-4516-85c2-7c994330e0e4 | rbd://374de550-a32a-11e9-b0f4-fa163e55087b/glance/26e83cd0-1692-4516-85c2-7c994330e0e4/snap | 2019-07-10 19:33:25 | 2019-07-10 19:33:25 | NULL | 0 | {} | active |
+----+--------------------------------------+---------------------------------------------------------------------------------------------+---------------------+---------------------+------------+---------+-----------+--------+

Revision history for this message
Edward Hope-Morley (hopem) wrote :

Cosmic verification test output ^^

Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Since this is an SRU of the python-glance-store we'd normally need to know the version number of python-glance-store that was used for verification, not glance-api (as it's built from a different package). That being said, the test output looks otherwise good, so I'll conditionally accept this verification.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-glance-store - 0.26.1-0ubuntu2.2

---------------
python-glance-store (0.26.1-0ubuntu2.2) cosmic; urgency=medium

  * d/p/fix-python3-compatibility-of-rbd-get_fsid.patch: Cherry-picked
    from https://review.opendev.org/#/c/663857 to prevent invalid
    characters from being stored in database (LP: #1816721).

 -- Corey Bryant <email address hidden> Mon, 10 Jun 2019 10:53:32 -0400

Changed in python-glance-store (Ubuntu Cosmic):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for python-glance-store has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Corey Bryant (corey.bryant) wrote :

The verification of the Stable Release Update for python-glance-store has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Corey Bryant (corey.bryant) wrote :

This bug was fixed in the package python-glance-store - 0.26.1-0ubuntu2.2~cloud0
---------------

 python-glance-store (0.26.1-0ubuntu2.2~cloud0) bionic-rocky; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 python-glance-store (0.26.1-0ubuntu2.2) cosmic; urgency=medium
 .
   * d/p/fix-python3-compatibility-of-rbd-get_fsid.patch: Cherry-picked
     from https://review.opendev.org/#/c/663857 to prevent invalid
     characters from being stored in database (LP: #1816721).

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance_store (stable/rocky)

Reviewed: https://review.opendev.org/663857
Committed: https://git.openstack.org/cgit/openstack/glance_store/commit/?id=7880a5bfa7639fd6aa07d96b32da594c76940e38
Submitter: Zuul
Branch: stable/rocky

commit 7880a5bfa7639fd6aa07d96b32da594c76940e38
Author: Michal Arbet <email address hidden>
Date: Wed Feb 6 18:38:22 2019 +0100

    Fix python3 compatibility of rbd get_fsid

    In python3 conn.get_fsid() is represented as binary.
    Because of this, direct_url is corrupted in DB.
    This patch is fixing this. More informations in
    closing bug.

    Closes-Bug: #1816721

    Change-Id: I6a89042fae63bf5c9096a6bee6774aebf3f5864b
    (cherry picked from commit 9c73370358c02fc7f308d88f4563f22969a42c96)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance_store 0.26.2

This issue was fixed in the openstack/glance_store 0.26.2 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.