| 2023-05-19 00:30:01 |
Nobuto Murata |
bug |
|
|
added bug |
| 2023-05-19 00:41:35 |
OpenStack Infra |
python-glanceclient: status |
New |
In Progress |
|
| 2023-05-19 00:45:11 |
Nobuto Murata |
description |
When Cinder volume service creates a volume from an image, it will download an image from Glance to a cinder-volume unit (to be precise, when CoW is not available by the backend between Glance and Cinder).
The image fetch speed is not super fast as of today:
2023-05-17 02:52:28.275 158391 DEBUG cinder.image.image_utils [req-e4234307-ae7c-4220-a10a-15760679ac01 a32b8e6ed20f484eba8ddae8f7b6eb1e 08dd29cd8676470c8e0b73d2667f8cf0 - - -] Image fetch details: dest /var/lib/cinder/conversion/image_fetch_741953c3-08f7-43b6-b423-2be938ac8d02_bc1hvgd7cinder@cinder-ceph, size 1907.35 MB, duration 27.80 sec fetch /usr/lib/python3/dist-packages/cinder/image/image_utils.py:637
2023-05-17 02:52:28.275 158391 INFO cinder.image.image_utils [req-e4234307-ae7c-4220-a10a-15760679ac01 a32b8e6ed20f484eba8ddae8f7b6eb1e 08dd29cd8676470c8e0b73d2667f8cf0 - - -] Image download 1907.35 MB at 68.61 MB/s
-> ~ 549 Mbps
By running a profiler, it looks like serious_integrity_iter (glanceclient/common/utils.py) is consuming a lot of CPU cycles on the client side (in this case the cinder-volume unit). |
When Cinder volume service creates a volume from an image, it will download an image from Glance to a cinder-volume unit (to be precise, when CoW is not available by the backend between Glance and Cinder).
The image fetch speed is not super fast as of today:
2023-05-17 02:52:28.275 158391 DEBUG cinder.image.image_utils [req-e4234307-ae7c-4220-a10a-15760679ac01 a32b8e6ed20f484eba8ddae8f7b6eb1e 08dd29cd8676470c8e0b73d2667f8cf0 - - -] Image fetch details: dest /var/lib/cinder/conversion/image_fetch_741953c3-08f7-43b6-b423-2be938ac8d02_bc1hvgd7cinder@cinder-ceph, size 1907.35 MB, duration 27.80 sec fetch /usr/lib/python3/dist-packages/cinder/image/image_utils.py:637
2023-05-17 02:52:28.275 158391 INFO cinder.image.image_utils [req-e4234307-ae7c-4220-a10a-15760679ac01 a32b8e6ed20f484eba8ddae8f7b6eb1e 08dd29cd8676470c8e0b73d2667f8cf0 - - -] Image download 1907.35 MB at 68.61 MB/s
-> ~ 549 Mbps
By running a profiler, it looks like serious_integrity_iter (glanceclient/common/utils.py) is consuming a lot of CPU cycles on the client side (in this case the cinder-volume unit).
Image caching can overcome this kind of challenges but this report is for when we need to download images (new images, updated images, etc.).
https://docs.openstack.org/cinder/latest/admin/image-volume-cache.html |
|
| 2023-07-07 13:59:18 |
Nobuto Murata |
python-glanceclient: status |
In Progress |
Fix Committed |
|
| 2025-05-19 18:17:50 |
Rodrigo Barbieri |
bug task added |
|
python-glanceclient (Ubuntu) |
|
| 2025-05-19 18:18:06 |
Rodrigo Barbieri |
bug task added |
|
python-openstacksdk (Ubuntu) |
|
| 2025-05-19 18:20:09 |
Rodrigo Barbieri |
bug task added |
|
cloud-archive |
|
| 2025-05-19 18:20:21 |
Rodrigo Barbieri |
summary |
Image fetch speed from Glance to Cinder volume service can be slow and capped by client CPU |
[SRU] Image fetch speed from Glance to Cinder volume service can be slow and capped by client CPU |
|
| 2025-05-19 18:30:47 |
Rodrigo Barbieri |
nominated for series |
|
cloud-archive/yoga |
|
| 2025-05-19 18:30:47 |
Rodrigo Barbieri |
bug task added |
|
cloud-archive/yoga |
|
| 2025-05-19 18:30:47 |
Rodrigo Barbieri |
nominated for series |
|
cloud-archive/caracal |
|
| 2025-05-19 18:30:47 |
Rodrigo Barbieri |
bug task added |
|
cloud-archive/caracal |
|
| 2025-05-19 18:30:47 |
Rodrigo Barbieri |
nominated for series |
|
cloud-archive/bobcat |
|
| 2025-05-19 18:30:47 |
Rodrigo Barbieri |
bug task added |
|
cloud-archive/bobcat |
|
| 2025-05-19 18:30:58 |
Rodrigo Barbieri |
cloud-archive/bobcat: status |
New |
Fix Released |
|
| 2025-05-19 18:31:03 |
Rodrigo Barbieri |
cloud-archive/caracal: status |
New |
Fix Released |
|
| 2025-05-19 18:31:20 |
Rodrigo Barbieri |
cloud-archive: status |
New |
Fix Released |
|
| 2025-05-19 18:31:37 |
Rodrigo Barbieri |
cloud-archive/yoga: status |
New |
In Progress |
|
| 2025-05-19 18:31:45 |
Rodrigo Barbieri |
cloud-archive/yoga: assignee |
|
Rodrigo Barbieri (rodrigo-barbieri2010) |
|
| 2025-05-19 18:31:53 |
Rodrigo Barbieri |
python-glanceclient (Ubuntu): status |
New |
Fix Released |
|
| 2025-05-19 18:31:58 |
Rodrigo Barbieri |
python-openstacksdk (Ubuntu): status |
New |
Fix Released |
|
| 2025-05-19 18:32:54 |
Rodrigo Barbieri |
nominated for series |
|
Ubuntu Jammy |
|
| 2025-05-19 18:32:54 |
Rodrigo Barbieri |
bug task added |
|
python-glanceclient (Ubuntu Jammy) |
|
| 2025-05-19 18:32:54 |
Rodrigo Barbieri |
bug task added |
|
python-openstacksdk (Ubuntu Jammy) |
|
| 2025-05-19 18:32:54 |
Rodrigo Barbieri |
nominated for series |
|
Ubuntu Noble |
|
| 2025-05-19 18:32:54 |
Rodrigo Barbieri |
bug task added |
|
python-glanceclient (Ubuntu Noble) |
|
| 2025-05-19 18:32:54 |
Rodrigo Barbieri |
bug task added |
|
python-openstacksdk (Ubuntu Noble) |
|
| 2025-05-19 18:33:14 |
Rodrigo Barbieri |
python-glanceclient (Ubuntu Jammy): status |
New |
In Progress |
|
| 2025-05-19 18:33:14 |
Rodrigo Barbieri |
python-glanceclient (Ubuntu Jammy): assignee |
|
Rodrigo Barbieri (rodrigo-barbieri2010) |
|
| 2025-05-19 18:33:32 |
Rodrigo Barbieri |
python-glanceclient (Ubuntu Noble): status |
New |
Fix Released |
|
| 2025-05-19 18:34:07 |
Rodrigo Barbieri |
python-openstacksdk (Ubuntu Noble): status |
New |
Fix Released |
|
| 2025-05-19 18:34:25 |
Rodrigo Barbieri |
python-openstacksdk (Ubuntu Jammy): status |
New |
In Progress |
|
| 2025-05-19 18:34:25 |
Rodrigo Barbieri |
python-openstacksdk (Ubuntu Jammy): assignee |
|
Rodrigo Barbieri (rodrigo-barbieri2010) |
|
| 2025-05-19 18:36:03 |
Rodrigo Barbieri |
description |
When Cinder volume service creates a volume from an image, it will download an image from Glance to a cinder-volume unit (to be precise, when CoW is not available by the backend between Glance and Cinder).
The image fetch speed is not super fast as of today:
2023-05-17 02:52:28.275 158391 DEBUG cinder.image.image_utils [req-e4234307-ae7c-4220-a10a-15760679ac01 a32b8e6ed20f484eba8ddae8f7b6eb1e 08dd29cd8676470c8e0b73d2667f8cf0 - - -] Image fetch details: dest /var/lib/cinder/conversion/image_fetch_741953c3-08f7-43b6-b423-2be938ac8d02_bc1hvgd7cinder@cinder-ceph, size 1907.35 MB, duration 27.80 sec fetch /usr/lib/python3/dist-packages/cinder/image/image_utils.py:637
2023-05-17 02:52:28.275 158391 INFO cinder.image.image_utils [req-e4234307-ae7c-4220-a10a-15760679ac01 a32b8e6ed20f484eba8ddae8f7b6eb1e 08dd29cd8676470c8e0b73d2667f8cf0 - - -] Image download 1907.35 MB at 68.61 MB/s
-> ~ 549 Mbps
By running a profiler, it looks like serious_integrity_iter (glanceclient/common/utils.py) is consuming a lot of CPU cycles on the client side (in this case the cinder-volume unit).
Image caching can overcome this kind of challenges but this report is for when we need to download images (new images, updated images, etc.).
https://docs.openstack.org/cinder/latest/admin/image-volume-cache.html |
*********** SRU TEMPLATE AT THE BOTTOM **************
When Cinder volume service creates a volume from an image, it will download an image from Glance to a cinder-volume unit (to be precise, when CoW is not available by the backend between Glance and Cinder).
The image fetch speed is not super fast as of today:
2023-05-17 02:52:28.275 158391 DEBUG cinder.image.image_utils [req-e4234307-ae7c-4220-a10a-15760679ac01 a32b8e6ed20f484eba8ddae8f7b6eb1e 08dd29cd8676470c8e0b73d2667f8cf0 - - -] Image fetch details: dest /var/lib/cinder/conversion/image_fetch_741953c3-08f7-43b6-b423-2be938ac8d02_bc1hvgd7cinder@cinder-ceph, size 1907.35 MB, duration 27.80 sec fetch /usr/lib/python3/dist-packages/cinder/image/image_utils.py:637
2023-05-17 02:52:28.275 158391 INFO cinder.image.image_utils [req-e4234307-ae7c-4220-a10a-15760679ac01 a32b8e6ed20f484eba8ddae8f7b6eb1e 08dd29cd8676470c8e0b73d2667f8cf0 - - -] Image download 1907.35 MB at 68.61 MB/s
-> ~ 549 Mbps
By running a profiler, it looks like serious_integrity_iter (glanceclient/common/utils.py) is consuming a lot of CPU cycles on the client side (in this case the cinder-volume unit).
Image caching can overcome this kind of challenges but this report is for when we need to download images (new images, updated images, etc.).
https://docs.openstack.org/cinder/latest/admin/image-volume-cache.html
===============
SRU DESCRIPTION
===============
[Impact]
TBD
[Test case]
TBD
[Where problems could occur]
TBD
[Other Info]
TBD |
|
| 2025-05-19 18:54:43 |
Rodrigo Barbieri |
description |
*********** SRU TEMPLATE AT THE BOTTOM **************
When Cinder volume service creates a volume from an image, it will download an image from Glance to a cinder-volume unit (to be precise, when CoW is not available by the backend between Glance and Cinder).
The image fetch speed is not super fast as of today:
2023-05-17 02:52:28.275 158391 DEBUG cinder.image.image_utils [req-e4234307-ae7c-4220-a10a-15760679ac01 a32b8e6ed20f484eba8ddae8f7b6eb1e 08dd29cd8676470c8e0b73d2667f8cf0 - - -] Image fetch details: dest /var/lib/cinder/conversion/image_fetch_741953c3-08f7-43b6-b423-2be938ac8d02_bc1hvgd7cinder@cinder-ceph, size 1907.35 MB, duration 27.80 sec fetch /usr/lib/python3/dist-packages/cinder/image/image_utils.py:637
2023-05-17 02:52:28.275 158391 INFO cinder.image.image_utils [req-e4234307-ae7c-4220-a10a-15760679ac01 a32b8e6ed20f484eba8ddae8f7b6eb1e 08dd29cd8676470c8e0b73d2667f8cf0 - - -] Image download 1907.35 MB at 68.61 MB/s
-> ~ 549 Mbps
By running a profiler, it looks like serious_integrity_iter (glanceclient/common/utils.py) is consuming a lot of CPU cycles on the client side (in this case the cinder-volume unit).
Image caching can overcome this kind of challenges but this report is for when we need to download images (new images, updated images, etc.).
https://docs.openstack.org/cinder/latest/admin/image-volume-cache.html
===============
SRU DESCRIPTION
===============
[Impact]
TBD
[Test case]
TBD
[Where problems could occur]
TBD
[Other Info]
TBD |
*********** SRU TEMPLATE AT THE BOTTOM **************
When Cinder volume service creates a volume from an image, it will download an image from Glance to a cinder-volume unit (to be precise, when CoW is not available by the backend between Glance and Cinder).
The image fetch speed is not super fast as of today:
2023-05-17 02:52:28.275 158391 DEBUG cinder.image.image_utils [req-e4234307-ae7c-4220-a10a-15760679ac01 a32b8e6ed20f484eba8ddae8f7b6eb1e 08dd29cd8676470c8e0b73d2667f8cf0 - - -] Image fetch details: dest /var/lib/cinder/conversion/image_fetch_741953c3-08f7-43b6-b423-2be938ac8d02_bc1hvgd7cinder@cinder-ceph, size 1907.35 MB, duration 27.80 sec fetch /usr/lib/python3/dist-packages/cinder/image/image_utils.py:637
2023-05-17 02:52:28.275 158391 INFO cinder.image.image_utils [req-e4234307-ae7c-4220-a10a-15760679ac01 a32b8e6ed20f484eba8ddae8f7b6eb1e 08dd29cd8676470c8e0b73d2667f8cf0 - - -] Image download 1907.35 MB at 68.61 MB/s
-> ~ 549 Mbps
By running a profiler, it looks like serious_integrity_iter (glanceclient/common/utils.py) is consuming a lot of CPU cycles on the client side (in this case the cinder-volume unit).
Image caching can overcome this kind of challenges but this report is for when we need to download images (new images, updated images, etc.).
https://docs.openstack.org/cinder/latest/admin/image-volume-cache.html
===============
SRU DESCRIPTION
===============
[Impact]
This issue causes images to be download much slower than they should, causing VM creation to potentially time out if the image is too big and takes too long. The fix proposed greatly improves performance and prevents timeouts in most cases where it would timeout without the fix.
[Test case]
TBD
[Where problems could occur]
TBD
[Other Info]
TBD |
|
| 2025-05-19 18:55:00 |
Rodrigo Barbieri |
attachment added |
|
lp2020139_jammy.debdiff https://bugs.launchpad.net/ubuntu/jammy/+source/python-openstacksdk/+bug/2020139/+attachment/5878881/+files/lp2020139_jammy.debdiff |
|
| 2025-05-19 18:55:25 |
Rodrigo Barbieri |
attachment added |
|
lp2020139_jammy_openstacksdk.debdiff https://bugs.launchpad.net/ubuntu/jammy/+source/python-openstacksdk/+bug/2020139/+attachment/5878882/+files/lp2020139_jammy_openstacksdk.debdiff |
|
| 2025-05-20 15:50:01 |
Rodrigo Barbieri |
description |
*********** SRU TEMPLATE AT THE BOTTOM **************
When Cinder volume service creates a volume from an image, it will download an image from Glance to a cinder-volume unit (to be precise, when CoW is not available by the backend between Glance and Cinder).
The image fetch speed is not super fast as of today:
2023-05-17 02:52:28.275 158391 DEBUG cinder.image.image_utils [req-e4234307-ae7c-4220-a10a-15760679ac01 a32b8e6ed20f484eba8ddae8f7b6eb1e 08dd29cd8676470c8e0b73d2667f8cf0 - - -] Image fetch details: dest /var/lib/cinder/conversion/image_fetch_741953c3-08f7-43b6-b423-2be938ac8d02_bc1hvgd7cinder@cinder-ceph, size 1907.35 MB, duration 27.80 sec fetch /usr/lib/python3/dist-packages/cinder/image/image_utils.py:637
2023-05-17 02:52:28.275 158391 INFO cinder.image.image_utils [req-e4234307-ae7c-4220-a10a-15760679ac01 a32b8e6ed20f484eba8ddae8f7b6eb1e 08dd29cd8676470c8e0b73d2667f8cf0 - - -] Image download 1907.35 MB at 68.61 MB/s
-> ~ 549 Mbps
By running a profiler, it looks like serious_integrity_iter (glanceclient/common/utils.py) is consuming a lot of CPU cycles on the client side (in this case the cinder-volume unit).
Image caching can overcome this kind of challenges but this report is for when we need to download images (new images, updated images, etc.).
https://docs.openstack.org/cinder/latest/admin/image-volume-cache.html
===============
SRU DESCRIPTION
===============
[Impact]
This issue causes images to be download much slower than they should, causing VM creation to potentially time out if the image is too big and takes too long. The fix proposed greatly improves performance and prevents timeouts in most cases where it would timeout without the fix.
[Test case]
TBD
[Where problems could occur]
TBD
[Other Info]
TBD |
*********** SRU TEMPLATE AT THE BOTTOM **************
When Cinder volume service creates a volume from an image, it will download an image from Glance to a cinder-volume unit (to be precise, when CoW is not available by the backend between Glance and Cinder).
The image fetch speed is not super fast as of today:
2023-05-17 02:52:28.275 158391 DEBUG cinder.image.image_utils [req-e4234307-ae7c-4220-a10a-15760679ac01 a32b8e6ed20f484eba8ddae8f7b6eb1e 08dd29cd8676470c8e0b73d2667f8cf0 - - -] Image fetch details: dest /var/lib/cinder/conversion/image_fetch_741953c3-08f7-43b6-b423-2be938ac8d02_bc1hvgd7cinder@cinder-ceph, size 1907.35 MB, duration 27.80 sec fetch /usr/lib/python3/dist-packages/cinder/image/image_utils.py:637
2023-05-17 02:52:28.275 158391 INFO cinder.image.image_utils [req-e4234307-ae7c-4220-a10a-15760679ac01 a32b8e6ed20f484eba8ddae8f7b6eb1e 08dd29cd8676470c8e0b73d2667f8cf0 - - -] Image download 1907.35 MB at 68.61 MB/s
-> ~ 549 Mbps
By running a profiler, it looks like serious_integrity_iter (glanceclient/common/utils.py) is consuming a lot of CPU cycles on the client side (in this case the cinder-volume unit).
Image caching can overcome this kind of challenges but this report is for when we need to download images (new images, updated images, etc.).
https://docs.openstack.org/cinder/latest/admin/image-volume-cache.html
===============
SRU DESCRIPTION
===============
[Impact]
This issue causes images to be download much slower than they should, causing VM creation to potentially time out if the image is too big and takes too long. The fix proposed greatly improves performance and prevents timeouts in most cases where it would timeout without the fix.
[Test case]
1. Deploy OpenStack with Cinder and Glance
2. Upload a resonable large image, like jammy ~650mb is enough.
3. Create a volume from the image
openstack volume create --size 3 --image jammy v1
4. Check the logs for messages of time to fetch/download the image:
... cinder.image.image_utils ... Image fetch details: dest /var/lib/cinder/conversion/image_fetch_ed429779-0d11-45cb-8ee5-fee7c2003d86_rubea933juju-056b6e-jy3-0@LVM-default, size 645.16 MB, duration 3.59 sec fetch ...
... cinder.image.image_utils ... Image download 645.16 MB at 179.58 MB/s
5. Repeat steps (3) and (4) 3 times just to make sure it is consistent and to have an average (avoid underlying infra fluctuations)
6. Install fix and restart all cinder services (such as apache2, cinder-scheduler and cinder-volume)
7. Repeat steps (3), (4) and (5), and make sure the new duration and download rate is at least about 50% faster.
[Where problems could occur]
If there are issues with the fix, it could impact the ability to download images at all, therefore impacting the ability to create volumes from images, and create VMs. Reverting the package is enough to undo the changes and restore previous functionality.
[Other Info]
Both python-glanceclient and openstacksdk fixes listed below are necessary.
https://review.opendev.org/c/openstack/openstacksdk/+/883461
https://review.opendev.org/c/openstack/python-glanceclient/+/924060 |
|
| 2025-05-21 17:14:54 |
Heitor Alves de Siqueira |
python-openstacksdk (Ubuntu Jammy): status |
In Progress |
Incomplete |
|
| 2025-05-21 17:37:02 |
Heitor Alves de Siqueira |
bug |
|
|
added subscriber Heitor Alves de Siqueira |
| 2025-05-22 13:33:02 |
Rodrigo Barbieri |
attachment added |
|
lp2020139_jammy_openstacksdk_v2.debdiff https://bugs.launchpad.net/ubuntu/jammy/+source/python-openstacksdk/+bug/2020139/+attachment/5879750/+files/lp2020139_jammy_openstacksdk_v2.debdiff |
|
| 2025-05-22 13:33:55 |
Rodrigo Barbieri |
python-openstacksdk (Ubuntu Jammy): status |
Incomplete |
In Progress |
|
