glance_store

glance backend is in crazy resize when an image is uploading

Bug #1792710 reported by Liang Fang
20
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Cinder
In Progress
Medium
Liang Fang
Glance
New
Undecided
Unassigned
glance_store
In Progress
High
Erno Kuvaja

Bug Description

When uploading a volume to glance as an image, the glance server don't know the image size, so the backend storage server(such as ceph) need to resize the image every time it received new chunk of data(by default 64K). So there will be huge times of resize operations that will impact the performance.

- regarding cinder, it has not calculate the image size and pass the correct size to glance
- regarding glance, it should allow the client to set image size.

This is an known issue which can be found in driver files of all kinds of backend storage system:
In file: glance_store/_drivers/rbd.py, function: add
In file: glance_store/_drivers/cinder.py, function: add
In file: glance_store/_drivers/sheepdog.py, function: add
In all these files, there're comments like below:
    # If the image size provided is zero we need to do
    # a resize for the amount we are writing. This will
    # be slower so setting a higher chunk size may
    # speed things up a bit.

See original description
Revision history for this message
Sean McGinnis (sean-mcginnis) wrote :

What is the Cinder issue here?

Changed in cinder:
status: New → Incomplete
Liang Fang (liangfang)
Changed in cinder:
assignee: nobody → Liang Fang (liangfang)
Changed in glance:
assignee: nobody → Liang Fang (liangfang)
Changed in python-glanceclient:
assignee: nobody → Liang Fang (liangfang)
Revision history for this message
Liang Fang (liangfang) wrote :

cinder need to get the image size and send the size to glance at the beginning of the uploading. I will point the review to this bug soon.

description: updated
OpenStack Infra (hudson-openstack)
Changed in cinder:
status: Incomplete → In Progress
Revision history for this message
Sean McGinnis (sean-mcginnis) wrote :

Glance and glanceclient take an optional size as input. What is the bug there? This needs better detail.

OpenStack Infra (hudson-openstack)
Changed in glance:
status: New → In Progress
OpenStack Infra (hudson-openstack)
Changed in python-glanceclient:
status: New → In Progress
Revision history for this message
Liang Fang (liangfang) wrote :

@Sean, more information provided in "Bug Description", thanks

description: updated
Jay Bryant (jsbryant)
Changed in cinder:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/609994

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance (master)

Fix proposed to branch: master
Review: https://review.openstack.org/609997

Liang Fang (liangfang)
no longer affects: python-glanceclient
description: updated
Revision history for this message
Erno Kuvaja (jokke) wrote :

PTG May 2nd Cinder/Glance session:

Lets not make the size user settable and instead just resize bigger chunks at the time and shrink back after EOF.

Changed in glance:
importance: Undecided → High
milestone: none → next
Changed in glance-store:
importance: Undecided → High
no longer affects: glance
Revision history for this message
Liang Fang (liangfang) wrote :

Thanks Jokke. I know ceph support shrink, but not sure if other storage backend support shrink or not. It's a good idea, but may need to check if backends support shrink when doing resize. if not support shrink, then may need to keep current way - resize in a small step.

Erno Kuvaja (jokke)
Changed in glance-store:
assignee: nobody → Erno Kuvaja (jokke)
status: New → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance_store (master)

Fix proposed to branch: master
Review: https://review.opendev.org/746579

Changed in glance-store:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance_store (master)

Reviewed: https://review.opendev.org/746579
Committed: https://git.openstack.org/cgit/openstack/glance_store/commit/?id=c43f19e8456b9e20f03709773fb2ffdb94807a0a
Submitter: Zuul
Branch: master

commit c43f19e8456b9e20f03709773fb2ffdb94807a0a
Author: Erno Kuvaja <email address hidden>
Date: Mon Aug 17 16:35:03 2020 +0100

    Ramp up rbd resize to avoid excessive calls

    Change the RBD store to resize the image by up to 8GiB
    at the time to not resize on every write.

    Trim the image in Ceph after all data has been written
    to the actual size in case we overshot the resize.

    Partial-Bug: #1792710
    Related-to: spec-lite-Ceph-Store-Optimization

    Change-Id: I7f0bffda222b663d4316c5d6c03fdbd0d3337035

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance_store (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/762498

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance_store (stable/ussuri)

Reviewed: https://review.opendev.org/762498
Committed: https://opendev.org/openstack/glance_store/commit/ca0c58b52756058b6d51bf6a47aeac3d525c1e16
Submitter: Zuul
Branch: stable/ussuri

commit ca0c58b52756058b6d51bf6a47aeac3d525c1e16
Author: Erno Kuvaja <email address hidden>
Date: Mon Aug 17 16:35:03 2020 +0100

    Ramp up rbd resize to avoid excessive calls

    Change the RBD store to resize the image by up to 8GiB
    at the time to not resize on every write.

    Trim the image in Ceph after all data has been written
    to the actual size in case we overshot the resize.

    Partial-Bug: #1792710
    Related-to: spec-lite-Ceph-Store-Optimization

    Change-Id: I7f0bffda222b663d4316c5d6c03fdbd0d3337035
    (cherry picked from commit c43f19e8456b9e20f03709773fb2ffdb94807a0a)

tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance_store (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/763570

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.