Cinder

Simultaneous volume creation with the same image in multi-attach mode returns error

Bug #1969373 reported by Rajat Dhasmana on 2022-04-18

This bug affects 1 person

	Status	Importance	Assigned to
Cinder	In Progress	Low	Unassigned
glance_store	Fix Released	Undecided	Rajat Dhasmana
Xena	Fix Released	Undecided	Unassigned
Yoga	Fix Released	Undecided	Unassigned

Bug Description

Glance backend is Cinder.
Multi-attachment is enabled.

1. launch two commands one after another and don’t wait responses (in 2 terminal tabs)

$ openstack volume create --size 50 --image Ubuntu-18.04 volume_1

$ openstack volume create --size 50 --image Ubuntu-18.04 volume_2

2.see volumes statuses

3. see cinder logs

ERROR cinder.volume.api [req-41b9c765-827d-4c75-ac95-331c8fa0c840 cd15c27229234675ac78f3f83af1c60c 3a2aec7e5c5b46c2a366e471fa133066 - 3f2f1264e29e4cc9913aa2a63197866a 3f2f1264e29e4cc9913aa2a63197866a] Volume status must be available or in-use to reserve, but the status is attaching.

or similar error message, but the status in detaching.

expected behavior:
two volumes created successfully

actual behavior:
the second volume was not created because the first volume creation task reserved volume.

Tags:

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-04-18: Fix proposed to glance_store (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/glance_store/+/838326

Changed in glance-store:
status:	New → In Progress

Rajat Dhasmana (whoami-rajat) on 2022-04-18

Changed in glance-store:
assignee:	nobody → Rajat Dhasmana (whoami-rajat)

Revision history for this message

Bence Romsics (bence-romsics) wrote on 2022-04-19:

Just for the record: there's some history (e.g. alternative reproduction steps) to this bug in another bug report:
https://bugs.launchpad.net/cinder/+bug/1940466

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-04-21: Fix merged to glance_store (master)

Reviewed: https://review.opendev.org/c/openstack/glance_store/+/838326
Committed: https://opendev.org/openstack/glance_store/commit/ba4af147fb93d522f8cf396098891f4cbe7b30fb
Submitter: "Zuul (22348)"
Branch: master

commit ba4af147fb93d522f8cf396098891f4cbe7b30fb
Author: whoami-rajat <email address hidden>
Date: Mon Apr 18 21:43:14 2022 +0530

Correct retry interval during attach volume

    When we try to simultaneously attach same volume multiple times
    (multiattach), there are multiple attachments created, suppose
    attachA and attachB. If attachA marks the volume "reserved" then
    attachB can't proceed until the volume is in "in-use" or "available"
    state. We retry until we reach any of these states for which we use
    the retrying library.

    The retrying library defaults to 1 second retry[1] (5 times) which causes
    the original failure as the volume takes time to transition between
    "reserved" -> "in-use" state. This patch corrects it by adding an
    exponential retry time and max exponential retry i.e.

    1: 2 ** 1 = 2 seconds
    2: 2 ** 2 = 4 seconds
    3: 2 ** 3 = 8 seconds
    4: 2 ** 4 = 16 seconds (but max wait time is 10 seconds)
    5: 2 ** 5 = 32 seconds (but max wait time is 10 seconds)

[1] https://github.com/rholder/retrying/blob/master/retrying.py#L84

Closes-Bug: #1969373

Change-Id: I0094b044085d7f92b07ea86236de3b6efd7d67ea

Changed in glance-store:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-04-25: Fix proposed to glance_store (stable/yoga)

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/glance_store/+/839203

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-04-25: Fix proposed to glance_store (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/glance_store/+/839204

Sofia Enriquez (lsofia-enriquez) on 2022-04-27

Changed in cinder:
status:	New → In Progress
importance:	Undecided → Medium
tags:	added: glance-store image multiattach

Sofia Enriquez (lsofia-enriquez) on 2022-04-27

Changed in cinder:
importance:	Medium → Low

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-05-05: Fix merged to glance_store (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/glance_store/+/839203
Committed: https://opendev.org/openstack/glance_store/commit/c6716cb8fbbdd75b3c5ce36f5fc9cede8b83bc0d
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit c6716cb8fbbdd75b3c5ce36f5fc9cede8b83bc0d
Author: whoami-rajat <email address hidden>
Date: Mon Apr 18 21:43:14 2022 +0530

Correct retry interval during attach volume

    1: 2 ** 1 = 2 seconds
    2: 2 ** 2 = 4 seconds
    3: 2 ** 3 = 8 seconds
    4: 2 ** 4 = 16 seconds (but max wait time is 10 seconds)
    5: 2 ** 5 = 32 seconds (but max wait time is 10 seconds)

[1] https://github.com/rholder/retrying/blob/master/retrying.py#L84

Closes-Bug: #1969373

Change-Id: I0094b044085d7f92b07ea86236de3b6efd7d67ea
(cherry picked from commit ba4af147fb93d522f8cf396098891f4cbe7b30fb)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-05-09: Fix merged to glance_store (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/glance_store/+/839204
Committed: https://opendev.org/openstack/glance_store/commit/a4ba0c4fded13190f9161acedf6240f025718a77
Submitter: "Zuul (22348)"
Branch: stable/xena

commit a4ba0c4fded13190f9161acedf6240f025718a77
Author: whoami-rajat <email address hidden>
Date: Mon Apr 18 21:43:14 2022 +0530

Correct retry interval during attach volume

    1: 2 ** 1 = 2 seconds
    2: 2 ** 2 = 4 seconds
    3: 2 ** 3 = 8 seconds
    4: 2 ** 4 = 16 seconds (but max wait time is 10 seconds)
    5: 2 ** 5 = 32 seconds (but max wait time is 10 seconds)

[1] https://github.com/rholder/retrying/blob/master/retrying.py#L84

Closes-Bug: #1969373

    Change-Id: I0094b044085d7f92b07ea86236de3b6efd7d67ea
    (cherry picked from commit ba4af147fb93d522f8cf396098891f4cbe7b30fb)
    (cherry picked from commit c6716cb8fbbdd75b3c5ce36f5fc9cede8b83bc0d)