os-brick

Failure in writing to multipath device as it's not ready for I/O

Bug #2067949 reported by Rajat Dhasmana on 2024-06-03

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	os-brick	Fix Released	Undecided	Rajat Dhasmana

Bug Description

When we try to create a volume from image, we attach the volume to the host and perform qemu-img convert command to convert the image and write it into the volume.

In a iSCSI environment, the following error is observed when performing the qemu-img operation.

2024-05-02 16:40:38.189 8 ERROR cinder.volume.volume_utils Stderr: 'qemu-img: /dev/dm-7: error while converting raw: Cannot grow device files\n'

This happens because the multipath device is not ready for I/O and we try to write data into it.

We require a check in os-brick to verify the multipath device is ready for read/write before returning the call to cinder/nova/glance for writing data into the volume.

See original description

OpenStack Infra (hudson-openstack) on 2024-06-03

Changed in os-brick:
status:	New → In Progress

Rajat Dhasmana (whoami-rajat) on 2024-06-13

Changed in os-brick:
assignee:	nobody → Rajat Dhasmana (whoami-rajat)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-08-06: Fix merged to os-brick (master)

Reviewed: https://review.opendev.org/c/openstack/os-brick/+/920516
Committed: https://opendev.org/openstack/os-brick/commit/639f953194cdc07b1e6aff1fc1ca2a7bc9d28536
Submitter: "Zuul (22348)"
Branch: master

commit 639f953194cdc07b1e6aff1fc1ca2a7bc9d28536
Author: Rajat Dhasmana <email address hidden>
Date: Mon May 27 18:33:54 2024 +0530

Wait for multipath device to be ready for I/O

    The "multipath -C <mpath_name>" command waits for the multipath
    device map to be ready for I/O. This is useful in preventing race
    conditions when we are trying to write to the multipath device
    before it is ready for I/O.

We added 2 new config options to make the wait time configurable,

1. wait_mpath_device_attempts - defaults to 4 attempts
2. wait_mpath_device_interval - defaults to 1 second

Closes-Bug: #2067949

Change-Id: Ib075ec62a2bf993615c5c802f34acd7838bfa2af

Changed in os-brick:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-08-07: Fix proposed to os-brick (stable/2024.1)

Fix proposed to branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/os-brick/+/925905

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-08-14: Fix merged to os-brick (stable/2024.1)

Reviewed: https://review.opendev.org/c/openstack/os-brick/+/925905
Committed: https://opendev.org/openstack/os-brick/commit/80da091dd3f949b3f06deef80b92061c7a487c58
Submitter: "Zuul (22348)"
Branch: stable/2024.1

commit 80da091dd3f949b3f06deef80b92061c7a487c58
Author: Rajat Dhasmana <email address hidden>
Date: Mon May 27 18:33:54 2024 +0530

Wait for multipath device to be ready for I/O

We added 2 new config options to make the wait time configurable,

1. wait_mpath_device_attempts - defaults to 4 attempts
2. wait_mpath_device_interval - defaults to 1 second

Closes-Bug: #2067949

Change-Id: Ib075ec62a2bf993615c5c802f34acd7838bfa2af
(cherry picked from commit 639f953194cdc07b1e6aff1fc1ca2a7bc9d28536)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-08-14: Fix proposed to os-brick (stable/2023.2)

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/os-brick/+/926355

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-08-19: Fix merged to os-brick (stable/2023.2)

Reviewed: https://review.opendev.org/c/openstack/os-brick/+/926355
Committed: https://opendev.org/openstack/os-brick/commit/a6fc5132c5786c11cdc94bc8dc44bad80843f2cc
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit a6fc5132c5786c11cdc94bc8dc44bad80843f2cc
Author: Rajat Dhasmana <email address hidden>
Date: Mon May 27 18:33:54 2024 +0530

Wait for multipath device to be ready for I/O

We added 2 new config options to make the wait time configurable,

1. wait_mpath_device_attempts - defaults to 4 attempts
2. wait_mpath_device_interval - defaults to 1 second

Closes-Bug: #2067949

    Change-Id: Ib075ec62a2bf993615c5c802f34acd7838bfa2af
    (cherry picked from commit 639f953194cdc07b1e6aff1fc1ca2a7bc9d28536)
    (cherry picked from commit 80da091dd3f949b3f06deef80b92061c7a487c58)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-08-22: Fix proposed to os-brick (stable/2023.1)

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/os-brick/+/926918

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-08-22: Fix merged to os-brick (stable/2023.1)

Reviewed: https://review.opendev.org/c/openstack/os-brick/+/926918
Committed: https://opendev.org/openstack/os-brick/commit/7ad0ed6e5be4443cf2578ff073f01966729275d4
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 7ad0ed6e5be4443cf2578ff073f01966729275d4
Author: Rajat Dhasmana <email address hidden>
Date: Mon May 27 18:33:54 2024 +0530

Wait for multipath device to be ready for I/O

We added 2 new config options to make the wait time configurable,

1. wait_mpath_device_attempts - defaults to 4 attempts
2. wait_mpath_device_interval - defaults to 1 second

Closes-Bug: #2067949

    Change-Id: Ib075ec62a2bf993615c5c802f34acd7838bfa2af
    (cherry picked from commit 639f953194cdc07b1e6aff1fc1ca2a7bc9d28536)
    (cherry picked from commit 80da091dd3f949b3f06deef80b92061c7a487c58)
    (cherry picked from commit a6fc5132c5786c11cdc94bc8dc44bad80843f2cc)