Cinder fails to create image-based volume if mirroring is enabled

Our environment has two ceph clusters (az1 and az2) with a separate ceph-rbd-mirror charm deployed to each. The ceph-rbd-mirror charm enables pool mirroring on all rbd pools.

Glance is backed by only one of the ceph clusters, az1.

We have volume types named after the ceph cluster, arbor-az1 and arbor-az2.

When we copy an image to cinder using the following openstack cli, it's fast and reliable when copying from az1 to az1, but we have greater than 50% failure when copying from az1 to az2.

  openstack volume create --image bionic --size 3 --type arbor-az1 volume-az1 # works
  openstack volume create --image bionic --size 3 --type arbor-az2 volume-az2 # fails >50%

When the az2 image upload fails, it's always with "cinder.exception.VolumeIsBusy: deleting volume volume-<id> that has snapshot"

The full log from the cinder-volume log follows.

2020-10-19 10:15:35.871 45458 INFO cinder.volume.flows.manager.create_volume [req-a4905f43-c6f8-42e4-a602-a00238068492 f2ee9e8060e54d058060d220fab84088 0c0bb65d03f446ca8a845d1ae9229790 - 1e59a4057fb24003a902bc4df6247d3c 1e59a4057fb24003a902bc4df6247d3c] Volume 63e4d69c-0722-40fc-b44f-cb59fca0d435: being created as image with specification: {'status': 'creating', 'volume_name': 'volume-63e4d69c-0722-40fc-b44f-cb59fca0d435', 'volume_size': 20, 'image_id': 'e93f97e0-f514-435d-a277-0ac288ed0c6c', 'image_location': ('rbd://36bd979e-0511-11eb-bced-ecebb88db476/glance/e93f97e0-f514-435d-a277-0ac288ed0c6c/snap', [{'url': 'rbd://36bd979e-0511-11eb-bced-ecebb88db476/glance/e93f97e0-f514-435d-a277-0ac288ed0c6c/snap', 'metadata': {}}]), 'image_meta': {'name': 'UBUNTU-18.04', 'disk_format': 'qcow2', 'container_format': 'bare', 'visibility': 'public', 'size': 359923712, 'virtual_size': None, 'status': 'active', 'checksum': '9aa011b2b79b1fe42a7c306555923b1b', 'protected': False, 'min_ram': 0, 'min_disk': 0, 'owner': '86fcc0b3839b45029dd325641ddc2a09', 'os_hidden': False, 'os_hash_algo': 'sha512', 'os_hash_value': '03786c51866c1c6c50ef671502b265afb71084e0d018748ca9b29f871ca5445c9eefffffdf927d540bddaf0e8597adc673e218e785fcf16013fba7e7d9898e6e', 'id': 'e93f97e0-f514-435d-a277-0ac288ed0c6c', 'created_at': datetime.datetime(2020, 10, 15, 16, 49, tzinfo=<iso8601.Utc>), 'updated_at': datetime.datetime(2020, 10, 15, 16, 49, 18, tzinfo=<iso8601.Utc>), 'locations': [{'url': 'rbd://36bd979e-0511-11eb-bced-ecebb88db476/glance/e93f97e0-f514-435d-a277-0ac288ed0c6c/snap', 'metadata': {}}], 'direct_url': 'rbd://36bd979e-0511-11eb-bced-ecebb88db476/glance/e93f97e0-f514-435d-a277-0ac288ed0c6c/snap', 'tags': [], 'file': '/v2/images/e93f97e0-f514-435d-a277-0ac288ed0c6c/file', 'properties': {}}, 'image_service': <cinder.image.glance.GlanceImageService object at 0x7fb92712cda0>}
2020-10-19 10:15:40.810 45458 INFO cinder.image.image_utils [req-a4905f43-c6f8-42e4-a602-a00238068492 f2ee9e8060e54d058060d220fab84088 0c0bb65d03f446ca8a845d1ae9229790 - 1e59a4057fb24003a902bc4df6247d3c 1e59a4057fb24003a902bc4df6247d3c] Image download 343.25 MB at 70.20 MB/s
2020-10-19 10:15:44.685 45458 INFO cinder.image.image_utils [req-a4905f43-c6f8-42e4-a602-a00238068492 f2ee9e8060e54d058060d220fab84088 0c0bb65d03f446ca8a845d1ae9229790 - 1e59a4057fb24003a902bc...

IMHO this is a bug in https://opendev.org/openstack/cinder . And I agree with the previous analysis:

1. volume/flows/manager/create_volume.py:_create_from_image_download() creates a volume, then
2. calls volume_utils.copy_image_to_volume(), which
    a. fetches the image,
    b. deletes the volume, and
    c. runs `rbd import`.

If a snapshot of the volume got created between 1 and 2b (e.g. by rbd mirror), 2b will fail. So I think we need one of these solutions:

1. Instead of creating a volume and immediately deleting it, let's not create it in the first place.
2. Is there a way to delete a volume and all its snapshot in an atomic way?
3. Can we force the deletion of a volume even if it has snapshots?

It's worth noting that the ceph-rbd-mirror charm uses RBD journaling to mirror RBD images between two Ceph clusters. [1] The ceph-rbd-mirror charm doesn't use the snapshot-based approach to RBD Mirroring.

[1] https://jaas.ai/ceph-rbd-mirror

Documentation on ceph RBD mirroring modes (journal-based and snapshot-based): https://docs.ceph.com/en/latest/rbd/rbd-mirroring/

This is correct, however while watching the content of the snap list filtering by the volume ID it is clear that the snapshot does get created for this empty volume, having a name of "rbd_mirror.<volume_id>", and it stays there for about a second. Maybe this is how the images of the volume gets created on the other side of the mirror (list a metadata sync), it is more a question to the behaviour of ceph-rbd-mirror daemon itself.

Andrey/Vern, can you help us understand where the snapshot is coming from? Perhaps we could also get confirmation that journaling is in fact configured for the rbd mirroring; I'm not sure how we can get that information, perhaps 'rbd mirror pool status' will show details.

I have a bionic-train ceph-rbd-mirror deployment up (the smoke test from charm-ceph-rbd-mirror). I need to pick this up tomorrow but the next step is to create a volume from an image and hopefully I can add a sleep to the cinder code to recreate the reported bug. Listing some relevant commands below that I think will be useful, in case someone else picks this up or someone has better commands. Note: I'm listing the glance pool below because it has a snapshot.

ceph-mon/0
----------
$ sudo ceph osd lspools
1 glance
2 cinder-ceph

ceph-rbd-mirror/0
-----------------
$ ls /etc/ceph
ceph.client.rbd-mirror.juju-2e2313-zaza-940825664f98-12.keyring ceph.conf rbdmap remote.client.rbd-mirror.juju-2e2313-zaza-940825664f98-12.keyring remote.conf
$ sudo rbd --id rbd-mirror.juju-2e2313-zaza-940825664f98-12 pool stats glance
Total Images: 1
Total Snapshots: 1
Provisioned Size: 342 MiB
$ sudo rbd --id rbd-mirror.juju-2e2313-zaza-940825664f98-12 pool stats cinder-ceph
Total Images: 1
Total Snapshots: 0
Provisioned Size: 8 GiB
$ sudo rbd --id rbd-mirror.juju-2e2313-zaza-940825664f98-12 mirror pool status glance
health: OK
images: 1 total
    1 stopped
$ sudo rbd --id rbd-mirror.juju-2e2313-zaza-940825664f98-12 mirror pool status cinder-ceph
health: OK
images: 1 total
    1 stopped
$ sudo rbd --id rbd-mirror.juju-2e2313-zaza-940825664f98-12 ls -l glance
NAME SIZE PARENT FMT PROT LOCK
db355184-09a7-4ecc-869e-5a99ef0ef97e 342 MiB 2
db355184-09a7-4ecc-869e-5a99ef0ef97e@snap 342 MiB 2 yes
$ sudo rbd --id rbd-mirror.juju-2e2313-zaza-940825664f98-12 ls -l cinder-ceph
NAME SIZE PARENT FMT PROT LOCK
volume-b0ea5477-d218-409e-b608-dc992a659c0a 8 GiB 2
$ sudo rbd --id rbd-mirror.juju-2e2313-zaza-940825664f98-12 snap ls glance/db355184-09a7-4ecc-869e-5a99ef0ef97e
SNAPID NAME SIZE PROTECTED TIMESTAMP
    24 snap 342 MiB yes Wed Oct 21 20:29:56 2020

Corey, as I mentioned, the snapshot lives for only about 1 second. In order to reproduce that:
1. Create a pool
2. Set up mirror
3. Create a volume from the image
4. IMMEDIATELY grab the volume id
5. run "watch -n0.2 rbd -p <pool> snap ls|grep <vol_id>" (alternatively just run watch without grep permanently). You will see the snapshot come and go.

Adding sleep to the cinder code can only help to overcome the issue (though will be dirty hack and will never be accepted to upstream), and won't help to watch the snapshot lifecycle.
The replication indeed is configured with journals because it is Nautilus (no snapshot-based mirroring yet).

The exception is raised in the following code block:

https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/rbd.py#L1145

(maybe).

As this is as an expected situation then having some sort of retry around this might work OK

@Andrey you're right, I think the sleep would have to be in ceph. @James Yes we might be able to do something like:

diff --git a/cinder/volume/drivers/rbd.py b/cinder/volume/drivers/rbd.py
index 5b868e4fe..d0b556a94 100644
--- a/cinder/volume/drivers/rbd.py
+++ b/cinder/volume/drivers/rbd.py
@@ -1594,6 +1594,9 @@ class RBDDriver(driver.CloneableImageVD, driver.MigrateVD,
             finally:
                 fileutils.delete_if_exists(dest_image_path)

+ @utils.retry(exception.VolumeIsBusy,
+ self.configuration.rados_connection_interval,
+ self.configuration.rados_connection_retries)
     def _copy_image_to_volume(self, context, volume, image_service, image_id,
                               encrypted=False):

and the sleep was just a way to force recreation, not a fix :)

Fix proposed to branch: master
Review: https://review.opendev.org/759315

I tested Corey's fix (via ppa) side-by-side with a cinder unit with version 2:15.4.0-0ubuntu1~cloud0 (from bionic-proposed/train).

I used a bionic qcow2 image with:
  openstack volume create --image bionic --size 3 --type stable2-az2 vernvol

Again, glance is backended by the az1 ceph cluster so this is a cross-cluster copy. In our previous tests, qcow2 images tend to fail at a much higher rate than raw images.

On the unpatched cinder, 77% failure with 23 out of 30 tries erring with:
  Exception during message handling: cinder.exception.ImageCopyFailure: Failed to copy image to volume: deleting volume volume-1b4d5231-2baf-4880-a380-f1dc937b20bd that has snapshot

On Corey's patched version, 0% failure. 30 out of 30 volumes succeeded.

So, in this deployment, this simple retry solves the problem.

Vern, Thanks very much for testing. I'm glad it worked. I just wanted to give you an update on this. I've been monitoring the upstream gate status which has been broken and blocking all new patches. We want to land patches upstream first so they can get a good upstream review before starting SRUs. This helps reduce regression potential. I'll continue to monitor the gate status and once the gate is back in order I'll work on pushing to get a review of the patch.

Bumping this down to high as a workaround exists

This fix has been merged into upstream master. I will proposed backports now.

https://review.opendev.org/c/openstack/cinder/+/759315/

This in Ubuntu hirsute (wallaby) in cinder version 2:17.0.1+git2021010614.a9c922ab7-0ubuntu1.

For ubuntu and the cloud-archive I've uploaded package for all affected releases. The ubuntu uploads are awiting SRU team approval.

Correction, I've not uploaded a fix to xenial/mitaka. If fixes for these releases are needed, please let me know.

Hello Andrey, or anyone else affected,

Accepted cinder into train-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:train-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-train-needed to verification-train-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-train-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Andrey, or anyone else affected,

Accepted cinder into stein-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:stein-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-stein-needed to verification-stein-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-stein-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Andrey, or anyone else affected,

Accepted cinder into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cinder/2:17.0.1-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Andrey, or anyone else affected,

Accepted cinder into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cinder/2:16.2.1-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Andrey, or anyone else affected,

Accepted cinder into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cinder/2:12.0.10-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Andrey, or anyone else affected,

Accepted cinder into victoria-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:victoria-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-victoria-needed to verification-victoria-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-victoria-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Andrey, or anyone else affected,

Accepted cinder into queens-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:queens-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-queens-needed to verification-queens-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-queens-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

The verification of the Stable Release Update for cinder has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package cinder - 2:17.0.1-0ubuntu2~cloud0
---------------

 cinder (2:17.0.1-0ubuntu2~cloud0) focal-victoria; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 cinder (2:17.0.1-0ubuntu2) groovy; urgency=medium
 .
   * d/p/rbd-retry-delete.patch: Fix RBD mirroring race by retrying volume delete
     if VolumeIsBusy in _copy_image_to_volume (LP: #1900775).
   * d/p/add-mock-psutil-in-quobyte-tests.patch: Add a mock of psutil
     disk_partitions to fix failing unit test (LP: #1913607).

Hello Andrey, or anyone else affected,

Accepted cinder into ussuri-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ussuri-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ussuri-needed to verification-ussuri-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ussuri-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

This issue was fixed in the openstack/cinder 18.0.0.0b1 development milestone.

verified successfully

== groovy-proposed ==
root@g1:/usr/lib/python3/dist-packages# python3 -m unittest cinder.tests.unit.volume.drivers.test_rbd.RBDTestCase.test_copy_image_busy_volume

/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py:359: OsloDBDeprecationWarning: EngineFacade is deprecated; please use oslo_db.sqlalchemy.enginefacade
  self._legacy_facade = LegacyEngineFacade(None, _factory=self)
.
----------------------------------------------------------------------
Ran 1 test in 1.403s

OK
root@g1:/usr/lib/python3/dist-packages#
root@g1:/usr/lib/python3/dist-packages# apt policy python3-cinder
python3-cinder:
  Installed: 2:17.0.1-0ubuntu2
  Candidate: 2:17.0.1-0ubuntu2
  Version table:
 *** 2:17.0.1-0ubuntu2 500
        500 http://archive.ubuntu.com/ubuntu groovy-proposed/main amd64 Packages

== focal-proposed ==
root@f1:/usr/lib/python3/dist-packages# python3 -m unittest cinder.tests.unit.volume.drivers.test_rbd.RBDTestCase.test_copy_image_busy_volume
/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py:359: OsloDBDeprecationWarning: EngineFacade is deprecated; please use oslo_db.sqlalchemy.enginefacade
  self._legacy_facade = LegacyEngineFacade(None, _factory=self)
.
----------------------------------------------------------------------
Ran 1 test in 0.901s

OK
root@f1:/usr/lib/python3/dist-packages# apt policy python3-cinder
python3-cinder:
  Installed: 2:16.2.1-0ubuntu2
  Candidate: 2:16.2.1-0ubuntu2
  Version table:
 *** 2:16.2.1-0ubuntu2 500
        500 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages

== ussuri-proposed ==
root@bu:/usr/lib/python3/dist-packages# python3 -m unittest cinder.tests.unit.volume.drivers.test_rbd.RBDTestCase.test_copy_image_busy_volume
/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py:359: OsloDBDeprecationWarning: EngineFacade is deprecated; please use oslo_db.sqlalchemy.enginefacade
  self._legacy_facade = LegacyEngineFacade(None, _factory=self)
.
----------------------------------------------------------------------
Ran 1 test in 1.201s

OK
root@bu:/usr/lib/python3/dist-packages# apt policy python3-cinder
python3-cinder:
  Installed: 2:16.2.1-0ubuntu2~cloud0
  Candidate: 2:16.2.1-0ubuntu2~cloud0
  Version table:
 *** 2:16.2.1-0ubuntu2~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/ussuri/main amd64 Packages

== train-proposed ==
root@bt:/usr/lib/python3/dist-packages# python3 -m unittest cinder.tests.unit.volume.drivers.test_rbd.RBDTestCase.test_copy_image_busy_volume
/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py:359: OsloDBDeprecationWarning: EngineFacade is deprecated; please use oslo_db.sqlalchemy.enginefacade
  self._legacy_facade = LegacyEngineFacade(None, _factory=self)
/usr/lib/python3/dist-packages/migrate/versioning/script/py.py:144: DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature() or inspect.getfullargspec()
  if not inspect.getargspec(script_func)[0]:
.
----------------------------------------------------------------------
Ran 1 test in 0.733s

OK
root@bt:/usr/lib/python3/dist-packages# apt policy p...

This bug was fixed in the package cinder - 2:17.0.1-0ubuntu2

---------------
cinder (2:17.0.1-0ubuntu2) groovy; urgency=medium

  * d/p/rbd-retry-delete.patch: Fix RBD mirroring race by retrying volume delete
    if VolumeIsBusy in _copy_image_to_volume (LP: #1900775).
  * d/p/add-mock-psutil-in-quobyte-tests.patch: Add a mock of psutil
    disk_partitions to fix failing unit test (LP: #1913607).

 -- Corey Bryant <email address hidden> Tue, 26 Jan 2021 15:12:48 -0500

This bug was fixed in the package cinder - 2:16.2.1-0ubuntu2

---------------
cinder (2:16.2.1-0ubuntu2) focal; urgency=medium

  * d/p/rbd-retry-delete.patch: Fix RBD mirroring race by retrying volume delete
    if VolumeIsBusy in _copy_image_to_volume (LP: #1900775).
  * d/p/add-mock-psutil-in-quobyte-tests.patch: Add a mock of psutil
    disk_partitions to fix failing unit test (LP: #1913607).

 -- Corey Bryant <email address hidden> Tue, 26 Jan 2021 15:28:53 -0500

This bug was fixed in the package cinder - 2:12.0.10-0ubuntu2

---------------
cinder (2:12.0.10-0ubuntu2) bionic; urgency=medium

  [ Chris MacNaughton ]
  * d/control: Update VCS paths for move to lp:~ubuntu-openstack-dev.

  [ Corey Bryant ]
  * d/p/rbd-retry-delete.patch: Fix RBD mirroring race by retrying volume delete
    if VolumeIsBusy in _copy_image_to_volume (LP: #1900775).

 -- Corey Bryant <email address hidden> Tue, 26 Jan 2021 15:56:11 -0500

The verification of the Stable Release Update for cinder has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package cinder - 2:16.2.1-0ubuntu2~cloud0
---------------

 cinder (2:16.2.1-0ubuntu2~cloud0) bionic-ussuri; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 cinder (2:16.2.1-0ubuntu2) focal; urgency=medium
 .
   * d/p/rbd-retry-delete.patch: Fix RBD mirroring race by retrying volume delete
     if VolumeIsBusy in _copy_image_to_volume (LP: #1900775).
   * d/p/add-mock-psutil-in-quobyte-tests.patch: Add a mock of psutil
     disk_partitions to fix failing unit test (LP: #1913607).

The verification of the Stable Release Update for cinder has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package cinder - 2:15.4.1-0ubuntu1~cloud1
---------------

 cinder (2:15.4.1-0ubuntu1~cloud1) bionic-train; urgency=medium
 .
   * d/p/rbd-retry-delete.patch: Fix RBD mirroring race by retrying volume delete
     if VolumeIsBusy in _copy_image_to_volume (LP: #1900775).
   * d/p/add-mock-psutil-in-quobyte-tests.patch: Add a mock of psutil
     disk_partitions to fix failing unit test (LP: #1913607).

This bug was fixed in the package cinder - 2:14.3.1-0ubuntu1~cloud1
---------------

 cinder (2:14.3.1-0ubuntu1~cloud1) bionic-stein; urgency=medium
 .
   * d/p/rbd-retry-delete.patch: Fix RBD mirroring race by retrying volume delete
     if VolumeIsBusy in _copy_image_to_volume (LP: #1900775).
   * d/p/add-mock-psutil-in-quobyte-tests.patch: Add a mock of psutil
     disk_partitions to fix failing unit test (LP: #1913607).

The verification of the Stable Release Update for cinder has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package cinder - 2:12.0.10-0ubuntu2~cloud0
---------------

 cinder (2:12.0.10-0ubuntu2~cloud0) xenial-queens; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 cinder (2:12.0.10-0ubuntu2) bionic; urgency=medium
 .
   [ Chris MacNaughton ]
   * d/control: Update VCS paths for move to lp:~ubuntu-openstack-dev.
 .
   [ Corey Bryant ]
   * d/p/rbd-retry-delete.patch: Fix RBD mirroring race by retrying volume delete
     if VolumeIsBusy in _copy_image_to_volume (LP: #1900775).

This issue was fixed in the openstack/cinder 17.1.0 release.

This issue was fixed in the openstack/cinder 16.3.0 release.

trusty-mitaka is EOL

Xenial is nearing ESM and this isn't critical enough to backport at this time.

This issue was fixed in the openstack/cinder 15.5.0 release.

Change abandoned by "Brian Rosmaita <email address hidden>" on branch: stable/stein
Review: https://review.opendev.org/c/openstack/cinder/+/769583
Reason: Stein transitioned to End of Life by change Icf9a539a7b8b and is accepting no more changes.