OpenStack Compute (nova)

Nova assisted volume snapshots fails to find BDM in multi-cell

Bug #1713735 reported by Lucian Petrut
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Lucian Petrut
Pike
Fix Committed
High
Matt Riedemann

Bug Description

When using separate databases for each Nova cell, nova assisted volume snapshots always fail with the following error:
BadRequest: No volume Block Device Mapping with id a10bd120-9b88-4710-bf6e-f1d34de87da2. (HTTP 400)

The reason is that the according API call does not include an instance id, which is fetched from the BDM. At the same time, the BDM cannot properly be retrieved since Nova doesn't know which cell to use, looking for the BDM in the wrong DB.

Cinder trace: http://paste.openstack.org/raw/619767/

Among others, Cinder NFS backends are affected by this, as per the following Cinder NFS CI logs:
http://logs.openstack.org/21/498321/5/check/gate-tempest-dsvm-full-devstack-plugin-nfs-nv/b8bca96/logs/screen-c-vol.txt.gz?level=ERROR

Tags: api cells volumes
Lucian Petrut (petrutlucian94)
tags: added: api cells volumes
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/498845

Changed in nova:
assignee: nobody → Lucian Petrut (petrutlucian94)
status: New → In Progress
Revision history for this message
Matt Riedemann (mriedem) wrote :

Aug 28 10:22:51.522922 ubuntu-xenial-rax-ord-10656600 cinder-volume[2128]: ERROR cinder.volume.drivers.remotefs [None req-3022f65f-b7e0-43b1-8258-e41f75a9f106 tempest-VolumesSnapshotTestJSON-799368090 None] Call to Nova to create snapshot failed: BadRequest: No volume Block Device Mapping with id a10bd120-9b88-4710-bf6e-f1d34de87da2. (HTTP 400) (Request-ID: req-14d58a0b-bc8c-4521-908e-84cfadd7033a)
Aug 28 10:22:51.523057 ubuntu-xenial-rax-ord-10656600 cinder-volume[2128]: ERROR cinder.volume.drivers.remotefs Traceback (most recent call last):
Aug 28 10:22:51.523171 ubuntu-xenial-rax-ord-10656600 cinder-volume[2128]: ERROR cinder.volume.drivers.remotefs File "/opt/stack/new/cinder/cinder/volume/drivers/remotefs.py", line 1459, in _create_snapshot_online
Aug 28 10:22:51.523282 ubuntu-xenial-rax-ord-10656600 cinder-volume[2128]: ERROR cinder.volume.drivers.remotefs connection_info)
Aug 28 10:22:51.523407 ubuntu-xenial-rax-ord-10656600 cinder-volume[2128]: ERROR cinder.volume.drivers.remotefs File "/opt/stack/new/cinder/cinder/compute/nova.py", line 225, in create_volume_snapshot
Aug 28 10:22:51.523516 ubuntu-xenial-rax-ord-10656600 cinder-volume[2128]: ERROR cinder.volume.drivers.remotefs create_info=create_info)
Aug 28 10:22:51.523623 ubuntu-xenial-rax-ord-10656600 cinder-volume[2128]: ERROR cinder.volume.drivers.remotefs File "/usr/local/lib/python2.7/dist-packages/novaclient/v2/assisted_volume_snapshots.py", line 43, in create
Aug 28 10:22:51.523728 ubuntu-xenial-rax-ord-10656600 cinder-volume[2128]: ERROR cinder.volume.drivers.remotefs return self._create('/os-assisted-volume-snapshots', body, 'snapshot')
Aug 28 10:22:51.523831 ubuntu-xenial-rax-ord-10656600 cinder-volume[2128]: ERROR cinder.volume.drivers.remotefs File "/usr/local/lib/python2.7/dist-packages/novaclient/base.py", line 361, in _create
Aug 28 10:22:51.523949 ubuntu-xenial-rax-ord-10656600 cinder-volume[2128]: ERROR cinder.volume.drivers.remotefs resp, body = self.api.client.post(url, body=body)
Aug 28 10:22:51.524098 ubuntu-xenial-rax-ord-10656600 cinder-volume[2128]: ERROR cinder.volume.drivers.remotefs File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 294, in post
Aug 28 10:22:51.524215 ubuntu-xenial-rax-ord-10656600 cinder-volume[2128]: ERROR cinder.volume.drivers.remotefs return self.request(url, 'POST', **kwargs)
Aug 28 10:22:51.524320 ubuntu-xenial-rax-ord-10656600 cinder-volume[2128]: ERROR cinder.volume.drivers.remotefs File "/usr/local/lib/python2.7/dist-packages/novaclient/client.py", line 83, in request
Aug 28 10:22:51.524424 ubuntu-xenial-rax-ord-10656600 cinder-volume[2128]: ERROR cinder.volume.drivers.remotefs raise exceptions.from_response(resp, body, url, method)
Aug 28 10:22:51.524542 ubuntu-xenial-rax-ord-10656600 cinder-volume[2128]: ERROR cinder.volume.drivers.remotefs BadRequest: No volume Block Device Mapping with id a10bd120-9b88-4710-bf6e-f1d34de87da2. (HTTP 400) (Request-ID: req-14d58a0b-bc8c-4521-908e-84cfadd7033a)
Aug 28 10:22:51.524648 ubuntu-xenial-rax-ord-10656600 cinder-volume[2128]: ERROR cinder.volume.drivers.remotefs

Changed in nova:
importance: Undecided → High
summary: - Nova assisted volume snapshots issue
+ Nova assisted volume snapshots fails to find BDM in multi-cell
OpenStack Infra (hudson-openstack)
Changed in nova:
assignee: Lucian Petrut (petrutlucian94) → Matt Riedemann (mriedem)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/498979

Matt Riedemann (mriedem)
Changed in nova:
assignee: Matt Riedemann (mriedem) → Lucian Petrut (petrutlucian94)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/498845
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=074614f07751f3a10e630fc7344a0794516bb32b
Submitter: Jenkins
Branch: master

commit 074614f07751f3a10e630fc7344a0794516bb32b
Author: Lucian Petrut <email address hidden>
Date: Tue Aug 29 17:30:18 2017 +0300

    Fix nova assisted volume snapshots

    When performing nova assisted volume snapshots, the nova api does
    not receive an instance id, so there is not the usual instance
    lookup from the cell which automatically targets the context for the
    cell that the instance was pulled from, which is also where we'd get
    the BDM.

    Since we do not know which cell to target when fetching the BDM, we
    have to iterate through *all* of them.

    Closes-Bug: #1713735

    Change-Id: Id2e3d3f177739a31d63790e4a1ae6ac41f438ddd

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/pike)

Reviewed: https://review.openstack.org/498979
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b514f93c404b3054d0caff94f37d87cd49d9071b
Submitter: Jenkins
Branch: stable/pike

commit b514f93c404b3054d0caff94f37d87cd49d9071b
Author: Lucian Petrut <email address hidden>
Date: Tue Aug 29 17:30:18 2017 +0300

    Fix nova assisted volume snapshots

    When performing nova assisted volume snapshots, the nova api does
    not receive an instance id, so there is not the usual instance
    lookup from the cell which automatically targets the context for the
    cell that the instance was pulled from, which is also where we'd get
    the BDM.

    Since we do not know which cell to target when fetching the BDM, we
    have to iterate through *all* of them.

    Closes-Bug: #1713735

    Change-Id: Id2e3d3f177739a31d63790e4a1ae6ac41f438ddd
    (cherry picked from commit 074614f07751f3a10e630fc7344a0794516bb32b)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 16.0.1

This issue was fixed in the openstack/nova 16.0.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 17.0.0.0b1

This issue was fixed in the openstack/nova 17.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.