Cinder

Volume retype fails when migration occurs

Bug #1740544 reported by Mohammed Naser on 2017-12-29

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Cinder	New	Undecided	Unassigned
	tempest	Fix Released	Medium	Unassigned

Bug Description

While running Tempest internally, I've discovered the following test was failing:

tempest.scenario.test_volume_migrate_attached.TestVolumeMigrateRetypeAttached.test_volume_migrate_attached[compute,id-deadd2c2-beef-4dce-98be-f86765ff311b,slow,volume]

Upon further debugging, I've discovered that cinder-volume has the following traceback:

2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server [req-7b3d3148-cdbb-485d-9dd4-3ff7a24e59ad 7088d6afa8a44ef386333b30ff6d62e0 ef87e48ff4604c6aabdead43ca5cb6ca - default default] Exception during message handling: Forbidden: Policy doesn't allow os_compute_api:os-volumes-attachments:update to be performed. (HTTP 403) (Request-ID: req-d5fb2992-48ba-4459-a214-f2cceb8bb0ac)
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/cinder/volume/manager.py", line 2745, in retype
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server new_reservations, status_update)
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server self.force_reraise()
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/cinder/volume/manager.py", line 2741, in retype
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server new_type_id=new_type_id)
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/cinder/volume/manager.py", line 2348, in migrate_volume
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server volume.save()
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server self.force_reraise()
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/cinder/volume/manager.py", line 2341, in migrate_volume
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server self._migrate_volume_generic(ctxt, volume, host, new_type_id)
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/cinder/volume/manager.py", line 2107, in _migrate_volume_generic
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server new_volume)
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server self.force_reraise()
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/cinder/volume/manager.py", line 2100, in _migrate_volume_generic
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server new_volume.id)
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/cinder/compute/nova.py", line 217, in update_server_volume
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server new_volume_id)
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/novaclient/v2/volumes.py", line 97, in update_server_volume
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server body, "volumeAttachment")
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/novaclient/base.py", line 375, in _update
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server resp, body = self.api.client.put(url, body=body)
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 297, in put
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server return self.request(url, 'PUT', **kwargs)
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server File "/openstack/venvs/pike/lib/python2.7/site-packages/novaclient/client.py", line 83, in request
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server raise exceptions.from_response(resp, body, url, method)
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server Forbidden: Policy doesn't allow os_compute_api:os-volumes-attachments:update to be performed. (HTTP 403) (Request-ID: req-d5fb2992-48ba-4459-a214-f2cceb8bb0ac)
2017-12-29 20:38:01.565 32587 ERROR oslo_messaging.rpc.server
2017-12-29 20:38:01.661 32585 INFO cinder.volume.manager [req-7b3d3148-cdbb-485d-9dd4-3ff7a24e59ad 7088d6afa8a44ef386333b30ff6d62e0 ef87e48ff4604c6aabdead43ca5cb6ca - default default] Deleted volume successfully.

Upon further checks, I've found out that during the retype process, it can fallback to migrating the volume in the event that it is not able to retype. In the migrate process, the notification of swapping the volume is sent with the same auth credentials of the user that requested this (which isn't an admin) and it fails.

I'm not sure if Nova's policy should be adjusted for this, or if Cinder should send the request with elevated context, but I believe that's where the issue stands.

Tags:

Revision history for this message

Ghanshyam Mann (ghanshyammann) wrote on 2018-01-01:

yes, with migration_policy as 'on-demand'(this is what test do) it has to be admin user. This is fixed in your patch - https://review.openstack.org/#/c/530508/

Also i am much interested to test retype with migration_policy as 'never' so that we can check the auth behavior in retype operation. That should be success with non admin user. Let's add that test also.

Revision history for this message

Ghanshyam Mann (ghanshyammann) wrote on 2018-01-01:

++ tempest in this bug to add a test the retype with 'never' migration policy.

Changed in tempest:
status:	New → Triaged
importance:	Undecided → Medium

Revision history for this message

Eric Harney (eharney) wrote on 2018-01-03:

This may be related to bug 1698224.

chandan kumar (chkumar246) on 2018-01-04

Changed in tempest:
status:	Triaged → Confirmed
tags:	added: low-hanging-fruit

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-01-07: Related fix merged to tempest (master)

Reviewed: https://review.openstack.org/530508
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=65e4f9b9d1f0c61636895cc133062bf0a6263848
Submitter: Zuul
Branch: master

commit 65e4f9b9d1f0c61636895cc133062bf0a6263848
Author: Mohammed Naser <email address hidden>
Date: Fri Dec 29 21:22:24 2017 +0000

Revert "Use a non admin privileges for retyping a volume"

This reverts commit f5869b4557855af9450501c65e03bd16fd269036.

    When retyping a volume, it is possible that Cinder falls back to
    migrating a volume in order to complete the retype which is admin-only
    operation.

    This will result in a failure because Cinder will attempt to contact
    Nova using the credentials of the user which are not elevated, with
    Nova refusing to do the volume swap.

    Therefore, in order to effectively test this, we should stick to using
    the admin API as a retype can require a migration which is an
    admin-only operation.

Related-Bug: #1740544
Change-Id: I3e4853146bb6e2a62205ffe690da081229215f54

Rajat Dhasmana (whoami-rajat) on 2018-05-23

Changed in tempest:
assignee:	nobody → Rajat Dhasmana (whoami-rajat)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-06-05: Related fix proposed to tempest (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/572287

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-07-03: Related fix merged to tempest (master)

Reviewed: https://review.openstack.org/572287
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=a3a67d476132fe5f55020cf2dc1f672c17be6eb3
Submitter: Zuul
Branch: master

commit a3a67d476132fe5f55020cf2dc1f672c17be6eb3
Author: whoami-rajat <email address hidden>
Date: Tue Jun 5 07:44:35 2018 +0000

Adding test case for migration policy 'never'

There doesn't exist any test case in tempest for retype
volume without migration, so adding new test for it.

Related-Bug: #1740544

Change-Id: I58c348b954a679bc809906b2c73df9b1ab4459c5

Revision history for this message

Martin Kopec (mkopec) wrote on 2020-10-26:

Unassigning due to a long inactivity.

Changed in tempest:
assignee:	Rajat Dhasmana (whoami-rajat) → nobody

Revision history for this message

Martin Kopec (mkopec) wrote on 2022-03-24:

This was reported on tempest's side due to a missing test case - a new test case has been added by https://review.opendev.org/c/openstack/tempest/+/572287/ . I'm gonna close this as Fix Released for Tempest. If you feel like this needs to be addressed in Tempest further, please comment here and we'll reopen the issue (or fill a new one).

Changed in tempest:
status:	Confirmed → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.