Cinder

[RBD] Cinder fails to retype / migrate large volumes

Bug #1969408 reported by Jason Speckman on 2022-04-19

This bug affects 3 people

Affects		Status	Importance	Assigned to	Milestone
	Cinder	New	Medium	Unassigned

Bug Description

Cinder fails to complete large volume (1TB) retype w/ migration from one ceph custer to another.
Small volumes (30GB) complete migration.

Small volumes take a few minutes to complete. The large volume failure shows up a little over an hour after beginning the retype / migration.

The error occurs when the retype w/ migration is initiated from the cli or dashboard.
For good measure I bumped up the keystone timeout from 30 minutes to 8 hours.

From the command line, I use:
cinder retype --migration-policy on-demand <instance-volume> <new-volume-type>

The failure message only appears in the nova compute logs:

2022-04-18 23:38:09.652 1028283 ERROR nova.volume.cinder [req-5b02ec6b-e9cb-40af-bfe5-8a5fbe0a6219 fe66d45300e8433da89acf3ccfa239bd d6287309168b42cd8242ef41db87f998 - default default] Complete attachment failed for attachment 1530cbfd-486c-4b4c-9897-fdb038491ca5. Error: The request you have made requires authentication. (HTTP 401) Code: 401: cinderclient.exceptions.Unauthorized: The request you have made requires authentication. (HTTP 401)
2022-04-18 23:38:09.655 1028283 ERROR nova.compute.manager [req-5b02ec6b-e9cb-40af-bfe5-8a5fbe0a6219 fe66d45300e8433da89acf3ccfa239bd d6287309168b42cd8242ef41db87f998 - default default] [instance: e3248868-5cb3-4787-a47a-2611bae4fda8] Failed to swap volume 846390b2-b137-49d4-af5f-164c5015ce5a for 0b3ed898-9534-4b15-a250-8a11cfce55da: cinderclient.exceptions.Unauthorized: The request you have made requires authentication. (HTTP 401)
2022-04-18 23:38:09.655 1028283 ERROR nova.compute.manager [instance: e3248868-5cb3-4787-a47a-2611bae4fda8] Traceback (most recent call last):
2022-04-18 23:38:09.655 1028283 ERROR nova.compute.manager [instance: e3248868-5cb3-4787-a47a-2611bae4fda8] File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 7219, in _swap_volume
2022-04-18 23:38:09.655 1028283 ERROR nova.compute.manager [instance: e3248868-5cb3-4787-a47a-2611bae4fda8] self.volume_api.attachment_complete(context, new_attachment_id)
2022-04-18 23:38:09.655 1028283 ERROR nova.compute.manager [instance: e3248868-5cb3-4787-a47a-2611bae4fda8] File "/usr/lib/python3.6/site-packages/nova/volume/cinder.py", line 399, in wrapper
2022-04-18 23:38:09.655 1028283 ERROR nova.compute.manager [instance: e3248868-5cb3-4787-a47a-2611bae4fda8] res = method(self, ctx, *args, **kwargs)
2022-04-18 23:38:09.655 1028283 ERROR nova.compute.manager [instance: e3248868-5cb3-4787-a47a-2611bae4fda8] File "/usr/lib/python3.6/site-packages/nova/volume/cinder.py", line 450, in wrapper
2022-04-18 23:38:09.655 1028283 ERROR nova.compute.manager [instance: e3248868-5cb3-4787-a47a-2611bae4fda8] res = method(self, ctx, attachment_id, *args, **kwargs)
2022-04-18 23:38:09.655 1028283 ERROR nova.compute.manager [instance: e3248868-5cb3-4787-a47a-2611bae4fda8] File "/usr/lib/python3.6/site-packages/nova/volume/cinder.py", line 923, in attachment_complete
2022-04-18 23:38:09.655 1028283 ERROR nova.compute.manager [instance: e3248868-5cb3-4787-a47a-2611bae4fda8] 'code': getattr(ex, 'code', None)})
2022-04-18 23:38:09.655 1028283 ERROR nova.compute.manager [instance: e3248868-5cb3-4787-a47a-2611bae4fda8] File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2022-04-18 23:38:09.655 1028283 ERROR nova.compute.manager [instance: e3248868-5cb3-4787-a47a-2611bae4fda8] self.force_reraise()
2022-04-18 23:38:09.655 1028283 ERROR nova.compute.manager [instance: e3248868-5cb3-4787-a47a-2611bae4fda8] File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2022-04-18 23:38:09.655 1028283 ERROR nova.compute.manager [instance: e3248868-5cb3-4787-a47a-2611bae4fda8] six.reraise(self.type_, self.value, self.tb)
2022-04-18 23:38:09.655 1028283 ERROR nova.compute.manager [instance: e3248868-5cb3-4787-a47a-2611bae4fda8] File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2022-04-18 23:38:09.655 1028283 ERROR nova.compute.manager [instance: e3248868-5cb3-4787-a47a-2611bae4fda8] raise value
2022-04-18 23:38:09.655 1028283 ERROR nova.compute.manager [instance: e3248868-5cb3-4787-a47a-2611bae4fda8] File "/usr/lib/python3.6/site-packages/nova/volume/cinder.py", line 916, in attachment_complete
2022-04-18 23:38:09.655 1028283 ERROR nova.compute.manager [instance: e3248868-5cb3-4787-a47a-2611bae4fda8] attachment_id)
2022-04-18 23:38:09.655 1028283 ERROR nova.compute.manager [instance: e3248868-5cb3-4787-a47a-2611bae4fda8] File "/usr/lib/python3.6/site-packages/cinderclient/api_versions.py", line 423, in substitution
2022-04-18 23:38:09.655 1028283 ERROR nova.compute.manager [instance: e3248868-5cb3-4787-a47a-2611bae4fda8] return method.func(obj, *args, **kwargs)
2022-04-18 23:38:09.655 1028283 ERROR nova.compute.manager [instance: e3248868-5cb3-4787-a47a-2611bae4fda8] File "/usr/lib/python3.6/site-packages/cinderclient/v3/attachments.py", line 85, in complete
2022-04-18 23:38:09.655 1028283 ERROR nova.compute.manager [instance: e3248868-5cb3-4787-a47a-2611bae4fda8] None)
2022-04-18 23:38:09.655 1028283 ERROR nova.compute.manager [instance: e3248868-5cb3-4787-a47a-2611bae4fda8] File "/usr/lib/python3.6/site-packages/cinderclient/v3/attachments.py", line 96, in _action_return_resp_and_body
2022-04-18 23:38:09.655 1028283 ERROR nova.compute.manager [instance: e3248868-5cb3-4787-a47a-2611bae4fda8] return self.api.client.post(url, body=body)
2022-04-18 23:38:09.655 1028283 ERROR nova.compute.manager [instance: e3248868-5cb3-4787-a47a-2611bae4fda8] File "/usr/lib/python3.6/site-packages/cinderclient/client.py", line 215, in post
2022-04-18 23:38:09.655 1028283 ERROR nova.compute.manager [instance: e3248868-5cb3-4787-a47a-2611bae4fda8] return self._cs_request(url, 'POST', **kwargs)
2022-04-18 23:38:09.655 1028283 ERROR nova.compute.manager [instance: e3248868-5cb3-4787-a47a-2611bae4fda8] File "/usr/lib/python3.6/site-packages/cinderclient/client.py", line 203, in _cs_request
2022-04-18 23:38:09.655 1028283 ERROR nova.compute.manager [instance: e3248868-5cb3-4787-a47a-2611bae4fda8] return self.request(url, method, **kwargs)
2022-04-18 23:38:09.655 1028283 ERROR nova.compute.manager [instance: e3248868-5cb3-4787-a47a-2611bae4fda8] File "/usr/lib/python3.6/site-packages/cinderclient/client.py", line 189, in request
2022-04-18 23:38:09.655 1028283 ERROR nova.compute.manager [instance: e3248868-5cb3-4787-a47a-2611bae4fda8] raise exceptions.from_response(resp, body)
2022-04-18 23:38:09.655 1028283 ERROR nova.compute.manager [instance: e3248868-5cb3-4787-a47a-2611bae4fda8] cinderclient.exceptions.Unauthorized: The request you have made requires authentication. (HTTP 401)
2022-04-18 23:38:09.655 1028283 ERROR nova.compute.manager [instance: e3248868-5cb3-4787-a47a-2611bae4fda8]
2022-04-18 23:38:09.661 1028283 ERROR root [req-5b02ec6b-e9cb-40af-bfe5-8a5fbe0a6219 fe66d45300e8433da89acf3ccfa239bd d6287309168b42cd8242ef41db87f998 - default default] Original exception being dropped: ['Traceback (most recent call last):\n', ' File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 7219, in _swap_volume\n self.volume_api.attachment_complete(context, new_attachment_id)\n', ' File "/usr/lib/python3.6/site-packages/nova/volume/cinder.py", line 399, in wrapper\n res = method(self, ctx, *args, **kwargs)\n', ' File "/usr/lib/python3.6/site-packages/nova/volume/cinder.py", line 450, in wrapper\n res = method(self, ctx, attachment_id, *args, **kwargs)\n', ' File "/usr/lib/python3.6/site-packages/nova/volume/cinder.py", line 923, in attachment_complete\n \'code\': getattr(ex, \'code\', None)})\n', ' File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', ' File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise\n raise value\n', ' File "/usr/lib/python3.6/site-packages/nova/volume/cinder.py", line 916, in attachment_complete\n attachment_id)\n', ' File "/usr/lib/python3.6/site-packages/cinderclient/api_versions.py", line 423, in substitution\n return method.func(obj, *args, **kwargs)\n', ' File "/usr/lib/python3.6/site-packages/cinderclient/v3/attachments.py", line 85, in complete\n None)\n', ' File "/usr/lib/python3.6/site-packages/cinderclient/v3/attachments.py", line 96, in _action_return_resp_and_body\n return self.api.client.post(url, body=body)\n', ' File "/usr/lib/python3.6/site-packages/cinderclient/client.py", line 215, in post\n return self._cs_request(url, \'POST\', **kwargs)\n', ' File "/usr/lib/python3.6/site-packages/cinderclient/client.py", line 203, in _cs_request\n return self.request(url, method, **kwargs)\n', ' File "/usr/lib/python3.6/site-packages/cinderclient/client.py", line 189, in request\n raise exceptions.from_response(resp, body)\n', 'cinderclient.exceptions.Unauthorized: The request you have made requires authentication. (HTTP 401)\n']: cinderclient.exceptions.Unauthorized: The request you have made requires authentication. (HTTP 401)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server [req-5b02ec6b-e9cb-40af-bfe5-8a5fbe0a6219 fe66d45300e8433da89acf3ccfa239bd d6287309168b42cd8242ef41db87f998 - default default] Exception during message handling: cinderclient.exceptions.Unauthorized: The request you have made requires authentication. (HTTP 401)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 7219, in _swap_volume
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server self.volume_api.attachment_complete(context, new_attachment_id)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/volume/cinder.py", line 399, in wrapper
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server res = method(self, ctx, *args, **kwargs)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/volume/cinder.py", line 450, in wrapper
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server res = method(self, ctx, attachment_id, *args, **kwargs)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/volume/cinder.py", line 923, in attachment_complete
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server 'code': getattr(ex, 'code', None)})
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server self.force_reraise()
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server raise value
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/volume/cinder.py", line 916, in attachment_complete
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server attachment_id)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinderclient/api_versions.py", line 423, in substitution
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server return method.func(obj, *args, **kwargs)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinderclient/v3/attachments.py", line 85, in complete
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server None)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinderclient/v3/attachments.py", line 96, in _action_return_resp_and_body
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server return self.api.client.post(url, body=body)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinderclient/client.py", line 215, in post
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server return self._cs_request(url, 'POST', **kwargs)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinderclient/client.py", line 203, in _cs_request
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server return self.request(url, method, **kwargs)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinderclient/client.py", line 189, in request
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server raise exceptions.from_response(resp, body)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server cinderclient.exceptions.Unauthorized: The request you have made requires authentication. (HTTP 401)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 7248, in _swap_volume
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server self.volume_api.roll_detaching(context, old_volume_id)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/volume/cinder.py", line 399, in wrapper
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server res = method(self, ctx, *args, **kwargs)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/volume/cinder.py", line 435, in wrapper
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server res = method(self, ctx, volume_id, *args, **kwargs)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/volume/cinder.py", line 563, in roll_detaching
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server cinderclient(context).volumes.roll_detaching(volume_id)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinderclient/v2/volumes.py", line 397, in roll_detaching
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server return self._action('os-roll_detaching', volume)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinderclient/v2/volumes.py", line 336, in _action
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server resp, body = self.api.client.post(url, body=body)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinderclient/client.py", line 215, in post
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server return self._cs_request(url, 'POST', **kwargs)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinderclient/client.py", line 203, in _cs_request
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server return self.request(url, method, **kwargs)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinderclient/client.py", line 189, in request
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server raise exceptions.from_response(resp, body)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server cinderclient.exceptions.Unauthorized: The request you have made requires authentication. (HTTP 401)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/exception_wrapper.py", line 79, in wrapped
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server function_name, call_dict, binary, tb)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server self.force_reraise()
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server raise value
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/exception_wrapper.py", line 69, in wrapped
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/compute/utils.py", line 1456, in decorated_function
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 217, in decorated_function
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server self.force_reraise()
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server raise value
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 205, in decorated_function
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 7337, in swap_volume
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server new_attachment_id)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 359, in inner
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 7335, in _do_locked_swap_volume
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server instance, new_attachment_id)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 7386, in _do_swap_volume
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server is_cinder_migration)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 7311, in _swap_volume
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server error=failed)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/volume/cinder.py", line 399, in wrapper
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server res = method(self, ctx, *args, **kwargs)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/volume/cinder.py", line 648, in migrate_volume_completion
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server old_volume_id, new_volume_id, error)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinderclient/v2/volumes.py", line 556, in migrate_volume_completion
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server 'error': error})
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinderclient/v2/volumes.py", line 336, in _action
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server resp, body = self.api.client.post(url, body=body)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinderclient/client.py", line 215, in post
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server return self._cs_request(url, 'POST', **kwargs)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinderclient/client.py", line 203, in _cs_request
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server return self.request(url, method, **kwargs)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinderclient/client.py", line 189, in request
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server raise exceptions.from_response(resp, body)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server cinderclient.exceptions.Unauthorized: The request you have made requires authentication. (HTTP 401)
2022-04-18 23:38:09.693 1028283 ERROR oslo_messaging.rpc.server

Tags:

Revision history for this message

Sofia Enriquez (lsofia-enriquez) wrote on 2022-04-20:

Greetings,
Would you mind sharing the Cinder version or the openstack release you're referring to?
Cheers,

tags:	added: ceph migrate retype
summary:	- CInder fails to retype / migrate large volumes + Cinder fails to retype / migrate large volumes
Changed in cinder:
importance:	Undecided → Medium

Sofia Enriquez (lsofia-enriquez) on 2022-04-20

summary:

- Cinder fails to retype / migrate large volumes
+ [RBD] Cinder fails to retype / migrate large volumes

Revision history for this message

Sofia Enriquez (lsofia-enriquez) wrote on 2022-04-20:

Bug discussed on the weekly bug meeting https://meetings.opendev.org/meetings/cinder_bs/2022/cinder_bs.2022-04-20-15.04.log.html#l-28

On the one hand, looks like it's a token expiry issue for long running ops. There is a way to set up keystone tokens so you don't run into this: https://docs.openstack.org/cinder/latest/configuration/block-storage/service-token.html

On the other hand, is it possible that "bumped up the keystone timeout" wasn't done correctly and the default 30 minutes was still in effect when the 1+ hour migration occurred. Please update the case if so.

Cheers

Revision history for this message

Jason Speckman (jspeckman) wrote on 2022-04-22:

Sorry I have not had a chance to get back sooner...
Thank you for looking into this issue and your suggestions.

I double checked and I had the token expiration timeout set in the wrong spot.
After moving expiration to the token section of the keystone config and restarting, the timeout now shows correctly and I am able to migrate volumes that take over an hour.
Also, thank you for pointing out the service token documentation. I was not aware of the option and may implement it in the future for our installation.

Thanks again, this issue can be closed.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.