Description of the issue
When we do a volume-backed server rebuild, will throw ERROR about os-brick in nova 'os_brick.exception.BrickException: Error unmapping volume 6Vv5vqk/QK6vrBiU/agl6g==: The volume is not mapped to the Host'
Case Name - tempest.api.compute.servers.test_server_actions.ServerActionsV293TestJSON.test_rebuild_volume_backed_server
How often it happens?
every time
Environment with versions
Driver From OpenStack Master Branch (Caracal dev) Code
os-brick version - 6.5.0.dev9
Storage side - PowerFlex 4.5 GA
Steps to reproduce the issue, including commands
Rerun Tempest Case - tempest.api.compute.servers.test_server_actions.ServerActionsV293TestJSON.test_rebuild_volume_backed_server
Manual Steps:
Create a volume vol01 with an image
openstack volume create --image cirros-0.6.2-x86_64-disk --size 8 vol01
Create a server using vol01
openstack server create --flavor m1.small --network NETWORKID --volume vol01 server01
Rebuild the volume-backed server
openstack server rebuild --image cirros-0.6.2-x86_64-disk --reimage-boot-volume --os-compute-api-version 2.93 server01
will get the error from <email address hidden>
-------------------------------------------------------------------------------------------------
DEBUG os_brick.initiator.connectors.scaleio [None req-3048651c-934e-4988-9c2d-a91161cabedb demo admin] <== disconnect_volume: exception (459ms) BrickException('Error unmapping volume J0/LGiAMSHCitOvRrP1+Mw==: The volume is not mapped to the Host') (pid=2302993) trace_logging_wrapper /opt/stack/new/os-brick/os_brick/utils.py:189
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [None req-3048651c-934e-4988-9c2d-a91161cabedb demo admin] [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] Failed to detach volume 274fcb1a-200c-4870-a2b4-ebd1acfd7e33 from /dev/vda: os_brick.exception.BrickException: Error unmapping volume J0/LGiAMSHCitOvRrP1+Mw==: The volume is not mapped to the Host
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] Traceback (most recent call last):
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] File "/opt/stack/new/nova/nova/compute/manager.py", line 4131, in _do_rebuild_instance
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] self.driver.rebuild(**kwargs)
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] File "/opt/stack/new/nova/nova/virt/driver.py", line 384, in rebuild
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] raise NotImplementedError()
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] NotImplementedError
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892]
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] During handling of the above exception, another exception occurred:
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892]
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] Traceback (most recent call last):
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] File "/opt/stack/new/nova/nova/compute/manager.py", line 3555, in _detach_root_volume
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] self.driver.detach_volume(context, old_connection_info,
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2752, in detach_volume
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] self._disconnect_volume(context, connection_info, instance,
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2017, in _disconnect_volume
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] vol_driver.disconnect_volume(
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] File "/opt/stack/new/nova/nova/virt/libvirt/volume/scaleio.py", line 61, in disconnect_volume
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] self.connector.disconnect_volume(
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] File "/opt/stack/new/os-brick/os_brick/utils.py", line 186, in trace_logging_wrapper
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] result = f(*args, **kwargs)
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] File "/opt/stack/new/os-brick/os_brick/initiator/connectors/base.py", line 78, in inner
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] return f(*args, **kwargs)
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] File "/opt/stack/new/os-brick/os_brick/utils.py", line 405, in change_encrypted
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] res = func(**call_args)
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] File "/opt/stack/new/os-brick/os_brick/initiator/connectors/scaleio.py", line 529, in disconnect_volume
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] raise exception.BrickException(message=msg)
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] os_brick.exception.BrickException: Error unmapping volume J0/LGiAMSHCitOvRrP1+Mw==: The volume is not mapped to the Host
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892]
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: DEBUG nova.compute.utils [None req-3048651c-934e-4988-9c2d-a91161cabedb demo admin] [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] BrickException (pid=2302993) notify_about_instance_usage /opt/stack/new/nova/nova/compute/utils.py:430
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [None req-3048651c-934e-4988-9c2d-a91161cabedb demo admin] [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] Setting instance vm_state to ERROR: os_brick.exception.BrickException: Error unmapping volume J0/LGiAMSHCitOvRrP1+Mw==: The volume is not mapped to the Host
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] Traceback (most recent call last):
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] File "/opt/stack/new/nova/nova/compute/manager.py", line 4131, in _do_rebuild_instance
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] self.driver.rebuild(**kwargs)
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] File "/opt/stack/new/nova/nova/virt/driver.py", line 384, in rebuild
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] raise NotImplementedError()
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] NotImplementedError
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892]
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] During handling of the above exception, another exception occurred:
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892]
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] Traceback (most recent call last):
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] File "/opt/stack/new/nova/nova/compute/manager.py", line 10845, in _error_out_instance_on_exception
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] yield
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] File "/opt/stack/new/nova/nova/compute/manager.py", line 3857, in rebuild_instance
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] self._do_rebuild_instance_with_claim(
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] File "/opt/stack/new/nova/nova/compute/manager.py", line 3943, in _do_rebuild_instance_with_claim
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] self._do_rebuild_instance(
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] File "/opt/stack/new/nova/nova/compute/manager.py", line 4135, in _do_rebuild_instance
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] self._rebuild_default_impl(**kwargs)
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] File "/opt/stack/new/nova/nova/compute/manager.py", line 3713, in _rebuild_default_impl
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] self._rebuild_volume_backed_instance(
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] File "/opt/stack/new/nova/nova/compute/manager.py", line 3590, in _rebuild_volume_backed_instance
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] self._detach_root_volume(context, instance, root_bdm)
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] File "/opt/stack/new/nova/nova/compute/manager.py", line 3569, in _detach_root_volume
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] with excutils.save_and_reraise_exception():
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in _exit_
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] self.force_reraise()
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] raise self.value
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] File "/opt/stack/new/nova/nova/compute/manager.py", line 3555, in _detach_root_volume
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] self.driver.detach_volume(context, old_connection_info,
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2752, in detach_volume
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] self._disconnect_volume(context, connection_info, instance,
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2017, in _disconnect_volume
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] vol_driver.disconnect_volume(
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] File "/opt/stack/new/nova/nova/virt/libvirt/volume/scaleio.py", line 61, in disconnect_volume
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] self.connector.disconnect_volume(
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] File "/opt/stack/new/os-brick/os_brick/utils.py", line 186, in trace_logging_wrapper
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] result = f(*args, **kwargs)
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] File "/opt/stack/new/os-brick/os_brick/initiator/connectors/base.py", line 78, in inner
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] return f(*args, **kwargs)
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] File "/opt/stack/new/os-brick/os_brick/utils.py", line 405, in change_encrypted
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] res = func(**call_args)
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] File "/opt/stack/new/os-brick/os_brick/initiator/connectors/scaleio.py", line 529, in disconnect_volume
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] raise exception.BrickException(message=msg)
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892] os_brick.exception.BrickException: Error unmapping volume J0/LGiAMSHCitOvRrP1+Mw==: The volume is not mapped to the Host
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: ERROR nova.compute.manager [instance: 5ee7e0e2-5c0b-4786-9e41-78adf716c892]
Oct 20 21:37:06 pflex237188 nova-compute[2302993]: DEBUG oslo_concurrency.lockutils [None req-3048651c-934e-4988-9c2d-a91161cabedb demo admin] Acquiring lock "compute_resources" by "nova.compute.resource_tracker.ResourceTracker.update_usage" (pid=2302993) inner /usr/local/lib/python3.10/dist-packages/oslo_concurrency/lockutils.py:404
-------------------------------------------------------------------------------------------------
The reason why it’s failing against v4 is because they changed the expected error_code that the code is looking for.
https:/ /opendev. org/openstack/ os-brick/ src/commit/ fb23a40a65f1a6e 06b4580ca309469 f9ae2cced6/ os_brick/ initiator/ connectors/ scaleio. py#L519
the code is looking for VOLUME_ NOT_MAPPED_ ERROR = 84 as show here: https:/ /opendev. org/openstack/ os-brick/ src/commit/ fb23a40a65f1a6e 06b4580ca309469 f9ae2cced6/ os_brick/ initiator/ connectors/ scaleio. py#L50
but the response given by the API is 4039.