[huawei] wrong hostname passed to os_brick

Stack trace:

2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server [req-47821d5d-88da-42fe-b57d-313e28d8e601 106fd23f1fbb4d96aca2c727bb20f6fa c1524133a19945fc9f59708819277bc9 - 86de275b9fa1445c8642cabcc1dd698b 86de275b9fa1445c8642cabcc1dd698b] Exception during message handling: cinder.exception.ImageCopyFailure: Failed to copy image to volume: Bad or unexpected response from the storage volume backend API: Unable to fetch connection information from backend: Bad or unexpected response from the storage volume backend API: Initiator iqn.1993-08.org.debian:01:26abafb22f4 has been added to another host 7f7afc77863a34e8e2803135be19e31.
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/driver.py", line 1045, in _attach_volume
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server conn = self.initialize_connection(volume, properties)
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server File "<decorator-gen-731>", line 2, in initialize_connection
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/coordination.py", line 151, in _synchronized
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server return f(*a, **k)
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/drivers/huawei/huawei_driver.py", line 104, in initialize_connection
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server self.client.ensure_initiator_added(initiator_name,
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/drivers/huawei/rest_client.py", line 504, in ensure_initiator_added
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server if not self.is_initiator_associated_to_host(initiator_name, host_id):
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/drivers/huawei/rest_client.py", line 783, in is_initiator_associated_to_host
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server raise exception.VolumeBackendAPIException(data=msg)
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server cinder.exception.VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Initiator iqn.1993-08.org.debian:01:26abafb22f4 has been added to another host 7f7afc77863a34e8e2803135be19e31.
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/volume_utils.py", line 1152, in copy_image_to_volume
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server ...

Hello Rodrigo Barbieri, I hope this message finds you doing well.
Would you kindly provide me with the following information:

- The steps required to reproduce the issue you are experiencing and which operations are impacted by it.
- Are you currently utilizing Cinder as a Glance backend? If not, which backend are you currently utilizing?
- Which version of OpenStack are you using?

Thank you in advance for your assistance.
Best regards,
Sofia

Hi Sofia

- Using stable/ussuri
- Step to reproduce is doing anything that requires Cinder to download an image from glance, such as creating an instance with a boot from volume, or just create a volume from image
- Cinder is not being used as Glance backend
- Using Huawei oceanstore backend for Cinder

Hi Rodrigo,

Nova passes a configured host value[1] which also defaults to socket.gethostname()[2].
You are correct that cinder doesn't pass any value for the host field[3] and os-brick uses socket.gethostname() again[4].
A similar case can be seen in the glance cinder store where we also pass socket.gethostname()[5]

One way to address this in cinder and glance_store is to add a config option for "host" field and operators can configure it in cinder.conf file. By default, the config option will use socket.gethostname() value.

[1] https://github.com/openstack/nova/blob/b9ac827491fff76eb6e4cd243d9be31304ea338d/nova/virt/libvirt/driver.py#L1799
[2] https://opendev.org/openstack/nova/src/commit/b9ac827491fff76eb6e4cd243d9be31304ea338d/nova/conf/netconf.py#L53
[3] https://github.com/openstack/cinder/blob/b75c29c7d8e0e6ac212b59f9ad8d140874e55251/cinder/volume/volume_utils.py#L1391-L1394
[4] https://github.com/openstack/os-brick/blob/b6f0f3c25ecda80861ff233ca8526d338319bd49/os_brick/initiator/connector.py#L224
[5] https://github.com/openstack/glance_store/blob/6741951591ca7d6144f6089678df8cee4f0a7030/glance_store/_drivers/cinder/store.py#L713

Looking at the error trace, this seems specific to the huawei driver since it creates a hostgroup from the "host" property in connector. Other drivers doing similarly can face it as well but I'm not aware of them.

2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server cinder.exception.VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Initiator iqn.1993-08.org.debian:01:26abafb22f4 has been added to another host 7f7afc77863a34e8e2803135be19e31.

here the host property is used
https://github.com/openstack/cinder/blob/40147e2489c1f4f9940dc3df2cfa0fa81b5070df/cinder/volume/drivers/huawei/huawei_driver.py#L101-L102

here we error out
https://github.com/openstack/cinder/blob/40147e2489c1f4f9940dc3df2cfa0fa81b5070df/cinder/volume/drivers/huawei/rest_client.py#L785-L787

This bug was discussed on May 31, 2023. You can refer to the discussion here: https://meetings.opendev.org/meetings/cinder_bs/2023/cinder_bs.2023-05-31-15.03.log.html#l-67 and https://meetings.opendev.org/irclogs/%23openstack-cinder/%23openstack-cinder.2023-06-01.log.html

15:27:32 <hillpd> Going forward, passing the cinder.conf "host" value to os-brick seems sensible. Unfortunately, it wouldn't be safe to backport this due to how the host value is used in existing deployments.
15:27:59 <hillpd> To address this in previous releases, we are considering the addition of a "[cinder-oceanstor]" config option, "use_host_fqdn" to handle this in the Huawei driver. Would that be a reasonable approach?

<whoami-rajat> hillpd, hey, I'm unsure how this would pan out. The code that establishes the connection via os-brick isn't part of the driver, so how would the Huawei driver utilize this config option? Are you suggesting we disregard the connector['host'] value and use the config option instead?

@hillpd, please respond here so that the conversation doesn't get lost. It would be even better if you could propose a WIP patch. This way, we can see how it fully functions and continue the discussion based on the actual code.

Thank you!

hillpd> whoami-rajat, howdy, yeah the idea is to intercept and modify the connector['host'] value that gets used by the huawei driver.