[huawei] wrong hostname passed to os_brick

Bug #2017575 reported by Rodrigo Barbieri
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Incomplete
Low
Unassigned
Ussuri
New
Low
Unassigned

Bug Description

When os-brick is invoked through get_connector_properties [1], it may or may not receive a host parameter. Nova sends the host parameter [1] according to the host config set, but cinder does not [2]. When the host parameter is not sent, os-brick will run socket.gethostname() and that may return a value that is different (such as the short hostname instead of the FQDN) than what Nova is passing, resulting in failures in certain workflows such as image-download followed by attach, because the iscsi initiator is saved with a different value than Nova's.

I believe Cinder should pass the configured "host" value the same way Nova does.

[1] https://github.com/openstack/nova/blob/d6aa8120995ab281b1b755776297c7385bee1c8f/nova/virt/libvirt/driver.py#L1799

[2] https://github.com/openstack/cinder/blob/40147e2489c1f4f9940dc3df2cfa0fa81b5070df/cinder/volume/volume_utils.py#L1391

[3] https://github.com/openstack/os-brick/blob/b6f0f3c25ecda80861ff233ca8526d338319bd49/os_brick/initiator/connector.py#L224

Revision history for this message
Rodrigo Barbieri (rodrigo-barbieri2010) wrote :
Download full text (10.9 KiB)

Stack trace:

2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server [req-47821d5d-88da-42fe-b57d-313e28d8e601 106fd23f1fbb4d96aca2c727bb20f6fa c1524133a19945fc9f59708819277bc9 - 86de275b9fa1445c8642cabcc1dd698b 86de275b9fa1445c8642cabcc1dd698b] Exception during message handling: cinder.exception.ImageCopyFailure: Failed to copy image to volume: Bad or unexpected response from the storage volume backend API: Unable to fetch connection information from backend: Bad or unexpected response from the storage volume backend API: Initiator iqn.1993-08.org.debian:01:26abafb22f4 has been added to another host 7f7afc77863a34e8e2803135be19e31.
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/driver.py", line 1045, in _attach_volume
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server conn = self.initialize_connection(volume, properties)
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server File "<decorator-gen-731>", line 2, in initialize_connection
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/coordination.py", line 151, in _synchronized
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server return f(*a, **k)
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/drivers/huawei/huawei_driver.py", line 104, in initialize_connection
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server self.client.ensure_initiator_added(initiator_name,
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/drivers/huawei/rest_client.py", line 504, in ensure_initiator_added
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server if not self.is_initiator_associated_to_host(initiator_name, host_id):
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/drivers/huawei/rest_client.py", line 783, in is_initiator_associated_to_host
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server raise exception.VolumeBackendAPIException(data=msg)
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server cinder.exception.VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Initiator iqn.1993-08.org.debian:01:26abafb22f4 has been added to another host 7f7afc77863a34e8e2803135be19e31.
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/volume_utils.py", line 1152, in copy_image_to_volume
2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server ...

tags: added: sts
Revision history for this message
Sofia Enriquez (lsofia-enriquez) wrote :

Hello Rodrigo Barbieri, I hope this message finds you doing well.
Would you kindly provide me with the following information:

- The steps required to reproduce the issue you are experiencing and which operations are impacted by it.
- Are you currently utilizing Cinder as a Glance backend? If not, which backend are you currently utilizing?
- Which version of OpenStack are you using?

Thank you in advance for your assistance.
Best regards,
Sofia

Changed in cinder:
status: New → Incomplete
Revision history for this message
Rodrigo Barbieri (rodrigo-barbieri2010) wrote :

Hi Sofia

- Using stable/ussuri
- Step to reproduce is doing anything that requires Cinder to download an image from glance, such as creating an instance with a boot from volume, or just create a volume from image
- Cinder is not being used as Glance backend
- Using Huawei oceanstore backend for Cinder

Changed in cinder:
status: Incomplete → New
Revision history for this message
Rajat Dhasmana (whoami-rajat) wrote :

Hi Rodrigo,

Nova passes a configured host value[1] which also defaults to socket.gethostname()[2].
You are correct that cinder doesn't pass any value for the host field[3] and os-brick uses socket.gethostname() again[4].
A similar case can be seen in the glance cinder store where we also pass socket.gethostname()[5]

One way to address this in cinder and glance_store is to add a config option for "host" field and operators can configure it in cinder.conf file. By default, the config option will use socket.gethostname() value.

[1] https://github.com/openstack/nova/blob/b9ac827491fff76eb6e4cd243d9be31304ea338d/nova/virt/libvirt/driver.py#L1799
[2] https://opendev.org/openstack/nova/src/commit/b9ac827491fff76eb6e4cd243d9be31304ea338d/nova/conf/netconf.py#L53
[3] https://github.com/openstack/cinder/blob/b75c29c7d8e0e6ac212b59f9ad8d140874e55251/cinder/volume/volume_utils.py#L1391-L1394
[4] https://github.com/openstack/os-brick/blob/b6f0f3c25ecda80861ff233ca8526d338319bd49/os_brick/initiator/connector.py#L224
[5] https://github.com/openstack/glance_store/blob/6741951591ca7d6144f6089678df8cee4f0a7030/glance_store/_drivers/cinder/store.py#L713

Revision history for this message
Rajat Dhasmana (whoami-rajat) wrote :

Looking at the error trace, this seems specific to the huawei driver since it creates a hostgroup from the "host" property in connector. Other drivers doing similarly can face it as well but I'm not aware of them.

2023-02-04 05:33:29.958 933946 ERROR oslo_messaging.rpc.server cinder.exception.VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Initiator iqn.1993-08.org.debian:01:26abafb22f4 has been added to another host 7f7afc77863a34e8e2803135be19e31.

here the host property is used
https://github.com/openstack/cinder/blob/40147e2489c1f4f9940dc3df2cfa0fa81b5070df/cinder/volume/drivers/huawei/huawei_driver.py#L101-L102

here we error out
https://github.com/openstack/cinder/blob/40147e2489c1f4f9940dc3df2cfa0fa81b5070df/cinder/volume/drivers/huawei/rest_client.py#L785-L787

Changed in cinder:
importance: Undecided → Low
tags: added: drivers huawei
summary: - wrong hostname passed to os_brick
+ [huawei] wrong hostname passed to os_brick
Revision history for this message
Sofia Enriquez (lsofia-enriquez) wrote :

This bug was discussed on May 31, 2023. You can refer to the discussion here: https://meetings.opendev.org/meetings/cinder_bs/2023/cinder_bs.2023-05-31-15.03.log.html#l-67 and https://meetings.opendev.org/irclogs/%23openstack-cinder/%23openstack-cinder.2023-06-01.log.html

15:27:32 <hillpd> Going forward, passing the cinder.conf "host" value to os-brick seems sensible. Unfortunately, it wouldn't be safe to backport this due to how the host value is used in existing deployments.
15:27:59 <hillpd> To address this in previous releases, we are considering the addition of a "[cinder-oceanstor]" config option, "use_host_fqdn" to handle this in the Huawei driver. Would that be a reasonable approach?

<whoami-rajat> hillpd, hey, I'm unsure how this would pan out. The code that establishes the connection via os-brick isn't part of the driver, so how would the Huawei driver utilize this config option? Are you suggesting we disregard the connector['host'] value and use the config option instead?

@hillpd, please respond here so that the conversation doesn't get lost. It would be even better if you could propose a WIP patch. This way, we can see how it fully functions and continue the discussion based on the actual code.

Thank you!

Changed in cinder:
status: New → Incomplete
Revision history for this message
Sofia Enriquez (lsofia-enriquez) wrote :

hillpd> whoami-rajat, howdy, yeah the idea is to intercept and modify the connector['host'] value that gets used by the huawei driver.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.