Failed to consume message from queue: 'NoneType' object has no attribute 'get'

Bug #2084168 reported by Vladimir Kozhukalov
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
oslo.messaging
Fix Released
Critical
Unassigned

Bug Description

Heat-api can not communicate with heat-engine due to error: Failed to consume message from queue: 'NoneType' object has no attribute 'get'. I use Heat 2024.2 (008c18f88c9c2fabae9dbf62d174a4d3b380010d) with oslo.messaging==14.9.0

If I put this [1] into the condition `if self.rabbit_stream_fanout:` it solves the issue, however I am not sure this is the correct solution. Possible solution [2].

[1] https://opendev.org/openstack/oslo.messaging/src/branch/master/oslo_messaging/_drivers/impl_rabbit.py#L635-L639
[2] https://review.opendev.org/c/openstack/oslo.messaging/+/932003

Stacktrace is as follows:

2024-10-09 18:49:58.528 1 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to consume message from queue: 'NoneType' object has no attribute 'get'
2024-10-09 18:49:58.530 1 DEBUG amqp [-] Closed channel #1 collect /var/lib/openstack/lib/python3.10/site-packages/amqp/channel.py:159
2024-10-09 18:49:58.530 1 ERROR oslo.messaging._drivers.impl_rabbit [-] Unable to connect to AMQP server on rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local:5672 after inf tries: 'NoneType' object has no attribute 'get'
2024-10-09 18:49:58.531 1 ERROR root [-] Unexpected exception occurred 1 time(s)... retrying.: oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on rabbitmq-rabbitmq-0.rabbitmq.openstack.svc.cluster.local:5672 after inf tries: 'NoneType' object has no attribute 'get'
2024-10-09 18:49:58.531 1 ERROR root Traceback (most recent call last):
2024-10-09 18:49:58.531 1 ERROR root File "/var/lib/openstack/lib/python3.10/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1163, in ensure
2024-10-09 18:49:58.531 1 ERROR root ret, channel = autoretry_method()
2024-10-09 18:49:58.531 1 ERROR root File "/var/lib/openstack/lib/python3.10/site-packages/kombu/connection.py", line 556, in _ensured
2024-10-09 18:49:58.531 1 ERROR root return fun(*args, **kwargs)
2024-10-09 18:49:58.531 1 ERROR root File "/var/lib/openstack/lib/python3.10/site-packages/kombu/connection.py", line 639, in __call__
2024-10-09 18:49:58.531 1 ERROR root return fun(*args, channel=channels[0], **kwargs), channels[0]
2024-10-09 18:49:58.531 1 ERROR root File "/var/lib/openstack/lib/python3.10/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1152, in execute_method
2024-10-09 18:49:58.531 1 ERROR root method()
2024-10-09 18:49:58.531 1 ERROR root File "/var/lib/openstack/lib/python3.10/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1454, in _consume
2024-10-09 18:49:58.531 1 ERROR root self.connection.drain_events(timeout=poll_timeout)
2024-10-09 18:49:58.531 1 ERROR root File "/var/lib/openstack/lib/python3.10/site-packages/kombu/connection.py", line 341, in drain_events
2024-10-09 18:49:58.531 1 ERROR root return self.transport.drain_events(self.connection, **kwargs)
2024-10-09 18:49:58.531 1 ERROR root File "/var/lib/openstack/lib/python3.10/site-packages/kombu/transport/pyamqp.py", line 171, in drain_events
2024-10-09 18:49:58.531 1 ERROR root return connection.drain_events(**kwargs)
2024-10-09 18:49:58.531 1 ERROR root File "/var/lib/openstack/lib/python3.10/site-packages/amqp/connection.py", line 526, in drain_events
2024-10-09 18:49:58.531 1 ERROR root while not self.blocking_read(timeout):
2024-10-09 18:49:58.531 1 ERROR root File "/var/lib/openstack/lib/python3.10/site-packages/amqp/connection.py", line 532, in blocking_read
2024-10-09 18:49:58.531 1 ERROR root return self.on_inbound_frame(frame)
2024-10-09 18:49:58.531 1 ERROR root File "/var/lib/openstack/lib/python3.10/site-packages/amqp/method_framing.py", line 77, in on_frame
2024-10-09 18:49:58.531 1 ERROR root callback(channel, msg.frame_method, msg.frame_args, msg)
2024-10-09 18:49:58.531 1 ERROR root File "/var/lib/openstack/lib/python3.10/site-packages/amqp/connection.py", line 538, in on_inbound_method
2024-10-09 18:49:58.531 1 ERROR root return self.channels[channel_id].dispatch_method(
2024-10-09 18:49:58.531 1 ERROR root File "/var/lib/openstack/lib/python3.10/site-packages/amqp/abstract_channel.py", line 156, in dispatch_method
2024-10-09 18:49:58.531 1 ERROR root listener(*args)
2024-10-09 18:49:58.531 1 ERROR root File "/var/lib/openstack/lib/python3.10/site-packages/amqp/channel.py", line 1629, in _on_basic_deliver
2024-10-09 18:49:58.531 1 ERROR root fun(msg)
2024-10-09 18:49:58.531 1 ERROR root File "/var/lib/openstack/lib/python3.10/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 626, in _callback
2024-10-09 18:49:58.531 1 ERROR root offset = message.headers.get("x-stream-offset")
2024-10-09 18:49:58.531 1 ERROR root AttributeError: 'NoneType' object has no attribute 'get'
2024-10-09 18:49:58.531 1 ERROR root
2024-10-09 18:49:58.531 1 ERROR root During handling of the above exception, another exception occurred:
2024-10-09 18:49:58.531 1 ERROR root
2024-10-09 18:49:58.531 1 ERROR root Traceback (most recent call last):
2024-10-09 18:49:58.531 1 ERROR root File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 257, in wrapper
2024-10-09 18:49:58.531 1 ERROR root return infunc(*args, **kwargs)
2024-10-09 18:49:58.531 1 ERROR root File "/var/lib/openstack/lib/python3.10/site-packages/oslo_messaging/_drivers/base.py", line 306, in _runner
2024-10-09 18:49:58.531 1 ERROR root incoming = self._poll_style_listener.poll(
2024-10-09 18:49:58.531 1 ERROR root File "/var/lib/openstack/lib/python3.10/site-packages/oslo_messaging/_drivers/base.py", line 54, in wrapper
2024-10-09 18:49:58.531 1 ERROR root message = func(in_self, timeout=watch.leftover(True))
2024-10-09 18:49:58.531 1 ERROR root File "/var/lib/openstack/lib/python3.10/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 417, in poll
2024-10-09 18:49:58.531 1 ERROR root self.conn.consume(timeout=min(self._current_timeout, left))
2024-10-09 18:49:58.531 1 ERROR root File "/var/lib/openstack/lib/python3.10/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1467, in consume
2024-10-09 18:49:58.531 1 ERROR root self.ensure(_consume,
2024-10-09 18:49:58.531 1 ERROR root File "/var/lib/openstack/lib/python3.10/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1185, in ensure
2024-10-09 18:49:58.531 1 ERROR root raise exceptions.MessageDeliveryFailure(msg)

Revision history for this message
Vladimir Kozhukalov (kozhukalov) wrote :

The same issue appears for nova-conductor and nova-scheduler

Changed in oslo.messaging:
importance: Undecided → Critical
Changed in oslo.messaging:
status: New → In Progress
Revision history for this message
Takashi Kajinami (kajinamit) wrote :

Is it possible that you share the rabbitmq related options (especially the ones in [oslo_messaging_rabbit]) you put in heat.conf , for record ?

Revision history for this message
Vladimir Kozhukalov (kozhukalov) wrote :

Here is my heat.conf file. This is the Openstack-Helm test job. The same configuration works for 2024.1 where oslo.messaging==14.7.1 is used.

```
[DEFAULT]
heat_metadata_server_url = http://cloudformation.openstack.svc.cluster.local
heat_waitcondition_server_url = http://cloudformation.openstack.svc.cluster.local/v1/waitcondition
heat_watch_server_url = http://cloudwatch.openstack.svc.cluster.local
host = heat-engine
log_config_append = /etc/heat/logging.conf
num_engine_workers = 1
stack_domain_admin = heat-domain
stack_domain_admin_password = password
stack_user_domain_name = heat
transport_url = rabbit://heat:<email address hidden>:5672/heat
trusts_delegated_roles =
[clients]
endpoint_type = internalURL
[clients_heat]
endpoint_type = publicURL
[clients_keystone]
auth_uri = http://keystone-api.openstack.svc.cluster.local:5000
endpoint_type = internalURL
[database]
connection = mysql+pymysql://heat:<email address hidden>:3306/heat
max_retries = -1
[heat_api]
bind_port = 8004
workers = 1
[heat_api_cfn]
bind_port = 8000
workers = 1
[heat_api_cloudwatch]
bind_port = 8003
workers = 1
[keystone_authtoken]
auth_type = password
auth_uri = http://keystone-api.openstack.svc.cluster.local:5000/v3
auth_url = http://keystone-api.openstack.svc.cluster.local:5000/v3
auth_version = v3
memcache_secret_key = WpcM2jgO3HkPt3URIpvq8zXtvkSL8BW5zfX5HjznZRv8QYM6lUbJC2qIMrZMRwpf
memcache_security_strategy = ENCRYPT
memcached_servers = memcached.openstack.svc.cluster.local:11211
password = password
project_domain_name = service
project_name = service
region_name = RegionOne
service_type = orchestration
user_domain_name = service
username = heat
[oslo_messaging_notifications]
driver = messagingv2
[oslo_messaging_rabbit]
rabbit_ha_queues = true
[oslo_middleware]
enable_proxy_headers_parsing = true
[oslo_policy]
policy_file = /etc/heat/policy.yaml
[paste_deploy]
api_paste_config = /etc/heat/api-paste.ini
[trustee]
auth_type = password
auth_url = http://keystone-api.openstack.svc.cluster.local:5000
auth_version = v3
password = password
region_name = RegionOne
user_domain_name = service
username = heat-trust

```

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to oslo.messaging (master)

Reviewed: https://review.opendev.org/c/openstack/oslo.messaging/+/932003
Committed: https://opendev.org/openstack/oslo.messaging/commit/930d97599c0048e84f7c47dca0d62d2cc0562166
Submitter: "Zuul (22348)"
Branch: master

commit 930d97599c0048e84f7c47dca0d62d2cc0562166
Author: Vladimir Kozhukalov <email address hidden>
Date: Wed Oct 9 18:51:35 2024 -0500

    [rabbit] Ignore stream offset header when stream queue is not used

    Consumer callback should look for x-stream-offset header only if
    the stream mode is used. Otherwise the message headers attribute could
    be None and the error appears when using get method.

    Closes-Bug: #2084168
    Change-Id: I7f9c742f8f557d9faae2cd749d34dcb15d8005c0

Changed in oslo.messaging:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo.messaging (stable/2024.2)

Fix proposed to branch: stable/2024.2
Review: https://review.opendev.org/c/openstack/oslo.messaging/+/932362

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to oslo.messaging (stable/2024.2)

Reviewed: https://review.opendev.org/c/openstack/oslo.messaging/+/932362
Committed: https://opendev.org/openstack/oslo.messaging/commit/d3892f6bac40a8d33d0ce7cd9e656c4cbdf2e0c5
Submitter: "Zuul (22348)"
Branch: stable/2024.2

commit d3892f6bac40a8d33d0ce7cd9e656c4cbdf2e0c5
Author: Vladimir Kozhukalov <email address hidden>
Date: Wed Oct 9 18:51:35 2024 -0500

    [rabbit] Ignore stream offset header when stream queue is not used

    Consumer callback should look for x-stream-offset header only if
    the stream mode is used. Otherwise the message headers attribute could
    be None and the error appears when using get method.

    Closes-Bug: #2084168
    Change-Id: I7f9c742f8f557d9faae2cd749d34dcb15d8005c0
    (cherry picked from commit 930d97599c0048e84f7c47dca0d62d2cc0562166)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo.messaging (stable/2024.1)

Fix proposed to branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/oslo.messaging/+/932519

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/oslo.messaging 14.9.1

This issue was fixed in the openstack/oslo.messaging 14.9.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/oslo.messaging 15.0.0

This issue was fixed in the openstack/oslo.messaging 15.0.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.