Queue Manager fails in podman containerised environments

Bug #2095178 reported by Matt Crees
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
oslo.messaging
In Progress
Medium
Matt Crees

Bug Description

We hit this when trying to use queue manager in Kolla-Ansible. An invocation of nova-manage fails with the following traceback:

root@primary:~# podman exec nova_conductor nova-manage cell_v2 discover_hosts --by-service --cell_uuid c321ca04-f6e5-44dc-b257-b9ebac11a5eb
Traceback (most recent call last):
  File "/var/lib/kolla/venv/bin/nova-manage", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/cmd/manage.py", line 3815, in main
    config.parse_args(sys.argv)
  File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/config.py", line 101, in parse_args
    rpc.init(CONF)
  File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/rpc.py", line 68, in init
    TRANSPORT = create_transport(get_transport_url())
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/kolla/venv/lib/python3.12/site-packages/nova/rpc.py", line 255, in create_transport
    return messaging.get_rpc_transport(CONF,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/rpc/transport.py", line 50, in get_rpc_transport
    return msg_transport._get_transport(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/transport.py", line 205, in _get_transport
    mgr = driver.DriverManager('oslo.messaging.drivers',
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/kolla/venv/lib/python3.12/site-packages/stevedore/driver.py", line 54, in __init__
    super().__init__(
  File "/var/lib/kolla/venv/lib/python3.12/site-packages/stevedore/named.py", line 78, in __init__
    extensions = self._load_plugins(invoke_on_load,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/kolla/venv/lib/python3.12/site-packages/stevedore/extension.py", line 218, in _load_plugins
    self._on_load_failure_callback(self, ep, err)
  File "/var/lib/kolla/venv/lib/python3.12/site-packages/stevedore/extension.py", line 206, in _load_plugins
    ext = self._load_one_plugin(ep,
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/kolla/venv/lib/python3.12/site-packages/stevedore/named.py", line 156, in _load_one_plugin
    return super()._load_one_plugin(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/kolla/venv/lib/python3.12/site-packages/stevedore/extension.py", line 242, in _load_one_plugin
    obj = plugin(*invoke_args, **invoke_kwds)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1787, in __init__
    super().__init__(
  File "/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 691, in __init__
    self._q_manager = QManager(
                      ^^^^^^^^^
  File "/var/lib/kolla/venv/lib/python3.12/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 73, in __init__
    with open(f'/proc/{self.pg}/stat') as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/proc/0/stat'

The issue here is that the group process id is zero when podman directly calls this command. /proc/0 doesn't exist, so the command fails.

podman exec nova_conductor ps xao pid,ppid,pgid,sid,comm
    PID PPID PGID SID COMMAND
      1 0 1 1 dumb-init
      2 1 1 1 nova-conductor
   2389 0 0 0 ps

As an aside, this doesn't affect docker as the process group id is not 0:

docker exec nova_conductor ps xao pid,ppid,pgid,sid,comm
    PID PPID PGID SID COMMAND
      1 0 1 1 dumb-init
      6 1 1 1 nova-conductor
     39 6 1 1 nova-conductor
     40 6 1 1 nova-conductor
   2418 0 2418 2418 ps

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo.messaging (master)
Changed in oslo.messaging:
status: New → In Progress
Changed in oslo.messaging:
importance: Undecided → Medium
assignee: nobody → Matt Crees (mattcrees)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.