Zun

On a particular zun compute node, zun_compute docker always restarts with the following error in the log file

Bug #2043368 reported by Debasis
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Zun
New
Undecided
Unassigned

Bug Description

On a particular zun compute node, zun_compute docker always restarts with the following error in the log file

urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='10.0.0.13', port=2375): Max retries exceeded with url: /v1.40/info (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd787f79990>: Failed to establish a new connection: [Errno 111] ECONNREFUSED'))
2023-11-13 15:02:39.415 7 ERROR zun
2023-11-13 15:02:39.415 7 ERROR zun During handling of the above exception, another exception occurred:
2023-11-13 15:02:39.415 7 ERROR zun
2023-11-13 15:02:39.415 7 ERROR zun Traceback (most recent call last):
2023-11-13 15:02:39.415 7 ERROR zun File "/var/lib/kolla/venv/bin/zun-compute", line 8, in <module>
2023-11-13 15:02:39.415 7 ERROR zun sys.exit(main())
2023-11-13 15:02:39.415 7 ERROR zun File "/var/lib/kolla/venv/lib/python3.10/site-packages/zun/cmd/compute.py", line 43, in main
2023-11-13 15:02:39.415 7 ERROR zun compute_manager.Manager(),
2023-11-13 15:02:39.415 7 ERROR zun File "/var/lib/kolla/venv/lib/python3.10/site-packages/zun/compute/manager.py", line 52, in __init__
2023-11-13 15:02:39.415 7 ERROR zun self.driver = driver_module.load_container_driver(container_driver)
2023-11-13 15:02:39.415 7 ERROR zun File "/var/lib/kolla/venv/lib/python3.10/site-packages/zun/container/driver.py", line 71, in load_container_driver
2023-11-13 15:02:39.415 7 ERROR zun driver = stevedore_driver.DriverManager(
2023-11-13 15:02:39.415 7 ERROR zun File "/var/lib/kolla/venv/lib/python3.10/site-packages/stevedore/driver.py", line 54, in __init__
2023-11-13 15:02:39.415 7 ERROR zun super(DriverManager, self).__init__(
2023-11-13 15:02:39.415 7 ERROR zun File "/var/lib/kolla/venv/lib/python3.10/site-packages/stevedore/named.py", line 78, in __init__
2023-11-13 15:02:39.415 7 ERROR zun extensions = self._load_plugins(invoke_on_load,
2023-11-13 15:02:39.415 7 ERROR zun File "/var/lib/kolla/venv/lib/python3.10/site-packages/stevedore/extension.py", line 218, in _load_plugins
2023-11-13 15:02:39.415 7 ERROR zun self._on_load_failure_callback(self, ep, err)
2023-11-13 15:02:39.415 7 ERROR zun File "/var/lib/kolla/venv/lib/python3.10/site-packages/stevedore/extension.py", line 206, in _load_plugins
2023-11-13 15:02:39.415 7 ERROR zun ext = self._load_one_plugin(ep,
2023-11-13 15:02:39.415 7 ERROR zun File "/var/lib/kolla/venv/lib/python3.10/site-packages/stevedore/named.py", line 156, in _load_one_plugin
2023-11-13 15:02:39.415 7 ERROR zun return super(NamedExtensionManager, self)._load_one_plugin(
2023-11-13 15:02:39.415 7 ERROR zun File "/var/lib/kolla/venv/lib/python3.10/site-packages/stevedore/extension.py", line 242, in _load_one_plugin
2023-11-13 15:02:39.415 7 ERROR zun obj = plugin(*invoke_args, **invoke_kwds)
2023-11-13 15:02:39.415 7 ERROR zun File "/var/lib/kolla/venv/lib/python3.10/site-packages/zun/container/docker/driver.py", line 114, in __init__
2023-11-13 15:02:39.415 7 ERROR zun self._get_host_storage_info()
2023-11-13 15:02:39.415 7 ERROR zun File "/var/lib/kolla/venv/lib/python3.10/site-packages/zun/container/docker/driver.py", line 121, in _get_host_storage_info
2023-11-13 15:02:39.415 7 ERROR zun host_info = self.get_host_info()
2023-11-13 15:02:39.415 7 ERROR zun File "/var/lib/kolla/venv/lib/python3.10/site-packages/zun/container/docker/driver.py", line 1047, in get_host_info
2023-11-13 15:02:39.415 7 ERROR zun info = docker.info()
2023-11-13 15:02:39.415 7 ERROR zun File "/var/lib/kolla/venv/lib/python3.10/site-packages/docker/api/daemon.py", line 95, in info
2023-11-13 15:02:39.415 7 ERROR zun return self._result(self._get(self._url("/info")), True)
2023-11-13 15:02:39.415 7 ERROR zun File "/var/lib/kolla/venv/lib/python3.10/site-packages/docker/utils/decorators.py", line 46, in inner
2023-11-13 15:02:39.415 7 ERROR zun return f(self, *args, **kwargs)
2023-11-13 15:02:39.415 7 ERROR zun File "/var/lib/kolla/venv/lib/python3.10/site-packages/docker/api/client.py", line 237, in _get
2023-11-13 15:02:39.415 7 ERROR zun return self.get(url, **self._set_request_timeout(kwargs))
2023-11-13 15:02:39.415 7 ERROR zun File "/var/lib/kolla/venv/lib/python3.10/site-packages/requests/sessions.py", line 600, in get
2023-11-13 15:02:39.415 7 ERROR zun return self.request("GET", url, **kwargs)
2023-11-13 15:02:39.415 7 ERROR zun File "/var/lib/kolla/venv/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
2023-11-13 15:02:39.415 7 ERROR zun resp = self.send(prep, **send_kwargs)
2023-11-13 15:02:39.415 7 ERROR zun File "/var/lib/kolla/venv/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
2023-11-13 15:02:39.415 7 ERROR zun r = adapter.send(request, **kwargs)
2023-11-13 15:02:39.415 7 ERROR zun File "/var/lib/kolla/venv/lib/python3.10/site-packages/requests/adapters.py", line 565, in send
2023-11-13 15:02:39.415 7 ERROR zun raise ConnectionError(e, request=request)
2023-11-13 15:02:39.415 7 ERROR zun requests.exceptions.ConnectionError: HTTPConnectionPool(host='10.0.0.13', port=2375): Max retries exceeded with url: /v1.40/info (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd787f79990>: Failed to establish a new connection: [Errno 111] ECONNREFUSED'))
2023-11-13 15:02:39.415 7 ERROR zun

When I try to create a docker using zun service from horizon, on the controller node, in the zun-api.log, I see the following

Filtering removed all hosts for the request with container ID 'b3c71d19-7740-4da0-9b32-1989cabfb5a6'. Filter results: ['AvailabilityZoneFilter: (start: 4, end: 4)', 'ComputeFilter: (start: 4, end: 3)', 'RuntimeFilter: (start: 3, end: 0)']

All other zun_compute containers on other compute nodes, running just fine without any error.

Could any suggest me what's the issue?

Revision history for this message
Debasis (debamondal) wrote :

On Horizon, I also see the following error while creating a container

Unexpected error while make filesystem. Devpath: /dev/rbd/volumes/volume-4926f7ec-ee7e-4b68-83f3-c108b4a709d7, Fstype: ext4, Error: The command: mkfs -t ext4 -F /dev/rbd/volumes/volume-4926f7ec-ee7e-4b68-83f3-c108b4a709d7 failed on the system, due to Unexpected error while running command. Command: mkfs -t ext4 -F /dev/rbd/volumes/volume-4926f7ec-ee7e-4b68-83f3-c108b4a709d7 Exit code: 1 Stdout: '' Stderr: 'mke2fs 1.46.5 (30-Dec-2021)\nThe file /dev/rbd/volumes/volume-4926f7ec-ee7e-4b68-83f3-c108b4a709d7 does not exist and no size was specified.\n'

Revision history for this message
Debasis (debamondal) wrote :
Download full text (15.3 KiB)

corresponding zun-compute.logs

2023-11-13 21:04:32.009 7 WARNING os_brick.initiator.connectors.nvmeof [None req-4ad939f9-7245-4f42-9625-e77a8c9a8109 8fa8f9ae1cc1985e470217478e05a06309c5c27d53b025f897fc70088fefbe5b 211000e0a2f34f6aae9f8c130f4ff5bf dac85865914d48b6acce6b98d5e32bd6 - - -] Could not find nvme_core/parameters/multipath: FileNotFoundError: [Errno 2] No such file or directory: '/sys/module/nvme_core/parameters/multipath'
2023-11-13 21:04:32.025 7 WARNING os_brick.initiator.connectors.nvmeof [None req-4ad939f9-7245-4f42-9625-e77a8c9a8109 8fa8f9ae1cc1985e470217478e05a06309c5c27d53b025f897fc70088fefbe5b 211000e0a2f34f6aae9f8c130f4ff5bf dac85865914d48b6acce6b98d5e32bd6 - - -] Process execution error in _get_host_uuid: Unexpected error while running command.
Command: blkid overlay -s UUID -o value
Exit code: 2
Stdout: ''
Stderr: '': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2023-11-13 21:04:32.060 47179 WARNING os_brick.privileged.nvmeof [-] Could not generate host nqn: [Errno 2] No such file or directory: 'nvme'
2023-11-13 21:04:33.169 7 INFO zun.volume.cinder_workflow [None req-4ad939f9-7245-4f42-9625-e77a8c9a8109 8fa8f9ae1cc1985e470217478e05a06309c5c27d53b025f897fc70088fefbe5b 211000e0a2f34f6aae9f8c130f4ff5bf dac85865914d48b6acce6b98d5e32bd6 - - -] Get connection information {'driver_volume_type': 'rbd', 'data': {'name': 'volumes/volume-be72dfcd-8582-48ee-8d19-e762b0e8410d', 'hosts': ['20.0.0.13', '20.0.0.18', '20.0.0.17', '20.0.0.15', '20.0.0.14'], 'ports': ['6789', '6789', '6789', '6789', '6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'cinder', 'secret_type': 'ceph', 'secret_uuid': 'a43734cd-c930-45ee-b4b2-2a0655160146', 'volume_id': 'be72dfcd-8582-48ee-8d19-e762b0e8410d', 'discard': True, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': False, 'cacheable': False}}
2023-11-13 21:04:33.394 7 WARNING os_brick.initiator.connectors.rbd [None req-4ad939f9-7245-4f42-9625-e77a8c9a8109 8fa8f9ae1cc1985e470217478e05a06309c5c27d53b025f897fc70088fefbe5b 211000e0a2f34f6aae9f8c130f4ff5bf dac85865914d48b6acce6b98d5e32bd6 - - -] Volume volume-be72dfcd-8582-48ee-8d19-e762b0e8410d has not been mapped to local device /dev/rbd/volumes/volume-be72dfcd-8582-48ee-8d19-e762b0e8410d; is the udev daemon running and are the ceph-renamer udev rules configured? See bug #1884114 for more information.
2023-11-13 21:04:33.395 7 INFO zun.volume.cinder_workflow [None req-4ad939f9-7245-4f42-9625-e77a8c9a8109 8fa8f9ae1cc1985e470217478e05a06309c5c27d53b025f897fc70088fefbe5b 211000e0a2f34f6aae9f8c130f4ff5bf dac85865914d48b6acce6b98d5e32bd6 - - -] Get device_info after connect to volume {'path': '/dev/rbd/volumes/volume-be72dfcd-8582-48ee-8d19-e762b0e8410d', 'type': 'block'}
2023-11-13 21:04:33.672 7 INFO zun.volume.cinder_workflow [None req-4ad939f9-7245-4f42-9625-e77a8c9a8109 8fa8f9ae1cc1985e470217478e05a06309c5c27d53b025f897fc70088fefbe5b 211000e0a2f34f6aae9f8c130f4ff5bf dac85865914d48b6acce6b98d5e32bd6 - - -] Attach volume to this server successfully
2023-11-13 21:04:33.697 7 INFO oslo.privsep.daemon [None req-4ad939f9-7245-4f42-9625-e77a8c9a8109 8fa8f9ae1cc1985e470217478e05a06309c5c27d...

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.