Mistral

Mistral not starting on devstack

Bug #2040048 reported by Arnaud Morin on 2023-10-21

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Mistral	Fix Released	Undecided	Arnaud Morin

Bug Description

When running devstack, following the documentation:
https://docs.openstack.org/mistral/latest/contributor/devstack.html

It fails with a stacktrace in systemd:

Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 ERROR oslo_service.service [-] Error starting thread.: etcd3gw.exceptions.Etcd3Exception: Not Found
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service Traceback (most recent call last):
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_service/service.py", line 806, in run_service
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service service.start()
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service File "/opt/stack/mistral/mistral/engine/engine_server.py", line 60, in start
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service super(EngineServer, self).start()
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service File "/opt/stack/mistral/mistral/service/base.py", line 46, in start
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service self.cluster_member.register_membership()
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py", line 414, in inner
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service return f(*args, **kwargs)
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service File "/opt/stack/mistral/mistral/service/coordination.py", line 174, in register_membership
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service service_coordinator = get_service_coordinator()
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service File "/opt/stack/mistral/mistral/service/coordination.py", line 156, in get_service_coordinator
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service _SERVICE_COORDINATOR.start()
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service File "/opt/stack/mistral/mistral/service/coordination.py", line 54, in start
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service self._coordinator.start(start_heart=True)
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/tooz/coordination.py", line 689, in start
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service super(CoordinationDriverWithExecutor, self).start(start_heart)
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/tooz/coordination.py", line 426, in start
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service self._start()
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/tooz/drivers/etcd3gw.py", line 235, in _start
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service self._membership_lease = self.client.lease(self.membership_timeout)
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/etcd3gw/client.py", line 122, in lease
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service result = self.post(self.get_url("/lease/grant"),
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service File "/opt/stack/data/venv/lib/python3.10/site-packages/etcd3gw/client.py", line 92, in post
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service raise exceptions.Etcd3Exception(resp.text, resp.reason)
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service etcd3gw.exceptions.Etcd3Exception: Not Found
Oct 20 21:56:40 arnaud-devstack mistral-server[115482]: 2023-10-20 21:56:40.669 TRACE oslo_service.service

This seems to be the root cause of the gate failing recently

Revision history for this message

Arnaud Morin (arnaud-morin) wrote on 2023-10-23:

I found the root cause, etcd3 endpoint is not configured correctly in mistral config on devstack.

This is the same issue as this one for cinder:

https://review.opendev.org/c/openstack/devstack/+/891353

Changed in mistral:
assignee:	nobody → Arnaud Morin (arnaud-morin)
status:	New → Confirmed

Revision history for this message

Jan Gutter (jangutter) wrote on 2023-10-23:

Apologies, yeah, etcdv3 has dropped support for the deprecated endpoint and tooz couldn't be updated in time to make the default transparent.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-10-24: Fix proposed to mistral (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/mistral/+/899169

Changed in mistral:
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-11-27: Fix merged to mistral (master)

Reviewed: https://review.opendev.org/c/openstack/mistral/+/899169
Committed: https://opendev.org/openstack/mistral/commit/e343ccb078d8ba261ac70afca93f4358589730d3
Submitter: "Zuul (22348)"
Branch: master

commit e343ccb078d8ba261ac70afca93f4358589730d3
Author: Arnaud Morin <email address hidden>
Date: Tue Oct 24 15:14:53 2023 +0200

Add api_version for etcd3 coordination URL

    Since upgrade of etcd3 (see [1]), the v3aplha endpoint has been
    deprecated but tooz commit (see [2]) is not yet released.
    Until then, we should force the api_version=v3 to make sure the
    coordination between engine is working correctly.

Closes-Bug: #2040048

Signed-off-by: Arnaud Morin <email address hidden>
Change-Id: Ieade7a30a192910fc3870b173017efd481f0b18f

Changed in mistral:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-03-18: Fix included in openstack/mistral 18.0.0.0rc1

This issue was fixed in the openstack/mistral 18.0.0.0rc1 release candidate.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.