Since tooz 4.3.0 was released, grenade-skip-level-always is broken[1].
[1] https://zuul.opendev.org/t/openstack/builds?job_name=grenade-skip-level-always&project=openstack/nova
The root cause is that cinder fails to connect the etcd server because of the following error.
2023-11-17 06:29:24.781 | Nov 17 06:29:20 np0035791906 <email address hidden>[169832]: ERROR cinder File "/usr/local/lib/python3.10/dist-packages/tooz/drivers/etcd3gw.py", line 235, in _start
2023-11-17 06:29:24.781 | Nov 17 06:29:20 np0035791906 <email address hidden>[169832]: ERROR cinder self._membership_lease = self.client.lease(self.membership_timeout)
2023-11-17 06:29:24.781 | Nov 17 06:29:20 np0035791906 <email address hidden>[169832]: ERROR cinder File "/usr/local/lib/python3.10/dist-packages/etcd3gw/client.py", line 123, in lease
2023-11-17 06:29:24.781 | Nov 17 06:29:20 np0035791906 <email address hidden>[169832]: ERROR cinder result = self.post(self.get_url("/lease/grant"),
2023-11-17 06:29:24.781 | Nov 17 06:29:20 np0035791906 <email address hidden>[169832]: ERROR cinder File "/usr/local/lib/python3.10/dist-packages/etcd3gw/client.py", line 93, in post
2023-11-17 06:29:24.781 | Nov 17 06:29:20 np0035791906 <email address hidden>[169832]: ERROR cinder raise exceptions.Etcd3Exception(resp.text, resp.reason)
2023-11-17 06:29:24.781 | Nov 17 06:29:20 np0035791906 <email address hidden>[169832]: ERROR cinder etcd3gw.exceptions.Etcd3Exception: Not Found
2023-11-17 06:29:24.781 | Nov 17 06:29:20 np0035791906 <email address hidden>[169832]: ERROR cinder
2023-11-17 06:29:24.781 | Nov 17 06:29:20 np0035791906 <email address hidden>[169832]: unable to load app 0 (mountpoint='') (callable not found or import error)
When the node is deployed initially it is deployed with etcd v3.3.12.
The coordination_url parameter in cinder.conf does not contain api_version but tooz 3.2.0 used v1alpha by default which is available in etcd v3.3.12.
The problem arises when all components, including cinder are upgraded while etcd is kept at v3.3.12.
The coordination_url in cinder does not contain api_version at this point.
However the tooz 4.3.0 uses v3 API by default but this is not available in etcd v3.3.12 and eventually results in the above failure.
Although we added a release note to explain this requirement during upgrade, it's much better if we can implement some logic to avoid needing operators to update the config file.
Reviewed: https:/
Committed: https:/
Submitter: "Zuul (22348)"
Branch: master
commit e5faa53eada8b17
Author: Takashi Kajinami <email address hidden>
Date: Fri Nov 17 22:53:59 2023 +0900
Discover API version automatically
etcd changed API version from v3alpha to v3beta, then v3, and because
of transition available api versions are different according to
the etcd server available.
This change implements the mechanism to detect the current api version
according to the etcd version obtained via the version API. Discovery
is executed at the first request and the result is stored.
Closes-Bug: #2043810
Change-Id: I08bc429dd8c59e