On a deployment running Ussuri Bionic, 2 of the 3 designate units stayed blocked waiting for the "identity-service" relation for 3 hours.
The designate central log shows that it is failing to connect to MySQL, and the other designate services are failing to connect to rabbit.
2022-03-27 12:52:32.887 14100 ERROR oslo.service.loopingcall [req-d8171106-9f24-48f3-b4c4-6efbde4af37a - - - - -] Fixed interval looping call 'designate.heartbeat_emitter.HeartbeatEmitter._emit_heartbeat' failed: oslo_db.exception.CantStartEngineError: No sql_connection parameter is established
2022-03-27 12:52:32.887 14100 ERROR oslo.service.loopingcall Traceback (most recent call last):
2022-03-27 12:52:32.887 14100 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/oslo_service/loopingcall.py", line 67, in func
2022-03-27 12:52:32.887 14100 ERROR oslo.service.loopingcall return f(*args, **kwargs)
2022-03-27 12:52:32.887 14100 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/designate/heartbeat_emitter.py", line 89, in _emit_heartbeat
2022-03-27 12:52:32.887 14100 ERROR oslo.service.loopingcall self.transmit(service_status)
2022-03-27 12:52:32.887 14100 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/designate/heartbeat_emitter.py", line 109, in transmit
2022-03-27 12:52:32.887 14100 ERROR oslo.service.loopingcall api.update_service_status(admin_context, status)
2022-03-27 12:52:32.887 14100 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/designate/rpc.py", line 238, in exception_wrapper
2022-03-27 12:52:32.887 14100 ERROR oslo.service.loopingcall return f(self, *args, **kwargs)
2022-03-27 12:52:32.887 14100 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/designate/central/service.py", line 2962, in update_service_status
2022-03-27 12:52:32.887 14100 ERROR oslo.service.loopingcall db_status = self.storage.find_service_status(
2022-03-27 12:52:32.887 14100 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/designate/central/service.py", line 226, in storage
2022-03-27 12:52:32.887 14100 ERROR oslo.service.loopingcall self._storage = storage.get_storage(storage_driver)
2022-03-27 12:52:32.887 14100 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/designate/storage/__init__.py", line 36, in get_storage
2022-03-27 12:52:32.887 14100 ERROR oslo.service.loopingcall return cls()
2022-03-27 12:52:32.887 14100 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/designate/storage/impl_sqlalchemy/__init__.py", line 40, in __init__
2022-03-27 12:52:32.887 14100 ERROR oslo.service.loopingcall super(SQLAlchemyStorage, self).__init__()
2022-03-27 12:52:32.887 14100 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/designate/sqlalchemy/base.py", line 72, in __init__
2022-03-27 12:52:32.887 14100 ERROR oslo.service.loopingcall self.engine = session.get_engine(self.get_name())
2022-03-27 12:52:32.887 14100 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/designate/sqlalchemy/session.py", line 47, in get_engine
2022-03-27 12:52:32.887 14100 ERROR oslo.service.loopingcall facade = _create_facade_lazily(cfg_group)
2022-03-27 12:52:32.887 14100 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/designate/sqlalchemy/session.py", line 40, in _create_facade_lazily
2022-03-27 12:52:32.887 14100 ERROR oslo.service.loopingcall **conf
2022-03-27 12:52:32.887 14100 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1295, in __init__
2022-03-27 12:52:32.887 14100 ERROR oslo.service.loopingcall slave_connection=slave_connection)
2022-03-27 12:52:32.887 14100 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 510, in _start
2022-03-27 12:52:32.887 14100 ERROR oslo.service.loopingcall engine_args, maker_args)
2022-03-27 12:52:32.887 14100 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 532, in _setup_for_connection
2022-03-27 12:52:32.887 14100 ERROR oslo.service.loopingcall "No sql_connection parameter is established")
2022-03-27 12:52:32.887 14100 ERROR oslo.service.loopingcall oslo_db.exception.CantStartEngineError: No sql_connection parameter is established
2022-03-27 12:52:32.887 14100 ERROR oslo.service.loopingcall
Rabbit error:
2022-03-27 12:52:34.119 14301 ERROR oslo.messaging._drivers.impl_rabbit [req-535bcb89-9261-4f4d-b080-cfbbcc06de6c - - - - -] Connection failed: [Errno 111] ECONNREFUSED (retrying in 32.0 seconds): ConnectionRefusedError: [Errno 111] ECONNREFUSED
Testrun can be found at:
https://solutions.qa.canonical.com/testruns/testRun/3da7cd55-de25-4250-9158-1e3f8cb56741
Future occurrences of this bug can be found at:
https://solutions.qa.canonical.com/bugs/bugs/bug/1966779
