While testing the quincy/edge/chrome0 channel, we noticed a similar issue except that the failed hook is "radosgw-dashboard-relation-changed" (we can open a new bug if you prefer). In the logs we see a couple of ceph commands failing with exist status 5, ending with: ============== Traceback (most recent call last): File "./src/charm.py", line 378, in _run_cmd output = subprocess.check_output(cmd, stderr=subprocess.STDOUT) File "/usr/lib/python3.8/subprocess.py", line 415, in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, File "/usr/lib/python3.8/subprocess.py", line 516, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['ceph', 'dashboard', 'set-audit-api-log-payload', 'True']' returned non-zero exit status 5. 2023-06-10 02:37:29 ERROR unit.ceph-dashboard/2.juju-log server.go:316 dashboard:103: Command failed: b"Error EIO: Module 'dashboard' has experienced an error and cannot handle commands: [('x509 certificate routines', 'X509_check_private_key', 'key values mismatch')]\n" Traceback (most recent call last): File "./src/charm.py", line 378, in _run_cmd output = subprocess.check_output(cmd, stderr=subprocess.STDOUT) File "/usr/lib/python3.8/subprocess.py", line 415, in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, File "/usr/lib/python3.8/subprocess.py", line 516, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['ceph', 'dashboard', 'motd', 'clear']' returned non-zero exit status 5. 2023-06-10 02:37:29 DEBUG unit.ceph-dashboard/2.juju-log server.go:316 dashboard:103: Attempting to collect TLS config from relation 2023-06-10 02:37:29 DEBUG unit.ceph-dashboard/2.dashboard-relation-changed logger.go:60 Updating certificates in /etc/ssl/certs... 2023-06-10 02:37:30 DEBUG unit.ceph-dashboard/2.dashboard-relation-changed logger.go:60 0 added, 0 removed; done. 2023-06-10 02:37:30 DEBUG unit.ceph-dashboard/2.dashboard-relation-changed logger.go:60 Running hooks in /etc/ca-certificates/update.d... 2023-06-10 02:37:30 DEBUG unit.ceph-dashboard/2.dashboard-relation-changed logger.go:60 done. 2023-06-10 02:37:30 DEBUG unit.ceph-dashboard/2.juju-log server.go:316 dashboard:103: ['ceph', 'dashboard', 'set-ssl-certificate', 'juju-f98180-0-lxd-1', '-i', PosixPath('/etc/ceph/ceph-dashboard.crt')] 2023-06-10 02:37:31 WARNING unit.ceph-dashboard/2.dashboard-relation-changed logger.go:60 Error EIO: Module 'dashboard' has experienced an error and cannot handle commands: [('x509 certificate routines', 'X509_check_private_key', 'key values mismatch')] 2023-06-10 02:37:31 ERROR unit.ceph-dashboard/2.juju-log server.go:316 dashboard:103: Uncaught exception while in charm code: Traceback (most recent call last): File "./src/charm.py", line 632, in main(CephDashboardCharm) File "/var/lib/juju/agents/unit-ceph-dashboard-2/charm/venv/ops/main.py", line 431, in main _emit_charm_event(charm, dispatcher.event_name) File "/var/lib/juju/agents/unit-ceph-dashboard-2/charm/venv/ops/main.py", line 142, in _emit_charm_event event_to_emit.emit(*args, **kwargs) File "/var/lib/juju/agents/unit-ceph-dashboard-2/charm/venv/ops/framework.py", line 316, in emit framework._emit(event) File "/var/lib/juju/agents/unit-ceph-dashboard-2/charm/venv/ops/framework.py", line 784, in _emit self._reemit(event_path) File "/var/lib/juju/agents/unit-ceph-dashboard-2/charm/venv/ops/framework.py", line 857, in _reemit custom_handler(event) File "/var/lib/juju/agents/unit-ceph-dashboard-2/charm/src/interface_dashboard.py", line 50, in on_changed self.on.mon_ready.emit() File "/var/lib/juju/agents/unit-ceph-dashboard-2/charm/venv/ops/framework.py", line 316, in emit framework._emit(event) File "/var/lib/juju/agents/unit-ceph-dashboard-2/charm/venv/ops/framework.py", line 784, in _emit self._reemit(event_path) File "/var/lib/juju/agents/unit-ceph-dashboard-2/charm/venv/ops/framework.py", line 857, in _reemit custom_handler(event) File "./src/charm.py", line 432, in _configure_dashboard self._configure_tls() File "./src/charm.py", line 557, in _configure_tls ceph_utils.dashboard_set_ssl_certificate( File "/var/lib/juju/agents/unit-ceph-dashboard-2/charm/venv/charms_ceph/utils.py", line 3599, in _dashboard_set_ssl_artifact subprocess.check_call(cmd) File "/usr/lib/python3.8/subprocess.py", line 364, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['ceph', 'dashboard', 'set-ssl-certificate', 'juju-f98180-0-lxd-1', '-i', PosixPath('/etc/ceph/ceph-dashboard.crt')]' returned non-zero exit status 5. ============== `ceph dashboard debug disable` is one of them, just like the original hook failure. Crashdumps and configs can be found here: https://oil-jenkins.canonical.com/artifacts/db5586e7-b34e-4e8f-ab6a-09f6a98070f5/index.html