Key repository setup fails after simultaneously removing several ceph-mon units

Ceph-mon will reach a blocked state with a status of "Unit not clustered (no quorum)" if several clustered ceph-mon units are simultaneously removed while leaving enough units for quorum. This seems to be caused by the key repository setup failing in keystone.

This problem involves bad paths to keyrings on ceph-mon units. The expected keyring path is /var/lib/ceph/mon/ceph-<juju-machine>/keyring but the units are using /var/lib/ceph/mon/ceph-/keyring. An abbreviated portion of unit logs which shows this is included below.

WARNING osd-relation-changed 2021-03-26T23:33:15.200+0000 7f07e7300700 -1 auth: unable to find a keyring on /var/lib/ceph/mon/ceph-/keyring: (2) No such file or directory
WARNING osd-relation-changed 2021-03-26T23:33:15.200+0000 7f07e7300700 -1 AuthRegistry(0x7f07e0058960) no keyring found at /var/lib/ceph/mon/ceph-/keyring, disabling cephx
WARNING osd-relation-changed exported keyring for client.rgw.juju-81dd13-bug-hunt2-6

This bug is observed in a healthy test deployment which initially includes keystone, three ceph-mon units, among others. To reproduce this bug, four mon units are added then later removed after full deployment. The remaining ceph-mon units enter a blocked state after losing quorum.

On keystone, the following error is found:

INFO juju-log Checking no pids for apache2 exist
INFO juju-log Installing [] with options: ['--option=Dpkg::Options::=--force-confold']
ERROR juju-log Key repository setup failed, will retry in config-changed hook: Command '['sudo', '-u', 'keystone', 'keystone-manage', 'credential_migrate']' returned non-zero exit status 1.
WARNING config-changed ERROR no relation id specified

keystone version: 18.0.0
ceph-mon version: 15.2.8
Juju version: 2.8.10-focal-amd64
OpenStack version: openstack 5.2.0