Certain functions run more than once when upgrading the keystone charm

When upgrading the keystone charm, certain charm functions run more than once. In particular the update_all_identity_relation_units function runs 3 times (twice during the upgrade-charm hook and once in the config-changed hook).

This update_all_identity_relation_units function can take a long time to run in larger environments (see https://bugs.launchpad.net/charm-keystone/+bug/1782004 where the bug reporter says update_all_identity_relation_units can take 15-30 minutes to run).

The aim of this bug is to see if there's a way we can reduce the number of times certain functions run (specifically update_all_identity_relation_units) during the upgrade-charm hook and the subsequent config-changed hook that gets called following the upgrade-charm hook.

I added debug status_set messages after each function in keystone_hooks.py (https://pastebin.ubuntu.com/p/KMTcVpdbGb/) to show the issue. Here is what it looks like:

$ juju upgrade-charm --path ~/keystone-debugs/charm-keystone keystone

$ juju show-status-log -n 100 keystone/0

18 Apr 2019 14:19:58Z juju-unit executing running upgrade-charm hook
18 Apr 2019 14:20:01Z workload maintenance DEBUG Running main
18 Apr 2019 14:20:04Z workload maintenance DEBUG Running upgrade_charm
18 Apr 2019 14:20:04Z workload maintenance Installing apt packages
18 Apr 2019 14:20:14Z workload maintenance DEBUG Running leader_init_db_if_ready
18 Apr 2019 14:20:14Z workload maintenance DEBUG Running update_all_identity_relation_units
18 Apr 2019 14:20:29Z workload maintenance DEBUG Running identity_changed
18 Apr 2019 14:20:34Z workload maintenance DEBUG Running update_nrpe_config
18 Apr 2019 14:20:35Z workload maintenance DEBUG Running update_all_identity_relation_units
18 Apr 2019 14:20:49Z workload maintenance DEBUG Running identity_changed
18 Apr 2019 14:20:55Z workload active Unit is ready
18 Apr 2019 14:20:57Z juju-unit executing running commands
18 Apr 2019 14:21:02Z juju-unit executing running config-changed hook
18 Apr 2019 14:21:04Z workload maintenance DEBUG Running main
18 Apr 2019 14:21:07Z workload maintenance DEBUG Running config_changed
18 Apr 2019 14:21:08Z workload maintenance DEBUG Running cluster_joined
18 Apr 2019 14:21:09Z workload maintenance DEBUG Running config_changed_postupgrade
18 Apr 2019 14:21:16Z workload maintenance DEBUG Running configure_https
18 Apr 2019 14:21:22Z workload maintenance DEBUG Running update_nrpe_config
18 Apr 2019 14:21:35Z workload maintenance DEBUG Running update_all_identity_relation_units
18 Apr 2019 14:21:50Z workload maintenance DEBUG Running identity_changed
18 Apr 2019 14:21:55Z workload maintenance DEBUG Running update_all_domain_backends
18 Apr 2019 14:21:55Z workload maintenance DEBUG Running update_all_fid_backends
18 Apr 2019 14:21:55Z workload maintenance DEBUG Running notify_middleware_with_release_version
18 Apr 2019 14:21:57Z workload active Unit is ready
18 Apr 2019 14:21:58Z juju-unit idle

I use a freshly cloned version of https://github.com/openstack/charm-keystone with my added debugs in each function. I was also able to reproduce this behaviour by going from the keystone charm version 288 to 296. I'm running juju version 2.5.4-bionic-amd64. Attaching juju-crashdump tarball as well (juju-crashdump-c3165783-c53c-4c6f-983a-d3e41122c9df.tar.xz).