Clear configured and clustered flags for removed instances

Secondary issue:

It turned out clearing the flag turned out to be more difficult than anticipated. After much trial and error figured out that the issue is leader set with a key with dots in it.

You should be able to do the following:

# leader-set key=value
# leader-get
key=value

# leader-set key=
# leader-get
{}

However when a dot is in the key name this does not work:

# leader-set cluster-instance-clustered-10.5.0.10=
# leader-set key.with.dots="True"
# leader-get
cluster-instance-clustered-10.5.0.10: "True"
key.with.dots: "True"

# leader-set key.with.dots=
# leader-get
cluster-instance-clustered-10.5.0.10: "True"
cluster-with.dots: "True"

TRIAGE:

Change flags from IP address to IP separated by dashes:
       leadership.leader_set({
            "cluster-instance-clustered-{}"
            .format(self.cluster_address.replace(".", "-")): True})

The next step is to translate this into the leader layer settings.

The remove-instance action should clear flags and bonus points for having the leader remove instance when in a departing hook.

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-mysql-innodb-cluster/+/786514

Reviewed: https://review.opendev.org/c/openstack/charm-mysql-innodb-cluster/+/786514
Committed: https://opendev.org/openstack/charm-mysql-innodb-cluster/commit/f22ca3b5b4dde7f92edb3a9b1e17835555590d1a
Submitter: "Zuul (22348)"
Branch: master

commit f22ca3b5b4dde7f92edb3a9b1e17835555590d1a
Author: David Ames <email address hidden>
Date: Wed Apr 14 11:42:18 2021 -0700

    Remove instance flags when instance removed

    Previously when an instance was removed the leadership settings and
    charms.reactive flags remained for that instance's IP address. If a new
    instance was subsequently added and happened to have the same IP address
    the charm would never add the new instance to the cluster because it
    believed the instance was already configured and clustered based on
    leader settings.

    Clear leader settings flags for instance cluster configured and
    clustered.

    Due to a bug in Juju the previous use of IP addresses with '.' were
    unable to be unset. Transform dotted flags to use '-' instead.

    func-test-pr: https://github.com/openstack-charmers/zaza-openstack-tests/pull/565

    Change-Id: If3ffa9e9191c057ac7e3d96bfcf84d8a3a2ad45a
    Closes-Bug: #1922394
    Related-Bug: #1889792

Fix proposed to branch: stable/21.04
Review: https://review.opendev.org/c/openstack/charm-mysql-innodb-cluster/+/799953

Reviewed: https://review.opendev.org/c/openstack/charm-mysql-innodb-cluster/+/799953
Committed: https://opendev.org/openstack/charm-mysql-innodb-cluster/commit/6d9a50774c4a1c7875bd2d90fe66b5cb5dafe63a
Submitter: "Zuul (22348)"
Branch: stable/21.04

commit 6d9a50774c4a1c7875bd2d90fe66b5cb5dafe63a
Author: David Ames <email address hidden>
Date: Wed Apr 14 11:42:18 2021 -0700

    Remove instance flags when instance removed

    Previously when an instance was removed the leadership settings and
    charms.reactive flags remained for that instance's IP address. If a new
    instance was subsequently added and happened to have the same IP address
    the charm would never add the new instance to the cluster because it
    believed the instance was already configured and clustered based on
    leader settings.

    Clear leader settings flags for instance cluster configured and
    clustered.

    Due to a bug in Juju the previous use of IP addresses with '.' were
    unable to be unset. Transform dotted flags to use '-' instead.

    Change-Id: If3ffa9e9191c057ac7e3d96bfcf84d8a3a2ad45a
    Closes-Bug: #1922394
    Related-Bug: #1889792
    (cherry picked from commit f22ca3b5b4dde7f92edb3a9b1e17835555590d1a)