Cluster binding change results in config-changed hook error

Bug #2011609 reported by Márton Kiss
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
memcached-charm
Triaged
Low
Unassigned

Bug Description

This is slightly a corner case, however when the unit's cluster binding is getting updated into a different network space with the "juju bind" command, the config-changed hook is failing with the following stack trace:

unit-memcached-0: 14:26:42 DEBUG unit.memcached/0.config-changed active
unit-memcached-0: 14:26:42 WARNING unit.memcached/0.config-changed Traceback (most recent call last):
unit-memcached-0: 14:26:42 WARNING unit.memcached/0.config-changed File "/var/lib/juju/agents/unit-memcached-0/charm/hooks/config-changed", line 384, in <module>
unit-memcached-0: 14:26:42 WARNING unit.memcached/0.config-changed main()
unit-memcached-0: 14:26:42 WARNING unit.memcached/0.config-changed File "/var/lib/juju/agents/unit-memcached-0/charm/hooks/config-changed", line 378, in main
unit-memcached-0: 14:26:42 WARNING unit.memcached/0.config-changed hooks.execute(sys.argv)
unit-memcached-0: 14:26:42 WARNING unit.memcached/0.config-changed File "/var/lib/juju/agents/unit-memcached-0/charm/hooks/charmhelpers/core/hookenv.py", line 731, in execute
unit-memcached-0: 14:26:42 WARNING unit.memcached/0.config-changed self._hooks[hook_name]()
unit-memcached-0: 14:26:42 WARNING unit.memcached/0.config-changed File "/var/lib/juju/agents/unit-memcached-0/charm/hooks/charmhelpers/core/host.py", line 637, in wrapped_f
unit-memcached-0: 14:26:42 WARNING unit.memcached/0.config-changed restart_functions)
unit-memcached-0: 14:26:42 WARNING unit.memcached/0.config-changed File "/var/lib/juju/agents/unit-memcached-0/charm/hooks/charmhelpers/core/host.py", line 659, in restart_on_change_helper
unit-memcached-0: 14:26:42 WARNING unit.memcached/0.config-changed r = lambda_f()
unit-memcached-0: 14:26:42 WARNING unit.memcached/0.config-changed File "/var/lib/juju/agents/unit-memcached-0/charm/hooks/charmhelpers/core/host.py", line 636, in <lambda>
unit-memcached-0: 14:26:42 WARNING unit.memcached/0.config-changed (lambda: f(*args, **kwargs)), restart_map, stopstart,
unit-memcached-0: 14:26:42 WARNING unit.memcached/0.config-changed File "/var/lib/juju/agents/unit-memcached-0/charm/hooks/config-changed", line 274, in config_changed
unit-memcached-0: 14:26:42 WARNING unit.memcached/0.config-changed update_status()
unit-memcached-0: 14:26:42 WARNING unit.memcached/0.config-changed File "/var/lib/juju/agents/unit-memcached-0/charm/hooks/config-changed", line 352, in update_status
unit-memcached-0: 14:26:42 WARNING unit.memcached/0.config-changed if replication.get_current_replica():
unit-memcached-0: 14:26:42 WARNING unit.memcached/0.config-changed File "/var/lib/juju/agents/unit-memcached-0/charm/hooks/replication.py", line 61, in get_current_replica
unit-memcached-0: 14:26:42 WARNING unit.memcached/0.config-changed log("Current replica value: %s" % replica, level="INFO")
unit-memcached-0: 14:26:42 WARNING unit.memcached/0.config-changed TypeError: not all arguments converted during string formatting
unit-memcached-0: 14:26:42 ERROR juju.worker.uniter.operation hook "config-changed" (via explicit, bespoke hook script) failed: exit status

Memcached is keeping the cluster ip values from the former network space in the /var/run/repcached_replica file:
$ juju run -a memcached "cat /var/run/repcached_replica; echo"
- Stdout: |
    10.99.43.67,10.99.43.119
  UnitId: memcached/0
- Stdout: |
    10.99.43.67,10.99.43.119
  UnitId: memcached/1

and the charm cannot decide about the master and secondary roles, as the cluster ip address from the new network space, won't match the ips stored in repcached_replica:

def get_current_replica():
    replica = get_repcached_replica()

    if replica:
        master, secondary = replica
        ip_addr = get_relation_ip('cluster')
        if ip_addr == master:
            replica = secondary
        elif ip_addr == secondary:
            replica = master

Workaround: update the repcached_replica content on each memcached unit using the unit ip addresses from the new network space:

identify ip addresses of the cluster binding:
$ juju run -a memcached "network-get --primary-address cluster"
- Stdout: |
    10.99.40.102
  UnitId: memcached/0
- Stdout: |
    10.99.40.156
  UnitId: memcached/1

upgrade the content of the /var/run/repcached_replica
$ juju run -a memcached 'echo -n "10.99.40.102,10.99.40.156" | tee /var/run/repcached_replica'

Resolve the units:
$ juju resolve memcached/0
$ juju resolve memcached/1

Eric Chen (eric-chen)
Changed in charm-memcached:
importance: Undecided → Low
status: New → Triaged
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.