swift proxy has inconsistent ring builder across units

Using stable charms and this procedure:

$ juju deploy --to 1 --repository . --config=config.yaml local:trusty/swift-storage swift-zone1
$ juju deploy --to 2 --repository . --config=config.yaml local:trusty/swift-storage swift-zone2
$ juju deploy --to 3 --repository . --config=config.yaml local:trusty/swift-storage swift-zone3
$ juju deploy --to lxc:1 --repository . --config=config.yaml local:trusty/swift-proxy
$ juju add-unit --to lxc:2 swift-proxy
$ juju add-unit --to lxc:3 swift-proxy
$ juju add-relation swift-proxy swift-hacluster
$ juju add-relation swift-proxy swift-zone1
$ juju add-relation swift-proxy swift-zone2
$ juju add-relation swift-proxy swift-zone3
$ juju add-relation swift-proxy glance
$ juju add-relation swift-proxy keystone

Behavior is inconsistent; most of the time it works, but sometimes we have inconsistency across proxies. This is relevant config from yaml:

swift-proxy:
  auth-type: 'keystone'
  use-https: 'False'
  vip: '192.168.0.10'
  vip_iface: eth0
  zone-assignment: 'manual'
  swift-hash: 'random_hash'
swift-zone1:
   block-device: '/dev/sdc /dev/sde /dev/sdg /dev/sdi /dev/sdk'
   overwrite: 'true'
   zone: 1
swift-zone2:
   block-device: '/dev/sdc /dev/sde /dev/sdg /dev/sdi /dev/sdk'
   overwrite: 'true'
   zone: 2
swift-zone3:
   block-device: '/dev/sdc /dev/sde /dev/sdg /dev/sdi /dev/sdk'
   overwrite: 'true'
   zone: 3

We've also tried with manual zone assignment with the same result. Always only one of swift-proxies is able to actually communicate with storage zones.

Ante, quick thought, how many of block-device: '/dev/sdc /dev/sde /dev/sdg /dev/sdi /dev/sdk' actually exist? Is each storage host equal in what devices it has? And is it possible that devices were added and/or removed from any of the storage? Just trying to narrow down the scenario a bit here. This really smells like a case of storage nodes changing (e.g. new devices coming online) but the update not reaching all proxy nodes (so clearly some kind of race condition). We do definitely have a problem with the way we manage .builder files on proxy nodes and need to find a safer way to do it.

All the nodes are exactly the same - same number of disks, same disk sizes, same disk model.

Possibly related to https://bugs.launchpad.net/charms/+source/swift-proxy/+bug/1337765

Still investigating this. I have managed to reproduce inconsistent .builder files and it looks like even if swift is given the same parameters when building .builder files on multiple proxies, the builder files will not necessarily be the same. So either this is a bug in swift or we just need to build on one proxy only and share across units. Still digging...