Juju doesn't disable dhclient in KVMs

It seems that just doing "ifdown" isn't sufficient to stop dhclient. We may need to add a "dhclient -r" just before we do ifdown. I'm pretty sure the ifup will start it again if we stay in DHCP mode. (needs investigation to be sure of that.)

Note bugs like https://bugs.launchpad.net/ubuntu/+source/dhcp3/+bug/38140 that has been open for a long time.

So, trying to dig into this, I'm not sure that 'disable dhclient' is actually the bug. It might still be, and it might only show up with multiple network interfaces, or some particular configuration. The way Juju sets up network interfaces for KVM containers, is that it lets cloud-init do its standard process to start (where it sets up a /etc/network/interfaces that sources /etc/network/interfaces.d/50-cloud-init.conf)

It then asks Cloud init to write a concrete file name /etc/network/interfaces-juju which contains our expected configuration for the machine. (static addresses, routes once bug #1653708 is fixed, etc.) And then as part of the "run-cmd" portion of cloud-init it asks to do:

if [ -f "/etc/network/interfaces-juju" ]; then
    ifdown -a
    sleep 1.5
    if ifup -a --interfaces="/etc/network/interfaces-juju"; then
        cp /etc/network/interfaces /etc/network/interfaces-orig
        cp /etc/network/interfaces-juju /etc/network/interfaces
    else
        ifup -a
    fi
fi

So it tries to do 'ifdown -a' which should stop dhclient and release the leases. It then tries to use the "-juju" configuration, and if that succeeds, it then saves a backup, and puts the juju configuration into place. If it fails, then it restores networking using the default configuration.

And I can see the lease being released in cloud-init-output.log, there are these lines:
Cloud-init v. 0.7.5 running 'modules:final' at Sun, 26 Feb 2017 08:24:43 +0000. Up 26.78 seconds.
Internet Systems Consortium DHCP Client 4.2.4
Copyright 2004-2012 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/eth0/00:16:3e:c9:0b:5e
Sending on LPF/eth0/00:16:3e:c9:0b:5e
Sending on Socket/fallback
DHCPRELEASE on eth0 to 10.0.0.1 port 67 (xid=0x29fea88c)

On Xenial VMs, with 2.1.0, we run into a problem where the /e/n/interfaces-juju has hard-coded "eth0" as the name of the device, but on Xenial it is known as, eg, ens3. So we fail to bring up /e/n/i-juju and thus revert back to the normal /e/n/i.

Ideally, we would
a) Have a smarter script that could identify the device, say by its MAC address
b) Was run much earlier, because by the time we get here, we've already installed packages for cloud-init and for Juju dependencies. Which means we've already needed to have IP addresses and routes to our APT repository.

I started looking at this, but I'm unable to reproduce it. I'll need to work directly with Ante to understand what he's seeing before I can make concrete progress on it. (I was going to give it to Horacio, but he's off until Wed anyway.)

PR against develop (2.2): https://github.com/juju/juju/pull/7102

https://github.com/juju/juju/pull/7173

https://github.com/juju/juju/pull/7200