Stop and delete default libvirt network after installation

FYI this triggered a service outage (swift-storage) from conntrack exhaustion
on a deployment where we aggregate nova-compute and swift-storage services
into same units, with (default) net.nf_conntrack_max = 65536.

@jjo

Would increasing the net.nf_conntrack_max configuration option make sense as a good practice default? I could see how that might get exhausted in a number of ways.

I guess we could also stop and disable the default libvirt network; that would be sensible option.

We could also delete the default libvirt network which should have the right result as well.

I hit this today on a production system. One of the tenant networks was using 192.168.0.0/17 and eventually started allocating from 192.168.122.0. All traffic from those IPs get masqueraded which breaks all tenant traffic and the metadata service.

The traffic makes it over the GRE tunnel due to the rule being POSTROUTING but the public IP of the compute node is on the packet, and the return traffic never gets back to the compute node.

Deleting the default network with virsh will immediately delete the MASQUERADE rules and prevent it occurring on reboot. I suggest that the charm checks for existence of /etc/libvirt/qemu/networks/default.xml and then executes virsh net-destroy default if it does.

See also related bug about having a default in libvirt at all: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1567466

Change submitted for review: https://review.openstack.org/340085

Reviewed: https://review.openstack.org/340085
Committed: https://git.openstack.org/cgit/openstack/charm-nova-compute/commit/?id=b533a3fbfb96bd5939b3e790f67964ce3ba8f7fd
Submitter: Jenkins
Branch: master

commit b533a3fbfb96bd5939b3e790f67964ce3ba8f7fd
Author: Trent Lloyd <email address hidden>
Date: Sun Jul 10 10:59:47 2016 +0800

    Delete libvirt-bin default network to avoid MASQUERADE rules

    libvirt-bin installs a 192.168.122.0/24 default network and creates
    MASQUERADE rules for it on boot. These rules will effect and break
    instance traffic including GRE tenant networks.

    Check if the network exists and then destroy it with virsh net-destroy
    which both immediately removes the MASQUERADE rules and the network so
    it is not applied after reboot.

    Change-Id: Ia79aea6ef889d1ef58f903f967bea37dc07fd160
    Closes-Bug: #1387390

Reopening, today we had a tenant outage because of this:
libvirt-bin package was (auto)upgraded on 2017-03-20,
re-instating the 192.168.122.0/24 network, interferring
with any tenant VM with a neutron port IP inside that CIDR.

Observed behavior FTR: packets leaving the neutron bridge
thru qvb interface getting MASQ'd by libvirt nat rules.

Details from the debugging session:
"af06e846-37" is the common id of the VMs ports, for iface purposes

1) OK: tap iface egress from the VM:
root@compute-031:~# tcpdump -n -i tapaf06e846-37 icmp
18:05:52.656501 IP 192.168.122.10 > 8.8.8.8: ICMP echo request, id 1, seq 387, length 40

2) OK: as seen at qbr neutron bridge
root@compute-031:~# tcpdump -n -i qbraf06e846-37 icmp
18:06:22.656489 IP 192.168.122.10 > 8.8.8.8: ICMP echo request, id 1, seq 393, length 40

3) BAD!: getting SNATted/MASQ-d when leaving the bridge:
root@compute-031:~# tcpdump -n -i qvbaf06e846-37 icmp
18:06:52.656471 IP 10.182.255.57 > 8.8.8.8: ICMP echo request, id 1, seq 399, length 40

Sticky workaround: after removing the network,
use dpkg-divert to void a default.xml re-placement:

juju run --application=nova-compute 'rm -f /etc/libvirt/qemu/networks/default.xml;
  virsh net-destroy default;
  dpkg-divert --package libvirt-bin --add --rename --divert /etc/libvirt/qemu/networks/default.xml{.disabled,}'

It looks like the original patch only does `virsh net-destroy default`. But no `virsh net-autostart --disable default`. That could be a reason to show up 192.168.122.0/24 again after daemon restart or machine reboot.

Yeah my original patch was faulty and only did net-destroy and not net-undefine.

I thought it had been fixed since but it turns out I started fixing it in my local branch and never pushed it up and I was looking at that code. I should probably look to submit that change for review.

I assume this is a conffile and dpkg shouldn't reinstall the file on upgrade?

On a xenial-ocata environment, a nova-compute unit shows:
"""
$ sudo iptables -S -t nat
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-A POSTROUTING -s 192.168.122.0/24 -d 224.0.0.0/24 -j RETURN
-A POSTROUTING -s 192.168.122.0/24 -d 255.255.255.255/32 -j RETURN
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE
-A POSTROUTING -s 10.0.121.0/24 ! -d 10.0.121.0/24 -m comment --comment "managed by lxd-bridge" -j MASQUERADE
"""

Pools are from virbr0 and lxdbr0.

Should we file a new bug to track the need of a new fix?

For 192.168.122.0/24 this is the same issue, I'll investigate that in this bug.

For the lxd-bridge 10.0.121.0/24 that may need a new bug but I'll take an initial look at that one.

Is there any update on this bug? Milestone seems it needs to be updated, and future outages could happen as they've occurred in the past.

The net-undefine was fixed in Bug #1800160 - so putting this bug back into fix released