Backport netplan to xenial

With the backported nplan one test case fails:

======================================================================
FAIL: test_manual_addresses (__main__.TestNetworkd)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/autopkgtest.RBASGN/build.kj6/nplan-0.12~16.04/tests/integration.py", line 458, in test_manual_addresses
    'inet6 1234']) # old static IPv6
  File "/tmp/autopkgtest.RBASGN/build.kj6/nplan-0.12~16.04/tests/integration.py", line 303, in assert_iface_up
    self.assertNotRegex(out, r, out)
AssertionError: Regex matched: 'inet 192.168.5' matches 'inet 192.168.5'

I. e. after "netplan apply" the old DHCP address from the interface does not disappear when switching to a pure "static addresses" config. This works in yakkety, and I need to investigate this first (might need a backported fix to networkd).

We need to backport a networkd fix: https://github.com/systemd/systemd/commit/6fc2549711 . This will clean up old addresses from interfaces when restarting networkd, so that changed netplan configuration will actually reflect reality.

Hello Martin, or anyone else affected,

Accepted systemd into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/229-4ubuntu9 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Martin, or anyone else affected,

Accepted network-manager into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/network-manager/1.2.2-0ubuntu0.16.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Martin, or anyone else affected,

Accepted nplan into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nplan/0.12~16.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Martin, or anyone else affected,

Accepted network-manager into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/network-manager/1.2.2-0ubuntu0.16.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

http://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#network-manager and http://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#systemd tests look fine, but http://autopkgtest.ubuntu.com/packages/n/nplan/xenial/amd64 fails on some race condition. I'm investigating.

This isn't reproducible in yakkety, but it does reproduce locally in a QEMU xenial instance (although a lot harder than on the infra, which is why I didn't see it at first). It seems systemd 229's systemd-networkd-wait-online has a bug that it does not actually wait for the interfaces to be fully configured. When it exits, the status is

● 412: eth42
       Link File: /lib/systemd/network/99-default.link
    Network File: /run/systemd/network/10-netplan-eth42.network
            Type: ether
           State: degraded (configured)
          Driver: veth
      HW Address: 0a:9b:04:93:f9:c0
             MTU: 1500
         Address: fe80::89b:4ff:fe93:f9c0
             DNS: 2600:0000:0000:0000:0000:0000:0000:0001

i. e. the interface is still being set up. I would not like to work around that in the tests, because if we are actually going to use networkd on xenial the integration with network-online.target must work.

I understand this better now. networkd actually went to handling DHCPv6/RA in userspace (https://github.com/systemd/systemd/commit/3b015d40c19) so that it actually *can* wait for IPv6 RA addresses -- if the kernel handles them, it does not know what to expect.

In yakkety we let networkd do that, but in xenial we revert it (https://anonscm.debian.org/cgit/pkg-systemd/systemd.git/tree/debian/patches/Revert-Revert-networkd-ndisc-revert-to-letting-the-k.patch?h=ubuntu-xenial) as networkd v229 has some regressions wrt. IPv6 handling compared to the kernel. These are fixed in 231, so we could drop the Debian reversion patch and instead backport these fixes.

nplan and wait-online works fine with the patch dropped, for the record.

I just realized that the DHCPv6 failure that netplan's tests detect were already known in xenial's networkd tests; these tests got marked as "expected failure".

I started to backport individual networkd fixes, but this quickly became a frankensoftware which has never been tested in that form, and it would take prohibitively long to finish. Since 231, networkd development has slowed down considerably and it's by and large stable (judging by Debian bug reports a lot of people actually use it, and we did not get complaints since 231 any more).

So in summary I think it is better to backport networkd 231 wholesale. In the running system it is completely independent of systemd and other tools (standalone binary), it has good test coverage, has been tried and tested in yakkety (unlike xenial when we did not yet use/support networkd), so IMHO the risk of this is lower than spending days on reengineering fixes on top of 229.

The main noise of the backport is that this also requires backporting some common utility code.

Hello Martin, or anyone else affected,

Accepted systemd into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/229-4ubuntu11 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

systemd's and netplan's tests now work:
  http://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#systemd
  http://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#nplan

nplan's tests still show a race condition on ppc64el, but this isn't a regression.

network-manager's tests also all work:
  http://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#network-manager

I ran the above test cases on a 16.04.1 desktop (--print-config is identical, verifying existing connections still work), and NM still generally works.

This bug was fixed in the package network-manager - 1.2.2-0ubuntu0.16.04.3

---------------
network-manager (1.2.2-0ubuntu0.16.04.3) xenial; urgency=medium

  * debian/tests/wpa-dhclient: Don't assume that the IPv6 prefix length from
    the DHCP server is /64. (LP: #1609898)

network-manager (1.2.2-0ubuntu0.16.04.2) xenial; urgency=medium

  [ Martin Pitt ]
  * Read config and system connections from /run/NetworkManager/ to support
    netplan (LP: #1627641)
  * debian/gbp.conf: Set debian-branch to xenial

  [ Mathieu Trudel-Lapierre ]
  * Add dns-manager-don-t-merge-split-DNS-search-domains.patch: do not add
    split DNS search domains to resolv.conf; doing so would risk leaking names
    to non-VPN DNS nameservers when attempting to resolve non- FQDN names.
    (LP: #1592721)

 -- Martin Pitt <email address hidden> Tue, 27 Sep 2016 16:29:22 +0200

The verification of the Stable Release Update for network-manager has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package systemd - 229-4ubuntu11

---------------
systemd (229-4ubuntu11) xenial; urgency=medium

  * 73-usb-net-by-mac.rules: Split kernel command line import line.
    Reportedly this makes the rule actually work on some platforms. Thanks
    Alp Toker! (LP: #1593379)
  * fsckd: Do not exit on idle timeout if there are still clients connected
    (Closes: #788050, LP: #1547844)
  * libnss-*.prerm: Remove possible [key=value] options from NSS modules as
    well. (LP: #1625584)
  * Backport networkd 231. Compared to 229 this has a lot of fixes, some of
    which we need for good netplan support. Backporting them individually
    would be a lot more work and a lot less robust, and we did not use/support
    networkd in 16.04 so far. Drop the other network related patches as they
    are included in this backport now. (LP: #1627641)
  * debian/tests/networkd: Re-enable the the DHCPv6 tests. The DHCPv6
    behaviour is fixed with the above backport now.
  * pid1: process zero-length notification messages again. Just remove the
    assertion, the "n" value was not used anyway. This fixes a local DoS due
    to unprocessed/unclosed fds which got introduced by the previous fix.
    (LP: #1628687)
  * pid1: Robustify manager_dispatch_notify_fd(). If
    manager_dispatch_notify_fd() fails and returns an error then the handling
    of service notifications will be disabled entirely leading to a
    compromised system. (side issue of LP: #1628687)

 -- Martin Pitt <email address hidden> Tue, 04 Oct 2016 21:43:04 +0200

This bug was fixed in the package nplan - 0.12~16.04

---------------
nplan (0.12~16.04) xenial; urgency=medium

  [ Martin Pitt ]
  * Backport to xenial. (LP: #1627641)
  * Adjust Breaks: network-manager to version in xenial that provides the
    "read config from /run" functionality.
  * src/netplan: Add hack to support current NetworkManager snap, which
    currently cannot provide "nmcli" and "NetworkManager.service". Note that
    this is meant to be temporary until snapd gets fixed, and will NOT be
    applied to yakkety or upstream. Patch by Simon Fels.

 -- Martin Pitt <email address hidden> Mon, 26 Sep 2016 11:06:57 +0200