Networkd fail to set ip address between leases if ip address changes on UbuntuCore

I spend a bit of time debugging this with xnox. Here some of the results:

# How to reproduce:
I setup a isc-dhcp-server with a fixed IP for my test rpi2 set to default/max lifetime of 120sec. The rpi2 was connected to the network with the isc-dhcp-server but unmodified otherwise. Once the pi2 connected and got an IP I switched the default IP to +1 and reloaded the isc-dhcp-server configuration. When the lease expired I could see the same as the OP - i.e. both IPs got removed.

# Related upstream bugs:
The patch from https://github.com/systemd/systemd/issues/5625 did not help with the issue

# Log
The full log of systemd-networkd is available here: http://paste.ubuntu.com/25686225/ - it was obtained by adding [Service]\nEnvironment=SYSTEMD_LOG_LEVEL=debug for the systemd-networkd.service.

The relevant lines:

Mar 14 21:02:52 localhost.localdomain systemd-networkd[7660]: eth0: Adding address: 192.168.1.124/24 (valid for 2min)
Mar 14 21:02:52 localhost.localdomain systemd-networkd[7660]: Sent message type=signal sender=n/a destination=n/a object=/org/freedesktop/network1/link/_33 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=16 reply_cookie=0 error=n/a
Mar 14 21:02:52 localhost.localdomain systemd-networkd[7660]: Sent message type=signal sender=n/a destination=n/a object=/org/freedesktop/network1 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=17 reply_cookie=0 error=n/a
Mar 14 21:02:52 localhost.localdomain systemd-timesyncd[995]: Network configuration changed, trying to establish connection.
Mar 14 21:02:52 localhost.localdomain systemd[1]: Starting Update resolvconf for networkd DNS...
Mar 14 21:02:56 localhost.localdomain systemd-networkd[7660]: NDISC: Sent Router Solicitation
Mar 14 21:03:00 localhost.localdomain systemd-networkd[7660]: NDISC: Sent Router Solicitation
Mar 14 21:03:04 localhost.localdomain systemd-networkd[7660]: NDISC: Invoking callback for 't'.
Mar 14 21:03:22 localhost.localdomain systemd[1]: Started Update resolvconf for networkd DNS.
Mar 14 21:03:51 localhost.localdomain systemd-networkd[7660]: DHCP CLIENT (0x22f0fd4b): REQUEST (renewing)
Mar 14 21:04:36 localhost.localdomain systemd-networkd[7660]: DHCP CLIENT (0x22f0fd4b): REQUEST (rebinding)
Mar 14 21:04:36 localhost.localdomain systemd-networkd[7660]: DHCP CLIENT (0x22f0fd4b): NAK: requested address is incorrect
Mar 14 21:04:36 localhost.localdomain systemd-networkd[7660]: DHCP CLIENT (0x4d15b90b): REBOOT in 0
Mar 14 21:04:36 localhost.localdomain systemd-networkd[7660]: DHCP CLIENT (0x4d15b90b): DISCOVER
Mar 14 21:04:36 localhost.localdomain systemd-networkd[7660]: DHCP CLIENT (0x4d15b90b): OFFER
Mar 14 21:04:36 localhost.localdomain systemd-networkd[7660]: DHCP CLIENT (0x4d15b90b): REQUEST (requesting)
Mar 14 21:04:36 localhost.localdomain systemd-networkd[7660]: DHCP CLIENT (0x4d15b90b): ACK
Mar 14 21:04:36 localhost.localdomain systemd-networkd[7660]: DHCP CLIENT (0x4d15b90b): lease expires in 1min 57s
Mar 14 21:04:36 localhost.localdomain systemd-networkd[7660]: DHCP CLIENT (0x4d15b90b): T2 expires in 1min 43s
Mar 14 21:04:36 localhost.localdomain systemd-networkd[7660]: DHCP CLIENT (0x4d15b90b): T1 expire...

Some more data point are the output of "ip addr": http://paste.ubuntu.com/25685527/

The interessting part if:
...
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether b8:27:eb:85:3f:94 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.116/24 brd 192.168.1.255 scope global dynamic eth0
       valid_lft 7sec preferred_lft 7sec
    inet 192.168.1.117/24 brd 192.168.1.255 scope global secondary dynamic eth0
       valid_lft 109sec preferred_lft 109sec
    inet6 fe80::ba27:ebff:fe85:3f94/64 scope link
       valid_lft forever preferred_lft forever
...
and then
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether b8:27:eb:85:3f:94 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::ba27:ebff:fe85:3f94/64 scope link
       valid_lft forever preferred_lft forever

And the output of "ip monitor eth0": http://paste.ubuntu.com/25686711/
...
2: eth0 inet 192.168.1.130/24 brd 192.168.1.255 scope global secondary dynamic eth0
       valid_lft 120sec preferred_lft 120sec
local 192.168.1.130 table local proto kernel scope host src 192.168.1.129

Deleted 2: eth0 inet 192.168.1.130/24 brd 192.168.1.255 scope global secondary dynamic eth0
       valid_lft 110sec preferred_lft 110sec
Deleted local 192.168.1.130 table local proto kernel scope host src 192.168.1.129
Deleted 2: eth0 inet 192.168.1.129/24 brd 192.168.1.255 scope global dynamic eth0
       valid_lft 0sec preferred_lft 0sec
Deleted 192.168.1.0/24 proto kernel scope link src 192.168.1.129
Deleted broadcast 192.168.1.255 table local proto kernel scope link src 192.168.1.129
Deleted broadcast 192.168.1.0 table local proto kernel scope link src 192.168.1.129
Deleted local 192.168.1.129 table local proto kernel scope host src 192.168.1.129
Deleted 192.168.1.114 lladdr 00:24:7e:11:b8:aa STALE

With ip monitor all: http://paste.ubuntu.com/25686763/

The output if dhclient is running with the same scenario: http://paste.ubuntu.com/25686791/

Fwiw, my current suspicion is that adding the new IP from the dhcp as a secondary IP is buggy. Secondary IPs will get removed when the kernel removes the primary IP (see kernels __inet_del_ifa() code).

As a workaround for now, it looks like with: "echo 1 > /proc/sys/net/ipv4/conf/all/promote_secondaries" on the device the problem goes away.

Or "net.ipv4.conf.all.promote_secondaries=1" in /etc/sysctl.d/50-ipv4.conf (or similar).

Xnox pointed out that the above net.ipv4.conf.all.promote_secondaries=1 is actually a default of systemd (added in https://github.com/systemd/systemd/commit/ad8bc9ea508740074cead005aa3cfd1ba10a5dac). This looks like we will enable it for everybody via a systemd SRU.

Setting snappy task to fix committed, core the ships the workaround from
https://github.com/snapcore/core/pull/60

...until systemd grows the correct defaults via an SRU

> This update also applies other safe-looking options, which are currently also already applied via sysctls shipped in other packages

SRUs are supposed to be minimal to minimise regression risk. Do you have an SRU justification for this set of changes, please? Are they related to this bug, and if not, shouldn't they be tracked in a separate bug?

I believe both promote_secondaries and default qdisc changes are good changes for our distribution to make.

I'd like to abstain from the judgment of making the change for existing releases via an SRU. (Be sure to make sure this change is obvious and easy to search for in package changelog please. Noting it in future release notes with statement that it has been rolled back to previous releases too would also be very kind.)

Thanks

What about server use cases that do not have networkd installed? There are complex network setups out there, such as HA with corosync/pacemaker, OpenStack Neutron, and that kind of thing. If this fix were SRU'd, will all of these things in the wild cope with this sysctl change?

Can networkd be made to work without relying on this sysctl tweak? If not, is it possible to refrain from tweaking this sysctl knob except for users affected by this bug (networkd users using DHCP)?

On 12 October 2017 at 17:31, Robie Basak <email address hidden> wrote:
> What about server use cases that do not have networkd installed? There
> are complex network setups out there, such as HA with
> corosync/pacemaker, OpenStack Neutron, and that kind of thing. If this
> fix were SRU'd, will all of these things in the wild cope with this
> sysctl change?
>
> Can networkd be made to work without relying on this sysctl tweak? If

No, as the alternative is what isc-dhcp /sbin/dhcp-client does that is
flush current dhcp lease IP and add a new IP, meaning that networking
is dropped =/
Which to me is horrifying.
Promotion of secondary addresses should actually reduce split-brain
situations. I don't buy the HA/Neutron argument, as in HA environment
the underlying nodes have static ip addresses and the management of
floating ip addresses is not managed by DHCP leases, given the DHCP
timings and fragility. Are you implying that HA and Neutron rely on
DHCP lease renew/rebind which can be initiated and spoofed by the
clients arbitrary?

> not, is it possible to refrain from tweaking this sysctl knob except for
> users affected by this bug (networkd users using DHCP)?

No, as that is racy.

Code wise it is a lot more risk-prone -> will require rewritting
networkd state machine.

Promoting secondary addresses has been around since 2.6.12 and I'm not
sure why that is not the default, given that if one wants to flush a
subset of ip addresses one can do so using the flush command. And
interent seems to be full of posts where people are surpised that
secondary ip addresses are removed when one removes the oldest one.
There is no kernel API to promote secondary ip address to primary; or
to have multiple primary ip addresses (such that removal / expiry of
one, doesn't affect others that happen to be from the same subnet) or
to explicitly add a secondary ip address.

--
Regards,

Dimitri.

On Fri, Oct 13, 2017 at 01:22:23AM -0000, Dimitri John Ledkov wrote:
> > Can networkd be made to work without relying on this sysctl tweak? If
>
> No, as the alternative is what isc-dhcp /sbin/dhcp-client does that is
> flush current dhcp lease IP and add a new IP, meaning that networking
> is dropped =/
> Which to me is horrifying.

This is not an SRU justification, however, as we have been doing this
for decades and it is not a problem in practice, nor a regression from
previous releases.

> Promotion of secondary addresses should actually reduce split-brain
> situations. I don't buy the HA/Neutron argument, as in HA environment
> the underlying nodes have static ip addresses and the management of
> floating ip addresses is not managed by DHCP leases, given the DHCP
> timings and fragility. Are you implying that HA and Neutron rely on
> DHCP lease renew/rebind which can be initiated and spoofed by the
> clients arbitrary?

I think you misunderstand my concern. I'm not talking about DHCP in my
HA use case. I'm talking about whether tweaking the particular sysctl
knob will cause any adverse affects in use cases unrelated to this bug.
All use cases that do unusual things with (layer 3) network
configuration need to be considered or eliminated.

I'm not claiming any particular regression. I'm trying to identify
unknown unknowns ("Regression Potential") here which might happen as a
consequence of tweaking a global sysctl value to change behaviour for
*all* existing users and corresponding use cases in a stable release.

Fwiw, I reported a RFE upstream to warn if networkd is used and promote_secondaries is unset (https://github.com/systemd/systemd/issues/7163)

This bug was fixed in the package systemd - 235-2ubuntu3

---------------
systemd (235-2ubuntu3) bionic; urgency=medium

  * Revert "Skip test-bpf in autopkgtest, currently is failing."
    This reverts commit 75cf986e450e062a3d5780d1976e9efef41e6c4c.
  * Fix test-bpf test case on ubuntu.
  * Skip rename tests in containers, crude fix for now.

 -- Dimitri John Ledkov <email address hidden> Mon, 13 Nov 2017 00:06:42 +0000

Fwiw, this is fixed upstream now with https://github.com/systemd/systemd/pull/7167

Hello Nicolino, or anyone else affected,

Accepted systemd into artful-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/234-2ubuntu12.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-artful to verification-done-artful. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-artful. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Nicolino, or anyone else affected,

Accepted systemd into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/229-4ubuntu21.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Starting with ephemeral lxd container, on a system with app/promote_secondaries set to 0:

root@systemd-sru-artful:~# grep -e 0 -e 1 /proc/sys/net/ipv4/conf/*/promote_secondaries
/proc/sys/net/ipv4/conf/all/promote_secondaries:0
/proc/sys/net/ipv4/conf/default/promote_secondaries:0
/proc/sys/net/ipv4/conf/eth0/promote_secondaries:0
/proc/sys/net/ipv4/conf/lo/promote_secondaries:0
root@systemd-sru-artful:~# dpkg-query -W systemd
systemd 234-2ubuntu12.1
root@systemd-sru-artful:~# grep DHCP= /run/systemd/network/10-netplan-eth0.network
DHCP=ipv4

And DHCP on eth0 interface.

Upgraded systemd to:
root@systemd-sru-artful:~# dpkg-query -W systemd
systemd 234-2ubuntu12.3

Observed that promote_secondaries, got enabled on networkd DHCP managed link:

root@systemd-sru-artful:~# grep -e 0 -e 1 /proc/sys/net/ipv4/conf/*/promote_secondaries
/proc/sys/net/ipv4/conf/all/promote_secondaries:0
/proc/sys/net/ipv4/conf/default/promote_secondaries:0
/proc/sys/net/ipv4/conf/eth0/promote_secondaries:1
/proc/sys/net/ipv4/conf/lo/promote_secondaries:0

root@systemd-sru-xenial:~# dpkg-query -W systemd
systemd 229-4ubuntu21.1

Installed netplan, executed ifupdown-upgrade, rebooted:

root@systemd-sru-xenial:~# grep -e 0 -e 1 /proc/sys/net/ipv4/conf/*/promote_secondaries
/proc/sys/net/ipv4/conf/all/promote_secondaries:0
/proc/sys/net/ipv4/conf/default/promote_secondaries:0
/proc/sys/net/ipv4/conf/eth0/promote_secondaries:0
/proc/sys/net/ipv4/conf/lo/promote_secondaries:0

Upgraded systemd to the one in proposed:

# dpkg-query -W systemd
systemd 229-4ubuntu21.2

# grep -e 0 -e 1 /proc/sys/net/ipv4/conf/*/promote_secondaries
/proc/sys/net/ipv4/conf/all/promote_secondaries:0
/proc/sys/net/ipv4/conf/default/promote_secondaries:0
/proc/sys/net/ipv4/conf/eth0/promote_secondaries:1
/proc/sys/net/ipv4/conf/lo/promote_secondaries:0

Observed change in promote_secondaries for eth0 device.

Note, when not migrating to netplan, promote_secondaries remains as 0, preserving the current xenial behaviour under ifupdown.

This bug was fixed in the package systemd - 234-2ubuntu12.3

---------------
systemd (234-2ubuntu12.3) artful; urgency=medium

  [ Dimitri John Ledkov ]
  * Fix test-functions failing with Ubuntu units. LP: #1750608
  * tests: switch to using ext4 by default, instead of ext3. LP: #1750608
  * Fix kdump service not starting, due to systemd not loading dropins.
    Cherrypick a fix from upstream. (LP: #1708409)
  * systemd-fsckd: Fix ADT tests to work on s390x too. (LP: #1736955)
  * netwokrd: add support for RequiredForOnline stanza. (LP: #1737570)
  * resolved.service: set DefaultDependencies=no (LP: #1734167)
  * systemd.postinst: enable persistent journal. (LP: #1618188)
  * core: add support for non-writable unified cgroup hierarchy for container support.
    Rebase and de-fuzz. (LP: #1734410)
  * Prevent MemoryDenyWriteExecution policy bypass, by disallowing pkey_mprotect when mprotect is disallowed.
    CVE-2017-15908 (LP: #1725348)
  * networkd: enable promote_secondaries on networkd managed dhcp links.
    This fixes failing to renew DHCP lease, on networkd managed devices.
    (LP: #1721223)

  [ Kleber Sacilotto de Souza ]
  * systemd-rfkill service times out when a new rfkill device is added
    - rfkill-fix-erroneous-behavior-when-polling-the-udev-.patch: Comparing
    udev_device_get_sysname(device) and sysname will always return true. We need to
    check the device received from udev monitor instead.
    - rfkill-fix-typo.patch: Fix typo in rfkill log message. (LP: #1734908)

 -- Dimitri John Ledkov <email address hidden> Tue, 20 Feb 2018 16:11:58 +0000

The verification of the Stable Release Update for systemd has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package systemd - 229-4ubuntu21.2

---------------
systemd (229-4ubuntu21.2) xenial; urgency=medium

  [ Dimitri John Ledkov ]
  * udev: Mark ndb devices as inactive until connected. (LP: #696435)
  * networkd: in dhcp, change UseMTU default to true, to accept DHCP provided MTU by default.
    (LP: #1717471)
  * sysctl: apply parameters in-order, instead of randomly. (LP: #1718444)
  * networkd: apply promote_secondaries, to make DHCP lease changes work.
    (LP: #1721223)
  * shutdown: sync filesystems, before going into a killing spree.
    (LP: #1722481)
  * sysctl: do not fail, when cannot apply sysctl changes due to read-only sysfs in containers.
    (LP: #1734409)
  * networkd,wait-online: add RequiredForOnline to mark mandatory/optional links for boot.
    (LP: #1737570)

  [ David Glasser ]
  * journald: don't reduce BurstRateLimit on low disk space (LP: #1732803)

 -- Dimitri John Ledkov <email address hidden> Wed, 21 Feb 2018 13:46:37 +0000

Want a seamless website to transfer money through mobile. You can also visit at: https://www.apponlinesolutions.com and get the resolution from our cash app technical number right away!