Comment 0 for bug 2036968

Following a recent change from linux-kvm kernel to linux-generic kernel in the mantic minimized images there is a reproducable bug where a guest VM does not have an IP address assigned as part of cloud-init provisioning.

This is easiest to reproduce when emulating arm64 on adm64 host. The bug is a race condition so there could exist fast enough virtualisation on fast enough hardware where this bug is not present but in all my testing I have been able to reproduce.

The latest mantic minimized images from http://cloud-images.ubuntu.com/minimal/daily/mantic/ have force initrdless boot and no initrd to fallback to.

This but is not present in the non minimized/base images @ http://cloud-images.ubuntu.com/mantic/ as these boot with initrd with the required drivers present for virtio-net.

Reproducer

```
wget -O "launch-qcow2-image-qemu-arm64.sh" https://people.canonical.com/~philroche/20230921-cloud-images-mantic-fail-to-provision/launch-qcow2-image-qemu-arm64.sh

chmod +x ./launch-qcow2-image-qemu-arm64.sh
wget https://people.canonical.com/~philroche/20230921-cloud-images-mantic-fail-to-provision/livecd.ubuntu-cpc.img
./launch-qcow2-image-qemu-arm64.sh --password passw0rd --image ./livecd.ubuntu-cpc.img
```

You will then be able to log in with user `ubuntu` and password `passw0rd`.

You can run `ip a` and see that there is a network interface present (separate to `lo`) but no IP address has been assigned.

```
ubuntu@cloudimg:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp0s1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff

```

This is because when cloud-init is trying to configure network interfaces it doesn't find any so it doesn't configure any. But by the time boot is complete the network interface is present but cloud-init provisioning has already completed.

You can verify this by running `sudo cloud-init clean && sudo cloud-init init`

You can then see a successfully configured network interface

```
ubuntu@cloudimg:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp0s1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 metric 100 brd 10.0.2.255 scope global dynamic enp0s1
       valid_lft 86391sec preferred_lft 86391sec
    inet6 fec0::5054:ff:fe12:3456/64 scope site dynamic mngtmpaddr noprefixroute
       valid_lft 86393sec preferred_lft 14393sec
    inet6 fe80::5054:ff:fe12:3456/64 scope link
       valid_lft forever preferred_lft forever

```

The bug is also reproducable with amd64 guest on adm64 host on older/slower hardware.

The suggested fixes while debugging this issue are:

* to include `virtio-net` as a built-in in the mantic generic kernel
* understand what needs to change in cloud-init so that it can react to late additions of network interfaces

I will file a separate bug against cloud-init to address the race condition on emulated guest/older hardware.