trusty: snapd deb package does not start systemd

Bug #1863816 reported by Chad Smith
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
snapd
Triaged
Medium
Michael Vogt

Bug Description

Installation snapd package on hwe kernel in Azure trusty Canonical Ubuntu vms does not start systemd. Likely some permissions errors. Seeing the following error messages (but no non-zero exit code). And systemd is service not started.

"""
All runlevel operations denied by policy
invoke-rc.d: policy-rc.d denied execution of start.
...
All runlevel operations denied by policy
invoke-rc.d: policy-rc.d denied execution of start.
All runlevel operations denied by policy
...
"""

$ uname -r
4.4.0-173-generic

$ sudo apt-get install --assume-yes snapd
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
  amd64-microcode cgroup-lite crda intel-microcode iucode-tool iw liblzo2-2
  libnl-3-200 libnl-genl-3-200 libseccomp2 libsystemd-journal0 linux-firmware
  linux-generic-lts-xenial linux-headers-4.4.0-174
  linux-headers-4.4.0-174-generic linux-headers-generic-lts-xenial
  linux-image-4.4.0-174-generic linux-image-generic-lts-xenial
  linux-modules-4.4.0-174-generic linux-modules-extra-4.4.0-174-generic
  squashfs-tools systemd thermald wireless-regdb
Suggested packages:
  fdutils linux-lts-xenial-doc-4.4.0 linux-lts-xenial-source-4.4.0
  linux-lts-xenial-tools systemd-ui
The following NEW packages will be installed:
  amd64-microcode cgroup-lite crda intel-microcode iucode-tool iw liblzo2-2
  libnl-3-200 libnl-genl-3-200 libseccomp2 libsystemd-journal0 linux-firmware
  linux-generic-lts-xenial linux-headers-4.4.0-174
  linux-headers-4.4.0-174-generic linux-headers-generic-lts-xenial
  linux-image-4.4.0-174-generic linux-image-generic-lts-xenial
  linux-modules-4.4.0-174-generic linux-modules-extra-4.4.0-174-generic snapd
  squashfs-tools systemd thermald wireless-regdb
0 upgraded, 25 newly installed, 0 to remove and 8 not upgraded.
Need to get 117 MB of archives.
After this operation, 514 MB of additional disk space will be used.
Get:1 http://azure.archive.ubuntu.com/ubuntu/ trusty-updates/main liblzo2-2 amd64 2.06-1.2ubuntu1.1 [46.1 kB]
Get:2 http://azure.archive.ubuntu.com/ubuntu/ trusty-updates/main libnl-3-200 amd64 3.2.21-1ubuntu4.1 [45.3 kB]
Get:3 http://azure.archive.ubuntu.com/ubuntu/ trusty-updates/main libnl-genl-3-200 amd64 3.2.21-1ubuntu4.1 [10.2 kB]
Get:4 http://azure.archive.ubuntu.com/ubuntu/ trusty-updates/main libsystemd-journal0 amd64 204-5ubuntu20.31 [50.5 kB]
Get:5 http://azure.archive.ubuntu.com/ubuntu/ trusty/multiverse iucode-tool amd64 1.0.1-1 [28.1 kB]
Get:6 http://azure.archive.ubuntu.com/ubuntu/ trusty/main wireless-regdb all 2013.02.13-1ubuntu1 [6,456 B]
Get:7 http://azure.archive.ubuntu.com/ubuntu/ trusty/main crda amd64 1.1.2-1ubuntu2 [15.2 kB]
Get:8 http://azure.archive.ubuntu.com/ubuntu/ trusty/main iw amd64 3.4-1 [51.7 kB]
Get:9 http://azure.archive.ubuntu.com/ubuntu/ trusty-updates/main linux-firmware all 1.127.24 [33.9 MB]
Get:10 https://esm.ubuntu.com/ubuntu/ trusty-infra-security/main libseccomp2 amd64 2.4.1-0ubuntu0.14.04.2 [39.6 kB]
Get:11 https://esm.ubuntu.com/ubuntu/ trusty-infra-security/main intel-microcode amd64 3.20191115.1ubuntu0.14.04.2 [2,428 kB]
Get:12 https://esm.ubuntu.com/ubuntu/ trusty-infra-security/main linux-modules-4.4.0-174-generic amd64 4.4.0-174.204~14.04.1 [11.2 MB]
Get:13 http://azure.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64-microcode amd64 3.20180524.1~ubuntu0.14.04.2+really20130710.1ubuntu1 [26.3 kB]
Get:14 http://azure.archive.ubuntu.com/ubuntu/ trusty/main cgroup-lite all 1.9 [3,918 B]
Get:15 http://azure.archive.ubuntu.com/ubuntu/ trusty-updates/main squashfs-tools amd64 1:4.2+20130409-2ubuntu0.14.04.3 [90.5 kB]
Get:16 http://azure.archive.ubuntu.com/ubuntu/ trusty-updates/main systemd amd64 204-5ubuntu20.31 [1,426 kB]
Get:17 http://azure.archive.ubuntu.com/ubuntu/ trusty-updates/main snapd amd64 2.38~14.04 [13.9 MB]
Get:18 http://azure.archive.ubuntu.com/ubuntu/ trusty-updates/main thermald amd64 1.4.3-5~14.04.4 [202 kB]
Get:19 https://esm.ubuntu.com/ubuntu/ trusty-infra-security/main linux-image-4.4.0-174-generic amd64 4.4.0-174.204~14.04.1 [6,846 kB]
Get:20 https://esm.ubuntu.com/ubuntu/ trusty-infra-security/main linux-modules-extra-4.4.0-174-generic amd64 4.4.0-174.204~14.04.1 [36.2 MB]
Get:21 https://esm.ubuntu.com/ubuntu/ trusty-infra-security/main linux-image-generic-lts-xenial amd64 4.4.0.174.153 [2,502 B]
Get:22 https://esm.ubuntu.com/ubuntu/ trusty-infra-security/main linux-headers-4.4.0-174 all 4.4.0-174.204~14.04.1 [10.1 MB]
Get:23 https://esm.ubuntu.com/ubuntu/ trusty-infra-security/main linux-headers-4.4.0-174-generic amd64 4.4.0-174.204~14.04.1 [811 kB]
Get:24 https://esm.ubuntu.com/ubuntu/ trusty-infra-security/main linux-headers-generic-lts-xenial amd64 4.4.0.174.153 [2,332 B]
Get:25 https://esm.ubuntu.com/ubuntu/ trusty-infra-security/main linux-generic-lts-xenial amd64 4.4.0.174.153 [1,802 B]
Fetched 117 MB in 10s (11.2 MB/s)
Selecting previously unselected package liblzo2-2:amd64.
(Reading database ... 29110 files and directories currently installed.)
Preparing to unpack .../liblzo2-2_2.06-1.2ubuntu1.1_amd64.deb ...
Unpacking liblzo2-2:amd64 (2.06-1.2ubuntu1.1) ...
Selecting previously unselected package libnl-3-200:amd64.
Preparing to unpack .../libnl-3-200_3.2.21-1ubuntu4.1_amd64.deb ...
Unpacking libnl-3-200:amd64 (3.2.21-1ubuntu4.1) ...
Selecting previously unselected package libnl-genl-3-200:amd64.
Preparing to unpack .../libnl-genl-3-200_3.2.21-1ubuntu4.1_amd64.deb ...
Unpacking libnl-genl-3-200:amd64 (3.2.21-1ubuntu4.1) ...
Selecting previously unselected package libseccomp2:amd64.
Preparing to unpack .../libseccomp2_2.4.1-0ubuntu0.14.04.2_amd64.deb ...
Unpacking libseccomp2:amd64 (2.4.1-0ubuntu0.14.04.2) ...
Selecting previously unselected package libsystemd-journal0:amd64.
Preparing to unpack .../libsystemd-journal0_204-5ubuntu20.31_amd64.deb ...
Unpacking libsystemd-journal0:amd64 (204-5ubuntu20.31) ...
Selecting previously unselected package iucode-tool.
Preparing to unpack .../iucode-tool_1.0.1-1_amd64.deb ...
Unpacking iucode-tool (1.0.1-1) ...
Selecting previously unselected package intel-microcode.
Preparing to unpack .../intel-microcode_3.20191115.1ubuntu0.14.04.2_amd64.deb ...
Unpacking intel-microcode (3.20191115.1ubuntu0.14.04.2) ...
Selecting previously unselected package wireless-regdb.
Preparing to unpack .../wireless-regdb_2013.02.13-1ubuntu1_all.deb ...
Unpacking wireless-regdb (2013.02.13-1ubuntu1) ...
Selecting previously unselected package crda.
Preparing to unpack .../crda_1.1.2-1ubuntu2_amd64.deb ...
Unpacking crda (1.1.2-1ubuntu2) ...
Selecting previously unselected package iw.
Preparing to unpack .../archives/iw_3.4-1_amd64.deb ...
Unpacking iw (3.4-1) ...
Selecting previously unselected package linux-firmware.
Preparing to unpack .../linux-firmware_1.127.24_all.deb ...
Unpacking linux-firmware (1.127.24) ...
Selecting previously unselected package linux-modules-4.4.0-174-generic.
Preparing to unpack .../linux-modules-4.4.0-174-generic_4.4.0-174.204~14.04.1_amd64.deb ...
Unpacking linux-modules-4.4.0-174-generic (4.4.0-174.204~14.04.1) ...
Selecting previously unselected package linux-image-4.4.0-174-generic.
Preparing to unpack .../linux-image-4.4.0-174-generic_4.4.0-174.204~14.04.1_amd64.deb ...
Unpacking linux-image-4.4.0-174-generic (4.4.0-174.204~14.04.1) ...
Selecting previously unselected package linux-modules-extra-4.4.0-174-generic.
Preparing to unpack .../linux-modules-extra-4.4.0-174-generic_4.4.0-174.204~14.04.1_amd64.deb ...
Unpacking linux-modules-extra-4.4.0-174-generic (4.4.0-174.204~14.04.1) ...
Selecting previously unselected package amd64-microcode.
Preparing to unpack .../amd64-microcode_3.20180524.1~ubuntu0.14.04.2+really20130710.1ubuntu1_amd64.deb ...
Unpacking amd64-microcode (3.20180524.1~ubuntu0.14.04.2+really20130710.1ubuntu1) ...
Selecting previously unselected package linux-image-generic-lts-xenial.
Preparing to unpack .../linux-image-generic-lts-xenial_4.4.0.174.153_amd64.deb ...
Unpacking linux-image-generic-lts-xenial (4.4.0.174.153) ...
Selecting previously unselected package linux-headers-4.4.0-174.
Preparing to unpack .../linux-headers-4.4.0-174_4.4.0-174.204~14.04.1_all.deb ...
Unpacking linux-headers-4.4.0-174 (4.4.0-174.204~14.04.1) ...
Selecting previously unselected package linux-headers-4.4.0-174-generic.
Preparing to unpack .../linux-headers-4.4.0-174-generic_4.4.0-174.204~14.04.1_amd64.deb ...
Unpacking linux-headers-4.4.0-174-generic (4.4.0-174.204~14.04.1) ...
Selecting previously unselected package linux-headers-generic-lts-xenial.
Preparing to unpack .../linux-headers-generic-lts-xenial_4.4.0.174.153_amd64.deb ...
Unpacking linux-headers-generic-lts-xenial (4.4.0.174.153) ...
Selecting previously unselected package linux-generic-lts-xenial.
Preparing to unpack .../linux-generic-lts-xenial_4.4.0.174.153_amd64.deb ...
Unpacking linux-generic-lts-xenial (4.4.0.174.153) ...
Selecting previously unselected package cgroup-lite.
Preparing to unpack .../cgroup-lite_1.9_all.deb ...
Unpacking cgroup-lite (1.9) ...
Selecting previously unselected package squashfs-tools.
Preparing to unpack .../squashfs-tools_1%3a4.2+20130409-2ubuntu0.14.04.3_amd64.deb ...
Unpacking squashfs-tools (1:4.2+20130409-2ubuntu0.14.04.3) ...
Selecting previously unselected package systemd.
Preparing to unpack .../systemd_204-5ubuntu20.31_amd64.deb ...
Unpacking systemd (204-5ubuntu20.31) ...
Selecting previously unselected package snapd.
Preparing to unpack .../snapd_2.38~14.04_amd64.deb ...
Unpacking snapd (2.38~14.04) ...
Selecting previously unselected package thermald.
Preparing to unpack .../thermald_1.4.3-5~14.04.4_amd64.deb ...
Unpacking thermald (1.4.3-5~14.04.4) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Processing triggers for ureadahead (0.100.0-16) ...
Processing triggers for mime-support (3.54ubuntu1.1) ...
Setting up liblzo2-2:amd64 (2.06-1.2ubuntu1.1) ...
Setting up libnl-3-200:amd64 (3.2.21-1ubuntu4.1) ...
Setting up libnl-genl-3-200:amd64 (3.2.21-1ubuntu4.1) ...
Setting up libseccomp2:amd64 (2.4.1-0ubuntu0.14.04.2) ...
Setting up libsystemd-journal0:amd64 (204-5ubuntu20.31) ...
Setting up iucode-tool (1.0.1-1) ...
Setting up intel-microcode (3.20191115.1ubuntu0.14.04.2) ...
update-initramfs: deferring update (trigger activated)
intel-microcode: microcode will be updated at next boot
Setting up wireless-regdb (2013.02.13-1ubuntu1) ...
Setting up crda (1.1.2-1ubuntu2) ...
Setting up iw (3.4-1) ...
Setting up linux-firmware (1.127.24) ...
update-initramfs: Generating /boot/initrd.img-4.4.0-173-generic
update-initramfs: Generating /boot/initrd.img-4.4.0-148-generic
Setting up linux-modules-4.4.0-174-generic (4.4.0-174.204~14.04.1) ...
Setting up linux-image-4.4.0-174-generic (4.4.0-174.204~14.04.1) ...
I: /vmlinuz.old is now a symlink to boot/vmlinuz-4.4.0-173-generic
I: /initrd.img.old is now a symlink to boot/initrd.img-4.4.0-173-generic
I: /vmlinuz is now a symlink to boot/vmlinuz-4.4.0-174-generic
I: /initrd.img is now a symlink to boot/initrd.img-4.4.0-174-generic
Setting up linux-modules-extra-4.4.0-174-generic (4.4.0-174.204~14.04.1) ...
Setting up amd64-microcode (3.20180524.1~ubuntu0.14.04.2+really20130710.1ubuntu1) ...
update-initramfs: deferring update (trigger activated)
Setting up linux-image-generic-lts-xenial (4.4.0.174.153) ...
Setting up linux-headers-4.4.0-174 (4.4.0-174.204~14.04.1) ...
Setting up linux-headers-4.4.0-174-generic (4.4.0-174.204~14.04.1) ...
Setting up linux-headers-generic-lts-xenial (4.4.0.174.153) ...
Setting up linux-generic-lts-xenial (4.4.0.174.153) ...
Setting up cgroup-lite (1.9) ...
All runlevel operations denied by policy
invoke-rc.d: policy-rc.d denied execution of start.
Setting up squashfs-tools (1:4.2+20130409-2ubuntu0.14.04.3) ...
Setting up systemd (204-5ubuntu20.31) ...
Initializing machine ID from D-Bus machine ID.
All runlevel operations denied by policy
invoke-rc.d: policy-rc.d denied execution of start.
Setting up thermald (1.4.3-5~14.04.4) ...
All runlevel operations denied by policy
invoke-rc.d: policy-rc.d denied execution of start.
Processing triggers for ureadahead (0.100.0-16) ...
Setting up snapd (2.38~14.04) ...
Processing triggers for libc-bin (2.19-0ubuntu6.15+esm1) ...
Processing triggers for initramfs-tools (0.103ubuntu4.11) ...
update-initramfs: Generating /boot/initrd.img-4.4.0-173-generic
Processing triggers for linux-image-4.4.0-174-generic (4.4.0-174.204~14.04.1) ...
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-4.4.0-174-generic
/etc/kernel/postinst.d/x-grub-legacy-ec2:
Searching for GRUB installation directory ... found: /boot/grub
Searching for default file ... found: /boot/grub/default
Testing for an existing GRUB menu.lst file ... found: /boot/grub/menu.lst
Searching for splash image ... none found, skipping ...
Found kernel: /boot/vmlinuz-4.4.0-174-generic
Found kernel: /boot/vmlinuz-4.4.0-173-generic
Found kernel: /boot/vmlinuz-4.4.0-148-generic
Replacing config file /run/grub/menu.lst with new version
Updating /boot/grub/menu.lst ... done

/etc/kernel/postinst.d/zz-update-grub:
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.4.0-174-generic
Found initrd image: /boot/initrd.img-4.4.0-174-generic
Found linux image: /boot/vmlinuz-4.4.0-173-generic
Found initrd image: /boot/initrd.img-4.4.0-173-generic
Found linux image: /boot/vmlinuz-4.4.0-148-generic
Found initrd image: /boot/initrd.img-4.4.0-148-generic
done

$ sudo service systemd status
systemd stop/waiting

$ sudo systemctl status snapd
Failed to get D-Bus connection: No connection to service manager.

$ snap version
snap 2.38~14.04
snapd unavailable
series -

Revision history for this message
Michael Vogt (mvo) wrote :

This is curious. The error message indicates that the "policy-rc.d" mechanism is used to prevent daemons from starting. And indeed:
```
# cat /usr/sbin/policy-rc.d
#!/bin/sh
echo "All runlevel operations denied by policy" >&2
exit 101
```

However this file is not owned by any package:
```
# dpkg -S /usr/sbin/policy-rc.d
dpkg-query: no path found matching pattern /usr/sbin/policy-rc.d
```
and nothing in /var/lib/dpkg/info seems to create policy-rc.d.

The file was last modified "Feb 11" but nothing else in {,/usr}/{,s}bin has a similar timestamp.
```
# stat /usr/sbin/policy-rc.d
  File: '/usr/sbin/policy-rc.d'
  Size: 71 Blocks: 8 IO Block: 4096 regular file
Device: 801h/2049d Inode: 2413 Links: 1
Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2020-02-19 10:51:55.427337996 +0000
Modify: 2020-02-11 14:54:19.833694933 +0000
Change: 2020-02-11 14:54:19.837694937 +0000
 Birth: -

root@snapdtrusty:~# find /{,usr}/{,s}bin/ -printf '%p %t\n' |grep "Feb 11"
/usr/sbin/policy-rc.d Tue Feb 11 14:54:19.0833694933 2020
root@snapdtrusty:~# find /{,usr}/{,s}bin/ -printf '%p %t\n' |wc -l
1169

But:
# find /usr -printf '%p %t\n' |grep "Feb 11"
/usr/lib/ubuntu-advantage Tue Feb 11 15:04:04.0982248752 2020
...

So maybe a side-effect of the ubuntu-advantage install or the script that did that install forgot to cleanup something?
```

Michael Vogt (mvo)
Changed in snapd:
status: New → Incomplete
Revision history for this message
Michael Vogt (mvo) wrote :

Fwiw - after moving /usr/sbin/policy-rc.d away and installing snapd I can:
```
root@snapdtrusty:~# snap install test-snapd-service --beta
2020-02-19T12:21:38Z INFO Waiting for restart...
test-snapd-service (beta) 1.0 from Canonical✓ installed
root@snapdtrusty:~# snap install go-example-webserver
go-example-webserver 16.04-9 from Canonical✓ installed
root@snapdtrusty:~# curl http://localhost:8081
Hello World
```
so things seems to be ok.

And re-exec also works:
```
root@snapdtrusty:~# snap version
snap 2.43.2
snapd 2.43.2
series 16
ubuntu 14.04
kernel 4.4.0-173-generic
```

Revision history for this message
Chad Smith (chad.smith) wrote :

mvo thanks for the input on this. Confirmed image creation bug which is being worked to avoid leaving /usr/sbin/policy-rc.d artifact around

Revision history for this message
Chad Smith (chad.smith) wrote :

Marking invalid because this is an image creation bug, not snapd

Changed in snapd:
status: Incomplete → Invalid
status: Invalid → New
Revision history for this message
Chad Smith (chad.smith) wrote :

Although... snapd not emitting a non-zero exit code when it cannot start systemd is a bug isn't it?

Marking 'New' so mvo can determine if not emitting a non-zero exit code due when failing to start systemd service is a bug that will be fixed in snapd.

Changed in snapd:
assignee: nobody → Michael Vogt (mvo)
Zygmunt Krynicki (zyga)
Changed in snapd:
status: New → Triaged
importance: Undecided → Medium
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.