systemd mount units fail during boot, while file system is correctly mounted

Hello,

from reading the upstream bug, it's unclear if the systemd commit actually did fix this; it seems like a kernel and/or util-linux patch is needed.

From your description, it sounds like you're not able to reliably reproduce this (only randomly), right? Have you tried the upstream reproducer? It would help a lot if we had a reliable way to reproduce and verify this.
https://github.com/systemd/systemd/issues/10872#issuecomment-523399087

Ping for reproducer, if you have one.

this appears to still be waiting for a fix upstream, so this bug will need to wait for upstream.

Just to follow up, a quick read of the upstream bug seems to indicate there are kernel patch(es) involved in fixing this as well as systemd patch(es). The upstream bug isn't yet marked fixed, so hopefully we can check back in the new year to start backporting.

The commits referenced in the upstream bug report are present in Lunar and newer.

I'm marking Bionic as Won't Fix, as this is no longer under standard support. The fixes for this bug will be made available under Ubuntu Pro for 18.04.

I wasn't able to reproduce this with the test script, so Jammy is likely unaffected by this. If this changes, please leave a comment so we can investigate.

I've now successfully tested this on Bionic, Focal and Jammy. Using a test script based on upstream's bug report, several systemd mount units are created that are tmpfs-backed for better performance. The basic routine of the script is as follows:

- mount the unit (or target mount point)
- sleep 1s
- verify whether the mount exists and is mounted correctly

This is done concurrently with 100 mountpoints, through 40 test loops by default. Additionally, the script covers all variants of this issue that were reported upstream:
1) install all systemd units at once, mount with 'systemctl start' all units
2) install each systemd unit when mounting, mount with 'systemctl start' each unit
3) mount directly with 'systemd-mount'
4) mount directly with 'mount'
5) install each systemd unit when mounting, mount with 'systemctl start',
   trigger multiple 'systemctl daemon-reload' throughout test loop
Each of these scenarios has an equivalent test "variant" in the script that can be invoked directly. Unless specified otherwise, only scenario 4 is tested.

Without the systemd and kernel fixes, at least one of the above scenarios seems to fail consistently in Bionic and Focal. After installing test packages from the PPAs [0] and [1], I've been able to run the test script without any errors for multiple iterations.

[0] https://launchpad.net/~halves/+archive/ubuntu/339757-test-kernel
[1] https://launchpad.net/~halves/+archive/ubuntu/339757-test-systemd

This bug is awaiting verification that the linux/5.4.0-164.181 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal-linux' to 'verification-done-focal-linux'. If the problem still exists, change the tag 'verification-needed-focal-linux' to 'verification-failed-focal-linux'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

I've tested the kernel from focal-proposed, with the systemd packages from my personal PPA (as the systemd patches aren't yet available in focal-proposed).

All test variants from the rep-tmpfs.sh script ran succesfully, and general smoke testing revealed no further issues.

ubuntu@z-rotomvm35:~$ uname -rv
5.4.0-164-generic #181-Ubuntu SMP Fri Sep 1 13:41:22 UTC 2023
ubuntu@z-rotomvm35:~$ dpkg -l systemd
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-==============================-============-=================================
ii systemd 245.4-4ubuntu3.22+20230626dbg2 amd64 system and service manager

This bug was fixed in the package linux - 5.4.0-164.181

---------------
linux (5.4.0-164.181) focal; urgency=medium

  * focal/linux: 5.4.0-164.181 -proposed tracker (LP: #2033867)

  * Please enable Renesas RZ platform serial installer (LP: #2022361)
    - [Config] enable hihope RZ/G2M serial console

  * Azure: hv_netvsc: add support for vlans in AF_PACKET mode (LP: #2030872)
    - hv_netvsc: add support for vlans in AF_PACKET mode

  * systemd mount units fail during boot, while file system is correctly mounted
    (LP: #1837227)
    - list: introduce list_for_each_continue()
    - proc/mounts: add cursor

  * CVE-2023-40283
    - Bluetooth: L2CAP: Fix use-after-free in l2cap_sock_ready_cb

  * CVE-2023-20588
    - x86/bugs: Increase the x86 bugs vector size to two u32s
    - x86/CPU/AMD: Do not leak quotient data after a division by 0
    - x86/CPU/AMD: Fix the DIV(0) initial fix attempt

  * CVE-2023-4194
    - net: tun_chr_open(): set sk_uid from current_fsuid()
    - net: tap_open(): set sk_uid from current_fsuid()

  * CVE-2023-1206
    - tcp: Reduce chance of collisions in inet6_hashfn().

  * CVE-2021-4001
    - bpf: Fix toctou on read-only map's constant scalar tracking

  * Focal update: v5.4.248 upstream stable release (LP: #2031121)
    - test_firmware: fix a memory leak with reqs buffer
    - KEYS: asymmetric: Copy sig and digest in public_key_verify_signature()
    - dasd: refactor dasd_ioctl_information
    - s390/dasd: Use correct lock while counting channel queue length
    - power: supply: ab8500: Fix external_power_changed race
    - power: supply: sc27xx: Fix external_power_changed race
    - power: supply: bq27xxx: Use mod_delayed_work() instead of cancel() +
      schedule()
    - ARM: dts: vexpress: add missing cache properties
    - power: supply: Ratelimit no data debug output
    - platform/x86: asus-wmi: Ignore WMI events with codes 0x7B, 0xC0
    - regulator: Fix error checking for debugfs_create_dir
    - irqchip/meson-gpio: Mark OF related data as maybe unused
    - power: supply: Fix logic checking if system is running from battery
    - btrfs: handle memory allocation failure in btrfs_csum_one_bio
    - parisc: Improve cache flushing for PCXL in arch_sync_dma_for_cpu()
    - parisc: Flush gatt writes and adjust gatt mask in parisc_agp_mask_memory()
    - MIPS: Alchemy: fix dbdma2
    - mips: Move initrd_start check after initrd address sanitisation.
    - xen/blkfront: Only check REQ_FUA for writes
    - drm:amd:amdgpu: Fix missing buffer object unlock in failure path
    - ocfs2: fix use-after-free when unmounting read-only filesystem
    - ocfs2: check new file size on fallocate call
    - nios2: dts: Fix tse_mac "max-frame-size" property
    - nilfs2: fix incomplete buffer cleanup in nilfs_btnode_abort_change_key()
    - nilfs2: fix possible out-of-bounds segment allocation in resize ioctl
    - kexec: support purgatories with .text.hot sections
    - powerpc/purgatory: remove PGO flags
    - nouveau: fix client work fence deletion race
    - RDMA/uverbs: Restrict usage of privileged QKEYs
    - net: usb: qmi_wwan: add support for Compal RXM-G1
    - ALSA: hda/realtek: Add a quirk for Compaq N14JP6
    - ...

This bug is awaiting verification that the linux-azure/5.4.0-1118.125 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal-linux-azure' to 'verification-done-focal-linux-azure'. If the problem still exists, change the tag 'verification-needed-focal-linux-azure' to 'verification-failed-focal-linux-azure'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Hello Guillaume, or anyone else affected,

Accepted systemd into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/245.4-4ubuntu3.23 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted systemd (245.4-4ubuntu3.23) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

casync/2+20190213-1 (armhf)
gvfs/1.44.1-1ubuntu1.2 (ppc64el)
linux-gcp-5.15/5.15.0-1048.56~20.04.1 (arm64)
linux-hwe-5.15/5.15.0-91.101~20.04.1 (armhf)
linux-oracle-5.15/5.15.0-1049.55~20.04.1 (arm64)
mariadb-10.3/1:10.3.38-0ubuntu0.20.04.1 (armhf)
netplan.io/0.104-0ubuntu2~20.04.4 (s390x)
puppet/5.5.10-4ubuntu3 (armhf)
upower/0.99.11-1build2 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

I've successfully confirmed that the mount issues seem to be fixed using systemd from focal-proposed. Using the 'rep-tmpfs.sh' script, all variants ran without any issues for multiple rounds. Basic testing on a VM also looks good. Below are the versions used for this test:

$ dpkg -l systemd
...
||/ Name Version Architecture Description
+++-==============-=================-============-=================================
ii systemd 245.4-4ubuntu3.23 amd64 system and service manager

$ uname -rv
5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023

I'll look into the reported regressions next, to confirm none of them are caused by this LP.

This bug was fixed in the package systemd - 245.4-4ubuntu3.23

---------------
systemd (245.4-4ubuntu3.23) focal; urgency=medium

  [ Nick Rosbrook ]
  * core/device: ignore DEVICE_FOUND_UDEV bit on switching root (LP: #2037281)
    File: debian/patches/lp2037281-core-device-ignore-DEVICE_FOUND_UDEV-bit-on-switching-roo.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=7793563bb38a84a3dc6bc0da1c08546c3b915ab8
  * dns-query: bump CNAME_MAX to 16 (LP: #2024009)
    File: debian/patches/lp2024009-dns-query-bump-CNAME_MAX-to-16.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=193899d103d44c642d362e9916b14df844ec702f
  * Fall back to kexec when no kexec binary exists (LP: #1969365)
    File: debian/patches/lp1969365-Fall-back-to-kexec-when-no-kexec-binary-exists.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=3934f3794427dee4e72824998dd4c6e6d5875289
  * test: ignore LXC filesystem when checking for writable locations (LP: #2029352)
    File: debian/patches/lp2029352-test-ignore-LXC-filesystem-when-checking-for-writable-loc.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=70facbfbf54c4ffb31ba392dbe3fec3084fdf3bc

  [ Heitor Alves de Siqueira ]
  * core/mount: adjust deserialized state based on /proc/self/mountinfo (LP: #1837227)
    Author: Heitor Alves de Siqueira
    File: debian/patches/lp1837227-core-mount-adjust-deserialized-state-based-on-proc-self-m.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=a0a749953d309f48bc45140102adf205d1071c4d

 -- Nick Rosbrook <email address hidden> Tue, 21 Nov 2023 16:10:21 -0500

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug has been fixed in systemd - 237-3ubuntu10.57+esm2

---------------
systemd (237-3ubuntu10.57+esm2) bionic; urgency=medium

  * Fix systemd mount units failing during boot (LP: #1837227)
    - d/p/lp1837227/0001-core-properly-reset-all-ExecStatus-structures-when-e.patch
    - d/p/lp1837227/0002-mount-flush-out-cycle-state-on-DEAD-MOUNTED-only-not.patch
    - d/p/lp1837227/0003-mount-rescan-proc-self-mountinfo-before-processing-w.patch
    - d/p/lp1837227/0004-mount-mark-an-existing-mounting-unit-from-proc-self-.patch
    - d/p/lp1837227/0005-core-mount-adjust-deserialized-state-based-on-proc-s.patch
  * Fix FTBFS for test-fs-util symlink failures (LP: #2077176)
    - d/p/lp2077176/0001-fs-util-introduce-fchmod_opath.patch
    - d/p/lp2077176/0002-fs-util-introduce-fchmod_and_chown.patch
    - d/p/lp2077176/0003-fs-util-beef-up-chmod_and_chown-a-bit.patch
    - d/p/lp2077176/0004-fs-util-change-chmod_and_chown-to-not-complain-if-st.patch
    - d/p/lp2077176/0005-fs-util-rewrite-chmod_and_chown.patch
    - d/p/lp2077176/0006-fs-util-no-need-for-fchmod_and_chown-to-access-proc-.patch
    - d/p/lp2077176/0007-tree-wide-port-various-places-over-to-use-chmod_and_.patch
    - d/p/lp2077176/0008-test-fs-util-don-t-validate-mode-of-symlinks.patch

 -- Heitor Alves de Siqueira <email address hidden> Fri, 16 Aug 2024 14:54:17 +0000