SRU of LXC 4.0.12 to focal (upstream bugfix release)

Bug #1959993 reported by Stéphane Graber
20
This bug affects 2 people
Affects Status Importance Assigned to Milestone
lxc (Ubuntu)
In Progress
Undecided
Stéphane Graber
Focal
Fix Released
Undecided
Unassigned
Impish
Fix Released
Undecided
Unassigned

Bug Description

LXC released 4.0.12 as a bugfix release and is now in jammy. We'd like to line things up in focal.

[Impact]
The proposed SRU will bump from 4.0.6 all the way to 4.0.12, lining it up with what's currently in jammy. We've been skipping a few of the bugfix releases in focal so far, mostly catching up when we're starting to see problems with the older version.

In this case, we've seen a number of issues when running with the HWE kernels as well as autopkgtest issues on foreign architectures (arm64 and s390x), all those will go away with this bump as we've confirmed everything is clean in jammy.

Changelog:

  * Cherry-pick upstream bugfixes (stable-4.0):
    - 0002-lxc-checkconfig-Fix-bashism.patch
    - 0003-doc-Fix-reverse-allowlist-denylist.patch

  * New upstream bugfix release (4.0.12):
    (https://discuss.linuxcontainers.org/t/lxc-4-0-12-has-been-released/13288)
    - Fixed CRIU restoration of containers with pre-created veth interfaces
    - Fixed issue with kernels lacking SMT support
    - Extended cgroup2 config options in lxc.mount.auto (cgroup2)
    - lxc-download now relies on HTTPS for validation (avoids GPG issues)

  * New upstream bugfix release (4.0.11):
    (https://discuss.linuxcontainers.org/t/lxc-4-0-11-has-been-released/12427)
    - Core scheduling support (lxc.sched.core)
    - riscv64 support in lxc.arch
    - Significantly improved bash completion profile
    - Greater use of the new VFS mount API (when supported by the kernel)
    - Fix containers with empty network namespaces
    - Handle kernels that lack TIOCGPTPEER
    - Improve CPU bitmask/id handling (handle skipped CPU numbers)
    - Reworked the tests to run offline

  * New upstream bugfix release (4.0.10):
    (https://discuss.linuxcontainers.org/t/lxc-4-0-10-has-been-released/11618)
    - Fix issues with less common architectures
    - Support for additional idmap mounts
    - nft support in lxc-net
    - Cleaner mount entries for sys:mixed
    - Switched GPG server to keyserver.ubuntu.com

  * New upstream bugfix release (4.0.9):
    (https://discuss.linuxcontainers.org/t/lxc-4-0-9-has-been-released/10999)
    - Fix incorrect personality setting when running 32bit containers on 64bit

  * New upstream bugfix release (4.0.8):
    - Fix CGroup attach against older running containers

  * New upstream bugfix release (4.0.7):
    - Testing improvements including fixes from oss-fuzz
    - Rework of the attach codepath
    - Cgroup handling rework

  * Bump to debhelper 12 (allows focal SRUs)
  * Bump standards to 4.6.0.1
  * Add lintian overrides for incorrect bashism detection
  * Remove bash completion install logic (now done upstream)

Just like Ubuntu itself, upstream releases long term support releases, e.g. 4.0, and then periodic point releases including all the accumulated bugfixes.

Only the latest upstream release gets full support from the upstream developers, everyone else is expected to first update to it before receiving any kind of support.

This should qualify under the minor/micro upstream bugfix release allowance of the SRU policy, letting us SRU this without paperwork for every single change included in this upstream release.

[Test Plan]
lxc has autopkgtests which will assert that the binaries built in -proposed are functional.

[Where problems could occur]
This is catching up a fair bit on recent kernel API changes, including cgroup1/cgroup2 support, handling of nftables, riscv64 and core scheduling which were all needed to properly handle the most recent HWE kernels especially as we're getting ready for Ubuntu 22.04's 5.15 to get pushed to focal.

We've had all that code running on well over a million of LXD snap users for a few months now without seeing any issues (or more precisely, those issues we found have been all been resolved as of 4.0.12).

However what LXD exercises isn't 100% of LXC and it's certainly possible that we missed a corner case in one of those changes.
The good news is that this would most likely be triggered by a HWE kernel, so a viable workaround in many cases would be to temporarily go back to the original kernel (5.4) while the issue is sorted out in a follow up SRU.

It's also worth noting that LXD CI runs daily tests against over a dozen different kernels coming from various distros which helps us identify such issues quite early on.

[Other Info]
Unless absolutely required, we're not intending to push for an SRU to impish as it has a reasonably recent LXC (4.0.10) and realistically, folks are quite a bit more likely to wait to upgrade from focal to jammy than jump through EOL releases from focal to impish.

Should someone do an upgrade to impish, we've confirmed that the upgrade is resolvable and that they'll just be left with a more recent version of LXC than that in the impish archive, until jammy releases and they upgrade to it.

Changed in lxc (Ubuntu):
status: New → Triaged
assignee: nobody → Stéphane Graber (stgraber)
Revision history for this message
Stéphane Graber (stgraber) wrote :

A build of the proposed SRU is available for all architectures (well, riscv64 is slowly building) at: https://launchpad.net/~stgraber/+archive/ubuntu/experimental-devirt/

description: updated
description: updated
Revision history for this message
Stéphane Graber (stgraber) wrote :

Uploaded to focal SRU queue.

Changed in lxc (Ubuntu):
status: Triaged → In Progress
Revision history for this message
Stéphane Graber (stgraber) wrote :

Worth noting that with this upload, the packaging gets virtually in sync with jammy minus a few artifacts of git-dpm in the patches, this should make maintenance a fair bit easier should a follow-up SRU be needed.

Upstream packaging delta:
https://github.com/lxc/lxc-pkg-ubuntu/compare/debian/1%254.0.6-0ubuntu1_20.04.1...debian/1%254.0.12-0ubuntu1_20.04.1

Revision history for this message
Stéphane Graber (stgraber) wrote :

Source package and all resulting binary packages are "lintian -iI" clean except for the warning caused by SRU version numbering.

Revision history for this message
Robie Basak (racb) wrote :

Thank you for the detailed SRU information! You anticipated most of the obvious questions and saved us a bunch of round trips.

> Should someone do an upgrade to impish, we've confirmed that the upgrade is resolvable and that they'll just be left with a more recent version of LXC than that in the impish archive, until jammy releases and they upgrade to it.

Wouldn't that leave them missing security updates that might be subsequently uploaded to Impish? I was under the impression that leaving the archive in this situation is always a hard no.

Revision history for this message
Stéphane Graber (stgraber) wrote :

In general, it's indeed a problem for such cases, though there are a few mitigating factors here:

 - This would only be a problem for those who upgrade from focal to impish. My understanding is that there is no direct upgrade path to achieve this, you'd need to go throughgroovy and hirsute, both of which are EOL, so I wouldn't really expect many/any user to do this.

 - The LXC team is responsible for self-assigning CVEs and preparing LXC security updates (~ubuntu-lxc-security). In such cases we usually prefer releasing a new upstream bugfix release (LXC 4.0.x is an LTS with a 5 years security commitment upstream) and then upload that to the -security pockets where applicable. So in that case, we'd bump jammy to match focal, saving us some effort on validating the security update by making things be identical on all releases shipping 4.0.x (jammy will be on 5.0.x).

That's generally how we've handled those micro release updates and so far haven't run into any problems but I'm also not necessarily against uploading the exact same source package as focal to impish, the preparation time would be pretty minimal, it's mostly on the review and testing side of things that things can increase significantly especially as we usually struggle to find users on non-LTS to perform more than a simple install/upgrade test.

Revision history for this message
Stéphane Graber (stgraber) wrote :

(We technically had the same issue with the previous upload as 4.0.6 was higher than what hirsute shipped at the time, though in that case the EOL for hirsute was just a month or so away making the case for skipping that SRU even stronger :))

Revision history for this message
Robie Basak (racb) wrote :

OK thanks. I'll consult with others on the SRU team on this - I've added it to our meeting agenda. If it's decided that this is OK then I'll make sure it's documented to avoid holding you up on it again.

Revision history for this message
Brian Murray (brian-murray) wrote :

As a point of clarification here the release upgrade process skips interim releases once they become end of life, so if one were to upgrade from Focal they would upgrade to Impish or Jammy (depending on how their system is configured). It looks like I made this change 7 years ago (gasp!).

https://bugs.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/+bug/1497024

Revision history for this message
Stéphane Graber (stgraber) wrote :

Thanks Brian, my memory of this whole thing clearly dates back a long time then ;)

I still remember some of the discussions of what we'd expect people to be doing in such cases and whether we'd ever officially support (as in test/validate) upgrade paths other than release to release+1 and LTS to LTS+1.

I remember us struggling to really validate those two more common paths so I wonder what's done today to validate the upgrade paths when one release goes EOL and the upgrade path changes to something different for those upgrading from the LTS.

Revision history for this message
Robie Basak (racb) wrote :

At yesterday's SRU team meeting we concluded that we don't want the archive to end up going down in versions as you upgrade from Focal to Impish. IOW, we aren't granting an exception in this case. Some reasons:

The time a security update is needed in the future would not be an appropriate time to be dealing with this complication - for example we wouldn't want to rule out a cherry-pick and a simple ".1" upload to Impish should that be the most appropriate thing at a later time, even if that isn't your normal intention.

I trust Stéphane to make sure the right thing happens in the future, but what if he's not around at the time? All general processes we have assume this situation never arises, so fixing it up properly later might get neglected, and it doesn't seem worth ensuring all relevant processes are adjusted to ensure this is done properly if we can just avoid the situation occurring in the first place.

Brian noted that the upgrade path from Focal to Impish _is_ supported.

It sounded like you can get an update for Impish prepared to match the one you have for Focal, and that wouldn't take you too much effort.

Please could you proceed on this basis?

Revision history for this message
Robie Basak (racb) wrote :

(note that I haven't actually reviewed the upload yet)

Revision history for this message
Stéphane Graber (stgraber) wrote :

Uploaded the exact same thing to impish queue.

Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Stéphane, or anyone else affected,

Accepted lxc into impish-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lxc/1:4.0.12-0ubuntu1~21.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-impish to verification-done-impish. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-impish. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in lxc (Ubuntu Impish):
status: New → Fix Committed
tags: added: verification-needed verification-needed-impish
Changed in lxc (Ubuntu Focal):
status: New → Fix Committed
tags: added: verification-needed-focal
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Stéphane, or anyone else affected,

Accepted lxc into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lxc/1:4.0.12-0ubuntu1~20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Stéphane Graber (stgraber) wrote :

Have confirmed the packages to be functional on both focal and impish.
Tried installation, upgrade and basic container creation, deletion and normal operations.

tags: added: verification-done-focal verification-done-impish
removed: verification-needed-focal verification-needed-impish
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package lxc - 1:4.0.12-0ubuntu1~21.10.1

---------------
lxc (1:4.0.12-0ubuntu1~21.10.1) impish; urgency=medium

  * Cherry-pick upstream bugfixes (stable-4.0):
    - 0002-lxc-checkconfig-Fix-bashism.patch
    - 0003-doc-Fix-reverse-allowlist-denylist.patch

  * New upstream bugfix release (4.0.12) (LP: #1959993):
    (https://discuss.linuxcontainers.org/t/lxc-4-0-12-has-been-released/13288)
    - Fixed CRIU restoration of containers with pre-created veth interfaces
    - Fixed issue with kernels lacking SMT support
    - Extended cgroup2 config options in lxc.mount.auto (cgroup2)
    - lxc-download now relies on HTTPS for validation (avoids GPG issues)

  * New upstream bugfix release (4.0.11):
    (https://discuss.linuxcontainers.org/t/lxc-4-0-11-has-been-released/12427)
    - Core scheduling support (lxc.sched.core)
    - riscv64 support in lxc.arch
    - Significantly improved bash completion profile
    - Greater use of the new VFS mount API (when supported by the kernel)
    - Fix containers with empty network namespaces
    - Handle kernels that lack TIOCGPTPEER
    - Improve CPU bitmask/id handling (handle skipped CPU numbers)
    - Reworked the tests to run offline

  * New upstream bugfix release (4.0.10):
    (https://discuss.linuxcontainers.org/t/lxc-4-0-10-has-been-released/11618)
    - Fix issues with less common architectures
    - Support for additional idmap mounts
    - nft support in lxc-net
    - Cleaner mount entries for sys:mixed
    - Switched GPG server to keyserver.ubuntu.com

  * New upstream bugfix release (4.0.9):
    (https://discuss.linuxcontainers.org/t/lxc-4-0-9-has-been-released/10999)
    - Fix incorrect personality setting when running 32bit containers on 64bit

  * New upstream bugfix release (4.0.8):
    - Fix CGroup attach against older running containers

  * New upstream bugfix release (4.0.7):
    - Testing improvements including fixes from oss-fuzz
    - Rework of the attach codepath
    - Cgroup handling rework

  * Bump to debhelper 12 (allows focal SRUs)
  * Bump standards to 4.6.0.1
  * Add lintian overrides for incorrect bashism detection
  * Remove bash completion install logic (now done upstream)

 -- Stéphane Graber <email address hidden> Thu, 03 Feb 2022 23:50:20 -0500

Changed in lxc (Ubuntu Impish):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for lxc has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package lxc - 1:4.0.12-0ubuntu1~20.04.1

---------------
lxc (1:4.0.12-0ubuntu1~20.04.1) focal; urgency=medium

  * Cherry-pick upstream bugfixes (stable-4.0):
    - 0002-lxc-checkconfig-Fix-bashism.patch
    - 0003-doc-Fix-reverse-allowlist-denylist.patch

  * New upstream bugfix release (4.0.12) (LP: #1959993):
    (https://discuss.linuxcontainers.org/t/lxc-4-0-12-has-been-released/13288)
    - Fixed CRIU restoration of containers with pre-created veth interfaces
    - Fixed issue with kernels lacking SMT support
    - Extended cgroup2 config options in lxc.mount.auto (cgroup2)
    - lxc-download now relies on HTTPS for validation (avoids GPG issues)

  * New upstream bugfix release (4.0.11):
    (https://discuss.linuxcontainers.org/t/lxc-4-0-11-has-been-released/12427)
    - Core scheduling support (lxc.sched.core)
    - riscv64 support in lxc.arch
    - Significantly improved bash completion profile
    - Greater use of the new VFS mount API (when supported by the kernel)
    - Fix containers with empty network namespaces
    - Handle kernels that lack TIOCGPTPEER
    - Improve CPU bitmask/id handling (handle skipped CPU numbers)
    - Reworked the tests to run offline

  * New upstream bugfix release (4.0.10):
    (https://discuss.linuxcontainers.org/t/lxc-4-0-10-has-been-released/11618)
    - Fix issues with less common architectures
    - Support for additional idmap mounts
    - nft support in lxc-net
    - Cleaner mount entries for sys:mixed
    - Switched GPG server to keyserver.ubuntu.com

  * New upstream bugfix release (4.0.9):
    (https://discuss.linuxcontainers.org/t/lxc-4-0-9-has-been-released/10999)
    - Fix incorrect personality setting when running 32bit containers on 64bit

  * New upstream bugfix release (4.0.8):
    - Fix CGroup attach against older running containers

  * New upstream bugfix release (4.0.7):
    - Testing improvements including fixes from oss-fuzz
    - Rework of the attach codepath
    - Cgroup handling rework

  * Bump to debhelper 12 (allows focal SRUs)
  * Bump standards to 4.6.0.1
  * Add lintian overrides for incorrect bashism detection
  * Remove bash completion install logic (now done upstream)

 -- Stéphane Graber <email address hidden> Thu, 03 Feb 2022 23:50:20 -0500

Changed in lxc (Ubuntu Focal):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.