TDX azure instances crash during boot because of glibc bug

@schopin fyi I'm preparing the patches for the SRU.

Jammy debdiff

Kinetic debdiff

Please don't upload this :)

This will get batched in the upcoming Jammy SRU (which will be actively worked on in the coming days). We try to limit the number of glibc uploads as each upload triggers autopkgtests for the entire archive.

The attachment "lp2011421_jammy.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

The kernel patch won't be released in an SRU until after the glibc fix has been released to updates. Otherwise there will be a window where TDX support is regressed.

The kernel patch is only for Kinetic linux-azure (5.19). The Jammy backport (linux-azure-5.19) will get it automatically.

This bug is marked as 'in progress' for glibc in both jammy and kinetic, but there is no glibc SRU in the kinetic queue. As the normal SRU process is to fix bugs in later releases before earlier releases, there needs to be some clarification about the plan here.

This is also the only one of the several bugs referenced in this jammy SRU that has a kinetic or lunar task; all the others have only the main task closed as fix released. So it would be helpful to clarify (perhaps by opening and then cosing additional per-series tasks) exactly when the bug was fixed post-jammy.

Sorry, my bad for the status, we discussed this offline with joalif.

I'm not planning on fixing this in Kinetic as the affected platform is more of an LTS user, as I understand, and the kinetic EOL is approaching. Given the time it can take for a glibc SRU to go through -proposed even when there's nothing breaking, I don't think it worth the trouble. I might be wrong, though :)

Hello Ioanna, or anyone else affected,

Accepted glibc into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/glibc/2.35-0ubuntu3.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted glibc (2.35-0ubuntu3.2) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

adsys/0.9.2~22.04.1 (armhf)
bbmap/38.95+dfsg-1 (armhf)
boost1.74/1.74.0-14ubuntu3 (armhf)
breezy/3.2.1+bzr7585-1build1 (armhf)
casync/2+20201210-1build1 (ppc64el)
cmake/3.22.1-1ubuntu1.22.04.1 (amd64, arm64, armhf)
cython/0.29.28-1ubuntu3 (i386)
dbus-test-runner/16.10.0~bzr100+repack1-4.1 (armhf)
exim4/4.95-4ubuntu2.2 (ppc64el)
fwupd/1.7.9-1~22.04.1 (armhf)
glib-networking/2.72.0-1 (i386)
golang-github-influxdata-tail/1.0.0+git20180327.c434825-4 (ppc64el)
golang-gogoprotobuf/1.3.2-1 (arm64)
grubzfs-testsuite/0.4.15build1 (amd64)
gtk+3.0/3.24.33-1ubuntu2 (i386)
gyoto/1.4.4-7build1 (amd64, arm64, s390x)
ksh93u+m/1.0.0~beta.2-1 (ppc64el)
libassuan/2.5.5-1build1 (amd64)
libflame/5.2.0-3ubuntu3 (s390x)
libite/2.5.1-1 (s390x)
linux-aws-5.19/5.19.0-1027.28~22.04.1 (amd64, arm64)
linux-azure-5.19/5.19.0-1027.30~22.04.2 (amd64)
linux-gcp-5.19/5.19.0-1026.28~22.04.1 (arm64)
linux-gke/5.15.0-1034.39 (arm64)
linux-hwe-5.19/5.19.0-44.45~22.04.1 (amd64)
linux-lowlatency-hwe-5.19/5.19.0-1026.27~22.04.1 (amd64)
linux-oracle-5.19/5.19.0-1025.28~22.04.1 (amd64)
mercurial/6.1.1-1ubuntu1 (amd64)
mtail/3.0.0~rc48-3 (ppc64el)
mutter/42.5-0ubuntu1 (amd64)
mypy/0.942-1ubuntu1 (armhf)
mysql-8.0/8.0.33-0ubuntu0.22.04.2 (s390x)
netgen/6.2.2006+really6.2.1905+dfsg-5build1 (armhf)
notcurses/3.0.6+dfsg.1-1 (armhf)
pango1.0/1.50.6+ds-2ubuntu1 (i386)
phcpack/2.4.85+dfsg-5build1 (arm64)
pinentry/1.1.1-1build2 (i386)
puma/5.5.2-2ubuntu2 (amd64, arm64)
pycryptodome/3.11.0+dfsg1-3build1 (i386)
pygobject/3.42.1-0ubuntu1 (i386)
python-evdev/1.4.0+dfsg-1build2 (i386)
python-llfuse/1.3.8+dfsg-2build1 (armhf)
python-lz4/3.1.3+dfsg-1build3 (i386)
python3.10/3.10.6-1~22.04.2ubuntu1.1 (arm64)
pyyaml/5.4.1-1ubuntu1 (i386)
r-bioc-rhdf5/2.38.0+dfsg-2 (amd64)
r-cran-randomfieldsutils/1.1.0-1 (armhf)
r-cran-spc/1:0.6.5-1 (armhf)
rsync/3.2.7-0ubuntu0.22.04.2 (i386)
ruby-nio4r/2.5.8-2 (amd64)
ruby-nokogiri/1.13.1+dfsg-2 (amd64)
ruby-prof/1.3.1-2build2 (amd64, ppc64el)
rustc/1.65.0+dfsg0ubuntu1-0ubuntu0.22.04.1 (arm64)
seqkit/2.1.0+ds-1 (arm64, s390x)
stunnel4/3:5.63-1build1 (i386)
swtpm/0.6.3-0ubuntu3.2 (s390x)
systemd/249.11-0ubuntu3.9 (amd64, arm64, ppc64el, s390x)
taptempo/1.4.5-1 (amd64)
texinfo/6.8-4build1 (armhf)
ubiquity/22.04.19 (amd64, arm64, armhf, ppc64el)
udisks2/2.9.4-1ubuntu2 (arm64)
utox/0.18.1-1build1 (armhf)
vlc/3.0.16-1build7 (i386)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#glibc

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

# VERIFICATION JAMMY - glibc

On an Azure TDX instance installed libc6 and libc-bin from -proposed and
rebooted. It boots successfully.

I also installed a test kernel with the kernel workaround reverted and it
also boots with no problems.

@schopin,

Do you have any ETA of when this will be released ?

Sorry, there were some potential regressions that needed investigating, and we're still waiting on a few benchmark results for bug 1999551. Other than that the overall autopkgtest results are in good shape, so assuming the benchmarks show what we want them to, this shouldn't be long.

Hello Ioanna, or anyone else affected,

Accepted glibc into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/glibc/2.35-0ubuntu3.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted glibc (2.35-0ubuntu3.2) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

linux-aws-6.2/6.2.0-1007.7~22.04.1 (arm64)
linux-azure-6.2/6.2.0-1007.7~22.04.1 (arm64)
linux-oracle-5.19/5.19.0-1025.28~22.04.1 (arm64)
mysql-8.0/8.0.33-0ubuntu0.22.04.2 (s390x)
ubiquity/22.04.19 (amd64, arm64, armhf, ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#glibc

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

The segfault no longer repros after I upgraded my glibc:

cat <<EOF >/etc/apt/sources.list.d/ubuntu-$(lsb_release -cs)-proposed.list
# Enable Ubuntu proposed archive
deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -cs)-proposed restricted main multiverse universe
EOF

apt update

dpkg -l |grep libc6
ii libc6:amd64 2.35-0ubuntu3.2 amd64 GNU C Library: Shared libraries

apt install libc6

root@ubuntu:~# dpkg -l |grep libc6
ii libc6:amd64 2.35-0ubuntu3.3 amd64 GNU C Library: Shared libraries

update-initramfs -u

All autopkgtests for the newly accepted glibc (2.35-0ubuntu3.3) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

adsys/0.9.2~22.04.1 (arm64)
datefudge/1.24 (arm64)
dbus/1.12.20-2ubuntu4.1 (armhf)
dpdk/21.11.3-0ubuntu0.22.04.1 (arm64)
ganeti/3.0.2-1ubuntu1 (armhf)
gcc-9/9.5.0-1ubuntu1~22.04 (armhf)
gjs/1.72.4-0ubuntu0.22.04.1 (amd64)
golang-github-canonical-go-dqlite/1.10.1-1 (arm64)
golang-github-influxdata-tail/1.0.0+git20180327.c434825-4 (amd64)
golang-github-xenolf-lego/4.1.3-3ubuntu1.22.04.1 (armhf)
golang-gogoprotobuf/1.3.2-1 (amd64, s390x)
google-osconfig-agent/20230504.00-0ubuntu1~22.04.0 (armhf)
gvfs/1.48.2-0ubuntu1 (amd64)
hilive/2.0a-3build3 (arm64)
hyphy/2.5.36+dfsg-1 (amd64)
kmediaplayer/5.92.0-0ubuntu1 (armhf)
libclass-methodmaker-perl/2.24-2build2 (armhf)
libflame/5.2.0-3ubuntu3 (arm64)
libimage-sane-perl/5-1build3 (s390x)
libuev/2.4.0-1.1 (s390x)
linux-aws-6.2/6.2.0-1009.9~22.04.2 (arm64)
linux-azure-5.19/5.19.0-1027.30~22.04.2 (arm64)
linux-azure-6.2/6.2.0-1009.9~22.04.2 (arm64)
linux-gcp-5.19/5.19.0-1030.32~22.04.1 (arm64)
linux-gke/5.15.0-1039.44 (arm64)
linux-hwe-5.19/5.19.0-50.50 (arm64)
linux-lowlatency/5.15.0-79.88 (arm64)
linux-nvidia-6.2/6.2.0-1006.6~22.04.2 (arm64)
linux-nvidia-tegra/5.15.0-1015.15 (arm64)
linux-oem-5.17/5.17.0-1035.36 (amd64)
linux-oracle-5.19/5.19.0-1027.30 (arm64)
linux-xilinx-zynqmp/5.15.0-1023.27 (arm64)
log4cxx/0.12.1-4 (armhf)
netplan.io/0.105-0ubuntu2~22.04.3 (arm64)
notcurses/3.0.6+dfsg.1-1 (armhf)
php-luasandbox/4.0.2-3build1 (ppc64el)
prometheus/2.31.2+ds1-1ubuntu1.22.04.2 (s390x)
qcustomplot/2.0.1+dfsg1-5 (armhf)
r-cran-mice/3.14.0-1 (armhf)
r-cran-randomfields/3.3.14-1 (armhf)
r-cran-sys/3.4-1 (s390x)
ruby-rblineprof/0.3.7-2build3 (arm64)
rustc/1.66.1+dfsg0ubuntu1-0ubuntu0.22.04.1 (arm64)
samba/2:4.15.13+dfsg-0ubuntu1.2 (s390x)
stunnel4/3:5.63-1build1 (ppc64el)
swtpm/0.6.3-0ubuntu3.2 (arm64)
tmux/3.2a-4ubuntu0.2 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#glibc

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Thanks for the tests/results, @decui!
Updated verification tags per comment #17.

Ubuntu 22.10 (Kinetic Kudu) has reached end of life, so this bug will not be fixed for that specific release.

This bug was fixed in the package glibc - 2.35-0ubuntu3.3

---------------
glibc (2.35-0ubuntu3.3) jammy; urgency=medium

  * Drop SVE patches due to kernal-related performance regression
  * Fix the armhf stripping exception for ld.so (LP: #1927192)

glibc (2.35-0ubuntu3.2) jammy; urgency=medium

  * d/rules.d/debhelper.mk: fix permissions of libc.so (LP: #1989082)
  * Cherry-picks from upstream:
    - d/p/lp1999551/*: arm64 memcpy optimization (LP: #1999551)
    - d/p/lp1995362*.patch: Fix ldd segfault with missing libs (LP: #1995362)
    - d/p/lp2007796*: Fix missing cancellation point in pthread (LP: #2007796)
    - d/p/lp2007599*: add new tunables for s390x (LP: #2007599)
    - d/p/lp2011421/*: Fix crash on TDX-enabled platforms (LP: #2011421)
    - d/p/lp1992159*: Fix socket.h headers for non-GNU compilers (LP: #1992159)

 -- Simon Chopin <email address hidden> Wed, 26 Jul 2023 10:27:54 +0200

The verification of the Stable Release Update for glibc has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

The glibc workaround has been removed from the kernel. This patch only affected users of TDX in CVM images on an Azure paravisor with HCL.

This bug was fixed in the package linux-azure-fde-6.2 - 6.2.0-1014.14~22.04.1.1

---------------
linux-azure-fde-6.2 (6.2.0-1014.14~22.04.1.1) jammy; urgency=medium

  * jammy/linux-azure-fde-6.2: 6.2.0-1014.14~22.04.1.1 -proposed tracker
    (LP: #2035364)

  * TDX azure instances crash during boot because of glibc bug (LP: #2011421)
    - [Packaging] Use latest glibc from ubuntu-core-initramfs

linux-azure-fde-6.2 (6.2.0-1013.13~22.04.1.1) jammy; urgency=medium

  * Master version: 6.2.0-1013.13~22.04.1

  * jammy/linux-azure-fde-6.2: 6.2.0-1013.13~22.04.1.1 -proposed tracker
    (LP: #2033764)

linux-azure-fde-6.2 (6.2.0-1012.12~22.04.1.1) jammy; urgency=medium

  * jammy/linux-azure-fde-6.2: 6.2.0-1012.12~22.04.1.1 -proposed tracker
    (LP: #2034142)

  * Packaging resync (LP: #1786013)
    - [Packaging] update helper scripts

linux-azure-fde-6.2 (6.2.0-1011.11~22.04.1.1) jammy; urgency=medium

  * Master version: 6.2.0-1011.11~22.04.1

  * jammy/linux-azure-fde-6.2: 6.2.0-1011.11~22.04.1.1 -proposed tracker
    (LP: #2030366)

linux-azure-fde-6.2 (6.2.0-1010.10~22.04.1.1) jammy; urgency=medium

  * jammy/linux-azure-fde-6.2: 6.2.0-1010.10~22.04.1.1 -proposed tracker
    (LP: #2030532)

linux-azure-fde-6.2 (6.2.0-1009.9~22.04.3.1) jammy; urgency=medium

  * jammy/linux-azure-fde-6.2: 6.2.0-1009.9~22.04.3.1 -proposed tracker
    (LP: #2029293)

  * Packaging resync (LP: #1786013)
    - [Packaging] update variants

linux-azure-fde-6.2 (6.2.0-1009.9~22.04.2.1) jammy; urgency=medium

  * Master version: 6.2.0-1009.9~22.04.2

  * jammy/linux-azure-fde-6.2: 6.2.0-1009.9~22.04.2.1 -proposed tracker
    (LP: #2026474)

  * Miscellaneous Ubuntu changes
    - generate-depends: do not silently ignore failures

linux-azure-fde-6.2 (6.2.0-1009.9~22.04.1.1) jammy; urgency=medium

  * Master version: 6.2.0-1009.9~22.04.1

  * jammy/linux-azure-fde-6.2: 6.2.0-1009.9~22.04.1.1 -proposed tracker
    (LP: #2026474)

linux-azure-fde-6.2 (6.2.0-1008.8~22.04.1.1) jammy; urgency=medium

  * jammy/linux-azure-fde-6.2: 6.2.0-1008.8~22.04.1.1 -proposed tracker
    (LP: #2026739)

linux-azure-fde-6.2 (6.2.0-1007.7~22.04.1.1) jammy; urgency=medium

  * Master version: 6.2.0-1007.7~22.04.1

  * jammy/linux-azure-fde-6.2: 6.2.0-1007.7~22.04.1.1 -proposed tracker
    (LP: #2024532)

 -- Tim Gardner <email address hidden> Thu, 14 Sep 2023 12:25:12 -0600

This bug was fixed in the package linux-azure - 6.2.0-1014.14

---------------
linux-azure (6.2.0-1014.14) lunar; urgency=medium

  * lunar/linux-azure: 6.2.0-1014.14 -proposed tracker (LP: #2035366)

  * TDX azure instances crash during boot because of glibc bug (LP: #2011421)
    - Revert "UBUNTU: SAUCE: TDX: Work around the segfault issue in glibc 2.35 in
      Ubuntu 22.04."

linux-azure (6.2.0-1013.13) lunar; urgency=medium

  * lunar/linux-azure: 6.2.0-1013.13 -proposed tracker (LP: #2033766)

  * Packaging resync (LP: #1786013)
    - [Packaging] update helper scripts

  * Azure: net: mana: Fix MANA VF unload when hardware is unresponsive
    (LP: #2033531)
    - net: mana: Fix MANA VF unload when hardware is unresponsive

  * Azure: net: mana: Add page pool for RX buffers (LP: #2034277)
    - net: mana: Add page pool for RX buffers

  [ Ubuntu: 6.2.0-34.34 ]

  * lunar/linux: 6.2.0-34.34 -proposed tracker (LP: #2033779)
  * CVE-2023-20569
    - x86/cpu, kvm: Add support for CPUID_80000021_EAX
    - tools headers x86 cpufeatures: Sync with the kernel sources
    - x86/alternative: Optimize returns patching
    - x86/retbleed: Add __x86_return_thunk alignment checks
    - x86/srso: Add a Speculative RAS Overflow mitigation
    - x86/srso: Add IBPB_BRTYPE support
    - x86/srso: Add SRSO_NO support
    - x86/srso: Add IBPB
    - x86/srso: Add IBPB on VMEXIT
    - x86/srso: Fix return thunks in generated code
    - x86/srso: Add a forgotten NOENDBR annotation
    - x86/srso: Tie SBPB bit setting to microcode patch detection
    - Documentation/hw-vuln: Unify filename specification in index
    - Documentation/srso: Document IBPB aspect and fix formatting
    - x86/srso: Fix build breakage with the LLVM linker
    - x86: Move gds_ucode_mitigated() declaration to header
    - x86/retpoline: Don't clobber RFLAGS during srso_safe_ret()
    - x86/srso: Disable the mitigation on unaffected configurations
    - x86/retpoline,kprobes: Fix position of thunk sections with CONFIG_LTO_CLANG
    - x86/retpoline,kprobes: Skip optprobe check for indirect jumps with
      retpolines and IBT
    - x86/cpu: Fix __x86_return_thunk symbol type
    - x86/cpu: Fix up srso_safe_ret() and __x86_return_thunk()
    - objtool/x86: Fix SRSO mess
    - x86/alternative: Make custom return thunk unconditional
    - x86/cpu: Clean up SRSO return thunk mess
    - x86/cpu: Rename original retbleed methods
    - x86/cpu: Rename srso_(.*)_alias to srso_alias_\1
    - x86/cpu: Cleanup the untrain mess
    - x86/srso: Explain the untraining sequences a bit more
    - objtool/x86: Fixup frame-pointer vs rethunk
    - x86/static_call: Fix __static_call_fixup()
    - x86/srso: Correct the mitigation status when SMT is disabled
    - Ubuntu: [Config]: enable Speculative Return Stack Overflow mitigation
  * Please enable Renesas RZ platform serial installer (LP: #2022361)
    - [Config] enable hihope RZ/G2M serial console
    - [Config] Mark sh-sci as built-in
  * dGPU cannot resume because system firmware stuck in IPCS method
    (LP: #2021572)
    - drm/i915/tc: Abort DP AUX transfer on a disconnected TC port
    - drm/i915/tc: switch to intel_de_* register accessors in display c...

This bug is awaiting verification that the linux-azure/6.2.0-1015.15 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-lunar-linux-azure' to 'verification-done-lunar-linux-azure'. If the problem still exists, change the tag 'verification-needed-lunar-linux-azure' to 'verification-failed-lunar-linux-azure'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!