[23.04 FEAT] [GNU2223] glibc - Influence hwcaps/stfle via GLIBC_TUNABLES glibc.cpu.hwcaps

This seems reasonable to add as an HWE feature, I'll include the patch in my next upload to lunar.

I'm not opposed to backporting it further down in the LTS SRUs, but I'd like a test plan a bit more involved than the `--list-tunables` if you don't mind?

------- Comment From <email address hidden> 2023-02-21 07:31 EDT-------
@Simon:
thanks for uploading this to Lunar.
Regarding our request for a backport / SRU in Jammy, this is the more detailed information you probably need. Just let us know if something is still missing.

SRU Justification:
==================
[Business Case]
The Eclipse OpenJ9 project is developing an "InstantOn" technology that leverages Linux's Checkpoint/Restore In Userspace (CRIU) feature. By restoring the JVM process from an existing CRIU snapshot, application can start-up up to 18x faster, without any compromises to throughput or JVM capabilities. This technology is ideal for serverless computing and scaling micro services in containerized deployments.
IBM WebSphere Liberty is one of the first exploiters of OpenJ9 InstantOn technology and have measured 10x faster start-up. In a 2021 survey with Java customers on IBM Z, 80% of users ranked initial start-up / ramp-up performance as an important metric. Other Java-based middleware and libraries can also take advantage of this technology for the significant start-up improvements. This GLIBC patch addresses a functional portability issue that is encountered when users create a container image with a CRIU snapshot on a newer hardware level and deploy this image against older hardware. GLIBC caches hardware facilities, which may not be available on the older deployment target, resulting in potential illegal instructions. With GLIBC_TUNABLES fixed to influence hwcaps and stfle bits on s390x, it allows the snapshot environment to mimic older hardware levels for the snapshot.

[Impact]
* This in a hardware enablement SRU, and mainly adds support for CryptoExpress 8S adapters to the s390-tools package.
* With that the new options 'show_serialnumbers', '--accelonly', '--ccaonly' and '--ep11only' are introduced to the lszcrypt tool.
* In addition lszcrypt now supports the checkstop state of a crypto card, that is provided by the 'chkstop' attribute in the sysfs of newer kernels.
* And lszcrypt now shows the AP bus msg size limit capability, which is needed for new adapter cards.
* New codes for zcryptstats are needed as well.

[Test Plan]
* Prepare an IBM z16 LPAR with Ubuntu 22.04 (incl. this patch) that has an CryptoExpress 8S adapter attached to it and at least one crypto domain online and available.
* Call 'lszcrypt -V' and check the 2nd column called 'type' and the last column called 'driver'.
* If both have entries that start with "cex8..." then the new CryptoExpress 8S driver is active and the new card is detected and can be used (and the new features exploited).
* If the driver listed there is older than 'cex8', than the new card is probably detected as an older type and it runs in toleration mode only.
* Try and test the new options.
* Run zcryptstats and with that make use of the new codes (which actually means add CEX8S support for zcryptstats).
* And finally extending lszcrypt's capabilities and make it aware of CEX8S.

[Where problems could occur]
* The new declarations, initializations or the scan for the serial numbers of the devices could fail, which would lead to a non-working or even erroneous new '-s' option.
* T...

------- Comment From <email address hidden> 2023-02-21 10:17 EDT-------
@Simon:
please ignore the SRU justification in my last comment - that was a copy/paste error on my end. Sorry about the confusion.
(You might have already wondered why the SRU justification - except the business case - does not fit at all to the glibc fix we are talking about in this bug / LP item.)

Please find below the CORRECTED VERSION of my comment:

SRU Justification:
==================
[Business Case]
The Eclipse OpenJ9 project is developing an "InstantOn" technology that leverages Linux's Checkpoint/Restore In Userspace (CRIU) feature. By restoring the JVM process from an existing CRIU snapshot, application can start-up up to 18x faster, without any compromises to throughput or JVM capabilities. This technology is ideal for serverless computing and scaling micro services in containerized deployments.
IBM WebSphere Liberty is one of the first exploiters of OpenJ9 InstantOn technology and have measured 10x faster start-up. In a 2021 survey with Java customers on IBM Z, 80% of users ranked initial start-up / ramp-up performance as an important metric. Other Java-based middleware and libraries can also take advantage of this technology for the significant start-up improvements. This GLIBC patch addresses a functional portability issue that is encountered when users create a container image with a CRIU snapshot on a newer hardware level and deploy this image against older hardware. GLIBC caches hardware facilities, which may not be available on the older deployment target, resulting in potential illegal instructions. With GLIBC_TUNABLES fixed to influence hwcaps and stfle bits on s390x, it allows the snapshot environment to mimic older hardware levels for the snapshot.

[Impact]
* For IFUNC'ed functions within glibc, the decision can now be influenced with GLIBC_TUNABLE: glibc.cpu.hwcaps
* This can be used as test vehicle
* This is a prerequisite for Eclipse OpenJ9 projects "InstantOn" technology which speeds up start-up times by factors.

[Test Plan]
* Prepare an IBM z15 LPAR with Ubuntu (incl. this patch)
* Check if patch is applied by running command "/usr/lib/s390x-linux-gnu/ld64.so.1 --list-tunables" and check if the new tunable "glibc.cpu.hwcaps" is listed
* Prepare a small test-program calling IFUNC-optimized functions: memmem, strstr and memmove.
* Start a gdb session and break at __memmem_vx, __memmem_arch13, __strstr_vx, __strstr_arch13, __memmove_z13 and __memmove_arch13. Run the program and you should always stop in the arch13=z15 functions. Now use gdb-command "set environment GLIBC_TUNABLES glibc.cpu.hwcaps=z13" and run again. Now you should always stop in the vx=z13 functions.

This bug was fixed in the package glibc - 2.37-0ubuntu2

---------------
glibc (2.37-0ubuntu2) lunar; urgency=medium

  * d/p/u/lp2007599*.patch: add tunables for s390x (LP: #2007599)
  * d/p/any/local-ldso-disable-hwcap: fix compilation error (LP: #2006485)
  * d/sysdeps/arm64.mk: enable Memory Tagging Extension (MTE) checking on arm64
    (LP: #2006739)
  * d/control: declare a Breaks on older cyrus-imapd (LP: #2011326)
  * d/control: Fix missing version bumps that could cause issues on upgrades
  * Cherry-pick patches from upstream maintenance branch:
    - 0001-cdefs-Limit-definition-of-fortification-macros.patch
    - 0002-LoongArch-Add-new-relocation-types.patch
    - 0003-Use-64-bit-time_t-interfaces-in-strftime-and-strptim.patch
    - 0004-Account-for-grouping-in-printf-width-bug-30068.patch
    - 0005-NEWS-Document-CVE-2023-25139.patch
    - 0006-elf-Smoke-test-ldconfig-p-against-system-etc-ld.so.c.patch
    - 0007-stdlib-Undo-post-review-change-to-16adc58e73f3-BZ-27.patch
    - 0008-elf-Restore-ldconfig-libc6-implicit-soname-logic-BZ-.patch

 -- Simon Chopin <email address hidden> Thu, 16 Mar 2023 09:44:01 +0100

------- Comment From <email address hidden> 2023-03-27 08:07 EDT-------
I've updated to libc6-2.37-0ubuntu2 and was able to influence memmove, strstr, memmem and __to_utf8_loop in UTF8_UTF16_Z9.so iconv module. Thanks for integration.

------- Comment From <email address hidden> 2023-03-29 19:36 EDT-------
Hello @Simon, thanks for getting this into Lunar.
Regarding the backport to Jammy, is there anything in addition that we need to provide? Thanks

This feature isn't planned to be backported to Kinetic.

------- Comment From <email address hidden> 2023-06-06 10:32 EDT-------
Hi Simon, yes, this makes perfect sense with Kinetic reaching its EOL in just a couple of weeks.

Hello bugproxy, or anyone else affected,

Accepted glibc into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/glibc/2.35-0ubuntu3.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted glibc (2.35-0ubuntu3.2) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

adsys/0.9.2~22.04.1 (armhf)
bbmap/38.95+dfsg-1 (armhf)
boost1.74/1.74.0-14ubuntu3 (armhf)
breezy/3.2.1+bzr7585-1build1 (armhf)
casync/2+20201210-1build1 (ppc64el)
cmake/3.22.1-1ubuntu1.22.04.1 (amd64, arm64, armhf)
cython/0.29.28-1ubuntu3 (i386)
dbus-test-runner/16.10.0~bzr100+repack1-4.1 (armhf)
exim4/4.95-4ubuntu2.2 (ppc64el)
fwupd/1.7.9-1~22.04.1 (armhf)
glib-networking/2.72.0-1 (i386)
golang-github-influxdata-tail/1.0.0+git20180327.c434825-4 (ppc64el)
golang-gogoprotobuf/1.3.2-1 (arm64)
grubzfs-testsuite/0.4.15build1 (amd64)
gtk+3.0/3.24.33-1ubuntu2 (i386)
gyoto/1.4.4-7build1 (amd64, arm64, s390x)
ksh93u+m/1.0.0~beta.2-1 (ppc64el)
libassuan/2.5.5-1build1 (amd64)
libflame/5.2.0-3ubuntu3 (s390x)
libite/2.5.1-1 (s390x)
linux-aws-5.19/5.19.0-1027.28~22.04.1 (amd64, arm64)
linux-azure-5.19/5.19.0-1027.30~22.04.2 (amd64)
linux-gcp-5.19/5.19.0-1026.28~22.04.1 (arm64)
linux-gke/5.15.0-1034.39 (arm64)
linux-hwe-5.19/5.19.0-44.45~22.04.1 (amd64)
linux-lowlatency-hwe-5.19/5.19.0-1026.27~22.04.1 (amd64)
linux-oracle-5.19/5.19.0-1025.28~22.04.1 (amd64)
mercurial/6.1.1-1ubuntu1 (amd64)
mtail/3.0.0~rc48-3 (ppc64el)
mutter/42.5-0ubuntu1 (amd64)
mypy/0.942-1ubuntu1 (armhf)
mysql-8.0/8.0.33-0ubuntu0.22.04.2 (s390x)
netgen/6.2.2006+really6.2.1905+dfsg-5build1 (armhf)
notcurses/3.0.6+dfsg.1-1 (armhf)
pango1.0/1.50.6+ds-2ubuntu1 (i386)
phcpack/2.4.85+dfsg-5build1 (arm64)
pinentry/1.1.1-1build2 (i386)
puma/5.5.2-2ubuntu2 (amd64, arm64)
pycryptodome/3.11.0+dfsg1-3build1 (i386)
pygobject/3.42.1-0ubuntu1 (i386)
python-evdev/1.4.0+dfsg-1build2 (i386)
python-llfuse/1.3.8+dfsg-2build1 (armhf)
python-lz4/3.1.3+dfsg-1build3 (i386)
python3.10/3.10.6-1~22.04.2ubuntu1.1 (arm64)
pyyaml/5.4.1-1ubuntu1 (i386)
r-bioc-rhdf5/2.38.0+dfsg-2 (amd64)
r-cran-randomfieldsutils/1.1.0-1 (armhf)
r-cran-spc/1:0.6.5-1 (armhf)
rsync/3.2.7-0ubuntu0.22.04.2 (i386)
ruby-nio4r/2.5.8-2 (amd64)
ruby-nokogiri/1.13.1+dfsg-2 (amd64)
ruby-prof/1.3.1-2build2 (amd64, ppc64el)
rustc/1.65.0+dfsg0ubuntu1-0ubuntu0.22.04.1 (arm64)
seqkit/2.1.0+ds-1 (arm64, s390x)
stunnel4/3:5.63-1build1 (i386)
swtpm/0.6.3-0ubuntu3.2 (s390x)
systemd/249.11-0ubuntu3.9 (amd64, arm64, ppc64el, s390x)
taptempo/1.4.5-1 (amd64)
texinfo/6.8-4build1 (armhf)
ubiquity/22.04.19 (amd64, arm64, armhf, ppc64el)
udisks2/2.9.4-1ubuntu2 (arm64)
utox/0.18.1-1build1 (armhf)
vlc/3.0.16-1build7 (i386)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#glibc

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

------- Comment From <email address hidden> 2023-06-12 03:52 EDT-------
Hi,
I've installed libc6:s390x (2.35-0ubuntu3.2) from jammy-proposed repository (for Ubuntu 22.04) and successfully tested the package:

$ /lib/ld64.so.1 --version
ld.so (Ubuntu GLIBC 2.35-0ubuntu3.2) stable release version 2.35.
...

$ /lib/ld64.so.1 --list-tunables | grep hwcaps
glibc.cpu.hwcaps:
=> Now this GLIBC_TUNABLES is also available on s390x.

Furthermore I've run a small test program which calls the IFUNC'ed functions memmove, memmem, strstr and UTF8_UTF16_Z9.so via iconv-conversion from UTF16BE to UTF8.
Running gdb with setting the GLIBC_TUNABLES in different flavors like this ...
gdb$ set environment GLIBC_TUNABLES glibc.cpu.hwcaps=z13
... leads to breaking in the different ifunc-variants of the mentioned functions.

Thanks.

------- Comment From <email address hidden> 2023-06-12 07:50 EDT-------
Thanks @Stefan for your successful verification on Jammy.
Setting the tags accordingly to verification-done and verification-done-jammy.

Hi, thanks for the verification!

For information, there's a chance that this caused an actual regression:

 libflame/5.2.0-3ubuntu3 (s390x)

I'm currently looking into it to try and find out more about what causes the timeout.

Hello bugproxy, or anyone else affected,

Accepted glibc into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/glibc/2.35-0ubuntu3.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted glibc (2.35-0ubuntu3.2) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

linux-aws-6.2/6.2.0-1007.7~22.04.1 (arm64)
linux-azure-6.2/6.2.0-1007.7~22.04.1 (arm64)
linux-oracle-5.19/5.19.0-1025.28~22.04.1 (arm64)
mysql-8.0/8.0.33-0ubuntu0.22.04.2 (s390x)
ubiquity/22.04.19 (amd64, arm64, armhf, ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#glibc

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

All autopkgtests for the newly accepted glibc (2.35-0ubuntu3.3) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

adsys/0.9.2~22.04.1 (arm64)
datefudge/1.24 (arm64)
dbus/1.12.20-2ubuntu4.1 (armhf)
dpdk/21.11.3-0ubuntu0.22.04.1 (arm64)
ganeti/3.0.2-1ubuntu1 (armhf)
gcc-9/9.5.0-1ubuntu1~22.04 (armhf)
gjs/1.72.4-0ubuntu0.22.04.1 (amd64)
golang-github-canonical-go-dqlite/1.10.1-1 (arm64)
golang-github-influxdata-tail/1.0.0+git20180327.c434825-4 (amd64)
golang-github-xenolf-lego/4.1.3-3ubuntu1.22.04.1 (armhf)
golang-gogoprotobuf/1.3.2-1 (amd64, s390x)
google-osconfig-agent/20230504.00-0ubuntu1~22.04.0 (armhf)
gvfs/1.48.2-0ubuntu1 (amd64)
hilive/2.0a-3build3 (arm64)
hyphy/2.5.36+dfsg-1 (amd64)
kmediaplayer/5.92.0-0ubuntu1 (armhf)
libclass-methodmaker-perl/2.24-2build2 (armhf)
libflame/5.2.0-3ubuntu3 (arm64)
libimage-sane-perl/5-1build3 (s390x)
libuev/2.4.0-1.1 (s390x)
linux-aws-6.2/6.2.0-1009.9~22.04.2 (arm64)
linux-azure-5.19/5.19.0-1027.30~22.04.2 (arm64)
linux-azure-6.2/6.2.0-1009.9~22.04.2 (arm64)
linux-gcp-5.19/5.19.0-1030.32~22.04.1 (arm64)
linux-gke/5.15.0-1039.44 (arm64)
linux-hwe-5.19/5.19.0-50.50 (arm64)
linux-lowlatency/5.15.0-79.88 (arm64)
linux-nvidia-6.2/6.2.0-1006.6~22.04.2 (arm64)
linux-nvidia-tegra/5.15.0-1015.15 (arm64)
linux-oem-5.17/5.17.0-1035.36 (amd64)
linux-oracle-5.19/5.19.0-1027.30 (arm64)
linux-xilinx-zynqmp/5.15.0-1023.27 (arm64)
log4cxx/0.12.1-4 (armhf)
netplan.io/0.105-0ubuntu2~22.04.3 (arm64)
notcurses/3.0.6+dfsg.1-1 (armhf)
php-luasandbox/4.0.2-3build1 (ppc64el)
prometheus/2.31.2+ds1-1ubuntu1.22.04.2 (s390x)
qcustomplot/2.0.1+dfsg1-5 (armhf)
r-cran-mice/3.14.0-1 (armhf)
r-cran-randomfields/3.3.14-1 (armhf)
r-cran-sys/3.4-1 (s390x)
ruby-rblineprof/0.3.7-2build3 (arm64)
rustc/1.66.1+dfsg0ubuntu1-0ubuntu0.22.04.1 (arm64)
samba/2:4.15.13+dfsg-0ubuntu1.2 (s390x)
stunnel4/3:5.63-1build1 (ppc64el)
swtpm/0.6.3-0ubuntu3.2 (arm64)
tmux/3.2a-4ubuntu0.2 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#glibc

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

I had a look at 2.35-0ubuntu3.3 and from all what I could check it looks ok.
But hoping that @Stefan can double-check as well.

------- Comment From <email address hidden> 2023-08-09 05:33 EDT-------
Hi Frank,

Sorry, I have not recognized that I should also check 2.35-0ubuntu3.3 package after my verification of 2.35-0ubuntu3.2 (see some comments above).
Nevertheless, I've installed the update and checked again:

$ /lib/ld64.so.1 --version
ld.so (Ubuntu GLIBC 2.35-0ubuntu3.3) stable release version 2.35.
...

$ /lib/ld64.so.1 --list-tunables | grep hwcaps
glibc.cpu.hwcaps:
=> Now this GLIBC_TUNABLES is also available on s390x.
When setting the tunables, you also see the value:
$ GLIBC_TUNABLES="glibc.cpu.hwcaps=z13" /lib/ld64.so.1 --list-tunables | grep hwcaps
glibc.cpu.hwcaps: z13

Furthermore I've run a small test program which calls the IFUNC'ed functions memmove, memmem, strstr and UTF8_UTF16_Z9.so via iconv-conversion from UTF16BE to UTF8.
Running gdb with setting the GLIBC_TUNABLES in different flavors like this ...
gdb$ set environment GLIBC_TUNABLES glibc.cpu.hwcaps=z13
... leads to breaking in the different ifunc-variants of the mentioned functions.

Thanks again for integration.

This bug was fixed in the package glibc - 2.35-0ubuntu3.3

---------------
glibc (2.35-0ubuntu3.3) jammy; urgency=medium

  * Drop SVE patches due to kernal-related performance regression
  * Fix the armhf stripping exception for ld.so (LP: #1927192)

glibc (2.35-0ubuntu3.2) jammy; urgency=medium

  * d/rules.d/debhelper.mk: fix permissions of libc.so (LP: #1989082)
  * Cherry-picks from upstream:
    - d/p/lp1999551/*: arm64 memcpy optimization (LP: #1999551)
    - d/p/lp1995362*.patch: Fix ldd segfault with missing libs (LP: #1995362)
    - d/p/lp2007796*: Fix missing cancellation point in pthread (LP: #2007796)
    - d/p/lp2007599*: add new tunables for s390x (LP: #2007599)
    - d/p/lp2011421/*: Fix crash on TDX-enabled platforms (LP: #2011421)
    - d/p/lp1992159*: Fix socket.h headers for non-GNU compilers (LP: #1992159)

 -- Simon Chopin <email address hidden> Wed, 26 Jul 2023 10:27:54 +0200

The verification of the Stable Release Update for glibc has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.