NFS deathlock with last Kernel 5.4.0-144.161 and 5.15.0-67.74

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 2009325

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

What I have noticed now only occurs at NFS clients, not with the NFS server.

NFS Client Server that create a large amount of simultaneous accesses have the problem.
NFS Client Server that create few accesses do not have the problem.

We have now booted our web servers on an older kernel version and the problem was gone.
Even if the NFS Server servers have the current kernel.

Same symptoms here.

In our case, five 22.04LTS servers with the 5.15.0-60-generic kernel running a web service, all five with an NFS-mounted volume.

After updating some of them with the 5.15.0-67-generic kernel, the traffic on the interface with which the NFS server is accessed increases for no apparent reason. Both output and input packets per second go from about 2k/s to a continuous 12k/s. The context switch rate per second increases by the same ratio. As a result, operations against the NFS volume are slowed down and the user experience is noticeably worse. In the meantime, systems running 5.15.0-60-generic are working as usual.

As in the case of the reporter, booting the systems with 5.15.0-60-generic returns us to the original state.

Hello,

We have the same behavior on 22.04 and 18.04 servers.

This behavior affect us on different on many of our customers web servers with different platforms AWS, Azure and Nutanix (KVM virt) VMs.

The used kernel was :

    linux-image-5.15.0-67-generic
    linux-image-5.4.0-1097-aws
    linux-image-5.4.0-1104-azure

We have also now booted our web servers on an older kernel version and the problem was mostly gone.

We have also tried with the 5.15.0-68-generic on a test server on the https://launchpad.net/%7Ecanonical-kernel-team/+archive/ubuntu/ppa this issue affect also this next kernel.

Best Regards,

We see this problem too on focal clients with ans without HWE.
The server seems to be not affected. Downgrading the clients to 5.15.0-60 or 5.4.0-139 solves the problem here.

Best regards.

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2003053/comments/21

We are seeing this issue with NFSv3. NFS ACCESS operations increased quadratically after upgrading to 5.4.0-144 and 5.15.0-67. Downgrading to either 5.4.0-139-generic or 5.15.0-60-generic resolves the issue again.

The commit you say that you have reverted references NFSv4, which we are not using. Therefore, combined with the massive increase in ACCESS operations, I am inclined to believe that the issue is related to the file access cache-related commits and not the retry lock commit.

In our case (https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2009325/comments/25), we also use NFSv3, and like Eirik Rye, we've seen this massive increase in ACCESS operations, from averages of 700/s (approx.) to peaks of 27000/s with 5.15.0-67.

In case it helps.

If the symptom observed is a massive increase in ACCESS operations,
it is more likely that the root cause is related to file access cache-related commits.
I will build a test kernel without those commits for testing.
Thanks for your help.

I have prepared a test kernel that reverts the file access cache-related commits.
Please follow these steps and give it a try:

1. sudo add-apt-repository ppa:chengendu/jammy-nfs-test
2. sudo apt update
3. sudo apt install linux-headers-5.15.0-67-generic linux-image-unsigned-5.15.0-67-generic linux-modules-5.15.0-67-generic linux-modules-extra-5.15.0-67-generic
4. reboot
5. uname -rv
   5.15.0-67-generic #74+test20230307b2h2cbb6062f8eb-Ubuntu SMP Tue Mar 7 23:16:16 UT

In the meantime, we will also work on figuring out the error.

I have sent a patch to fix the upstream error (https://lkml.org/lkml/2023/3/8/107),
which is intended to resolve the issue of a massive increase in ACCESS operations.
I have also prepared a test kernel that includes this patch and would appreciate it if you could use it for testing.

To install the test kernel, please follow these steps:

1. Run the following command: sudo add-apt-repository ppa:chengendu/jammy-nfs-patch
2. Update the packages: sudo apt update
3. Install the kernel packages: sudo apt install linux-headers-5.15.0-67-generic linux-image-unsigned-5.15.0-67-generic linux-modules-5.15.0-67-generic linux-modules-extra-5.15.0-67-generic
4. Reboot your system.
5. To check the installed kernel version, run the command 'uname -rv', which should output '5.15.0-67-generic #74+test20230308b0h1a13a615ee32-Ubuntu SMP Wed Mar 8 09:24:49 UT'.

Hello ChengEn, Du ,

Thank you for the quick response and fix .

We have quickly tried the test kernel ( 5.15.0-67-generic #74+test20230307b2h2cbb6062f8eb-Ubuntu ) on a test webserver (ubuntu 22.04.2 lts ) with an impacted configuration with nfs and db.

The test kernel seems for us to fix the problem, as we have returned to a normal behavior as the previous kernel (5.15.0-60-generic).

Best regards,

The first test kernel (5.15.0-67-generic #74+test20230307b2h2cbb6062f8eb-Ubuntu) merely reverts file access cache-related commits.
However, the second test kernel (5.15.0-67-generic #74+test20230308b0h1a13a615ee32-Ubuntu) truly identifies and resolves the problem.

The file access cache, which belongs to the user, will be cleared once the user logs out and logs in again.
After that, all behavior will be the same as usual.
We would appreciate it if you could use the second test kernel for testing and confirm that it has resolved the performance issue.

Hello ChengEn, Du ,
Thanks for the test Kernel.
I can no longer find a problem with this Kernel and NFS client.
I also think that the kernel has fixed the problem.

Best regards,

Hello ChengEn, Du ,

I have tested also the 2nd.

Same as the previous test kernel , the 2nd (5.15.0-67-generic #74+test20230308b0h1a13a615ee32-Ubuntu) seems to work as intended .

Same as Alexander , this patch seems to fix the problem.

Best regards,

Hi,

I believe I have seen the same problem on 5.19.0-35-generic

It was not present on 5.19.0-32-generic

And the nfs_access_add_cache function looks similar to 5.15.0 so the same fix will probably work.

A problematic patch has been identified in a bug report, which can lead to issues.
The bug report can be found at https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2003053.

This patch can affect several kernels, including:

linux/4.15.0-206.217 for bionic
linux/5.4.0-144.161 for focal
linux/5.15.0-66.73 for jammy
linux/5.19.0-35.36 for kinetic

The solution for this issue is the same for all the kernels mentioned above.
We are currently waiting for upstream to merge the patch.

We apologize for any inconvenience caused.

This bug is awaiting verification that the linux-gcp/5.15.0-1032.40 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-raspi/5.19.0-1016.23 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-kinetic' to 'verification-done-kinetic'. If the problem still exists, change the tag 'verification-needed-kinetic' to 'verification-failed-kinetic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

For Jammy

5.19.0-38-generic
and
5.15.0-70-generic

Fixed my problems with nfs client performance.

Thanks!

Should I change the tags the ubuntu-kernel-bot is talking about? verification-needed-jammy to verification-done-jammy?

I'm assuming not but I wouldn't want to see this bug closed and the fix dropped.

This bug was fixed in the package linux - 6.2.0-19.19

---------------
linux (6.2.0-19.19) lunar; urgency=medium

  * lunar/linux: 6.2.0-19.19 -proposed tracker (LP: #2012488)

  * Neuter signing tarballs (LP: #2012776)
    - [Packaging] neuter the signing tarball

  * LSM stacking and AppArmor refresh for 6.2 kernel (LP: #2012136)
    - Revert "UBUNTU: [Config] define CONFIG_SECURITY_APPARMOR_RESTRICT_USERNS"
    - Revert "UBUNTU: SAUCE: apparmor: add user namespace creation mediation"
    - Revert "UBUNTU: SAUCE: apparmor: Add fine grained mediation of posix
      mqueues"
    - Revert "UBUNTU: SAUCE: Revert "apparmor: make __aa_path_perm() static""
    - Revert "UBUNTU: SAUCE: LSM: Specify which LSM to display (using struct cred
      as input)"
    - Revert "UBUNTU: SAUCE: apparmor: Fix build error, make sk parameter const"
    - Revert "UBUNTU: SAUCE: LSM: Use lsmblob in smk_netlbl_mls()"
    - Revert "UBUNTU: SAUCE: LSM: change ima_read_file() to use lsmblob"
    - Revert "UBUNTU: SAUCE: apparmor: rename kzfree() to kfree_sensitive()"
    - Revert "UBUNTU: SAUCE: AppArmor: Remove the exclusive flag"
    - Revert "UBUNTU: SAUCE: LSM: Add /proc attr entry for full LSM context"
    - Revert "UBUNTU: SAUCE: Audit: Fix incorrect static inline function
      declration."
    - Revert "UBUNTU: SAUCE: Audit: Fix for missing NULL check"
    - Revert "UBUNTU: SAUCE: Audit: Add a new record for multiple object LSM
      attributes"
    - Revert "UBUNTU: SAUCE: Audit: Add new record for multiple process LSM
      attributes"
    - Revert "UBUNTU: SAUCE: NET: Store LSM netlabel data in a lsmblob"
    - Revert "UBUNTU: SAUCE: LSM: security_secid_to_secctx in netlink netfilter"
    - Revert "UBUNTU: SAUCE: LSM: Use lsmcontext in security_inode_getsecctx"
    - Revert "UBUNTU: SAUCE: LSM: Use lsmcontext in security_secid_to_secctx"
    - Revert "UBUNTU: SAUCE: LSM: Ensure the correct LSM context releaser"
    - Revert "UBUNTU: SAUCE: LSM: Specify which LSM to display"
    - Revert "UBUNTU: SAUCE: IMA: Change internal interfaces to use lsmblobs"
    - Revert "UBUNTU: SAUCE: LSM: Use lsmblob in security_cred_getsecid"
    - Revert "UBUNTU: SAUCE: LSM: Use lsmblob in security_inode_getsecid"
    - Revert "UBUNTU: SAUCE: LSM: Use lsmblob in security_task_getsecid"
    - Revert "UBUNTU: SAUCE: LSM: Use lsmblob in security_ipc_getsecid"
    - Revert "UBUNTU: SAUCE: LSM: Use lsmblob in security_secid_to_secctx"
    - Revert "UBUNTU: SAUCE: LSM: Use lsmblob in security_secctx_to_secid"
    - Revert "UBUNTU: SAUCE: net: Prepare UDS for security module stacking"
    - Revert "UBUNTU: SAUCE: LSM: Use lsmblob in security_kernel_act_as"
    - Revert "UBUNTU: SAUCE: LSM: Use lsmblob in security_audit_rule_match"
    - Revert "UBUNTU: SAUCE: LSM: Create and manage the lsmblob data structure."
    - Revert "UBUNTU: SAUCE: LSM: Infrastructure management of the sock security"
    - Revert "UBUNTU: SAUCE: apparmor: LSM stacking: switch from SK_CTX() to
      aa_sock()"
    - Revert "UBUNTU: SAUCE: apparmor: rename aa_sock() to aa_unix_sk()"
    - Revert "UBUNTU: SAUCE: apparmor: disable showing the mode as part of a secid
      to secctx"
    - Revert "UBUNTU: SAUCE: app...

This bug is awaiting verification that the linux-kvm/4.15.0-1138.143 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-raspi/5.4.0-1083.94 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-raspi/5.15.0-1027.29 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-oracle/5.19.0-1020.23 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-kinetic' to 'verification-done-kinetic'. If the problem still exists, change the tag 'verification-needed-kinetic' to 'verification-failed-kinetic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-gkeop/5.15.0-1018.23 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-ibm/5.4.0-1047.52 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-gcp/5.19.0-1020.22 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-kinetic' to 'verification-done-kinetic'. If the problem still exists, change the tag 'verification-needed-kinetic' to 'verification-failed-kinetic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-oracle/4.15.0-1117.128 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-raspi2/4.15.0-1130.138 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-ibm/5.15.0-1028.31 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-oracle/5.15.0-1033.39 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Hi ChengEn, Du,

Do we have another issue opened to address the NFS mechanism and the performance impacts it has on the kernel? We cannot move to the new kernel currently because of these issues so I just want to make sure we are continuing to push a solution forward. Thank you!

This bug is awaiting verification that the linux-riscv/5.19.0-1016.17 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-kinetic' to 'verification-done-kinetic'. If the problem still exists, change the tag 'verification-needed-kinetic' to 'verification-failed-kinetic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-starfive/5.19.0-1015.17 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-kinetic' to 'verification-done-kinetic'. If the problem still exists, change the tag 'verification-needed-kinetic' to 'verification-failed-kinetic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-allwinner/5.19.0-1010.10 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-kinetic' to 'verification-done-kinetic'. If the problem still exists, change the tag 'verification-needed-kinetic' to 'verification-failed-kinetic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-gke/5.15.0-1031.36 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Hi @krglosse,

We have opened a bug to address the performance issue related to NFS.
You can find more details about the bug at this link: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2015827.

We have also sent a proposal to upstream for their consideration, but unfortunately, we have not received a response yet.
As a result, we may need to send a patch directly and await upstream's feedback and judgement.

This bug is awaiting verification that the linux-kvm/5.19.0-1021.22 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-kinetic' to 'verification-done-kinetic'. If the problem still exists, change the tag 'verification-needed-kinetic' to 'verification-failed-kinetic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Hi @chengendu,

Thanks.

I have posted a comment to the effect of backporting changes in new (and at the time, unreleased and not yet stable) Linux kernels to LTS releases. There is no need to await upstream's feedback and judgment, this change belongs to Linux 6.2, not old LTS kernels.

This bug is awaiting verification that the linux-ibm/5.19.0-1020.22 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-kinetic' to 'verification-done-kinetic'. If the problem still exists, change the tag 'verification-needed-kinetic' to 'verification-failed-kinetic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-gkeop/5.4.0-1067.71 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-gcp-4.15/4.15.0-1148.164 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-gke/5.4.0-1097.104 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-nvidia-5.19/5.19.0-1009.9 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug was fixed in the package linux-aws - 6.2.0-1003.3

---------------
linux-aws (6.2.0-1003.3) lunar; urgency=medium

  * lunar/linux-aws: 6.2.0-1003.3 -proposed tracker (LP: #2015430)

  * Packaging resync (LP: #1786013)
    - debian/dkms-versions -- update from kernel-versions (main/master)

  * Miscellaneous Ubuntu changes
    - [Config] aws: update annotations after rebase to the latest 6.2

  [ Ubuntu: 6.2.0-20.20 ]

  * lunar/linux: 6.2.0-20.20 -proposed tracker (LP: #2015429)
  * Packaging resync (LP: #1786013)
    - debian/dkms-versions -- update from kernel-versions (main/master)
  * FTBFS with different dkms or when makeflags are set (LP: #2015361)
    - [Packaging] FTBFS with different dkms or when makeflags are set
  * expoline.o is packaged unconditionally for s390x (LP: #2013209)
    - [Packaging] Copy expoline.o only when produced by the build
  * net:l2tp.sh failure with lunar:linux 6.2 (LP: #2013014)
    - SAUCE: l2tp: generate correct module alias strings
  * Miscellaneous Ubuntu changes
    - [Packaging] annotations: prevent duplicate include lines

  [ Ubuntu: 6.2.0-19.19 ]

  * lunar/linux: 6.2.0-19.19 -proposed tracker (LP: #2012488)
  * Neuter signing tarballs (LP: #2012776)
    - [Packaging] neuter the signing tarball
  * LSM stacking and AppArmor refresh for 6.2 kernel (LP: #2012136)
    - Revert "UBUNTU: [Config] define CONFIG_SECURITY_APPARMOR_RESTRICT_USERNS"
    - Revert "UBUNTU: SAUCE: apparmor: add user namespace creation mediation"
    - Revert "UBUNTU: SAUCE: apparmor: Add fine grained mediation of posix
      mqueues"
    - Revert "UBUNTU: SAUCE: Revert "apparmor: make __aa_path_perm() static""
    - Revert "UBUNTU: SAUCE: LSM: Specify which LSM to display (using struct cred
      as input)"
    - Revert "UBUNTU: SAUCE: apparmor: Fix build error, make sk parameter const"
    - Revert "UBUNTU: SAUCE: LSM: Use lsmblob in smk_netlbl_mls()"
    - Revert "UBUNTU: SAUCE: LSM: change ima_read_file() to use lsmblob"
    - Revert "UBUNTU: SAUCE: apparmor: rename kzfree() to kfree_sensitive()"
    - Revert "UBUNTU: SAUCE: AppArmor: Remove the exclusive flag"
    - Revert "UBUNTU: SAUCE: LSM: Add /proc attr entry for full LSM context"
    - Revert "UBUNTU: SAUCE: Audit: Fix incorrect static inline function
      declration."
    - Revert "UBUNTU: SAUCE: Audit: Fix for missing NULL check"
    - Revert "UBUNTU: SAUCE: Audit: Add a new record for multiple object LSM
      attributes"
    - Revert "UBUNTU: SAUCE: Audit: Add new record for multiple process LSM
      attributes"
    - Revert "UBUNTU: SAUCE: NET: Store LSM netlabel data in a lsmblob"
    - Revert "UBUNTU: SAUCE: LSM: security_secid_to_secctx in netlink netfilter"
    - Revert "UBUNTU: SAUCE: LSM: Use lsmcontext in security_inode_getsecctx"
    - Revert "UBUNTU: SAUCE: LSM: Use lsmcontext in security_secid_to_secctx"
    - Revert "UBUNTU: SAUCE: LSM: Ensure the correct LSM context releaser"
    - Revert "UBUNTU: SAUCE: LSM: Specify which LSM to display"
    - Revert "UBUNTU: SAUCE: IMA: Change internal interfaces to use lsmblobs"
    - Revert "UBUNTU: SAUCE: LSM: Use lsmblob in security_cred_getsecid"
    - Revert "UBUNTU: SAUCE: LSM: Use lsmblob in secur...

Judging from previous comments this change was verified.
Verification is requested again because the initial change was pushed in the last cycle and non aws derivatives that were not respun introduced this change in this cycle.
Since this is blocking the release of a lot of kernels for cycle 20230320, I will tag verification-done.

This bug is awaiting verification that the linux-intel-iotg/5.15.0-1028.33 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-bluefield/5.4.0-1061.67 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-riscv-5.15/5.15.0-1031.35~20.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-intel-iotg-5.15/5.15.0-1029.34~20.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-xilinx-zynqmp/5.15.0-1021.25 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Hi,

I can also confirm that kernel 5.4.0-149.166 brings CPU on the 20.04.6 NFS server back from 90-100% to normal, but i observed the following.

After reboot of the NFS server all CPU are going quickly to 100% load, to get load back to normal I had to do a `exportfs -ra' else CPU stays even with the new kernel above 90%

This bug is awaiting verification that the linux-azure/6.2.0-1009.9 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-lunar' to 'verification-done-lunar'. If the problem still exists, change the tag 'verification-needed-lunar' to 'verification-failed-lunar'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-aws-5.15/5.15.0-1046.51~20.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal-linux-aws-5.15' to 'verification-done-focal-linux-aws-5.15'. If the problem still exists, change the tag 'verification-needed-focal-linux-aws-5.15' to 'verification-failed-focal-linux-aws-5.15'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the linux-mtk/5.15.0-1030.34 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-mtk' to 'verification-done-jammy-linux-mtk'. If the problem still exists, change the tag 'verification-needed-jammy-linux-mtk' to 'verification-failed-jammy-linux-mtk'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!