GRUB menu loading failure via HTTP Boot on BlueField

Reassigning to boot engineering. BlueField seems to be arm64? I reassigned this to grub2-unsigned accordingly.

This is not something you should be touching yourselves, really.

Bluefield is arm64, and the patch has undergone successful testing.
I've already initiated a bug report in grub2-unsigned. Should this be marked as a duplicate?
The bug report is available at: https://bugs.launchpad.net/ubuntu/+source/grub2-unsigned/+bug/2043085.
Additionally, I'm inquiring whether I should provide the debdiff for this issue or await action from the boot engineering.

Mate is merging the fixes with some others and we should have an update available in the next days.

@chengendu https://code.launchpad.net/~mkukri/grub/+git/grub/+merge/455398 (sorry first MP had a typo'd branch name)

Marking mantic as invalid because it doesn't have the http code yet, it may get it back with noble, in which case the bug will become fix comitted/released again to make it clear that the re-added network patch set has the fix.

This bug was fixed in the package grub2-unsigned - 2.12~rc1-12ubuntu2

---------------
grub2-unsigned (2.12~rc1-12ubuntu2) noble; urgency=medium

  * Merge from Debian unstable; remaining changes:
    - Add Ubuntu sbat data
    - build-efi-images: do not produce -installer.efi.signed. LP: 1863994
    - grub-common: Install canonical-uefi-ca.crt
    - Check signatures
    - Support installing to multiple ESP (LP: 1871821)
    - Disable various bits on i386
    - Split out unsigned artefacts into grub2-unsigned
    - Vcs-Git: Point to ubuntu packaging branch
    - Relax dependencies on grub-common and grub2-common
    - grub-pc: Avoid the possibility of breaking grub on SRU update due
      to ABI change
    - UBUNTU: Default timeout changes
    - Revert "Add jfs module to signed UEFI images. Closes: #950959"
    - Revert "Add f2fs module to signed UEFI images"
    - Install grub-initrd-fallback.service again
    - Build using -O1 on s390x to avoid misoptimization
    - grub-check-signatures: Support gzip compressed kernels (LP: #1954683)
    - grub-multi-install: Reset partition type between partitions (LP: #1997795)
    - Drop i386 from grub-efi-amd64* (LP: #2020907)
    - Turn depends on grub-efi-amd64/arm64 unversioned
    - forward port fix for LP: #1926748
    - Make the grub2/no_efi_extra_removable setting work correctly
    - Forward port the fix for LP: #1930742 and make it conditional (xenial/bionic only)
    - Build grub2-unsigned packages with xz compression
    - Revert: "Have -bin packages Break pre-2.12 -signed packages.", this is not
      compatible with our versioning schemes.
    - Install a /usr/lib/grub/grub-sort-version and use that to sort versions as
      it respects GRUB_FLAVOUR_ORDER. Depend on python3 to do so.
    - rules: Add DPKG_BUILDPACKAGE_OPTIONS to generate-grub2-unsigned
    - Replaced patches:
      - installe-signed.patched
      - grub-install-extra-removable.patch
      - grub-install-removable-shim.patch
    - Added patches:
      + rhboot-f34-dont-use-int-for-efi-status.patch
      + rhboot-f34-make-exit-take-a-return-code.patch
      + suse-grub.texi-add-net_bootp6-document.patch
      + ubuntu-add-devicetree-command-support.patch
      + ubuntu-add-initrd-less-boot-fallback.patch
      + ubuntu-add-initrd-less-boot-messages.patch
      + ubuntu-boot-from-multipath-dependent-symlink.patch
      + ubuntu-dont-verify-loopback-images.patch
      + ubuntu-fix-lzma-decompressor-objcopy.patch
      + ubuntu-grub-install-extra-removable.patch
      + ubuntu-install-signed.patch
      + ubuntu-mkconfig-leave-breadcrumbs.patch
      + ubuntu-os-prober-auto.patch
      + ubuntu-recovery-dis_ucode_ldr.patch
      + ubuntu-resilient-boot-boot-order.patch
      + ubuntu-resilient-boot-ignore-alternative-esps.patch
      + ubuntu-shorter-version-info.patch
      + ubuntu-speed-zsys-history.patch
      + ubuntu-support-initrd-less-boot.patch
      + ubuntu-verifiers-last.patch
      + ubuntu-zfs-enhance-support.patch
      + ubuntu-zfs-gfxpayload-dynamic.patch
      + ubuntu-zfs-gfxpayload-keep-default.patch
      + ubuntu-zfs-insmod-xzio-and-lzopio-on-xen.patch
      + ubuntu-zfs-mkconfig-recovery-title.patch
     ...

@mkukri, I apologize for my lack of familiarity with the grub EFI release process. Could you confirm if the arm64 testing version will be available for download at http://ports.ubuntu.com/ubuntu-ports/dists/?

Hi @chengendu,

For Noble, I believe it already is available there.

Mantic is not affected as it is missing the patches needed for UEFI HTTP booting (might be SRUd later).

We've decided against spending time fixing Lunar unless it is known to be required.

Jammy and Focal have been signed and waiting to be released. I believe they will appear in the proposed pocket after that and will get eventually promoted to updates.

For stable releases, I'm unembargoing the signed binaries now, they'll be available in

https://launchpad.net/~ubuntu-uefi-team/+archive/ubuntu/proposed-public/

shortly (well mantic and jammy are, focal is missing).

After that

- I'll copy the binary to the main archive
- SRU team accepts it into -proposed
- Verification needs to be done and it needs to age 7 days
- SRU team releases it to -updates

> The patch only updates a member variable type to comply with UEFI 2.9, preserving the original behavior.

Please explain what "preserving the original behavior" is supposed to mean here. You are clearly changing the size of a struct element on all 64-bit architectures, and this struct is being passed to a UEFI function. So how can unconditionally changing the size of the struct elements, and the offsets of all elements after it in the struct, "preserve the original behavior"?

@vorlon I'll have to cross-reference more spec versions next week, but I suspect what's going in is the size of those fields were incorrect, but it happened to sometimes work (when the allocated memory didnt contain garbage) due to the offsets matching because of C struct packing rules adding padding after the 32-bit ints because they are followed by 64-bit pointers (and on 32-bit arches this is a no-op change).

@vorlon The upstream commit message is confusing and incorrect, what this patch actually does is fix the type of those fields that were always incorrect in GRUB.

I have cross-referenced UEFI specification starting from the introduction of HTTP boot, and I can confirm those fields always had type UINTN. These all have type UINTN in these fields:
- UEFI 2.5 (this seems to be the version that specificed HTTP boot first)
- UEFI 2.8
- UEFI 2.9
- and the current UEFI 2.10.

And I suppose the reason this wasn't caught earlier is because as I have mentioned yesterday, C struct packing rules made the offset of fields still correct despite them being the wrong size. However 64-bit firmware still read them as 64-bit qwords and I guess if the padding contained garbage it would cause bad behavior.

Can the test plan please be extended to include tests on hardware other than the customer hardware?

@vorlon I have confirmed that this fixes UEFI HTTP boot on arm64 qemu too, and can repeat the same test for SRU verification.
I don't have any UEFI native arm64 hardware to test with however.

@vorlon Spoke to dannf, and there is a plan now for testing this on hyperscale lab machines.

Hello Chengen, or anyone else affected,

Accepted grub2-unsigned into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-unsigned/2.06-2ubuntu14.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Chengen, or anyone else affected,

Accepted grub2-signed into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-signed/1.187.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Chengen, or anyone else affected,

Accepted grub2-unsigned into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-unsigned/2.06-2ubuntu14.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Chengen, or anyone else affected,

Accepted grub2-signed into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-signed/1.187.8~20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted grub2-signed (1.187.8) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

edk2/2022.02-3ubuntu0.22.04.2 (arm64, armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#grub2-signed

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Verification on this is waiting for dannf being back.

Chengen verifying that the -proposed update fixes their issue would also be nice.