Support for grub upgrades with bios+uefi bootloader targets

Performing some test builds before submitting packages to the SRU queue: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3308/+packages

It looks like this was fixed in cosmic but referenced a different bug number.
grub-installer (1.128ubuntu9) cosmic; urgency=medium

  * grub-installer: install grub-pc for EFI setups and make sure we're
    not purging it earlier in case it got installed. It's needed to make sure
    the right maintainer scripts are run and the ESP populated. (LP: #1775743)

 -- Łukasz 'sil2100' Zemczak <email address hidden> Tue, 19 Jun 2018 10:25:14 +0200

Hello Łukasz, or anyone else affected,

Accepted grub-installer into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub-installer/1.128ubuntu8.18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

This too was fixed in cosmic.

shim-signed (1.36) cosmic; urgency=medium

  * debian/shim-signed.postinst: use --auto-nvram with grub-install in case
    we're installing on a NVRAM-unavailable platform.
  * debian/control: bump the dependency for grub2-common to make sure
    grub-install supports --auto-nvram.
  * debian/control: switch the grub-efi-amd64-bin dependency to
    grub-efi-amd64-signed.

 -- Łukasz 'sil2100' Zemczak <email address hidden> Wed, 06 Jun 2018 20:25:57 +0200

Hello Łukasz, or anyone else affected,

Accepted shim-signed into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/shim-signed/1.34.9.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Also fixed in cosmic.grub2-signed (1.96) cosmic; urgency=medium

  * debian/control: switch the grub-efi-amd64 dependency of
    grub-efi-amd64-signed to grub-efi-amd64-bin.
  * debian/grub-efi-amd64-signed.postinst: invoke grub-install with
    --auto-nvram and pass the x86_64-efi target to it, making sure we always
    install the right target.

 -- Łukasz 'sil2100' Zemczak <email address hidden> Wed, 06 Jun 2018 20:09:57 +0200

Hello Łukasz, or anyone else affected,

Accepted grub2-signed into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2-signed/1.93.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

This too was fixed in cosmic.
grub2 (2.02-2ubuntu9) cosmic; urgency=medium

  * debian/patches/add-an-auto-nvram-option-to-grub-install.patch: Add the
    --auto-nvram option to grub-install for auto-detecting NVRAM availability
    before attempting NVRAM updates.

 -- Łukasz 'sil2100' Zemczak <email address hidden> Tue, 05 Jun 2018 00:34:38 +0200

Hello Łukasz, or anyone else affected,

Accepted grub2 into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2/2.02-2ubuntu8.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Uploaded ubiquity with the new -proposed packages updated. That is necessary for the fixes to work on desktop.

Hello Łukasz, or anyone else affected,

Accepted ubiquity into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubiquity/18.04.14.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Why would you want to install grub-efi if you can't update the EFI boot catalog? Without that, it can't be booted in EFI mode.

On Mon, Jul 02, 2018 at 12:14:40AM -0000, Phillip Susi wrote:
> Why would you want to install grub-efi if you can't update the EFI boot
> catalog? Without that, it can't be booted in EFI mode.

Because with shim's 'fallback' implementation, a system that could not
update the boot catalog in nvram because it was not booted under UEFI
initially still stands a chance of auto-configuring itself to be able to
boot Ubuntu, via selecting the disk for boot from the firmware boot
selector.

Verification for bionic, basic grub2 grub-install test (using bionic ubuntu-server daily image 20180704):

I had a legacy BIOS system on a virtual HDD. Using kvm, I booted the bionic daily image in UEFI mode and installed a new Ubuntu system next to it (with the ESP created etc.). I have then booted the UEFI system and ran `grub-install --target=x86_64-efi --auto-nvram` - the command returned with return value of 0.
Then I switched to legacy BIOS mode and booted into the UEFI-installed system. I have run the `grub-install --target=x86_64-efi --auto-nvram` command there again and the command also returned with the return value of 0 (success).

Proceeding with installation tests.

ubuntu-server daily (d-i) (20180704):

Confirmed in the image build logs that the image contains the right versions of grub2, grub2-signed, shim-signed and grub-installer [1].

 UEFI install:
 * Performed a full-disk installation that resulted in a bootable system.
 BIOS install:
 * Performed a full-disk installation that resulted in a bootable system.

ubuntu-desktop daily-live (ubiquity) (20180704):

Confirmed in the image build logs that the image contains the right versions of grub2, grub2-signed, shim-signed and ubiquity [2]

 UEFI install:
 * Performed a full-disk installation that resulted in a bootable system.
 BIOS install:
 * Performed a full-disk installation that resulted in a bootable system.

[1] http://people.canonical.com/~ubuntu-archive/cd-build-logs/ubuntu-server/bionic/daily-20180704.log
[2] http://people.canonical.com/~ubuntu-archive/cd-build-logs/ubuntu/bionic/daily-live-20180704.log

I am unable to verify the ubuntu-server subiquity based images as the installation currently is failing. It looks to me unrelated to the proposed grub changes as it's a problem with getting the right kernel installed - I'll poke the subiquity team about this.

While the complete verification of this is not yet done with respect to the server live image, the ubiquity+grub-installer pieces (affecting d-i and ubiquity install paths) have been verified, and there is another grub-installer SRU waiting in the unapproved queue; so I'm going to go ahead with releasing these two packages since they don't need to go in at the same time as shim+grub2. The others can be released on Monday after verification is completed.

This bug was fixed in the package grub-installer - 1.128ubuntu8.18.04.1

---------------
grub-installer (1.128ubuntu8.18.04.1) bionic; urgency=medium

  * grub-installer: install grub-pc for EFI setups and make sure we're
    not purging it earlier in case it got installed. It's needed to make sure
    the right maintainer scripts are run and the ESP populated.
    Required as part of the dependency-chain switch for dual EFI/BIOS support.
    (LP: #1778848)

 -- Łukasz 'sil2100' Zemczak <email address hidden> Wed, 27 Jun 2018 10:27:18 +0200

This bug was fixed in the package ubiquity - 18.04.14.3

---------------
ubiquity (18.04.14.3) bionic; urgency=medium

  * Automatic update of included source packages: grub-installer
    1.128ubuntu8.18.04.1, partman-auto 134ubuntu8.1, partman-efi
    71ubuntu2.1. (LP: #1766945, LP: #1778848)

 -- Łukasz 'sil2100' Zemczak <email address hidden> Fri, 29 Jun 2018 09:47:18 +0200

ubuntu-server daily-live (subiquity) (20180709):

Confirmed in the image build logs that the image contains the right versions of grub2, grub2-signed and shim-signed [1]. The previous issues with subiquity were related to the kernel binaries being stuck in NEW (so unrelated). By the time of 20180709 everything was good and testing could be performed.

 UEFI install:
 * Performed a full-disk installation that resulted in a bootable system.
 BIOS install:
 * Performed a full-disk installation that resulted in a bootable system.

[1] http://people.canonical.com/~ubuntu-archive/cd-build-logs/ubuntu-server/bionic/daily-live-20180709.log

This basically concludes the testing from my side.

This bug was fixed in the package grub2 - 2.02-2ubuntu8.1

---------------
grub2 (2.02-2ubuntu8.1) bionic; urgency=medium

  * debian/patches/add-an-auto-nvram-option-to-grub-install.patch: Add the
    --auto-nvram option to grub-install for auto-detecting NVRAM availability
    before attempting NVRAM updates. (LP: #1778848)

 -- Łukasz 'sil2100' Zemczak <email address hidden> Tue, 05 Jun 2018 00:34:38 +0200

The verification of the Stable Release Update for grub2 has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package grub2-signed - 1.93.1

---------------
grub2-signed (1.93.1) bionic; urgency=medium

  * debian/control: switch the grub-efi-amd64 dependency of
    grub-efi-amd64-signed to grub-efi-amd64-bin. (LP: #1778848)
  * debian/grub-efi-amd64-signed.postinst: invoke grub-install with
    --auto-nvram and pass the x86_64-efi target to it, making sure we always
    install the right target. (LP: #1778848)
  * debian/control: rebuild against grub2 2.02-2ubuntu8.1.

 -- Łukasz 'sil2100' Zemczak <email address hidden> Wed, 27 Jun 2018 09:01:13 +0200

This bug was fixed in the package shim-signed - 1.34.9.2

---------------
shim-signed (1.34.9.2) bionic; urgency=medium

  * debian/shim-signed.postinst: use --auto-nvram with grub-install in case
    we're installing on a NVRAM-unavailable platform. (LP: #1778848)
  * debian/control: bump the dependency for grub2-common to make sure
    grub-install supports --auto-nvram. (LP: #1778848)
  * debian/control: switch the grub-efi-amd64-bin dependency to
    grub-efi-amd64-signed. (LP: #1778848)

 -- Łukasz 'sil2100' Zemczak <email address hidden> Wed, 27 Jun 2018 10:14:24 +0200

On 07/02/2018 12:51 PM, Steve Langasek wrote:
> On Mon, Jul 02, 2018 at 12:14:40AM -0000, Phillip Susi wrote:
>> Why would you want to install grub-efi if you can't update the EFI boot
>> catalog? Without that, it can't be booted in EFI mode.
> Because with shim's 'fallback' implementation, a system that could not
> update the boot catalog in nvram because it was not booted under UEFI
> initially still stands a chance of auto-configuring itself to be able to
> boot Ubuntu, via selecting the disk for boot from the firmware boot
> selector.
>
You mean we are now installing a "removable media" boot loader in
EFI/BOOT that chainloads the one in EFI/Ubuntu?

On Tue, Jul 17, 2018 at 07:15:06PM -0000, Phillip Susi wrote:
> On 07/02/2018 12:51 PM, Steve Langasek wrote:
> > On Mon, Jul 02, 2018 at 12:14:40AM -0000, Phillip Susi wrote:
> >> Why would you want to install grub-efi if you can't update the EFI boot
> >> catalog? Without that, it can't be booted in EFI mode.
> > Because with shim's 'fallback' implementation, a system that could not
> > update the boot catalog in nvram because it was not booted under UEFI
> > initially still stands a chance of auto-configuring itself to be able to
> > boot Ubuntu, via selecting the disk for boot from the firmware boot
> > selector.

> You mean we are now installing a "removable media" boot loader in
> EFI/BOOT that chainloads the one in EFI/Ubuntu?

The "removable media" bootloader does not chainload the one in EFI\Ubuntu;
it chainloads fbx64.efi, whose sole function is to (re)populate the nvram
boot settings from the set of BOOT.CSV files it discovers on disk.

https://github.com/rhboot/shim/blob/master/README.fallback

So even if the we cannot populate nvram at install time (e.g. because the
system is not booted in EFI mode at that point), it is useful to initialize
and populate an ESP for future EFI compatibility.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer https://www.debian.org/
<email address hidden> <email address hidden>