System doesn't boot after installation - Legacy mode / 2 disks

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1847898

This bug was fixed in the package ubiquity - 19.10.20

---------------
ubiquity (19.10.20) eoan; urgency=medium

  [ Jean-Baptiste Lallement ]
  * zsys-setup: Force a re-read of the partition table before using the disk
    (LP: #1847748)

  [ Iain Lane ]
  * zsys-setup: Mark packages to kept in the init phase, rather than finalize.
    Finalize runs after ubiquity has removed files from packages which it has
    decided the target system isn't going to need. It's too late for us to try
    to save things then. (LP: #1847898)

  [ Dimitri John Ledkov ]
  * Automatic update of included source packages: partman-auto
    134ubuntu12, partman-base 206ubuntu6 (LP: #1847944)

 -- Dimitri John Ledkov <email address hidden> Mon, 14 Oct 2019 14:31:19 +0100

Reopening, the wrong bug has been closed.

We reproduced this in eoan, and then went on to also reproduce it in disco, meaning it's not a regression, and therefor not RC for eoan.

It is, however, really bad UX, and I want to make sure we do this better for 20.04.

For future reference qemu reproducer is:

qemu-system-x86_64 -m 4096 -cdrom ./ubuntu-19.04-desktop-amd64.iso -drive file=generic.qcow2,if=none,id=D22 -device nvme,drive=D22,serial=1234 --enable-kvm -smp 2 -drive id=disk,file=generic-uefi-1.qcow2,if=none -device ahci,id=ahci -device ide-drive,drive=disk,bus=ahci.0 -boot menu=on

It appears that we go by alphabetical order for both select_disk option and the bootdev, and never update grub-installer/bootdev to partman-auto/select_disk value post the "choose disk" screen.

Related bugs:

Ubuntu Installer uses wrong bootloader location for USB/sdb UEFI installs
https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/1173457
Posted work around to manually unmount & mount correct ESP during install
https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/1396379
https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1229488
https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/1229488
EFI doesn't show multiple installs of the same operating system
https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1561712

My primary nvme drive shows up as sdb. Installing with 2 drives, and the nvme as sdb fails. Machine will not boot, post install. Pulled the secondary drive, reinstalled, and it does boot.

I think the confusion is in this function (and the similarly shaped kde one):

    def get_grub_choice(self):
        i = self.grub_device_entry.get_active_iter()
        if i:
            return self.grub_device_entry.get_model().get_value(i, 0)
        else:
            self.debug('No active iterator for grub device entry.')
            disk = self.get_current_disk_partman_id()
            if isinstance(disk, str) and disk:
                disk_path = disk.replace("=", "/")
                if os.path.exists(disk_path):
                    return misc.grub_default(boot=disk_path)

            return misc.grub_default()

(or https://git.launchpad.net/ubiquity/tree/ubiquity/plugins/ubi-partman.py#n681)

This figures out which device has been selected as the target disk and passes it as the boot= argument to misc.grub_default(), clearly expecting that grub_default() will return an appropriate path to install grub too. But grub_default() does not interpret the boot= argument in that way at all, and (modulo some fussing about installing from writable media) returns the first line from grub-mkdevicemap.

Untested maybe-fix at https://code.launchpad.net/~mwhudson/ubiquity/+git/ubiquity/+merge/383762

This bug was fixed in the package ubiquity - 20.10.1

---------------
ubiquity (20.10.1) groovy; urgency=medium

  [ Sebastien Bacher ]
  * debian/real-po:
    - updated translations from launchpad, including strings from the new
      subpage for RST (lp: #1874103)

  [ Jean-Baptiste Lallement ]
  * zsys-setup: Enable autotrim on all pools.

  [ Dimitri John Ledkov ]
  * Automatic update of included source packages: console-setup
    1.195ubuntu2, localechooser 2.71ubuntu5, partman-efi 84ubuntu1,
    partman-lvm 133.

  [ Michael Hudson-Doyle ]
  * make grub_default consider the boot argument in the non-removable case
    (LP: #1847898)

 -- Dimitri John Ledkov <email address hidden> Wed, 27 May 2020 11:32:08 +0100

This is now on daily groovy.

Jibel can you please test this?

Hello Jean-Baptiste, or anyone else affected,

Accepted ubiquity into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubiquity/20.04.15.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

SRU verification for Focal:
I have reproduced the problem with ubiquity 20.04.15 in focal and have verified that the version of ubiquity 20.04.15.1 in -proposed fixes the issue.

I tested on a VM with 3 disks, and it boots from 3rd disk when I set it as the first disk to try. Otherwise it doesn't find any bootable device but I suppose it's because the VM doesn't try the other disks in sequence.

Marking as verification-done

This bug was fixed in the package ubiquity - 20.04.15.1

---------------
ubiquity (20.04.15.1) focal; urgency=medium

  [ Sebastien Bacher ]
  * debian/real-po: updated translations from launchpad, including strings
    from the new subpage for RST (LP: #1874103)

  [ Michael Hudson-Doyle ]
  * make grub_default consider the boot argument in the non-removable case
    (LP: #1847898)

  [ Jean-Baptiste Lallement ]
  * Enable ZFS autotrim (LP: #1883686)
  * Use persistent path for pool's vdevs (LP: #1880869)
  * Do not export all pools (LP: #1875045)

  [ Dimitri John Ledkov ]
  * ubiquity-dm: start gsd-keyboard for keyboard indicator (LP: #1847307)

 -- Dimitri John Ledkov <email address hidden> Tue, 16 Jun 2020 11:17:42 +0100

The verification of the Stable Release Update for ubiquity has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.