18.04 install stuck on grub install when a zfs disk is present

Status changed to 'Confirmed' because the bug affects multiple users.

What sort of system is this on and can you reproduce it? If so can you edit /usr/bin/os-prober and add an x to the set -e line at the top of it then reproduce the issue to get more information?

Yes, its reproducible.

Dell PowerEdge T110 (circa ~2009). I can give more details if it would be helpful.

I found a /bin/os-prober (no /usr/bin/os-prober), and added the x option. Resulting syslog is the same as before, no additional information. Is there another file I should be looking in?

Hrm.. if the second line shows "set -ex" then running os-prober should produce a lot of debug output showing every line of the script that is executed. Oh, try running it yourself instead of letting the installer do it.

I've attached the os-prober output when run as the install was waiting for me to confirm installing grub on the master boot record.

Did it get stuck that time?

Sorry, this is my first time trying to debug os-level installer. I don't know how to switch the install from auto to manual, then back to auto.

I ran the auto install through to the point where it was waiting for me to confirm the grub install on the master boot record, where I would normally say yes. At that point, I switched over to a console (Alt-F2), and ran the os-prober manually and captured the output. When switching back to the auto installer (Alt-F1), I answered the question (yes), and it gave me a different failure.

I wasn't focusing on that failure, as I was thinking the os-prober output was the interesting part. What I do remember is that when running it straight through automatically, it put up a list of my drives asking which one I wanted to install grub on (mbr on sda, partition on sdb, or sdc, or sdd). This time, when switching back from the command line console to the installer console, it didn't give me my list of drives to choose from, and it popped up a text field for me to enter the drive name to do the grub install on. When I entered /dev/sda, it gave the failure.

I'll redo it tonight and get more specific detail of the failure.

The interesting information I'm trying to get is exactly where os-prober froze, so if it finishes correctly then that information isn't going to help. I think at that point in the installer if you look, you will see the hard disk mounted under /target, and it has its own copy of os-prober, so make sure that it has the set -ex and let the installer install grub, and if it freezes there again ( and you should also check ps aux | grep os-prober to see if the process is still running ), then at that point all of the debug output should be in /var/log/syslog and we should be able to see where it was when it froze.

Ok, I understand now what your previous question meant.

Yes, I believe the os-prober completed normally (returned to the command prompt) with the output as in the previously attached os-prober.log. I ran it with 2>&1, so both stdout and stderr would be included in that file, and I didn't notice any obvious error messages. I will re-run the process tonight to make sure I didn't miss anything we've discussed here.

If os-prober did complete properly (even in the auto install), do you have a guess as to what else might be the hang-up? I'll do the ps aux as you suggested to see what else is going on at that point in the process.

thanks for the diagnostic help

If it isn't os-prober that hung, then I'll have to look up the next step the installer takes. Or you might see some other program in ps that is hung and the installer is waiting on.

Problem solved. There was a USB stick plugged directly in to the motherboard which I had forgotten about since changing this box over from a freeNAS install. Once that was removed, the install completed with no hangup.

I'll include the output from the point where it was hung. Both from ps, and from the full os-prober output in syslog. At this point, I'm happy to close the issue, but if you would like any further testing on this setup, I'd be willing to support that. I'm not sure if you would expect a USB drive to hang up the install or not.

The USB stick was the boot drive for freeNAS, so the primary partition was formatted as ZFS.

Strange, yes, a USB stick should not cause it to hang. I'm still not seeing the output of os-prober with the set -x option working. Also you were able to manually run it with this zfs usb stick plugged in and it did not hang right?

-- I'm still not seeing the output of os-prober with the set -x option working.

I updated /target/usr/bin/os-prober with the -x option added. I was expecting it to show up in the syslog. Apparently still having trouble following the twisty maze of passages all alike. I will try again.

-- Also you were able to manually run it with this zfs usb stick plugged in and it did not hang right?

That it correct, if all the calls in os-prober are synchronous, then it did not hang. It returned to the command line with no error output. I did not check ps at the same time I ran os-prober manually however, so I don't know what might have still been running. I assumed everything was synchronous, so I didn't bother.

Ok, new experiment. I'm including the steps here and in an attached README along with outputs captured at various points in the process in a tar.gz file.

I've moved drives around throughout this debugging process, but for this experiment they were;
/dev/sda - 80GB spinner (os install drive)
/dev/sdb - ZFS USB drive (problem drive)
/dev/sdc - ext4 USB drive (data drive to transfer output files)

Let me know if there are variations in the process I should try.

miket$ cat README
Auto Install up to "Install the GRUB boot loader on a hard disk" step
Alt-F2
nano /target/usr/bin/os-prober (-ex)
nano /bin/os-prober (-ex)

./step1
/target/usr/bin/os-prober > os-prober.log 2>&1
ps > ps.log
cp /var/log/syslog .

Alt-F1
Proceed with auto install of GRUB to /dev/sda
Wait until it is stuck at 66% for about 1 minute
Alt-F2

./step2
ps > ps.log
cp /var/log/syslog .

./step3
/target/usr/bin/os-prober > os-prober.log 2>&1
ps > ps.log
cp /var/log/syslog .

So it looks like it is getting stuck running this command:

31197 root 26996 S dmsetup create -r osprober-linux-sdb1

It also looks like in the cases where it does not hang, there is a message about not recognizing what filesystem is in /dev/sdb1 and it can't be mounted, but in the one where it hangs, it seems like it does recognize it. I wonder if zfs isn't present in the installer and so it can't recognize and mount it, but it is installed in the target system, and it is somehow conflicting and causing the hang?

same bug, when I install http://cdimage.ubuntu.com/releases/18.04/release/ubuntu-18.04.1-server-amd64.iso with kinston datatraveler 2.0

I write ubuntu-18.04.1-server-amd64.iso into kinston with rufus-3.1p

The host machine is Think server ts250 with 2 1TB SATA hard disk, host raid 1.

tail /var/log/syslog, see the attachment

I have just run into this same problem on both 18.04 Server and 19.04 Server Beta. After finding this bug report, I unplugged the drive array full of ZFS disks and the local write cache disk only to still run into the issue due to my nvme used for read cache.

Even after removing all drive partitions from the ZFS drives, I am still having this problem.