Impossible to skip integrity test for ubuntu-server 20.04.1 iso

Status changed to 'Confirmed' because the bug affects multiple users.

Why is this bug not fixed since nearly two months? Its extremely annoying.

You can boot with a cmdline option to skip that.

http://manpages.ubuntu.com/manpages/hirsute/en/man7/casper.7.html#recognized%20boot%20options

fsck.mode=skip
              Let you skip the file system check on boot.

Can you ellaborate a bit more, and make sure that the console you are observing is the one connected to the grub/initrd?

Is this a serial console, tty1, hvc0 or something else?

live-server bugs go against subiquity project, rather than ubiquity.

and casper is the thing that does the integrity check.

separately remote iso mounting may be slow, thus you should try netbooting instead.

one can copy the kernel & initrd out of the iso. Boot them, with cmdline:
 ip=dhcp url=http://hostname/url/to/matching-full.iso

That way, initrd will establish networking and will download iso into ram, validate it very quickly, and boot.

This usually yields faster install times. If your target machine has enough RAM, something like more than 4GB.

Status changed to 'Confirmed' because the bug affects multiple users.

We are still missing the information which Dimitri asked for in comment #3. Please add some more details so that we can begin investigating this.

Seriously, why are you asking this?

To test it, just run the installation media on any computer or virtual machine, use real keyboard and mouse or virtual ones, this doesn't change anything.

I didn't test the "fsck.mode=skip" command line option but this should be an option that doesn't need the user to enter command line option to begin with.

I agree. I understand the situation, why this integrity check came up and is considered as helpful for users installing their OS with a broken USB Stick. But in case of Non-MAAS datacenter operations (SMC IPMI for me), this feature is an entire pain. Although connected to the ISO via an 1 GBps connection, it takes 25 minutes to complete the integrity check. After that, the entire setup is (almost) made within four minutes. Until... the unskippable "Installing security Updates" happens, which can be cancelled, when its finished. Srsly, I would highly appreciate any efforts to give one a chance to reliably skip integrity check and security patches during a server installation.

Netboot is not an option for me, as in a datacenter with public IPv4/v6 assignments, I prefer manual IP assignments. Therefore, no DHCP is available during setup.

IIRC I was able to use the parameter Dimitri suggested on Groovy ISO, but it didn't work in Focal 20.04.1 ISO. That said, we now have Ubuntu 20.04.2 ISO - can you give it a try Sebastian, with "fsck.mode=skip" set?
This is the link for the new ISO: http://releases.ubuntu.com/focal/ubuntu-20.04.2-live-server-amd64.iso

Thanks in advance for you help!

@gpiccoli: negative. 20.04.2-live iso doesn't honor this argument.

For the records: It is several minutes faster to install an ubuntu 18.04 iso (via IPMI virtual media) and then do a do-release-upgrade to ubuntu 20.40 than a clean Ubuntu 20.04 install via IPMI virtual media with integrity check.

Thanks a lot for your testing ! Do you mind in testing Groovy ISO [0] as well? I'm pretty sure it worked for me.
Thanks in advance!

[0] http://releases.ubuntu.com/groovy/ubuntu-20.10-live-server-amd64.iso

@gpiccoli: negative. 20.10-live-server-amd64.iso doesn't honor this argument, too.

Why the heck is a CTRL+C during integrity check not enough expression of my will to skip that process?

subscribed ~field-medium. This bug impacts deployments ands add very long delays for the installation of the infra nodes.

Sebastian / Camille, I've just tested in a virtual machine, and the parameter "fsck.mode=skip" worked for me in both cases (Groovy and Focal 20.04.2 images); I'll attach screenshots with the results (I needed to use a VNC connection since by default the images don't output to serial console, another point to improve if you ask me...)

So, I've added this parameter in the grub command-line after quiet and before the "---" - did you try adding there Sebastian?
Cheers,

Guilherme

On Thu, 25 Feb 2021 at 08:01, Guilherme G. Piccoli <
<email address hidden>> wrote:

> (I needed to
> use a VNC connection since by default the images don't output to serial
> console, another point to improve if you ask me...)
>

If you know a way to find out where the serial console _is_ on amd64 that
actually works, we're all ears!

Cool, I'm glad that it's being considered! I'd go with "ttyS0" - if it's not right, patience...won't work. But there's no downside in that case, it'll behave as it currently behaves, no output.

People got upset when I suggested that before...

Well...we can discuss that in a different LP heheh
I'm sure lots of people will love that idea, and a bunch of people will hate it.

I'll try to find some LP in which the discussion fits better - if I can't find, I'll open one and add you there! Thanks,

Guilherme

I was finally able to use the workaround on a Dell R740 remote console. On the Install Ubuntu screen, click e for options, add "fsck.mode=skip" before "quiet" on the 2nd line, then do ctrl+x to save the changes. It skipped the integrity check successfully.

Well, seems it's fixed for Focal/Groovy - so I marked as Fix Released.
Let us know if any issues are observed Camille and Sebastian.
Thanks,

Guilherme

Sorry, but that's not true. The integrity check, when booting from a virtual CD-ROM, as it is common with servers equipped with remote consoles on hardware management boards (iLO, iDRAC, CIMC tec.) still takes ages.

The integrity check is faster instead, when the operating system is installed in a VM and the ISO is mounted directly as a CD into the VM.

But the workaround Camille mentioned works. I've used this on a Cisco CIMC. So don't forget to hit "e" on the install screen. This definitely saves time.

Ulrich

Sorry Ulrich, I got a bit confused. What is not true?

This Launchpad bug is about users being unable to *skip* the integrity check. After the test from Camille and I (and you!), seems this is indeed fixed, users *can* skip the integrity check if they want, correct?

We are not discussing here if the integrity check is slow or not (it is, in Virtual ISO environments!), but the feasibility of skipping it.
Cheers,

Guilherme

This appears to be the case for the 20.04.2 ISO as well. There doesn't seem to be a way to cancel the integrity check once it's started. It would be nice to have that option so you don't have to manually edit the boot options to bypass it.

OK, I understand..the request then is for cancelling the integrity test *once it has started*. Can you open a new bug, with a very clear/direct title about this? Something like "Add a way to cancel integrity test on Ubuntu server ISO once it has started", so we can track this feature request over there.

Thanks in advance,

Guilherme

@gpiccoli

I've mentioned this bevore in #13

> Why the heck is a CTRL+C during integrity check not
> enough expression of my will to skip that process?

Just now I am in the same hazzle again. I've pressed E during boot, tried to apply fsck.skip=false, but via IPMI, the keyboard layout is somewhat dice-rolled. So again, the integrity check wasn't skipped and I had 40 minutes to think about restarting over or just get out of the house to have a walk with the dog. After returning, I tried to express another wish to the installer by hitting "Skip updates". Of course, the update after the setup wasn't skipped. Due to high IO (because of RAID), the update process takes another 15 minutes now and still isn't finished.

This installer is entirely unusable in manual datacenter operations. It is *that* unusable, that I think about creating my own installer ISO with customized scripts for the installation process, just for skipping integrity checks and autoupdates.

I also want to use the ISO over IPMI to enter a rescue console. Again, there must be an integrity check before I can use it. For now, I use Ubuntu 18.04 for that.

Please consider respecting the users commands given to the installer. There may be cases, where the user knows what he wants.

Thanks,
Sebastian

Fun fact: When trying to enter the "advanced options" in the welcome screen to notify the installer to not do an integrity check, Supermicro IPMI KVM-Viewer has assigned the F6 Button to "Macro for send CTRL-ALT-DELETE". Not the developers fault, I know. But you have to change this setting each and every time, because this setting in the KVM viewer is not persistent.

A honored CTRL+C during the integrity check would entirely fix this issue.

Since this is closed I created another issue for actually adding the option to skip the test like is done in the Desktop iso.

https://bugs.launchpad.net/ubuntu/+source/casper/+bug/1929935

Why is this still a problem with the 20.04.03 iso image? Seriously, are you under the impression that we all have nothing better to do with our very limited time on this earth, why the hell is an integrity check even the default in the first place? You can verify the checksum of the image before you mount the ISO image using your Baseboard Management Controller's Virtual Media.

This is still completely unworkable for me. I'm installing 20.04.3 on a rack server via the LOM, and at no time while the system was starting up did I see anywhere I could set command line options - it went from the firmware talking about scanning the bus to "No Signal" to the kernel starting to boot in the space of about 3 seconds.

And even if there was a splash screen that allowed me to press a key to set options as comments above imply there should be, it takes the firmware in this box two and half minutes to get to the point where the ISO is booting. So to make use of it, I'd have to be glued to my monitor for that long to make sure I didn't miss the window of how many seconds it's supposed to appear for. I could be spending more time waiting for the opportunity to press a key than I would spend answering the installer questions.

As things stands, #11 appears to be the best solution I've got and I think that's absolutely ridiculous.

Created an account to leave this update.

20.04.4 LTS (ubuntu-20.04.4-live-server-amd64.iso) still exhibits this behavior - adding the boot to the provided ISO does NOT skip the integrity check. Unfortunately, in my use case, I have to use 20.04 (because 18.04 is old enough it can't recognize my network devices), so can't use the FAR more efficient workaround provided in Comment #11. Would the installer team <i>please</i> address this:

1) (Preferred) Make the media check an opt-in feature of the ISO, or, failing that,
2) Provide a well-documented, functional command line parameter that skips the media check on the ISO

Because right now, "fsck.mode=skip" does NOT skip the media check, and I'm losing significant time trying to get this first node set up (so I can set up a local repository and not have to go through this time sink on <i>every other server I'm supposed to be deploying tonight</i>) because this issue that was initially raised 18 months ago, that was a functional regression from the previous LTS release ISO functionality, is still not resolved. :(

Thank you.

We're in the process of installing 20.04 on machines at the end of a very long VPN so the large ISO size is a real pain. To work-around the integrity check I cracked the ISO open and replaced md5sum.txt with my own that covered just the autoinstall files I wanted to track. I know this isn't a permanent solution but it worked for us to remove up to 2 hours of re-installation time from our process.

Issues reproducible with ubuntu-20.04.4-live-server-amd64.iso.

Apparently, ctrl-c does not stop the integrity check anymore
because casper-md5check runs as systemd service/disconnected
from plymouth. Per [1], iiuic. -- Well, this is _not_ a bug.

So, fsck.mode=skip is the only option available.

However, there's a bug in casper-md5check that ignores that
_if_ it is the last option in /proc/cmdline (likely, if the
user adds it after '---' or deletes '---' to insert it.)

Workaround: make sure to add fsck.mode=skip before '---'.

Patch proposed [2], with technical details and testing.

[1] https://launchpad.net/ubuntu/+source/casper/1.457
[2] https://code.launchpad.net/~mfo/casper/+git/casper/+merge/417120

Very very nice findings @mfo, thanks a lot! That explains everything - likely in my test I've added the parameter _before_ the line ending...

It's unfortunate we cannot break the test with ctrl_c, hopefully there's a way for that, but if not, at least the bug in the line parsing is fixed. Hopefully we can get your patch accepted quickly and have it backported to Focal.

Thanks again!

Unmarking as fixed in Subiquity due to the above reports.

This bug was fixed in the package casper - 1.468

---------------
casper (1.468) jammy; urgency=medium

  * casper-md5check: correctly detect fsck.mode=skip if last
    in /proc/cmdline, and add a hint for users (LP: #1892369)

 -- Mauricio Faria de Oliveira <email address hidden> Fri, 18 Mar 2022 18:02:14 -0300

The changes are working in jammy daily builds:
- jammy-live-server-amd64.iso @ 20220330.1
- jammy-desktop-amd64.iso @ 20220329.1

live-server
---

root@ubuntu-server:/# cat /proc/cmdline
BOOT_IMAGE=/casper/vmlinuz console=ttyS0

root@ubuntu-server:/# systemctl status -l --no-pager casper-md5check.service
...
Mar 30 14:48:56 ubuntu-server casper-md5check[1246]: Checking ./casper/ubuntu-server-minimal.ubuntu-server.installer.manifest..../casper/ubuntu-server-minimal.ubuntu-server.installer.manifest: OK
...
Mar 30 14:48:56 ubuntu-server casper-md5check[1246]: Check finished: no errors found.
Mar 30 14:48:56 ubuntu-server systemd[1]: Finished casper-md5check Verify Live ISO checksums.

...

root@ubuntu-server:/# cat /proc/cmdline
BOOT_IMAGE=/casper/vmlinuz console=ttyS0 fsck.mode=skip

root@ubuntu-server:/# systemctl status -l --no-pager casper-md5check.service
...
Mar 30 14:51:33 ubuntu-server systemd[1]: Starting casper-md5check Verify Live ISO checksums...
Mar 30 14:51:33 ubuntu-server casper-md5check[1245]: .
Mar 30 14:51:33 ubuntu-server casper-md5check[1245]: Check skipped.
Mar 30 14:51:33 ubuntu-server systemd[1]: Finished casper-md5check Verify Live ISO checksums.

desktop
---

ubuntu@ubuntu:~$ cat /proc/cmdline
BOOT_IMAGE=/casper/vmlinuz file=/cdrom/preseed/ubuntu.seed maybe-ubiquity console=ttyS0

ubuntu@ubuntu:~$ systemctl status -l --no-pager casper-md5check.service
...
Mar 30 14:40:20 ubuntu casper-md5check[1305]: Checking ./casper/filesystem.manifest..../casper/filesystem.manifest: OK
...
Mar 30 14:40:21 ubuntu casper-md5check[1305]: Check finished: no errors found.
Mar 30 14:40:21 ubuntu systemd[1]: Finished casper-md5check Verify Live ISO checksums.

...

ubuntu@ubuntu:~$ cat /proc/cmdline
BOOT_IMAGE=/casper/vmlinuz file=/cdrom/preseed/ubuntu.seed maybe-ubiquity console=ttyS0 fsck.mode=skip

ubuntu@ubuntu:~$ systemctl status -l --no-pager casper-md5check.service
...
Mar 30 14:46:15 ubuntu casper-md5check[1301]: .
Mar 30 14:46:15 ubuntu casper-md5check[1301]: Check skipped.
Mar 30 14:46:16 ubuntu systemd[1]: Finished casper-md5check Verify Live ISO checksums.

Test package results on focal: all good.

$ lsb_release -cs
focal

Before:
------

 $ sudo apt install --yes casper # focal-proposed

 $ dpkg -s casper | grep Version:
 Version: 1.445.2

1) Without fsck.mode=skip: check occurs (OK)

 $ cat /proc/cmdline
 BOOT_IMAGE=/vmlinuz-5.15.0-46-generic root=/dev/mapper/nvme--vg-root ro quiet splash mitigations=off vt.handoff=7

 $ sudo /usr/lib/casper/casper-md5check /tmp /dev/zero
 .
 Checking integrity, this may take some time
 ^C

2) With fsck.mode=skip last: check occurs (WRONG!)

 $ echo "$(cat /proc/cmdline) fsck.mode=skip" >/tmp/cmdline
 $ sudo mount --bind /tmp/cmdline /proc/cmdline

 $ cat /proc/cmdline
 BOOT_IMAGE=/vmlinuz-5.15.0-46-generic root=/dev/mapper/nvme--vg-root ro quiet splash mitigations=off vt.handoff=7 fsck.mode=skip

 $ sudo /usr/lib/casper/casper-md5check /tmp /dev/zero
 .
 Checking integrity, this may take some time
 ^C
 $ sudo umount /proc/cmdline

3) With fsck.mode=skip before last: check skipped (OK; workaround available)

 $ echo "$(cat /proc/cmdline) fsck.mode=skip workaround" >/tmp/cmdline
 $ sudo mount --bind /tmp/cmdline /proc/cmdline

 $ cat /proc/cmdline
 BOOT_IMAGE=/vmlinuz-5.15.0-46-generic root=/dev/mapper/nvme--vg-root ro quiet splash mitigations=off vt.handoff=7 fsck.mode=skip workaround

 $ sudo /usr/lib/casper/casper-md5check /tmp /dev/zero
 .
 Check skipped.
 ^C
 $ sudo umount /proc/cmdline

After:
-----

 $ sudo dpkg -i casper_1.445.3_amd64.deb

 $ dpkg -s casper | grep Version:
 Version: 1.445.3

1) Without fsck.mode=skip: check occurs (OK), and hint is provided

 $ cat /proc/cmdline
 BOOT_IMAGE=/vmlinuz-5.15.0-46-generic root=/dev/mapper/nvme--vg-root ro quiet splash mitigations=off vt.handoff=7

 $ sudo /usr/lib/casper/casper-md5check /tmp /dev/zero
 .
 Checking integrity, this may take some time (or try: fsck.mode=skip)
 ^C

2) With fsck.mode=skip last: check skipped (OK; FIXED!)

 $ echo "$(cat /proc/cmdline) fsck.mode=skip" >/tmp/cmdline
 $ sudo mount --bind /tmp/cmdline /proc/cmdline

 $ cat /proc/cmdline
 BOOT_IMAGE=/vmlinuz-5.15.0-46-generic root=/dev/mapper/nvme--vg-root ro quiet splash mitigations=off vt.handoff=7 fsck.mode=skip

 $ sudo /usr/lib/casper/casper-md5check /tmp /dev/zero
 .
 Check skipped.
 ^C
 $ sudo umount /proc/cmdline

3) With fsck.mode=skip before last: check skipped (OK; no regression)

 $ echo "$(cat /proc/cmdline) fsck.mode=skip workaround" >/tmp/cmdline
 $ sudo mount --bind /tmp/cmdline /proc/cmdline

 $ cat /proc/cmdline
 BOOT_IMAGE=/vmlinuz-5.15.0-46-generic root=/dev/mapper/nvme--vg-root ro quiet splash mitigations=off vt.handoff=7 fsck.mode=skip workaround

 $ sudo /usr/lib/casper/casper-md5check /tmp /dev/zero
 .
 Check skipped.
 ^C
 $ sudo umount /proc/cmdline

Submitted MR for Focal [1].

Sorry for the delay/little time for the 20.04.5 RC images.

Hopefully the detailed test coverage above provides enough
confidence if the change is desirable/in time, since it is
relatively simple.

I can test the RC images as soon as they're out, if needed.

cheers,
Mauricio

[1] https://code.launchpad.net/~mfo/casper/+git/casper/+merge/429028

Hello Slash, or anyone else affected,

Accepted casper into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/casper/1.445.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To properly test it you will need to obtain and boot a daily build of a Live CD for focal. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted casper (1.445.3) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

casper/1.445.3 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#casper

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

focal verification done; all good.

$ lxc launch ubuntu:focal casper
$ lxc shell casper

# echo 'deb http://archive.ubuntu.com/ubuntu focal-proposed main' >/etc/apt/sources.list.d/proposed.list
# apt-cache madison casper | grep proposed
    casper | 1.445.3 | http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages
# apt install --yes casper
# dpkg -s casper | grep ^Version:
Version: 1.445.3
# su - ubuntu

1) Without fsck.mode=skip: check occurs (OK)

$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-5.15.0-46-generic root=/dev/mapper/nvme--vg-root ro quiet splash mitigations=off vt.handoff=7
$ sudo /usr/lib/casper/casper-md5check /tmp /dev/zero
.
Checking integrity, this may take some time (or try: fsck.mode=skip)
^C

2) With fsck.mode=skip last: check skipped (OK; FIXED!)

$ echo "$(cat /proc/cmdline) fsck.mode=skip" >/tmp/cmdline
$ sudo mount --bind /tmp/cmdline /proc/cmdline

$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-5.15.0-46-generic root=/dev/mapper/nvme--vg-root ro quiet splash mitigations=off vt.handoff=7 fsck.mode=skip

$ sudo /usr/lib/casper/casper-md5check /tmp /dev/zero
.
Check skipped.
^C
$ sudo umount /proc/cmdline

3) With fsck.mode=skip before last: check skipped (OK; no regression)

$ echo "$(cat /proc/cmdline) fsck.mode=skip workaround" >/tmp/cmdline
$ sudo mount --bind /tmp/cmdline /proc/cmdline

$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-5.15.0-46-generic root=/dev/mapper/nvme--vg-root ro quiet splash mitigations=off vt.handoff=7 fsck.mode=skip workaround

$ sudo /usr/lib/casper/casper-md5check /tmp /dev/zero
.
Check skipped.
^C

This bug was fixed in the package casper - 1.445.3

---------------
casper (1.445.3) focal; urgency=medium

  * casper-md5check: correctly detect fsck.mode=skip if last
    in /proc/cmdline, and add a hint for users (LP: #1892369)

casper (1.445.2) focal; urgency=medium

  * Increase memory for qemu autopkgtest instances to 1G (LP: #1976287)

 -- Mauricio Faria de Oliveira <email address hidden> Sat, 27 Aug 2022 19:37:43 -0300

The verification of the Stable Release Update for casper has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

The fix is available and verified in the 20.04.5 RC images dated 20220831.

I tested both `focal-live-server-amd64.iso` and `focal-desktop-amd64.iso`.

The new "(or try fsck.mode=skip)" string is there, and `fsck.mode=skip`
as the last argument in the kernel cmdline string does skip the check.
The previous workaround of having it, but not as last arg still works.

In the desktop ISO, the behavior isn't as intuitive w/ plymouth/splash,
but the functionality is there and it works, just like in server/above.
(This is likely not an issue, the option is less relevant on desktop.)

In desktop, the message in splash doesn't mention the new string, and
if you append it, it stays at "0% complete" until the boot finishes.
Just remove `splash` to see the behavior as in server.

cheers,
Mauricio

P.S: the autopkgtest regression failure introduced by previous upload
(merged to this one) was fixed [1] (thanks, Steve, for review/merge!)

[1] https://code.launchpad.net/~mfo/autopkgtest-cloud/+git/autopkgtest-cloud/+merge/429182

I don't think it's a good fix

I do see the `or try: fsck.mode=skip` line after boot. But this means I need to reboot again to catch boot menu and (this time) remember to append this line.

A reboot will take 5+ minutes of my life (I don't know why is boot so slow -- maybe server tests all GPUs and all memory. Just want to mention that if your server boots in less than 30 seconds that's not the case for all of us)

Unfortunately, this seems to be as good as it can get
with the limitation on ctrl-c; please see comment #34.

Of course, suggestions for other methods are welcome.

And agreed on server boot time; usually deal w/ 5+min
(sometimes a lot more, depending on raid controllers).