No longer possible to use do-release-upgrade on Groovy

Status changed to 'Confirmed' because the bug affects multiple users.

@Brian Murray, first, thank you so much for confirming the issue and creating this bug report.

Just to be clear, I finally was able to upgrade from groovy to impish by following the instructions in this post: https://blog.invid.eu/2022/02/24/upgrade-eol-ubuntu-server-from-20-10-to-21-10-an-upgrade-from-groovy-to-impish-is-not-supported-with-this-tool-ubuntu-ubuntugeek-ubuntu-linux-eol-upgrade/ ; see also the official documentation: https://help.ubuntu.com/community/EOLUpgrades

To the people new to this thread: see also my comment (https://bugs.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/+bug/1745754/comments/6) in the other bug report: https://bugs.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/+bug/1745754 ; and my notes on the issue: https://gist.github.com/mikbuch/2c349e1281c69818b48893a460c5accb

@Brian Murray, your suggestion for the minimal fix is all OK, but I wonder if a more general mechanism could be implemented, to support upgrading from an EOL release X to a (EOL) release Y (as the `do-release-upgrade` already says/recognizes with which release there was a problem to which: "An upgrade from ‘groovy’ to ‘impish’ is not supported with this tool."). I don't fully grasp how the DistUpgrade.cfg work, maybe my issue is/was related only to groovy. Either way, if the current bug will be resolved, maybe it is also good idea to update the documentation on EOL upgrades, as well (https://help.ubuntu.com/community/EOLUpgrades).

I can help, and I will, as soon as I will get more familiar with the `do-release-upgrade` process.

@Brian Murray, if you don't mind, I will post my notes here, as comments, so that it will be easier for someone else to follow my thought process on getting familiar with the problem, i.e., getting to better know how does `do-release-upgrade` and `DistUpgrade.cfg` work.

It is very important to know all recent Ubuntu release names (https://wiki.ubuntu.com/Releases) in order to keep track with the analysis below.

Ok, so I inspected the `DistUpgrade.cfg` file (https://git.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/tree/data/DistUpgrade.cfg?h=ubuntu%2Fjammy#n109; stored in my file system at `/usr/share/ubuntu-release-upgrader/DistUpgrade.cfg`). It contains:

...
108 [Sources]
109 From=impish
110 To=jammy
...

So it represents my most recent update. (I finally did groovy=>impish, and impish=>jammy.) Then, during the next update, it would look like:

...
108 [Sources]
109 From=jammy
110 To=kinetic
...

(see: https://git.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/tree/data/DistUpgrade.cfg#n110)

I also found the `DistUpgrade.cfg.focal` (https://git.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/tree/data/DistUpgrade.cfg.focal), and it contains:

...
108 [Sources]
109 From=focal
110 To=jammy
...

Interestingly, the `DistUpgrade.cfg.focal` stored at my file system (at "~/.local/share/Trash/files/DistUpgrade.cfg.focal") contains:

...
108 [Sources]
109 From=focal
110 To=hirsute
...

E.g., it was moved to the trash when the `ubuntu-hirsute` branch was in use on my system, see: https://git.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/tree/data/DistUpgrade.cfg.focal?h=ubuntu%2Fhirsute#n109 .

---

@Brian Murray, your other suggestion was to create `demoted.cfg.groovy`. Indeed, in the https://git.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/tree/utils there is no `demoted.cfg.groovy`.

Maybe part of the simplest solution (for the others to use) is, as you suggest, to create "demoted.cfg.groovy" which contains basically the same packages as "demoted.cfg.focal".

---

Brian Murray wrote:

 > "For what its worth I tested an upgrade of a stock Ubuntu 20.10 system to Ubuntu 21.10 by copying DistUpgrade.cfg.focal to DistUpgrade.cfg.groovy and did not run into any issues"

It worked because of this piece of code: lines 44 & 45 here: https://git.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/tree/DistUpgrade/DistUpgradeConfigParser.py#n44
(
```
if os.path.exists(maincfg + "." + from_release):
    maincfg += "." + from_release
```
)

It is not a particularly neat solution, but it will work.

In the ideal world, there would be a mechanism similar to this: https://git.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/tree/DistUpgrade/DistUpgradeConfigParser.py#n39 (from_release = ...) in combination with the list of known releases (https://wiki.ubuntu.com/Releases) for the `DistUpgradeConfigParser.py` not to take fixed release names (from `DistUpgrade.cfg`/focal/groovy), but dynamically decide what is the release on the system, and what is the next available release, so that the `do-release-upgrade` would attempt to upgrade to the next available release. This way upgrades from the EOL systems would be supported, or...

Status changed to 'Confirmed' because the bug affects multiple users.

Hi, @Brian Murray, I tested the solution you suggested on a newly-installed Ubuntu 20.10 Groovy, and it worked. I described the steps to reproduce the problem, and the solution here: https://gist.github.com/mikbuch/2c349e1281c69818b48893a460c5accb?permalink_comment_id=4206418#gistcomment-4206418

I also created a merge request (https://code.launchpad.net/~mikbuch/ubuntu/+source/ubuntu-release-upgrader/+git/ubuntu-release-upgrader/+merge/425341), but I didn't know the patch version for the proper commit message. I am also not sure if I should have committed directly to ubuntu/impish, or rather, e.g., to the ubuntu/impish-devel. Hope that helps you somehow. Let me know if I could do something more to help you.

@Brian data/demoted.cfg.{bionic,focal,trusty,xenial} symlinks exist but one isn't added for the new utils/demoted.cfg.groovy. Is this OK?

Accepting anyway as it's probably fine, but I thought I'd mention it.

Hello Brian, or anyone else affected,

Accepted ubuntu-release-upgrader into impish-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:21.10.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-impish to verification-done-impish. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-impish. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Using the new version of the dist-upgrader I was successfully able to upgrade from Groovy to Impish.

bdmurray@clean-groovy-amd64:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=21.10
DISTRIB_CODENAME=impish
DISTRIB_DESCRIPTION="Ubuntu 21.10"
bdmurray@clean-groovy-amd64:~$ head -n20 /var/log/dist-upgrade/main.log
2022-07-06 08:12:44,568 INFO Using config files '['./DistUpgrade.cfg.groovy', '/etc/update-manager/release-upgrades.d/ubuntu-advantage-upgr
ades.cfg']'
2022-07-06 08:12:44,568 INFO uname information: 'Linux clean-groovy-amd64 5.8.0-63-generic #71-Ubuntu SMP Tue Jul 13 15:59:12 UTC 2021 x86_
64'
2022-07-06 08:12:44,852 INFO apt version: '2.1.10ubuntu0.3'
2022-07-06 08:12:44,852 INFO python version: '3.8.10 (default, Jun 2 2021, 10:49:15)
[GCC 10.3.0]'
2022-07-06 08:12:44,854 INFO release-upgrader version '21.10.10' started
2022-07-06 08:12:44,858 INFO locale: 'en_US' 'UTF-8'
2022-07-06 08:12:44,941 DEBUG Using 'DistUpgradeViewText' view
2022-07-06 08:12:44,984 DEBUG enable dpkg --force-overwrite
2022-07-06 08:12:45,029 DEBUG creating statefile: '/var/log/dist-upgrade/apt-clone_system_state.tar.gz'
2022-07-06 08:12:47,118 DEBUG lsb-release: 'groovy'
2022-07-06 08:12:47,119 DEBUG _pythonSymlinkCheck run
2022-07-06 08:12:47,124 DEBUG openCache()
2022-07-06 08:12:47,124 DEBUG quirks: running PreCacheOpen
...
bdmurray@clean-groovy-amd64:~$ tail -n10 /var/log/dist-upgrade/main.log
2022-07-06 08:37:51,460 DEBUG 'linux-image-5.8.0-63-generic' scheduled for remove but not safe to remove, skipping
2022-07-06 08:37:52,139 DEBUG skipping running kernel pkg 'linux-headers-5.8.0-63-generic'
2022-07-06 08:37:52,140 DEBUG 'linux-headers-5.8.0-63-generic' scheduled for remove but not safe to remove, skipping
2022-07-06 08:37:53,112 DEBUG Finish checking for obsolete pkgs
2022-07-06 08:37:53,165 DEBUG The following packages are marked for removal: ca-certificates-java libgssapi3-heimdal default-jre-headless l
iblibreoffice-java liborcus-parser-0.15-0 default-jre libedataserverui-1.2-2 libwind0-heimdal libridl-java libldap-2.4-2 libboost-iostreams
1.71.0 realmd libgdk-pixbuf2.0-0 libatk-wrapper-java linux-image-5.8.0-25-generic libpython3.8-minimal vino libraw19 gcc-10-base libjuh-jav
a libheimntlm0-heimdal xul-ext-ubufox libtracker-sparql-2.0-0 libamtk-5-common perl-modules-5.30 libheimbase1-heimdal libpython3.8 python3.
8 linux-modules-extra-5.8.0-25-generic libbrlapi0.7 linux-headers-5.8.0-25 linux-headers-5.8.0-63 libboost-locale1.71.0 libappindicator3-1
ure-java libhandy-0.0-0 openjdk-11-jre-headless linux-headers-5.8.0-63-generic libisl22 libboost-filesystem1.71.0 libtepl-5-0 lz4 linux-hea
ders-5.8.0-25-generic libtracker-miner-2.0-0 libatk-wrapper-java-jni sssd-dbus adcli libperl5.30 libllvm11 libsnmp35 libboost-date-time1.71
.0 libboost-thread1.71.0 libjurt-java libfwupdplugin1 linux-modules-5.8.0-25-generic python3.8-minimal openjdk-11-jre ubuntu-wallpapers-gro
ovy libhcrypto4-heimdal fonts-dejavu-extra sssd-tools liborcus-0.15-0 mysql-common libmysqlclient21 libphonenumber7 libpython3.8-stdlib lib
tracker-control-2.0-0 libroken18-heimdal libpoppler102 firefox gdbserver libasn1-8-heimdal libkrb5-26-heimdal python3-requests-unixsoc...

Additionally automatic upgrade testing from Groovy to Impish (for Ubuntu and flavors) is succeeding and the results are viewable to Canonical employees only (sorry!) here:

https://platform-qa-jenkins.ubuntu.com/view/Upgrade/

This bug was fixed in the package ubuntu-release-upgrader - 1:21.10.10

---------------
ubuntu-release-upgrader (1:21.10.10) impish; urgency=medium

  * Ship a DistUpgrade.cfg.groovy and demoted.cfg.groovy to allow upgrades
    from Groovy to Impish to be allowed. (LP: #1975533)
  * Run pre-build.sh: updating mirrors, demotions, and translations.

 -- Brian Murray <email address hidden> Mon, 27 Jun 2022 15:47:00 -0700

The verification of the Stable Release Update for ubuntu-release-upgrader has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Hello Brian, or anyone else affected,

Accepted ubuntu-release-upgrader into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:22.04.12 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted ubuntu-release-upgrader (1:22.04.12) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

ubuntu-release-upgrader/1:22.04.12 (ppc64el, s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#ubuntu-release-upgrader

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

The attached screenshot shows a successful upgrade from Groovy to Jammy!

bdmurray@clean-groovy-amd64:~$ head /var/log/dist-upgrade/main.log
2022-07-22 12:37:59,572 INFO Using config files '['./DistUpgrade.cfg.groovy', '/etc/update-manager/release-upgrades.d/ubuntu-advantage-upgrades.cfg']'
2022-07-22 12:37:59,572 INFO uname information: 'Linux clean-groovy-amd64 5.8.0-63-generic #71-Ubuntu SMP Tue Jul 13 15:59:12 UTC 2021 x86_64'
2022-07-22 12:37:59,881 INFO apt version: '2.1.10ubuntu0.3'
2022-07-22 12:37:59,881 INFO python version: '3.8.10 (default, Jun 2 2021, 10:49:15)
[GCC 10.3.0]'
2022-07-22 12:37:59,883 INFO release-upgrader version '22.04.12' started
2022-07-22 12:37:59,888 INFO locale: 'en_US' 'UTF-8'
2022-07-22 12:37:59,970 DEBUG Using 'DistUpgradeViewText' view
2022-07-22 12:38:00,024 DEBUG enable dpkg --force-overwrite
2022-07-22 12:38:00,067 DEBUG creating statefile: '/var/log/dist-upgrade/apt-clone_system_state.tar.gz'

With the version of ubuntu-release-upgrader in -proposed I was also able to conduct a successful upgrade from Hirsute to Jammy.

bdmurray@clean-hirsute-amd64:/tmp/uru$ sudo ./jammy --frontend DistUpgradeViewText
[screen is terminating]
bdmurray@clean-hirsute-amd64:/tmp/uru$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04 LTS"
bdmurray@clean-hirsute-amd64:/tmp/uru$ head /var/log/dist-upgrade/main.log
2022-07-22 18:18:41,143 INFO Using config files '['./DistUpgrade.cfg.hirsute', '/etc/update-manager/release-upgrades.d/ubuntu-advantage-upgrades.cfg']'
2022-07-22 18:18:41,144 INFO uname information: 'Linux clean-hirsute-amd64 5.11.0-49-generic #55-Ubuntu SMP Wed Jan 12 17:36:34 UTC 2022 x86_64'
2022-07-22 18:18:41,405 INFO apt version: '2.2.4ubuntu0.1'
2022-07-22 18:18:41,406 INFO python version: '3.9.5 (default, Nov 18 2021, 16:00:48)
[GCC 10.3.0]'
2022-07-22 18:18:41,408 INFO release-upgrader version '22.04.12' started
2022-07-22 18:18:41,411 INFO locale: 'en_US' 'UTF-8'
2022-07-22 18:18:41,484 DEBUG Using 'DistUpgradeViewText' view
2022-07-22 18:18:41,542 DEBUG enable dpkg --force-overwrite
2022-07-22 18:18:41,591 DEBUG creating statefile: '/var/log/dist-upgrade/apt-clone_system_state.tar.gz'

This bug was fixed in the package ubuntu-release-upgrader - 1:22.04.12

---------------
ubuntu-release-upgrader (1:22.04.12) jammy; urgency=medium

  [ Brian Murray ]
  * Add support for upgrading from End of Life releases (Ubuntu 20.10 and
    Ubuntu 21.04) to Ubuntu 22.04. (LP: #1975533)
  * DistUpgrade: Do not attempt to reboot in WSL. (LP: #1958668)
  * DistUpgrade/deb2snap.json: gnome-3-34-1804 is no longer a seeded snap and
    should not be refreshed. (LP: #1981485)
  * Run pre-build.sh: updating mirrors, demotions, and translations.

  [ William 'jawn-smith' Wilson ]
  * tests/test_quirks.py: Remove declaration of unused variables

 -- Brian Murray <email address hidden> Tue, 19 Jul 2022 06:52:28 -0700

Great news, thank you for the work on this issue!