bad interaction between snapd and update-notifier when snapd package is being upgraded

Hi Steve, thanks for reporting this, and for the hint on how to fix it. We currently have a shell script in our prerm script, which manually stops all our systemd units:

    https://github.com/snapcore/snapd/blob/master/packaging/ubuntu-16.04/snapd.prerm

Do I correctly understand that your suggestion is not to do that?

I'm actually a bit confused (though this is probably a question for my own team) why we have that prerm script, when we are already using dh_systemd_start in out debian/rules file:

==============
override_dh_systemd_start:
 # Due to https://github.com/systemd/systemd/issues/8102
 # debhelper supporting --remaining like behaviour here would
 # have been useful
 # (Re-)start socket first
 dh_systemd_start snapd.socket
 # Then service
 dh_systemd_start snapd.service
 # Then the rest
 dh_systemd_start $(filter-out snapd.socket snapd.service, $(shell ls debian/snapd/lib/systemd/system/))
==============

wouldn't this rule already cause the units to be stopped at the prerm step (we are using debhelper compatibility level 9)?

Anyway, I'm soon going to propose a PR, so we can discuss over it.

Here's the PR: https://github.com/snapcore/snapd/pull/11680

And please forget what I wrote about our prerm script: that script is not stopping snapd, but only all the systemd service installed *by snap packages*, so it should be fine.

Status changed to 'Confirmed' because the bug affects multiple users.

I uploaded a cherry pick of Albertos branch with the current jammy snapd plus the fix to https://launchpad.net/~mvo/+archive/ubuntu/tmp/+packages and will test once it's build in there.

On a minimal impish system (created with autopkgtest-buildvm) I ran:
$ sudo apt install --no-install--recommends snapd update-notifier
$ sudo snap install canonical-livepatch
$ sudo sed -i 's/impish/jammy/' /etc/apt/sources.list
$ sudo apt update
$ sudo apt full-upgrade -y
and with that I can reproduce the hang.

On a minimal impish system (created with autopkgtest-buildvm) I ran:
$ sudo apt install --no-install--recommends snapd update-notifier
$ sudo snap install canonical-livepatch
$ sudo apt-add-repository ppa:mvo/tmp
$ sudo sed -i 's/impish/jammy/' /etc/apt/sources.list
$ sudo sed -i 's/impish/jammy/' /etc/apt/sources.list.d/mvo-ubuntu-tmp-impish.list
$ sudo apt update
$ apt list -a snapd
[shows the PPA version as the one that will be upgraded to]
$ sudo apt full-upgrade -y

With that it still hangs, most likely because the *old* prerm is run which will stop snapd during the upgrade.

I uploaded a version of snapd with the changes from https://github.com/snapcore/snapd/pull/11680 and https://github.com/snapcore/snapd/pull/11287 to the my ppa:mvo/tmp for impish too.

On a minimal impish system (created with autopkgtest-buildvm) I ran:
$ sudo apt-add-repository ppa:mvo/tmp
$ sudo apt install --no-install--recommends snapd update-notifier
$ sudo snap install canonical-livepatch
$ apt list -a snapd
[shows the impish PPA version is used]
$ sudo sed -i 's/impish/jammy/' /etc/apt/sources.list
$ sudo sed -i 's/impish/jammy/' /etc/apt/sources.list.d/mvo-ubuntu-tmp-impish.list
$ sudo apt update
$ apt list -a snapd
[shows the PPA version as the one that will be upgraded to]
$ sudo apt full-upgrade -y

with that the upgrade does not hang anymore.

Accepting the jammy version into -proposed for a 0-day SRU.

Hello Steve, or anyone else affected,

Accepted snapd into impish-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/snapd/2.54.3+21.10.1ubuntu0.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-impish to verification-done-impish. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-impish. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

There is one thing I noticed - when transitioning from "dh_systemd_start" to "dh_systemd_start --restart-after-upgrade" at least on 16.04 snapd.service is stopped in the old prerm but when the new postinst runs it calls "systemctl restart snapd" but that only restarts a service if it was not stopped before by systemctl so one ends up without a running snapd. I need to check if that is just 16.04 (it was a failure in our CI) or if it also affects 21.10 and later (hope this makes sense)

It seems the above is not an issue on 21.10 at least.

ah whoops-please undo

I ran the spread test in the snapd PR https://github.com/snapcore/snapd/pull/11702 against this change in 21.10 and 22.04 and the restart behavior is correct: if snapd is upgraded from a version that does not yet use "dh_systemd --restart-after-upgrade" still keep running.

This bug was fixed in the package snapd - 2.55.3+22.04ubuntu1

---------------
snapd (2.55.3+22.04ubuntu1) jammy; urgency=medium

  * Cherry-pick https://github.com/snapcore/snapd/pull/11680
    - This fixes a bad interaction between snapd and update-notifier
      during a release upgrade (LP: #1969162)

 -- Michael Vogt <email address hidden> Thu, 21 Apr 2022 10:50:55 +0200

The verification of the Stable Release Update for snapd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package snapd - 2.54.3+21.10.1ubuntu0.3

---------------
snapd (2.54.3+21.10.1ubuntu0.3) impish; urgency=medium

  * Cherry-pick https://github.com/snapcore/snapd/pull/11680 and
    https://github.com/snapcore/snapd/pull/11287:
    - This fixes a bad interaction between snapd and update-notifier
      during a release upgrade (LP: #1969162)

 -- Michael Vogt <email address hidden> Thu, 21 Apr 2022 12:11:17 +0200

Now that a fix has been released for both affected versions, what's left before upgrades are enabled for 21.10→22.04?
(new to Ubuntu and Launchpad, so feel free to point out the seemingly obvious)

@mrconorae: I've just checked that the upgrade is enabled:
---------------------------
$ sudo do-release-upgrade -c
Checking for a new Ubuntu release
New release '22.04 LTS' available.
Run 'do-release-upgrade' to upgrade to it.

Hello Steve, or anyone else affected,

Accepted snapd into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/snapd/2.54.3+20.04.1ubuntu0.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Fwiw, I tested this also for focal->jammy on my laptop (real machien) and it worked fine there.

I followed the test case as detailed in the bug description and did not observe a 10 minute wait during the dbus package upgrade.

I also tested a distribution upgrade from Focal to Jammy with the canonical-livepatch snap installed and did not notice a 10 minute pause during the process.

This bug was fixed in the package snapd - 2.54.3+20.04.1ubuntu0.3

---------------
snapd (2.54.3+20.04.1ubuntu0.3) focal; urgency=medium

  * Cherry-pick https://github.com/snapcore/snapd/pull/11680 and
    https://github.com/snapcore/snapd/pull/11287:
    - This fixes a bad interaction between snapd and update-notifier
      during a release upgrade (LP: #1969162)

 -- Michael Vogt <email address hidden> Wed, 27 Apr 2022 18:12:51 +0200