package sosreport 4.4-1ubuntu0.18.04.1 failed to install/upgrade: installed sosreport package post-installation script subprocess returned error exit status 1

Thanks for reporting this bug.

This is an interesting one. The user is upgrading from xenial to bionic.

xenial has python 3.5, bionic has python 3.6.

During the postinst (included by dh_python), sosreport runs a python script containing an f-string. However f-strings were only introduced in python 3.6. In this case, the postinst ran with python 3.5, which results in a syntax error when the f-string is parsed.

This sounds like it needs a versioned depends on a higher version of Python, but this might mess with the ordering of package upgrades on release upgrade, so needs to be considered carefully.

Probably a fix isn't needed in series after Bionic? Needs checking.

High for Bionic because this affects release upgrades from Xenial to Bionic when sosreport is installed, which is something that Canonical recommends to its customers.

And indeed it's installed by default on server.

I looked into this issue, and also discussed with mfo.

The sos code that fails compilation due to the python 3.6 package not yet being installed is,

    @staticmethod
    def _query_address(url: str) -> HTTPResponse:
        """
        Query the given url address with headers required by Google Metadata
        Server.
        """
        try:
            req = request.Request(url, headers={'Metadata-Flavor': 'Google'})
            response = request.urlopen(req)
        except URLError as err:
            raise RuntimeError(
                "Failed to communicate with Metadata Server: " + str(err))
        if response.code != 200:
            raise RuntimeError(
                f"Failed to communicate with Metadata Server " <---
                f"(code: {response.code}): " + response.read().decode()) <----
        return response

We have decided to hold off the 4.5.6 release on bionic since 4.5.6 has many more plugins using f strings.

For this version, it's just the gcp plugin that has f strings in the bionic sos. But 4.5.6 source has more than hundred instances (https://pastebin.canonical.com/p/8DpdjwWKM4/)

Looking further,

The generated postinst tries to build py3compile sosreport, (Thanks mfo for this info!)

$ pull-lp-debs sosreport bionic 4.4-1ubuntu0.18.04.1
Found sosreport 4.4-1ubuntu0.18.04.1 in bionic
Downloading sosreport_4.4-1ubuntu0.18.04.1_amd64.deb from archive.ubuntu.com (0.291 MiB)
[=====================================================>]100%
:~/Downloads/dpkgtest$ mkdir ext
:~/Downloads/dpkgtest$ dpkg-deb -e
ext/ sosreport_4.4-1ubuntu0.18.04.1_amd64.deb
:~/Downloads/dpkgtest$ dpkg-deb -e sosreport_4.4-1ubuntu0.18.04.1_amd64.deb ext/
:~/Downloads/dpkgtest$ cd ext/
:~/Downloads/dpkgtest/ext$ ls
conffiles control md5sums postinst prerm
:~/Downloads/dpkgtest/ext$ cat postinst
#!/bin/sh
set -e

# Automatically added by dh_python3:
if which py3compile >/dev/null 2>&1; then
 py3compile -p sosreport <----
fi

# End automatically added section

There's a couple of things we can do for a fix.

Either have the depends in the control file using something like python3 (>= 3.6.5-3) which might be an easier fix, or try to generate a check additionally in the postinst that dh_python generates, that does not compile sos if the version is bionic and python 3.6 isn't available.

The series after bionic seem fine,

# rmadison python3
 python3 | 3.4.0-0ubuntu2 | trusty | amd64, arm64, armhf, i386, powerpc, ppc64el
 python3 | 3.5.1-3 | xenial | amd64, arm64, armhf, i386, powerpc, ppc64el, s390x
 python3 | 3.6.5-3 | bionic | amd64, arm64, armhf, i386, ppc64el, s390x
 python3 | 3.6.7-1~18.04 | bionic-updates | amd64, arm64, armhf, i386, ppc64el, s390x
 python3 | 3.8.2-0ubuntu2 | focal | amd64, arm64, armhf, i386, ppc64el, riscv64, s390x
 python3 | 3.10.4-0ubuntu2 | jammy | amd64, arm64, armhf, i386, ppc64el, riscv64, s390x
 python3 | 3.10.6-1~22.04 | jammy-security | amd64, arm64, armhf, i386, ppc64el, riscv64, s390x
 python3 | 3.10.6-1~22.04 | jammy-updates | amd64, arm64, armhf, i386, ppc64el, riscv64, s390x
 python3 | 3.11.2-1...

I could not reproduce this in a GCP VM or LXD VM.
(launch 16.04, apt update && apt -y upgrade, reboot, do-release-upgrade).

# grep -A1 'Setting up sosreport' /var/log/dist-upgrade/apt-term.log
Setting up sosreport (4.4-1ubuntu0.18.04.1) ...
Setting up xfsprogs (4.9.0+nmu1ubuntu2) ...

Hi Branimir,

Could you please upload /var/log/dist-upgrade/screenlog.0
(or a tarball with the full directory contents) ?

Thanks!

If it turns out to be needed, we could consider a minimal change to bionic-updates for now (details in .patch comment).

Hi Mauricio,

I could not reproduce this either,

root@dixie:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.6 LTS
Release: 18.04
Codename: bionic
root@dixie:~# dpkg -l | grep sos
ii sosreport 4.4-1ubuntu0.18.04.1 amd64 Set of tools to gather troubleshooting data from a system

Looking at /var/log/dist-upgrade/apt-term.log, python3 got installed and set up earlier than sos,

1704 Setting up python3.6 (3.6.9-1~18.04ubuntu1.12) ...^M
1705 Setting up libpython3-stdlib:amd64 (3.6.7-1~18.04) ...^M
1706 Setting up python3 (3.6.7-1~18.04) ...^M
1707 running python rtupdate hooks for python3.6...^M

2003 Setting up python3-pexpect (4.2.1-1) ...^M
2004 Setting up sosreport (4.4-1ubuntu0.18.04.1) ...^M
2005 Setting up xfsprogs (4.9.0+nmu1ubuntu2) ...^M

For bionic 4.5.6 to be able to release on bionic (https://warthogs.atlassian.net/browse/SEC-2624), we would need a solution for the more than hundred instances of formatted strings used in 4.5.6. (https://pastebin.canonical.com/p/8DpdjwWKM4/)

I noted "The optional X-Python3-Version field specifies the versions of Python 3 supported. ", but I am not sure if this would apply for the compilation in the postinst.

The Depends clause currently is,

Depends: ${python3:Depends}, ${misc:Depends}, python3-pexpect

Does adding python3 (>= 3.6.5-3) to Depends sound like a reasonable solution?

Hi Nikhil,

Thanks for checking!

> I noted "The optional X-Python3-Version field specifies the versions of Python 3 supported. ", but I am not sure if this would apply for the compilation in the postinst.

The version of the package providing py3compile at postinst time should be the key factor, IIUIC?

> Does adding python3 (>= 3.6.5-3) to Depends sound like a reasonable solution?

IMHO we first should understand the issue/criteria for reproducing the problem;
so I'd wait for Branimir to reply, or we try proactively to reproduce the issue,
e.g., should pro not be enabled? (there are no 'esm' versions in attached logs.)

When that is clear, we should be in a better position to assess the impact/pros/cons of different approaches to address it.

And as Robie mentioned in comment #3, if we change that, we should be clear on the changes/impacts to do-release-upgrade
(that's why I suggested in our chat to run do-release-upgrade before/after such a change, and compare the logs to look for differences, and also understand the steps done by do-release-ugprade in general w.r.t. python, if any).

Additionally, for the future, we could use this opportunity to further improve the sosreport SRU exception/test coverage [1,2] with release upgrade testing too (after understanding the details here; and it should be easy to automate).

Apparently this issue got in the SRU of sosreport 4.3 [3], but only got a bug report at the 4.4 timeframe.
(Ok, this means it isn't apparently widespread, which is good news, but we can always improve on catching issues early! :)

 --- sosreport-4.1/sos/report/plugins/gcp.py 1970-01-01 00:00:00.000000000 +0000
 +++ sosreport-4.3/sos/report/plugins/gcp.py 2022-02-15 04:20:20.000000000 +0000
 ...
 + f"Failed to communicate with Metadata Server "
 + f"(code: {response.code}): " + response.read().decode())
 ...

 ...

Thanks again for all your work on this!

[1] https://wiki.ubuntu.com/StableReleaseUpdates#sosreport
[2] https://wiki.ubuntu.com/SosreportUpdates
[3] https://launchpadlibrarian.net/588301517/sosreport_4.1-1ubuntu0.18.04.3_4.3-1ubuntu0.18.04.1.diff.gz

(comment #12)
> IMHO we first should understand the issue/criteria for reproducing the problem;
> so I'd wait for Branimir to reply, or we try proactively to reproduce the issue,
> e.g., should pro not be enabled? (there are no 'esm' versions in attached logs.)

I tried to reproduce this scenario too (or close to it): launched 16.04 Pro VM in GCP and downgraded the packages with 'esm' version suffix to the version in xenial-updates (this changes package status, so might not reflect the reporter's environment), then do-release-upgrade.

The issue didn't reproduce.

$ grep -A1 'Setting up sosreport' /var/log/dist-upgrade/apt-term.log
Setting up sosreport (4.4-1ubuntu0.18.04.1) ...
Setting up xfsprogs (4.9.0+nmu1ubuntu2) ...

I also tried to reproduce in a 16.04 LXD VM, which does not have any esm packages installed by default (so the package status change above wasn't an issue), and the issue didn't reproduce either.

# grep -A1 'Setting up sosreport' /var/log/dist-upgrade/apt-term.log
Setting up sosreport (4.4-1ubuntu0.18.04.1) ...
Setting up xfsprogs (4.9.0+nmu1ubuntu2) ...

So, after all these attempts to reproduce the issue, and failing to do so, it apparently indicates the issue might be related to something particular in this VM/system packages status?

Hopefully Branimir's logs requested in comment #9 may help understand this better.

(Updating sosreport/devel release to Invalid as this issue only affects Xenial -> Bionic upgrades.)