MAAS

MAAS 2.9.3 RC2 snap install fails with `pkg_resources.extern.packaging.version.InvalidVersion: Invalid version: '2.2.11.0ubuntu20.04'`

Bug #2030814 reported by Mauricio Faria de Oliveira
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Invalid
Undecided
Unassigned
2.9
Fix Released
Medium
Mauricio Faria de Oliveira
MAAS 2.9.3

Bug Description

Reproducer:

 $ snap info maas | fgrep 2.9/edge
   2.9/edge: 2.9.3~rc2-9274-g.9feb47e75 2023-06-08 (28600) 141MB -

 $ sudo snap install --channel=2.9/edge maas
 error: cannot perform the following tasks:
 - Run install hook of "maas" snap if present (run hook "install":
 -----
 Traceback (most recent call last):
   File "/snap/maas/28600/bin/maas", line 8, in <module>
     sys.exit(main())
   File "/snap/maas/28600/lib/python3.8/site-packages/maascli/__init__.py", line 39, in main
     parser = prepare_parser(argv)
   File "/snap/maas/28600/lib/python3.8/site-packages/maascli/parser.py", line 75, in prepare_parser
     register_cli_commands(parser)
   File "/snap/maas/28600/lib/python3.8/site-packages/maascli/cli.py", line 274, in register_cli_commands
     django_setup()
   File "/snap/maas/28600/usr/lib/python3/dist-packages/django/__init__.py", line 24, in setup
     apps.populate(settings.INSTALLED_APPS)
   File "/snap/maas/28600/usr/lib/python3/dist-packages/django/apps/registry.py", line 114, in populate
     app_config.import_models()
   File "/snap/maas/28600/usr/lib/python3/dist-packages/django/apps/config.py", line 211, in import_models
     self.models_module = import_module(models_module_name)
   File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
     return _bootstrap._gcd_import(name[level:], package, level)
   File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
   File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
   File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
   File "<frozen importlib._bootstrap_external>", line 848, in exec_module
   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
   File "/snap/maas/28600/lib/python3.8/site-packages/maasserver/models/__init__.py", line 107, in <module>
     from maasserver.models.bmc import (
   File "/snap/maas/28600/lib/python3.8/site-packages/maasserver/models/bmc.py", line 56, in <module>
     from maasserver.models.node import get_default_zone, Machine, Node
   File "/snap/maas/28600/lib/python3.8/site-packages/maasserver/models/node.py", line 194, in <module>
     from provisioningserver.drivers.power.registry import PowerDriverRegistry
   File "/snap/maas/28600/lib/python3.8/site-packages/provisioningserver/drivers/power/registry.py", line 9, in <module>
     from provisioningserver.drivers.pod.registry import PodDriverRegistry
   File "/snap/maas/28600/lib/python3.8/site-packages/provisioningserver/drivers/pod/registry.py", line 10, in <module>
     from provisioningserver.drivers.pod.lxd import LXDPodDriver
   File "/snap/maas/28600/lib/python3.8/site-packages/provisioningserver/drivers/pod/lxd.py", line 11, in <module>
     from pylxd import Client
   File "/snap/maas/28600/usr/lib/python3/dist-packages/pylxd/__init__.py", line 17, in <module>
     __version__ = pbr.version.VersionInfo('pylxd').version_string()
   File "/snap/maas/28600/usr/lib/python3/dist-packages/pbr/version.py", line 467, in version_string
     return self.semantic_version().brief_string()
   File "/snap/maas/28600/usr/lib/python3/dist-packages/pbr/version.py", line 462, in semantic_version
     self._semantic = self._get_version_from_pkg_resources()
   File "/snap/maas/28600/usr/lib/python3/dist-packages/pbr/version.py", line 442, in _get_version_from_pkg_resources
     provider = pkg_resources.get_provider(requirement)
   File "/snap/maas/28600/lib/python3.8/site-packages/pkg_resources/__init__.py", line 398, in get_provider
     return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
   File "/snap/maas/28600/lib/python3.8/site-packages/pkg_resources/__init__.py", line 695, in find
     if dist is not None and dist not in req:
   File "/snap/maas/28600/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3204, in __contains__
     return self.specifier.contains(item, prereleases=True)
   File "/snap/maas/28600/lib/python3.8/site-packages/pkg_resources/_vendor/packaging/specifiers.py", line 902, in contains
     item = Version(item)
   File "/snap/maas/28600/lib/python3.8/site-packages/pkg_resources/_vendor/packaging/version.py", line 197, in __init__
     raise InvalidVersion(f"Invalid version: '{version}'")
 pkg_resources.extern.packaging.version.InvalidVersion: Invalid version: '2.2.11.0ubuntu20.04'
 -----)

Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

The issue does not occur with DEB, only the SNAP.

Changed in maas:
status: New → Invalid
Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

The version string is from python3-pylxd in in ppa:maas/2.9{-next}.

 $ grep -Fr 2.2.11.0ubuntu20.04
 usr/lib/python3/dist-packages/pylxd-2.2.11.0ubuntu20.04.egg-info/PKG-INFO:Version: 2.2.11.0ubuntu20.04

It comes from python pkg-resources not being pinned at the snap,
and recently (early 2023) removed deprecated (non-PEP 440) type
'LegacyVersion' (which is more free form), leaving just 'Version'.

@ maas_*.snap:./lib/python3.8/site-packages/pkg_resources/_vendor/packaging/specifiers.py

The old 2.9.2 (2.9/stable) works:

    def contains(self, item, prereleases=None):
        # Ensure that our item is a Version or LegacyVersion instance.
        if not isinstance(item, (LegacyVersion, Version)):
            item = parse(item)

The new 2.9.3rc2 (2.9/edge) fails (no 'LegacyVersion' anymore)

        # Ensure that our item is a Version instance.
        if not isinstance(item, Version):
            item = Version(item)

The python3-pkg-resources package in the Ubuntu Archive,
which is used by the MAAS deb, works, as it's still old:

@ /usr/lib/python3/dist-packages/pkg_resources/_vendor/packaging/specifiers.py

    def contains(self, item, prereleases=None):
        # Ensure that our item is a Version or LegacyVersion instance.
        if not isinstance(item, (LegacyVersion, Version)):
            item = parse(item)

Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

This is the git commit in setuptools that introduced the regression:

 $ git describe --contains 2aad1a041e9c7
 v67.0.0~1^2~8

 $ git show v67.0.0
 ...
 Date: Sat Jan 28 12:58:37 2023 -0500
 ...

Interestingly enough, a warning about that issue is present in the
build log of python-pylxd in ppa:maas/2.9{-next}:

 ...
 running install_egg_info
 Copying pylxd.egg-info to /<<PKGBUILDDIR>>/debian/python3-pylxd/usr/lib/python3/dist-packages/pylxd-2.2.11.0ubuntu20.04.egg-info
 Skipping SOURCES.txt
 running install_scripts
 /usr/lib/python3/dist-packages/setuptools/dist.py:481: UserWarning: The version specified ('2.2.11.0ubuntu20.04') is an invalid version, this may not work as expected with newer versions of setuptools, pip, and PyPI. Please see PEP 440 for more details.
 ...

Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

The newer MAAS versions (3.0) and later do not have this issue,
as python-pylxd version in the egg (PKG-INFO file) is set to
'<version>a0' (i.e., alpha0) regardless of `debian/changelog`.

In MAAS 2.9 the packaging/build system uses `debian/changelog`,
and sets PBR_VERSION accordingly, that is used to generate the
egg info/PKG-INFO file.
(See /usr/share/perl5/Debian/Debhelper/Buildsystem/pybuild.pm,
in dh-python, which checks for a python3-pbr build dependency.)

Possible solutions include pinning python-pkg-resources in snap
build (ie, staging it for runtime, so the package in the archive
is used) or patching python-pylxd in ppa:maas/2.9{-next} to fix
the version.

The latter seems the best approach, since all later MAAS versions
do _not_ pin/stage this package as dependency, and python-pylxd
ships egg info/PKG-INFO file with Version with an 'a0' suffix
regardless, not based on changelog/package/git version at all.

So, we'll stick with the latter for consistency with later versions.

Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :
Download full text (4.5 KiB)

Comparison of MAAS versions (3.4 down to 2.9):

MAAS 3.4)

3.4$ pull-ppa-debs --ppa maas/3.4-next python3-pylxd
3.4$ dpkg-deb -x python3-pylxd_2.3.2~alpha1-435-10-g.43f5eb1~ubuntu22.04.1_all.deb deb
3.4$ grep ^Version: deb/usr/lib/python3/dist-packages/pylxd-*.egg-info/PKG-INFO
Version: 2.3.2a0

3.4$ snap download --channel=3.4/edge maas
3.4$ unsquashfs maas_*.snap
3.4$ grep ^Version: squashfs-root/usr/lib/python3/dist-packages/pylxd-*.egg-info/PKG-INFO
Version: 2.3.2a0

3.4$ grep -A2 '# Ensure that our item' squashfs-root/lib/python3.*/site-packages/pkg_resources/_vendor/packaging/specifiers.py
        # Ensure that our item is a Version instance.
        if not isinstance(item, Version):
            item = Version(item)

MAAS 3.3)

3.3$ pull-ppa-debs --ppa maas/3.3-next python3-pylxd
3.3$ dpkg-deb -x python3-pylxd_2.3.2~alpha1-420-10-g.72426bf~ubuntu22.04.1_all.deb deb
3.3$ grep ^Version: deb/usr/lib/python3/dist-packages/pylxd-*.egg-info/PKG-INFO
Version: 2.3.2a0

3.3$ snap download --channel=3.3/edge maas
3.3$ unsquashfs maas_*.snap
3.3$ grep ^Version: squashfs-root/usr/lib/python3/dist-packages/pylxd-*.egg-info/PKG-INFO
Version: 2.3.2a0

3.3$ grep -A2 '# Ensure that our item' squashfs-root/lib/python3.*/site-packages/pkg_resources/_vendor/packaging/specifiers.py
        # Ensure that our item is a Version instance.
        if not isinstance(item, Version):
            item = Version(item)

MAAS 3.2)

3.2$ pull-ppa-debs --ppa maas/3.2-next python3-pylxd
3.2$ dpkg-deb -x python3-pylxd_2.3.2~alpha1-426-10-g.9974257~ubuntu20.04.1_all.deb deb
3.2$ grep ^Version: deb/usr/lib/python3/dist-packages/pylxd-*.egg-info/PKG-INFO
Version: 2.3.2a0

3.2$ snap download --channel=3.2/edge maas
3.2$ unsquashfs maas_*.snap
3.2$ grep ^Version: squashfs-root/usr/lib/python3/dist-packages/pylxd-*.egg-info/PKG-INFO
Version: 2.3.2a0

3.2$ grep -A2 '# Ensure that our item' squashfs-root/lib/python3.*/site-packages/pkg_resources/_vendor/packaging/specifiers.py
        # Ensure that our item is a Version instance.
        if not isinstance(item, Version):
            item = Version(item)

MAAS 3.1)

3.1$ pull-ppa-debs --ppa maas/3.1-next python3-pylxd
3.1$ dpkg-deb -x python3-pylxd_2.3.2~alpha1-426-10-g.9974257~ubuntu20.04.1_all.deb deb
3.1$ grep ^Version: deb/usr/lib/python3/dist-packages/pylxd-*.egg-info/PKG-INFO
Version: 2.3.2a0

3.1$ snap download --channel=3.1/edge maas
3.1$ unsquashfs maas_*.snap
3.1$ grep ^Version: squashfs-root/usr/lib/python3/dist-packages/pylxd-*.egg-info/PKG-INFO
Version: 2.3.2a0

3.1$ grep -A2 '# Ensure that our item' squashfs-root/lib/python3.*/site-packages/pkg_resources/_vendor/packaging/specifiers.py
        # Ensure that our item is a Version instance.
        if not isinstance(item, Version):
            item = Version(item)

MAAS 3.0)

3.0$ pull-ppa-debs --ppa maas/3.0-next python3-pylxd
3.0$ dpkg-deb -x python3-pylxd_2.3.1~alpha1-386-8-g.b36a7ac~ubuntu20.04.1_all.deb deb
3.0$ grep ^Version: deb/usr/lib/python3/dist-packages/pylxd-*.egg-info/PKG-INFO
Version: 2.3.1a0

3.0$ snap download --channel=3.0/edge maas
3.0$ unsquashfs maas_*.snap
3.0$ grep ^Version: squashfs-root/usr/lib/python3/dist-packages/pylxd-*.egg-info/PKG-INFO
Ver...

Read more...

Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

Reproducer with MAAS 2.9 (2.9.3 on edge, 2.9.2 on stable)

New (fails)

2.9$ snap download --channel=2.9/edge maas
2.9$ unsquashfs maas_*.snap
2.9$ cd squashfs-root
2.9/squashfs-root$ PYTHONPATH=$PWD/usr/lib/python3/dist-packages:$PWD/usr/lib/python3.8/dist-packages:$PWD/lib/python3.8/site-packages python3 -q -c "from pkg_resources._vendor.packaging.specifiers import SpecifierSet; print(SpecifierSet('').contains('2.2.11.0ubuntu20.04'))"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/mfo/maas-pylxd/2.9/squashfs-root/lib/python3.8/site-packages/pkg_resources/_vendor/packaging/specifiers.py", line 902, in contains
    item = Version(item)
           ^^^^^^^^^^^^^
  File "/home/mfo/maas-pylxd/2.9/squashfs-root/lib/python3.8/site-packages/pkg_resources/_vendor/packaging/version.py", line 197, in __init__
    raise InvalidVersion(f"Invalid version: '{version}'")
pkg_resources._vendor.packaging.version.InvalidVersion: Invalid version: '2.2.11.0ubuntu20.04'

Old (works)

2.9$ snap download --channel=2.9/stable maas
2.9$ unsquashfs maas_*.snap
2.9$ cd squashfs-root
2.9-stable/squashfs-root$ PYTHONPATH=$PWD/usr/lib/python3/dist-packages:$PWD/usr/lib/python3.8/dist-packages:$PWD/lib/python3.8/site-packages python3 -q -c "from pkg_resources._vendor.packaging.specifiers import SpecifierSet; print(SpecifierSet('').contains('2.2.11.0ubuntu20.04'))"
True

Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

Attached packaging patch for reference.

The only difference in the old/new package is the PKG-INFO's Version: field, which is now '2.2.11a0' as in later releases of MAAS/pylxd.

The differences in the packages: (changelog and egg-info directories)

 $ dpkg-deb -x python3-pylxd_2.2.11~ubuntu20.04-1_all.deb deb-old
 $ dpkg-deb -x python3-pylxd_2.2.11~ubuntu20.04-2_all.deb deb-new

 $ diff -r deb-old/ deb-new/
 Only in deb-old/usr/lib/python3/dist-packages: pylxd-2.2.11.0ubuntu20.04.egg-info
 Only in deb-new/usr/lib/python3/dist-packages: pylxd-2.2.11a0.egg-info
 Binary files deb-old/usr/share/doc/python3-pylxd/changelog.Debian.gz and deb-new/usr/share/doc/python3-pylxd/changelog.Debian.gz differ

The differences in the egg-info directories: (just the version number)

 $ mv deb-old/usr/lib/python3/dist-packages/pylxd-2.2.11.0ubuntu20.04.egg-info egg-info-old
 $ mv deb-new/usr/lib/python3/dist-packages/pylxd-2.2.11a0.egg-info egg-info-new

 $ diff -ru egg-info-old/ egg-info-new/
 diff -ru egg-info-old/PKG-INFO egg-info-new/PKG-INFO
 --- egg-info-old/PKG-INFO 2020-07-15 11:53:01.000000000 -0300
 +++ egg-info-new/PKG-INFO 2023-08-08 18:37:44.000000000 -0300
 @@ -1,6 +1,6 @@
  Metadata-Version: 1.1
  Name: pylxd
 -Version: 2.2.11.0ubuntu20.04
 +Version: 2.2.11a0
  Summary: python library for lxd
  Home-page: http://www.linuxcontainers.org
  Author: Paul Hummer and others (see CONTRIBUTORS.rst)

Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

The '2.2.11a0' version works correctly in a test with the unsquashed snap:

Before:

$ PYTHONPATH=$PWD/usr/lib/python3/dist-packages:$PWD/usr/lib/python3.8/dist-packages:$PWD/lib/python3.8/site-packages ./bin/maas --help
...
pkg_resources.extern.packaging.version.InvalidVersion: Invalid version: '2.2.11.0ubuntu20.04'

After:

$ PYTHONPATH=$PWD/usr/lib/python3/dist-packages:$PWD/usr/lib/python3.8/dist-packages:$PWD/lib/python3.8/site-packages ./bin/maas --help
usage: maas [-h] COMMAND ...
...

Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

ppa:maas/2.9-next:

python-pylxd (2.2.11~ubuntu20.04-2) focal; urgency=medium

  * d/rules: set alpha version in PKG-INFO as later releases (LP: #2030814)

 -- Mauricio Faria de Oliveira <email address hidden> Tue, 08 Aug 2023 18:37:44 -0300

Revision history for this message
Mauricio Faria de Oliveira (mfo) wrote :

ppa:maas/2.9-next:

maas (1:2.9.3~rc3-9275-g.18d2d84af-0ubuntu1~20.04.1) focal; urgency=medium

  * New upstream release, MAAS 2.9.3 RC3.
  * No change rebuild to pick up python-pylxd in snap (LP: #2030814)

 -- Mauricio Faria de Oliveira <email address hidden> Wed, 09 Aug 2023 12:33:34 -0300

...

snap:2.9/edge:

$ snap info maas | fgrep 2.9/edge
  2.9/edge: 2.9.3~rc3-9275-g.18d2d84af 2023-08-09 (29955) 141MB -

$ sudo snap install --channel=2.9/edge maas
maas (2.9/edge) 2.9.3~rc3-9275-g.18d2d84af from Canonical✓ installed

$ which maas
/snap/bin/maas

$ maas --help
usage: maas [-h] COMMAND ...
...

@ https://launchpadlibrarian.net/681177409/buildlog_snap_ubuntu_focal_amd64_maas-2.9_BUILDING.txt.gz
Get:1 python3-pylxd_2.2.11~ubuntu20.04-2_all.deb [53.1 kB]

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.