Chains of parts with python contents can fail with pkg_resources import error

Note that this behaviour seems to have arisen in the last couple of months around the time of the 2.40 release to xenial. However, I haven't managed to establish exactly what changed to cause it.

I bisected it and this seemed to be the culprit:

4adec30131d8de1c8ea902c431c1b7e16bc4d8d5 is the first bad commit
commit 4adec30131d8de1c8ea902c431c1b7e16bc4d8d5
Author: Kyle Fazzari <email address hidden>
Date: Sat Nov 18 05:12:55 2017 -0800

    python plugin: use extracted pip (#1607)

    Simplify the Python plugin and its tests by using the independently
    tested pip that was extracted in previous commits. All functionality
    should remain the same.

    Signed-off-by: Kyle Fazzari <email address hidden>

:040000 040000 1d0e8b6a261d844846a90d13be8e08e4bd7463dd 2d56773183319e3dcfe93ad5f9f74b7a719856f0 M snapcraft

This seems to be due to the mix of stage-packages and python-packages, where pkg_resources from the staged python is loaded and the part's pip wanting to use pkg_resources, but instead of loading its vendored version it ends up using the already loaded one from stage, here's a workaround:

(snapcraft) ubuntu@snapcraft-xenial-dev:~/t$ cat snap/snapcraft.yaml
name: python-chained-parts
version: "0"
summary: test the python plugin
description: |
  This is a snap with a chain of python related parts
confinement: strict
grade: devel

parts:
  part1:
    plugin: python
    stage-packages:
      - python3-requests-unixsocket
  part2:
    plugin: nil
    stage-packages:
      - python3-yaml
    after:
      - part1
  part3:
    plugin: python
    python-packages:
      - bitstring
    stage-packages:
      - python3-pip
      - python3-wheel
      - python3-setuptools
    after:
      - part2

Or alternatively
(snapcraft) ubuntu@snapcraft-xenial-dev:~/t$ cat snap/snapcraft.yaml
name: python-chained-parts
version: "0"
summary: test the python plugin
description: |
  This is a snap with a chain of python related parts
confinement: strict
grade: devel

parts:
  part1:
    plugin: python
    stage-packages:
      - python3-requests-unixsocket
  part2:
    plugin: nil
    stage-packages:
      - python3-yaml
      - python3-pip
      - python3-wheel
      - python3-setuptools
    after:
      - part1
  part3:
    plugin: python
    python-packages:
      - bitstring
    after:
      - part2

where I said pip I meant setuptools and you only need python3-setuptools to workaround the issue

Thanks for the workaround suggestion. To relate this back to the specific problem I have been facing - the part2 from the example has actually been a plainbox_provider plugin part. Hence, I have tried locally making a change to the plugin itself and the following stage_packages as 'builtins' (only adding setuptools didnt seem to work for me):

    def __init__(self, name, options, project):
...
+ self.stage_packages.extend(['python3-pip', 'python3-wheel', 'python3-setuptools'])

    def build(self):
        super().build()

This gets our previous working builds going again, which is great. Does it seem reasonable to carry this in the plugin to you? I think it is acceptable.

On lun, may 14, 2018 at 7:14 AM, Jonathan Cave
<email address hidden> wrote:
> Thanks for the workaround suggestion. To relate this back to the
> specific problem I have been facing - the part2 from the example has
> actually been a plainbox_provider plugin part. Hence, I have tried
> locally making a change to the plugin itself and the following
> stage_packages as 'builtins' (only adding setuptools didnt seem to
> work
> for me):
>
> def __init__(self, name, options, project):
> ...
> + self.stage_packages.extend(['python3-pip', 'python3-wheel',
> 'python3-setuptools'])
>
> def build(self):
> super().build()
>
>
> This gets our previous working builds going again, which is great.
> Does it seem reasonable to carry this in the plugin to you? I think
> it is acceptable.

If you want to force the plainbox provider plugin to always use pip
from the archive, this seems reasonable. That said, this is still only
a workaround.

Pull request with workaround and integration test

https://github.com/snapcore/snapcraft/pull/2140