Puppet could not start Service[swift-proxy] because of pkg_resources.VersionConflict error

swift-proxy starts to work if we remove 'ceilometer' from pipeline in /etc/swift/proxy-server.conf. So it looks like this bug was introduced by https://review.openstack.org/154831 commit where we added 'ceilometer' to the pipeline, although according to the documentation pipeline is configured properly.

If we resolve conflict (http://paste.openstack.org/show/182435/) with python-oslo.rootwrap then appear new conflict (http://paste.openstack.org/show/182436/) with pyasn1.

1. In repository we have two versions of python-oslo.rootwrap: python-oslo.rootwrap_1.3.0-fuel6.0~mira16_all.deb and python-oslo.rootwrap_1.3.0.0a1-ubuntu2_all.deb. If compare this two versions with dpkg manager "if dpkg --compare-versions 1.3.0 lt 1.3.0.0a1; then echo true; fi" then we get that "first less than second". But pip compares this as "first greater than second" and we get version conflict. In repository http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.1-stable/ubuntu/all/ we have both versions of this package, so we need one == 1.3.0 because this version is newer.
2. pkg_resources.VersionConflict: (pyasn1 0.0.11a (/usr/lib/python2.7/dist-packages), Requirement.parse('pyasn1>=0.1.2'))
This conflict we get after resolve first conflict. Here we need newer version of pyasn1 than here http://fuel-repository.mirantis.com/fwm/6.1/ubuntu/pool/main/ because of this https://review.fuel-infra.org/gitweb?p=packages/precise/python-pysnmp.git;a=blob;f=pysnmp-4.2.5/pysnmp.egg-info/requires.txt;h=9b89a8d0e84e0c4d5a3370a2da916f6a2d6bd11e;hb=refs/heads/6.1. In global requirements (https://github.com/openstack/requirements/blob/stable/juno/global-requirements.txt) we have that pyasn1 <= 0.1.7, so we think that we could build pyasn1 == 0.1.2. This problem was hidden a lot of time, so now it is appear.

Sorry, second path will resolve in trusty, there will be pyasn1 0.1.7. Now we have only one problem with python-oslo.rootwrap.

Problem is in two packages: python-oslo.rootwrap and pyasn1. Pyasn1 is outdated package in precise (12.04 Ubuntu). When we'll be on Ubuntu 14.04 we'll have appropriate packages for this issue. It'll not reproduce

A customer met the issue on

  release: "6.0.1"
  api: "1.0"
  build_number: "21"
  build_id: "2015-02-19_16-29-42"
  astute_sha: "e14e0d9ee2dda2df9f6325163d45c304060677dd"
  fuellib_sha: "8249f235acfe1cf22e031d91a09d67f13f287416"
  ostf_sha: "3b57985d4d2155510894a1f6d03b478b201f7780"
  nailgun_sha: "baddb3b50181a4f4f1c2ec88dea13ace26421648"
  fuelmain_sha: "490d4b6688497a136c7e1a6dffe3d5d59d02da01"

This bug is fixed for 6.1 because we have Ubuntu 14.04 where python-pyasn1 version is 0.1.7 and oslo.rootwrap version is 1.5.0.

DEB package python-pysnmp has been built for project packages/precise/python-pysnmp
Package version == 4.2.5, package release == ubuntu2

Changeset: https://review.fuel-infra.org/5085
project: packages/precise/python-pysnmp
branch: 6.0.1
author: Ivan Berezovskiy
committer: Ivan Berezovskiy
subject: Fix requirements for pyasn1
status: patchset-created

Files placed on repository:
python-pysnmp4-doc_4.2.5-ubuntu2_all.deb
python-pysnmp4_4.2.5-ubuntu2_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0.1-stable-5085/ubuntu

This issue could be fixed by changing requires.txt file in python-pysnmp package (in egg.info folder). Package with fixed pyasn1 dependency is available here http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0.1-stable-5085/ubuntu . This fix will be prepared and merged in 6.0.2 branch later, when branch will be created

It' requires small fix for pysnmp library https://review.fuel-infra.org/#/c/5085/

DEB package python-pysnmp has been built for project packages/precise/python-pysnmp
Package version == 4.2.5, package release == ubuntu2

Changeset: https://review.fuel-infra.org/5085
project: packages/precise/python-pysnmp
branch: 6.0.1
author: Ivan Berezovskiy
committer: Ivan Berezovskiy
subject: Fix requirements for pyasn1
status: change-merged

Files placed on repository:
python-pysnmp4-doc_4.2.5-ubuntu2_all.deb
python-pysnmp4_4.2.5-ubuntu2_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0.1-stable/ubuntu

Verified on ISO #314

"build_id": "2015-04-15_23-18-33", "ostf_sha": "90f232a25d10f98db9d5531a3b133319529f57aa", "build_number": "314", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-04-15_23-18-33", "ostf_sha": "90f232a25d10f98db9d5531a3b133319529f57aa", "build_number": "314", "api": "1.0", "nailgun_sha": "24b56283fb4caaba6b6414056dad30e503a7c30f", "openstack_version": "2014.2-6.1", "production": "docker", "python-fuelclient_sha": "ff186de358cbda489a320723de47a9e61ae23364", "astute_sha": "36a3d54baf86480326866279778b867086aa4b05", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "d3aa72d814c3f18799b566363004a22591161cef", "fuellib_sha": "ab6dbba825a5e54f3183ed8582921a0328b211c7"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "24b56283fb4caaba6b6414056dad30e503a7c30f", "openstack_version": "2014.2-6.1", "production": "docker", "python-fuelclient_sha": "ff186de358cbda489a320723de47a9e61ae23364", "astute_sha": "36a3d54baf86480326866279778b867086aa4b05", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "d3aa72d814c3f18799b566363004a22591161cef", "fuellib_sha": "ab6dbba825a5e54f3183ed8582921a0328b211c7"

Bug reproduced on ISO #352.

Failed Jenkins job
https://jenkins-tpi.bud.mirantis.net/job/systest.vcenter/BRANCH=master,OPENSTACK_RELEASE=Ubuntu,TEST_GROUP=vcenter_ceilometer,label_exp=vsphere/273/console

Environment:
Ubuntu 14.04 as target OSt operating system
vCenter 5.5update2
test scenario (see test_vcenter.py in fuel-qa/):
        Scenario:
            1. Create cluster with Ceilometer support
            2. Add 3 nodes with controller+MongoDB roles
            3. Deploy the cluster
            4. Run OSTF

Puppet log snippet: http://paste.openstack.org/show/207959

/var/log/upstart/swift-proxy.log snippet:
http://paste.openstack.org/show/207960

Fuel diagnostic snapshot on reproduced Fuel ISO #352.

I hold a discussion with Ivan Berezovsky and according to his findings problem
that I faced is not connected to this bug. It must be filed as separate bug.

So marking this issue as 'Fix Released' and providing link to new bug:
https://bugs.launchpad.net/fuel/+bug/1449023

The main problem is in old version of oslo.utils: https://github.com/openstack/requirements/commit/efb080e744f7466942f977ae79ec2d74501494f6

our version is 1.2.1, but it should be >= 1.4.0. And when service swift-proxy start, it check requirements.txt file.

https://review.fuel-infra.org/gitweb?p=openstack/python-keystoneclient.git;a=blob;f=requirements.txt;h=b3d2d995bbca0e6831c1c0c2703828e230f91663;hb=refs/heads/openstack-ci/fuel-6.1/2014.2

Keystoneclient requires oslo.utils==1.4.0. That's why this problem happens. So, it should be another bug.