Test fail with import error and timeout exception

Bug #1637938 reported by PeterZhang
20
This bug affects 4 people
Affects Status Importance Assigned to Milestone
ceilometer-powervm
Fix Released
Undecided
Drew Thorstensen
networking-powervm
Fix Released
Undecided
Drew Thorstensen
nova-powervm
Fix Released
Undecided
Drew Thorstensen
oslo.messaging
Won't Fix
Undecided
Drew Thorstensen

Bug Description

Hello friends,

I cloned the master branch oslo.messaging and run tox -e py27.

it reports a lot of import errors (like below):

FAIL: oslo_messaging.tests.test_target.TargetReprTestCase.test_repr(fanout)
tags: worker-0
----------------------------------------------------------------------
Empty attachments:
  stderr
  stdout

Traceback (most recent call last):
  File "oslo_messaging/tests/utils.py", line 38, in setUp
    self.messaging_conf = self.useFixture(conffixture.ConfFixture(conf))
  File "oslo_messaging/conffixture.py", line 50, in __init__
    'oslo_messaging_rabbit')
  File "oslo_messaging/conffixture.py", line 25, in _import_opts
    __import__(module)
  File "oslo_messaging/_drivers/impl_rabbit.py", line 30, in <module>
    import kombu.connection
  File "/opt/stack/oslo.messaging/.tox/py27/local/lib/python2.7/site-packages/kombu/connection.py", line 15, in <module>
    from kombu import exceptions
  File "/opt/stack/oslo.messaging/.tox/py27/local/lib/python2.7/site-packages/kombu/exceptions.py", line 8, in <module>
    from kombu.five import python_2_unicode_compatible
  File "/opt/stack/oslo.messaging/.tox/py27/local/lib/python2.7/site-packages/kombu/five.py", line 6, in <module>
    import vine.five
ImportError: No module named vine.five

And some error like this:

FAIL: oslo_messaging.tests.drivers.test_impl_rabbit.TestRabbitPublisher.test_declared_queue_publisher
tags: worker-0
----------------------------------------------------------------------
Empty attachments:
  stderr
  stdout

Traceback (most recent call last):
  File "oslo_messaging/tests/drivers/test_impl_rabbit.py", line 282, in test_declared_queue_publisher
    self.assertRaises(exc, try_send, e_passive)
  File "/opt/stack/oslo.messaging/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 485, in assertRaises
    self.assertThat(our_callable, matcher)
  File "/opt/stack/oslo.messaging/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 496, in assertThat
    mismatch_error = self._matchHelper(matchee, matcher, message, verbose)
  File "/opt/stack/oslo.messaging/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 547, in _matchHelper
    mismatch = matcher.match(matchee)
  File "/opt/stack/oslo.messaging/.tox/py27/local/lib/python2.7/site-packages/testtools/matchers/_exception.py", line 108, in match
    mismatch = self.exception_matcher.match(exc_info)
  File "/opt/stack/oslo.messaging/.tox/py27/local/lib/python2.7/site-packages/testtools/matchers/_higherorder.py", line 62, in match
    mismatch = matcher.match(matchee)
  File "/opt/stack/oslo.messaging/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 475, in match
    reraise(*matchee)
  File "/opt/stack/oslo.messaging/.tox/py27/local/lib/python2.7/site-packages/testtools/matchers/_exception.py", line 101, in match
    result = matchee()
  File "/opt/stack/oslo.messaging/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 1049, in __call__
    return self._callable_object(*self._args, **self._kwargs)
  File "oslo_messaging/tests/drivers/test_impl_rabbit.py", line 278, in try_send
    exchange, {}, routing_key='foobar')
  File "oslo_messaging/_drivers/impl_rabbit.py", line 1170, in _ensure_publishing
    self.ensure(method, retry=retry, error_callback=_error_callback)
  File "oslo_messaging/_drivers/impl_rabbit.py", line 817, in ensure
    ret, channel = autoretry_method()
  File "/opt/stack/oslo.messaging/.tox/py27/local/lib/python2.7/site-packages/kombu/connection.py", line 527, in _ensured
    errback and errback(exc, 0)
  File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/opt/stack/oslo.messaging/.tox/py27/local/lib/python2.7/site-packages/kombu/connection.py", line 419, in _reraise_as_library_errors
    sys.exc_info()[2])
  File "/opt/stack/oslo.messaging/.tox/py27/local/lib/python2.7/site-packages/kombu/connection.py", line 414, in _reraise_as_library_errors
    yield
  File "/opt/stack/oslo.messaging/.tox/py27/local/lib/python2.7/site-packages/kombu/connection.py", line 494, in _ensured
    return fun(*args, **kwargs)
  File "/opt/stack/oslo.messaging/.tox/py27/local/lib/python2.7/site-packages/kombu/connection.py", line 570, in __call__
    return fun(*args, channel=channels[0], **kwargs), channels[0]
  File "oslo_messaging/_drivers/impl_rabbit.py", line 793, in execute_method
    method()
  File "oslo_messaging/_drivers/impl_rabbit.py", line 1234, in _publish_and_creates_default_queue
    queue.declare()
  File "/opt/stack/oslo.messaging/.tox/py27/local/lib/python2.7/site-packages/kombu/entity.py", line 604, in declare
    self._create_exchange(nowait=nowait, channel=channel)
  File "/opt/stack/oslo.messaging/.tox/py27/local/lib/python2.7/site-packages/kombu/entity.py", line 611, in _create_exchange
    self.exchange.declare(nowait=nowait, channel=channel)
  File "/opt/stack/oslo.messaging/.tox/py27/local/lib/python2.7/site-packages/kombu/entity.py", line 185, in declare
    nowait=nowait, passive=passive,
  File "/opt/stack/oslo.messaging/.tox/py27/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 496, in exchange_declare
    (50, 10), 'Channel.exchange_declare', '404',
kombu.exceptions.OperationalError: Channel.exchange_declare: (404) NOT_FOUND - no exchange 'foobar' in vhost u'/'
======================================================================
FAIL: oslo_messaging.tests.drivers.test_impl_rabbit.TestRabbitPublisher.test_send_with_timeout
tags: worker-0
----------------------------------------------------------------------
Empty attachments:
  stdout

stderr: {{{
/opt/stack/oslo.messaging/.tox/py27/local/lib/python2.7/site-packages/pkg_resources/__init__.py:187: RuntimeWarning: You have iterated over the result of pkg_resources.parse_version. This is a legacy behavior which is inconsistent with the new version class introduced in setuptools 8.0. In most cases, conversion to a tuple is unnecessary. For comparison of versions, sort the Version instances directly. If you have another use case requiring the tuple, please file a bug with the setuptools project describing that need.
  stacklevel=1,
}}}

Traceback (most recent call last):
  File "/opt/stack/oslo.messaging/.tox/py27/local/lib/python2.7/site-packages/mock/mock.py", line 1305, in patched
    return func(*args, **keywargs)
  File "oslo_messaging/tests/drivers/test_impl_rabbit.py", line 238, in test_send_with_timeout
    routing_key='routing_key')
  File "/opt/stack/oslo.messaging/.tox/py27/local/lib/python2.7/site-packages/mock/mock.py", line 937, in assert_called_with
    six.raise_from(AssertionError(_error_message(cause)), cause)
  File "/opt/stack/oslo.messaging/.tox/py27/local/lib/python2.7/site-packages/six.py", line 718, in raise_from
    raise value
AssertionError: Expected call: publish('msg', compression=None, exchange=<Mock id='140097101839760'>, expiration=1000, routing_key='routing_key')
Actual call: publish('msg', compression=None, exchange=<Mock id='140097101839760'>, expiration=1, routing_key='routing_key')
======================================================================
FAIL: oslo_messaging.tests.drivers.zmq.test_zmq_ttl_cache.TestZmqTTLCache.test_in_operator_with_executor
tags: worker-0
----------------------------------------------------------------------
Empty attachments:
  stderr
  stdout

Traceback (most recent call last):
  File "oslo_messaging/tests/drivers/zmq/test_zmq_ttl_cache.py", line 86, in test_in_operator_with_executor
    self._test_in_operator()
  File "oslo_messaging/tests/drivers/zmq/test_zmq_ttl_cache.py", line 77, in _test_in_operator
    self.assertIn(2, self.cache)
  File "/opt/stack/oslo.messaging/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 417, in assertIn
    self.assertThat(haystack, Contains(needle), message)
  File "/opt/stack/oslo.messaging/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: 2 not in <oslo_messaging._drivers.zmq_driver.server.zmq_ttl_cache.TTLCache object at 0x7f6ae416ac50>
Ran 1981 (+315) tests in 263.460s (+202.463s)
FAILED (id=1, failures=7 (-1626), skips=35)
error: testr failed (1)
ERROR: InvocationError: '/opt/stack/oslo.messaging/.tox/py27/bin/python setup.py testr --slowest --testr-args='
_______________________________________________ summary _______________________________________________
ERROR: py27: commands failed

I did not change any code on the master branch. just run tox -e py27. I am a bit confused why it happened, or I did something wrong?

Revision history for this message
Drew Thorstensen (thorst) wrote :

I'm seeing this too in a different project. I thought it may be related to this bug: https://bugs.launchpad.net/oslo.messaging/+bug/1587763

However, in my project, the amqp getting installed is 1.4.9. Which should be OK according to that bug.

Revision history for this message
Drew Thorstensen (thorst) wrote :

I think it was this commit that is causing it: https://github.com/celery/kombu/commit/5a76eddd2b0ee4670e565088c556c06c0ff740ab

And then when the version of kombu jumped (looks like it recently released 4.0.0), it started hitting others as 'five' is not a dependency in the global-requirements.

Revision history for this message
Drew Thorstensen (thorst) wrote :

Yeah - if you force kombu to be <4.0, you don't see this error.

Changed in nova-powervm:
assignee: nobody → Drew Thorstensen (thorst)
Changed in networking-powervm:
assignee: nobody → Drew Thorstensen (thorst)
Changed in ceilometer-powervm:
assignee: nobody → Drew Thorstensen (thorst)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo.messaging (master)

Fix proposed to branch: master
Review: https://review.openstack.org/391835

Changed in oslo.messaging:
assignee: nobody → Drew Thorstensen (thorst)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova-powervm (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/391837

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to networking-powervm (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/391838

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer-powervm (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/391839

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova-powervm (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/391840

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to networking-powervm (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/391841

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer-powervm (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/391842

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova-powervm (master)

Reviewed: https://review.openstack.org/391829
Committed: https://git.openstack.org/cgit/openstack/nova-powervm/commit/?id=986e9ebb24f0a6bb2ce38c3c183bcfb42fe30325
Submitter: Jenkins
Branch: master

commit 986e9ebb24f0a6bb2ce38c3c183bcfb42fe30325
Author: Drew Thorstensen <email address hidden>
Date: Mon Oct 31 09:33:38 2016 -0400

    Use upper constraints

    The tox.ini needs to make use of the upper constraints. Failure to do
    so is causing us to hit known issues with newer versions of packages.

    Change-Id: I065adf88c599ad91af3f29d02ccd225f5d07ffa9
    Closes-Bug: 1637938

Changed in nova-powervm:
status: New → Fix Released
Changed in ceilometer-powervm:
status: New → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer-powervm (master)

Reviewed: https://review.openstack.org/391833
Committed: https://git.openstack.org/cgit/openstack/ceilometer-powervm/commit/?id=95c789736705235298b30bb21087c9fb2282b290
Submitter: Jenkins
Branch: master

commit 95c789736705235298b30bb21087c9fb2282b290
Author: Drew Thorstensen <email address hidden>
Date: Mon Oct 31 09:37:03 2016 -0400

    Use upper constraints

    This changes the tox command to use the upper constraints. This will
    avoid known issues with newer versions of the library.

    Change-Id: I85a03fc40356ca1493d6dcf550f63f0deba8dd84
    Closes-Bug: 1637938

Changed in networking-powervm:
status: New → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to networking-powervm (master)

Reviewed: https://review.openstack.org/391832
Committed: https://git.openstack.org/cgit/openstack/networking-powervm/commit/?id=a6bded724ee5247e309a4836280f0fc3c024729e
Submitter: Jenkins
Branch: master

commit a6bded724ee5247e309a4836280f0fc3c024729e
Author: Drew Thorstensen <email address hidden>
Date: Mon Oct 31 09:35:44 2016 -0400

    Use upper constraints

    This changes the tox command to use the upper constraints. This avoids
    hitting known issues with newer versions of libraries.

    Change-Id: Ida4d9cabcff079a587d61b390cb50443a1cf7398
    Closes-Bug: 1637938

tags: added: in-stable-newton
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer-powervm (stable/newton)

Reviewed: https://review.openstack.org/391839
Committed: https://git.openstack.org/cgit/openstack/ceilometer-powervm/commit/?id=c44f7b5cbd224876f831ca05e4f571dfe44aae9a
Submitter: Jenkins
Branch: stable/newton

commit c44f7b5cbd224876f831ca05e4f571dfe44aae9a
Author: Drew Thorstensen <email address hidden>
Date: Mon Oct 31 09:37:03 2016 -0400

    Use upper constraints

    This changes the tox command to use the upper constraints. This will
    avoid known issues with newer versions of the library.

    Change-Id: I85a03fc40356ca1493d6dcf550f63f0deba8dd84
    Closes-Bug: 1637938

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova-powervm (stable/newton)

Reviewed: https://review.openstack.org/391837
Committed: https://git.openstack.org/cgit/openstack/nova-powervm/commit/?id=d4fd1bd9383349965954ab49c657ba48247223f7
Submitter: Jenkins
Branch: stable/newton

commit d4fd1bd9383349965954ab49c657ba48247223f7
Author: Drew Thorstensen <email address hidden>
Date: Mon Oct 31 09:33:38 2016 -0400

    Use upper constraints

    The tox.ini needs to make use of the upper constraints. Failure to do
    so is causing us to hit known issues with newer versions of packages.

    Change-Id: I065adf88c599ad91af3f29d02ccd225f5d07ffa9
    Closes-Bug: 1637938

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to networking-powervm (stable/newton)

Reviewed: https://review.openstack.org/391838
Committed: https://git.openstack.org/cgit/openstack/networking-powervm/commit/?id=3c4bbb2c620efc95d48d0fd85207bdd1d80eae82
Submitter: Jenkins
Branch: stable/newton

commit 3c4bbb2c620efc95d48d0fd85207bdd1d80eae82
Author: Drew Thorstensen <email address hidden>
Date: Mon Oct 31 09:35:44 2016 -0400

    Use upper constraints

    This changes the tox command to use the upper constraints. This avoids
    hitting known issues with newer versions of libraries.

    Change-Id: Ida4d9cabcff079a587d61b390cb50443a1cf7398
    Closes-Bug: 1637938

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova-powervm (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/391901

Amy Fong (afong2)
no longer affects: cinder
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova-powervm (stable/mitaka)

Change abandoned by Eric Fried (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/391840
Reason: Superseded by https://review.openstack.org/#/c/391901/

tags: added: in-stable-mitaka
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer-powervm (stable/mitaka)

Reviewed: https://review.openstack.org/391842
Committed: https://git.openstack.org/cgit/openstack/ceilometer-powervm/commit/?id=df401e5729ae873e19f12f0f19f1adc894bca30d
Submitter: Jenkins
Branch: stable/mitaka

commit df401e5729ae873e19f12f0f19f1adc894bca30d
Author: Drew Thorstensen <email address hidden>
Date: Mon Oct 31 09:37:03 2016 -0400

    Use upper constraints, back to mitaka ceilometer

    This changes the tox command to use the upper constraints. This will
    avoid known issues with newer versions of the library.

    It also properly sets the base ceilometer level for tox to the matching
    branch (stable/mitaka).

    Change-Id: I85a03fc40356ca1493d6dcf550f63f0deba8dd84
    Closes-Bug: 1637938

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova-powervm (stable/mitaka)

Reviewed: https://review.openstack.org/391901
Committed: https://git.openstack.org/cgit/openstack/nova-powervm/commit/?id=e11dab4ffcae21d9032c01b743dd168e742472fe
Submitter: Jenkins
Branch: stable/mitaka

commit e11dab4ffcae21d9032c01b743dd168e742472fe
Author: Eric Fried <email address hidden>
Date: Mon Oct 31 09:40:23 2016 -0500

    New pep8 rules, upper constraints

    This change updates the code to comply with a new pep8 rule to defer the
    string concatenation to the logger instead of doing it inline.

    It also adds an ignore rule for conf options being outside of nova.conf.
    Given that we are an out-of-tree driver, we can not put our conf options
    in tree. This is a temporary work around until the powervm driver is in
    tree.

    And the tox.ini needs to make use of the upper constraints. Failure to
    do so is causing us to hit known issues with newer versions of packages.

    And the tox.ini needs to be pulling the nova project from stable/mitaka.

    And having done that, older versions of test utils don't have some of
    the assertion methods we were using, so those had to be fixed.

    Change-Id: I8260acca9de5b21ff6539a509d6b29e9603b9d7f
    Closes-Bug: #1632441
    Closes-Bug: #1637938

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to networking-powervm (stable/mitaka)

Reviewed: https://review.openstack.org/391841
Committed: https://git.openstack.org/cgit/openstack/networking-powervm/commit/?id=4ae4c132522a28d63dc9208d04e23bea48d3f047
Submitter: Jenkins
Branch: stable/mitaka

commit 4ae4c132522a28d63dc9208d04e23bea48d3f047
Author: Drew Thorstensen <email address hidden>
Date: Mon Oct 31 09:35:44 2016 -0400

    Use upper constraints and mitaka neutron

    This changes the tox command to use the upper constraints. This avoids
    hitting known issues with newer versions of libraries.

    It also fixes the base neutron level to stable/mitaka to match the level
    of networking-powervm.

    Change-Id: Ida4d9cabcff079a587d61b390cb50443a1cf7398
    Closes-Bug: 1637938

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on oslo.messaging (master)

Change abandoned by Drew Thorstensen (thorst) (<email address hidden>) on branch: master
Review: https://review.openstack.org/391835
Reason: I believe that this has been fixed in another way.

Andy Smith (ansmith-t)
Changed in oslo.messaging:
status: In Progress → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.