OpenStack Compute (nova)

test_create_multiple_servers fails on master with venv

Bug #861440 reported by Mark McLoughlin
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
Brian Lamar
OpenStack Compute (nova) 2012.1 "essex"
kombu (Ubuntu)
Fix Released
Medium
Dave Walker
Ubuntu precise-alpha-1

Bug Description

Using venv, I'm seeing test_create_multiple_servers fail on master

The test was introduced by:

  commit 0fab78825ef06310926181f6f97d377058b56b97
  Author: Chris Behrens <email address hidden>
  Date: Sat Sep 24 23:49:36 2011 +0000

    compute_api create*() and schedulers refactoring

The error is:

2011-09-28 14:03:24,538 INFO nova.rpc [-] Connected to AMQP server on localhost:5672
2011-09-28 14:03:24,544 ERROR nova.rpc [-] Exception during message handling
(nova.rpc): TRACE: Traceback (most recent call last):
(nova.rpc): TRACE: File "/home/markmc/projects/openstack/git/nova/nova/rpc/impl_kombu.py", line 620, in _process_data
(nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args)
(nova.rpc): TRACE: File "/home/markmc/projects/openstack/git/nova/nova/exception.py", line 98, in wrapped
(nova.rpc): TRACE: return f(*args, **kw)
(nova.rpc): TRACE: File "/home/markmc/projects/openstack/git/nova/nova/compute/manager.py", line 119, in decorated_function
(nova.rpc): TRACE: function(self, context, instance_id, *args, **kwargs)
(nova.rpc): TRACE: File "/home/markmc/projects/openstack/git/nova/nova/compute/manager.py", line 512, in terminate_instance
(nova.rpc): TRACE: self._delete_instance(context, instance_id)
(nova.rpc): TRACE: File "/home/markmc/projects/openstack/git/nova/nova/compute/manager.py", line 493, in _delete_instance
(nova.rpc): TRACE: self._shutdown_instance(context, instance_id, 'Terminating')
(nova.rpc): TRACE: File "/home/markmc/projects/openstack/git/nova/nova/compute/manager.py", line 474, in _shutdown_instance
(nova.rpc): TRACE: network_info = self._get_instance_nw_info(context, instance)
(nova.rpc): TRACE: File "/home/markmc/projects/openstack/git/nova/nova/compute/manager.py", line 241, in _get_instance_nw_info
(nova.rpc): TRACE: instance)
(nova.rpc): TRACE: File "/home/markmc/projects/openstack/git/nova/nova/network/api.py", line 210, in get_instance_nw_info
(nova.rpc): TRACE: 'args': args})
(nova.rpc): TRACE: File "/home/markmc/projects/openstack/git/nova/nova/rpc/__init__.py", line 45, in call
(nova.rpc): TRACE: return get_impl().call(context, topic, msg)
(nova.rpc): TRACE: File "/home/markmc/projects/openstack/git/nova/nova/rpc/impl_kombu.py", line 738, in call
(nova.rpc): TRACE: rv = list(rv)
(nova.rpc): TRACE: File "/home/markmc/projects/openstack/git/nova/nova/rpc/impl_kombu.py", line 705, in __iter__
(nova.rpc): TRACE: self.done()
(nova.rpc): TRACE: File "/home/markmc/projects/openstack/git/nova/nova/rpc/impl_kombu.py", line 685, in done
(nova.rpc): TRACE: self._connection.close()
(nova.rpc): TRACE: File "/home/markmc/projects/openstack/git/nova/nova/rpc/impl_kombu.py", line 570, in close
(nova.rpc): TRACE: self._done()
(nova.rpc): TRACE: File "/home/markmc/projects/openstack/git/nova/nova/rpc/impl_kombu.py", line 547, in _done
(nova.rpc): TRACE: self.connection.reset()
(nova.rpc): TRACE: File "/home/markmc/projects/openstack/git/nova/nova/rpc/impl_kombu.py", line 382, in reset
(nova.rpc): TRACE: self.channel.close()
(nova.rpc): TRACE: File "/home/markmc/projects/openstack/git/nova/.nova-venv/lib/python2.7/site-packages/kombu/transport/virtual/__init__.py", line 560, in close
(nova.rpc): TRACE: self._qos.restore_unacked_once()
(nova.rpc): TRACE: File "/home/markmc/projects/openstack/git/nova/.nova-venv/lib/python2.7/site-packages/kombu/transport/virtual/__init__.py", line 165, in restore_unacked_once
(nova.rpc): TRACE: assert not state
(nova.rpc): TRACE: AssertionError

Revision history for this message
Chris Behrens (cbehrens) wrote :

It's a bug in kombu memory transport used for tests. Need to downgrade to version 1.0.4, which is the same version that's in our ppa until we can get a fix into newer kombu... or.. we stop using kombu's memory transport for tests in favor of a fakerabbit port.

Note that other tests will randomly fail with the same thing... some snapshot tests.

Lorin Hochstein (lorinh)
Changed in nova:
status: New → Fix Committed
Revision history for this message
Dave Walker (davewalker) wrote :

@Lorin, you've marked this Fix Committed for a nova task, can you comment where this is? I couldn't find the commit.

Thanks.

Dave Walker (davewalker)
Changed in kombu (Ubuntu):
milestone: none → precise-alpha-1
assignee: nobody → Dave Walker (davewalker)
importance: Undecided → Medium
status: New → Confirmed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package kombu - 1.2.1-1ubuntu1

---------------
kombu (1.2.1-1ubuntu1) precise; urgency=low

  * debian/patches/remove_assert_virtual_transport_restore_unacked_once.
    patch: Disabled seemingly invalid assert in virtual transport which
    was causing nova testsuite to fail. Pending better solution.
    - LP: #861440
 -- Dave Walker (Daviey) <email address hidden> Tue, 18 Oct 2011 15:16:16 +0100

Changed in kombu (Ubuntu):
status: Confirmed → Fix Released
Revision history for this message
Lorin Hochstein (lorinh) wrote :

@Dave:

I think it's eb4bd86f.

Revision history for this message
Lorin Hochstein (lorinh) wrote :

Here it is in Github: https://github.com/openstack/nova/commit/eb4bd86f

At the time, I didn't realize the Gerrit marks the bugs off as committed when they're linked, so I just manually set this one to fix committed when I found that it fixed the issue I was having.

Thierry Carrez (ttx)
Changed in nova:
milestone: none → essex-1
Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Mark McLoughlin (markmc)
Changed in nova:
assignee: nobody → Brian Lamar (blamar)
Thierry Carrez (ttx)
Changed in nova:
milestone: essex-1 → 2012.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.