Setting non blocking libvirt flag causes __str__ error on virConnect

Bug #962840 reported by Devin Carlen
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Yun Mao
Essex
Fix Released
High
Unassigned

Bug Description

If you set the --libvirt_nonblocking flag to true and attempt to launch an instance, the following exception is thrown:

(nova.rpc.amqp): TRACE: AttributeError: virConnect instance has no attribute '__str__'
(nova.rpc.amqp): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 219, in __str__
(nova.rpc.amqp): TRACE: return self._obj.__str__()

Revision history for this message
yong sheng gong (gongysh) wrote :

with trunk codes I failed to reproduce it on redhat 6.2:
[root@robinlinux ~]# rpm -qa | grep libvirt
libvirt-client-0.9.6-2.el6.x86_64
libvirt-java-devel-0.4.7-1.el6.noarch
libvirt-python-0.9.6-2.el6.x86_64
libvirt-0.9.6-2.el6.x86_64
libvirt-java-0.4.7-1.el6.noarch
libvirt-devel-0.9.6-2.el6.x86_64
[root@robinlinux ~]# python --version
Python 2.6.6

and devstack:
libvirt 0.9.2, python 2.7.2+

my nova.conf is:
[DEFAULT]
logdir=/var/log/nova
state_path=/var/lib/nova
lock_path=/var/lib/nova/tmp
instances_path=/var/lib/nova/instances
debug=True
scheduler_driver=nova.scheduler.simple.SimpleScheduler
dhcpbridge_flagfile=/etc/nova/nova.conf
fixed_range=10.10.10.0/24
s3_host=127.0.0.1
osapi_compute_extension=nova.api.openstack.compute.contrib.standard_extensions
my_ip=127.0.0.1
public_interface=br100
vlan_interface=eth0
sql_connection=mysql://nova:nova@localhost/nova
instance_name_template=instance-%08x
novncproxy_base_url=http://127.0.0.1:6080/vnc_auto.html
xvpvncproxy_base_url=http://127.0.0.1:6081/console
vncserver_listen=127.0.0.1
vncserver_proxyclient_address=127.0.0.1
api_paste_config=/etc/nova/api-paste.ini
image_service=nova.image.glance.GlanceImageService
ec2_dmz_host=127.0.0.1
rabbit_host=localhost
rabbit_password=guest
glance_api_servers=127.0.0.1:9292
connection_type=libvirt
libvirt_type=kvm
libvirt_nonblocking=true
.....

Revision history for this message
Thierry Carrez (ttx) wrote :

@Devin: Could you give a bit more detail on the environment you reproduce the issue on ?

Changed in nova:
importance: Critical → High
status: New → Incomplete
Revision history for this message
yong sheng gong (gongysh) wrote :
Download full text (4.0 KiB)

Haha, I reproduce it today.

My way is not to start nova-network, and then run nova boot command.

below is message:
2012-03-28 08:12:22 ERROR nova.compute.manager [req-6274f7dc-4815-4926-b913-b16cba0cc4d6 44c3d88f257a4dc39790c318fcb1a2cc 36cf36c3972a437890e5df7bf9805097] [instance: a26c20ed-72b1-45bc-a7c5-b321824b50b1] Instance failed network setup
(nova.compute.manager): TRACE: Traceback (most recent call last):
(nova.compute.manager): TRACE: File "/root/test/git/nova/nova/compute/manager.py", line 559, in _allocate_network
(nova.compute.manager): TRACE: requested_networks=requested_networks)
(nova.compute.manager): TRACE: File "/root/test/git/nova/nova/network/api.py", line 170, in allocate_for_instance
(nova.compute.manager): TRACE: 'args': args})
(nova.compute.manager): TRACE: File "/root/test/git/nova/nova/rpc/__init__.py", line 68, in call
(nova.compute.manager): TRACE: return _get_impl().call(context, topic, msg, timeout)
(nova.compute.manager): TRACE: File "/root/test/git/nova/nova/rpc/impl_kombu.py", line 674, in call
(nova.compute.manager): TRACE: return rpc_amqp.call(context, topic, msg, timeout, Connection.pool)
(nova.compute.manager): TRACE: File "/root/test/git/nova/nova/rpc/amqp.py", line 338, in call
(nova.compute.manager): TRACE: rv = list(rv)
(nova.compute.manager): TRACE: File "/root/test/git/nova/nova/rpc/amqp.py", line 299, in __iter__
(nova.compute.manager): TRACE: self._iterator.next()
(nova.compute.manager): TRACE: File "/root/test/git/nova/nova/rpc/impl_kombu.py", line 572, in iterconsume
(nova.compute.manager): TRACE: yield self.ensure(_error_callback, _consume)
(nova.compute.manager): TRACE: File "/root/test/git/nova/nova/rpc/impl_kombu.py", line 503, in ensure
(nova.compute.manager): TRACE: error_callback(e)
(nova.compute.manager): TRACE: File "/root/test/git/nova/nova/rpc/impl_kombu.py", line 553, in _error_callback
(nova.compute.manager): TRACE: raise rpc_common.Timeout()
(nova.compute.manager): TRACE: Timeout: Timeout while waiting on RPC response.
(nova.compute.manager): TRACE:
2012-03-28 08:12:22 ERROR nova.rpc.amqp [req-6274f7dc-4815-4926-b913-b16cba0cc4d6 44c3d88f257a4dc39790c318fcb1a2cc 36cf36c3972a437890e5df7bf9805097] Exception during message handling
(nova.rpc.amqp): TRACE: Traceback (most recent call last):
(nova.rpc.amqp): TRACE: File "/root/test/git/nova/nova/rpc/amqp.py", line 252, in _process_data
(nova.rpc.amqp): TRACE: rval = node_func(context=ctxt, **node_args)
(nova.rpc.amqp): TRACE: File "/root/test/git/nova/nova/exception.py", line 138, in wrapped
(nova.rpc.amqp): TRACE: payload)
(nova.rpc.amqp): TRACE: File "/root/test/git/nova/nova/notifier/api.py", line 118, in notify
(nova.rpc.amqp): TRACE: payload = utils.to_primitive(payload, convert_instances=True)
(nova.rpc.amqp): TRACE: File "/root/test/git/nova/nova/utils.py", line 798, in to_primitive
(nova.rpc.amqp): TRACE: level=level)
(nova.rpc.amqp): TRACE: File "/root/test/git/nova/nova/utils.py", line 792, in to_primitive
(nova.rpc.amqp): TRACE: level=level))
(nova.rpc.amqp): TRACE: File "/root/test/git/nova/nova/utils.py", line 813, in to_primitive
(nova.r...

Read more...

Thierry Carrez (ttx)
Changed in nova:
status: Incomplete → Confirmed
Revision history for this message
Yun Mao (yunmao) wrote :

This is due to libvirt using old-style class while the proxy_call method in eventlet tpool assumes all objects are in new style.

Changed in nova:
assignee: nobody → Yun Mao (yunmao)
Thierry Carrez (ttx)
tags: removed: essex-rc-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/6164

Changed in nova:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/6164
Committed: http://github.com/openstack/nova/commit/bc173ec44e6c3a42af22976122b46fd25546559b
Submitter: Jenkins
Branch: master

commit bc173ec44e6c3a42af22976122b46fd25546559b
Author: Yun Mao <email address hidden>
Date: Tue Apr 3 11:17:50 2012 -0400

    Fixed bug 962840, added a test case.

    eventlet.tpool.Proxy doesn't work with old-style class in __str__()
    or __repr__() calls. See bug #962840 for details.
    We perform a monkey patch to replace those two instance methods.

    Change-Id: Ia51bbd3e71cad7df45da5b3b27eef70f9d9e9002

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/essex)

Fix proposed to branch: stable/essex
Review: https://review.openstack.org/6309

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/essex)

Reviewed: https://review.openstack.org/6309
Committed: http://github.com/openstack/nova/commit/3b14c746910abf95e8136e409085873fd6ac970d
Submitter: Jenkins
Branch: stable/essex

commit 3b14c746910abf95e8136e409085873fd6ac970d
Author: Yun Mao <email address hidden>
Date: Tue Apr 3 11:17:50 2012 -0400

    Fixed bug 962840, added a test case.

    eventlet.tpool.Proxy doesn't work with old-style class in __str__()
    or __repr__() calls. See bug #962840 for details.
    We perform a monkey patch to replace those two instance methods.

    Change-Id: Ia51bbd3e71cad7df45da5b3b27eef70f9d9e9002

tags: added: in-stable-essex
Thierry Carrez (ttx)
Changed in nova:
milestone: none → folsom-1
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: folsom-1 → 2012.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.