Mirantis OpenStack

[Nova] UnexpectedTaskStateError: Conflict updating instance xxxx

Bug #1746424 reported by Alexander Rubtsov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Won't Fix
High
MOS Maintenance
Mirantis OpenStack 9.x-updates
9.x
Invalid
High
Alexander Rubtsov
Mirantis OpenStack 9.x-updates

Bug Description

MOS: 9.2

Sometimes evacuation of a VM fails with the following traceback:

[root@node-2 ~]$ nova evacuate VM44 compute-1-2.domain.tld

File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 139, in __call__
 return req.get_response(self.application)
File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
 application, catch_exc_info=False)
File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
 app_iter = application(self.environ, start_response)
File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
 return resp(environ, start_response)
File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
 resp = self.call_func(req, *args, **self.kwargs)
File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
 return self.func(req, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 467, in __call__
 response = req.get_response(self._app)
File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
 application, catch_exc_info=False)
File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
 app_iter = application(self.environ, start_response)
File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
 return resp(environ, start_response)
File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
 return resp(environ, start_response)
File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 136, in __call__
 response = self.app(environ, start_response)
File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
 return resp(environ, start_response)
File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
 resp = self.call_func(req, *args, **self.kwargs)
File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
 return self.func(req, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 672, in __call__
 content_type, body, accept)
File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 734, in _process_stack
 action_result = self.dispatch(meth, request, action_args)
File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 815, in dispatch
 return method(req=request, **action_args)
File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/forcemove.py", line 252, in forcemove
 return self._forcemove(context, instance, **fm_body)
File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/forcemove.py", line 310, in _forcemove
 on_shared_storage=on_shared_storage)
File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 142, in inner
 return f(self, context, instance, *args, **kw)
File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 3506, in evacuate
 instance.save(expected_task_state=[None])
File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 223, in wrapper
 return fn(self, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 709, in save
 columns_to_join=_expected_cols(expected_attrs))
File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 809, in instance_update_and_get_original
 expected=expected)
File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 265, in wrapper
 return f(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 148, in wrapper
 ectxt.value = e.inner_exc
File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
 self.force_reraise()
File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
 six.reraise(self.type_, self.value, self.tb)
File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper
 return f(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 336, in wrapped
 return f(context, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 2768, in instance_update_and_get_original
 context, instance_uuid, values, expected, original=instance_ref))
File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 2904, in _instance_update
 raise exc(**exc_props)

UnexpectedTaskStateError: Conflict updating instance a6bc46a7-7d1c-4214-8aa4-c96086c90f6e. Expected: {'task_state': [None]}. Actual: {'task_state': u'rebuilding'}

This upstream bug might be related:
https://bugs.launchpad.net/nova/+bug/1646841

Revision history for this message
Alexander Rubtsov (arubtsov) wrote :

sla1 for 9.0-updates

tags: added: customer-found sla1
Changed in mos:
importance: Undecided → High
Revision history for this message
Denis Meltsaykin (dmeltsaykin) wrote :

Please provide logs and steps to reproduce.

"Sometimes... fails..." is not a helpful diagnostic information.

Revision history for this message
Alexander Rubtsov (arubtsov) wrote :

I've provided the log files from the Controller node where the API request was processed and the destination Compute node directly.

The command "nova evacuate <VM> <host>" is the only step needed for reproduction. However, the certain circumstances under which the issue occurs are not known (the same situation is in the related upstream bug report). If we have some additional information that helps to narrow down the issue, I will provide you with it as well.

Revision history for this message
Denis Meltsaykin (dmeltsaykin) wrote :

Looking through the logs I can clearly see that there was a network failure around that time when you saw this error. This is visible in rabbitmq, nova, glance, haproxy, etc logs. So far it does seem more like a generic infrastructure issue rather than an OpenStack bug. I'm setting this as Incomplete until the issue occurs again without networking problems visible around.

Denis Meltsaykin (dmeltsaykin)
Changed in mos:
status: New → Won't Fix
Ilya Bumarskov (ibumarskov)
Changed in mos:
milestone: none → 9.x-updates
assignee: nobody → MOS Maintenance (mos-maintenance)
Revision history for this message
Denis Meltsaykin (dmeltsaykin) wrote :

There is no confirmation that the issue is caused by a bug not by a network failures. Closing as Invalid.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.