Fuel for OpenStack

[ironic] job master.fuel-agent.pkgs.ubuntu.review_fuel_agent_ironic_deploy failed: TimeoutError: Failed to update hypervisor details

Bug #1619263 reported by Alexander Gordeev on 2016-09-01

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	Fuel for OpenStack	Confirmed	High	MOS Ironic	Fuel for OpenStack 10.0

Bug Description

The CI job is continuing to failing with:

======================================================================
ERROR: Revert snapshot, update fuel-agent, bootstrap from review
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/master.fuel-agent.pkgs.ubuntu.review_fuel_agent_ironic_deploy/fuel-qa/fuelweb_test/helpers/decorators.py", line 121, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/master.fuel-agent.pkgs.ubuntu.review_fuel_agent_ironic_deploy/fuel-qa/gates_tests/tests/test_review_in_fuel_agent.py", line 126, in gate_patch_fuel_agent
    self._create_os_resources(ironic_conn)
  File "/home/jenkins/workspace/master.fuel-agent.pkgs.ubuntu.review_fuel_agent_ironic_deploy/fuel-qa/fuelweb_test/tests/test_ironic_base.py", line 139, in _create_os_resources
    ironic_conn.wait_for_ironic_hypervisors(ironic_conn, ironic_slaves)
  File "/home/jenkins/workspace/master.fuel-agent.pkgs.ubuntu.review_fuel_agent_ironic_deploy/fuel-qa/fuelweb_test/helpers/ironic_actions.py", line 94, in wait_for_ironic_hypervisors
    timeout_msg='Failed to update hypervisor details')
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 111, in wait
    raise TimeoutError(timeout_msg)
TimeoutError: Failed to update hypervisor details

----------------------------------------------------------------------

Eg.: https://ci.fuel-infra.org/job/master.fuel-agent.pkgs.ubuntu.review_fuel_agent_ironic_deploy/660/console

Tags:

Revision history for this message

Alexander Gordeev (a-gordeev) wrote on 2016-09-01:

Set to Crit just because it blocks the incoming CR queue for fuel-agent from merges.

Dmitry Pyzhov (dpyzhov) on 2016-09-06

Changed in fuel:
importance:	Critical → High
tags:	added: tech-debt

Dmitry Pyzhov (dpyzhov) on 2016-09-07

Changed in fuel:
assignee:	nobody → MOS Ironic (mos-ironic)

Revision history for this message

Vasyl Saienko (vsaienko) wrote on 2016-09-07:

Download full text (3.2 KiB)

nova-compute logs contain the following:

2016-08-31T15:20:54.427570+00:00 debug: 2016-08-31 15:20:54.425 30148 DEBUG keystoneauth.session [req-618ef27b-7261-41b5-9eb5-773cff188cac - - - - -] REQ: curl -g -i -X GET http://10.109.25.5:6385/v1/nodes/detail -H "X-OpenStack-Ironic-API-Version: 1.21" -H "User-Agent: python-ironicclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}8044ce8b8891579ba962dd2155492783c054e963" _http_log_request /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:337
2016-08-31T15:20:54.513042+00:00 debug: 2016-08-31 15:20:54.505 30148 DEBUG keystoneauth.session [req-618ef27b-7261-41b5-9eb5-773cff188cac - - - - -] RESP: [406] X-Openstack-Ironic-Api-Minimum-Version: 1.1 X-Openstack-Ironic-Api-Maximum-Version: 1.20 Content-Type: application/json Content-Length: 217 Date: Wed, 31 Aug 2016 15:20:54 GMT Connection: close
RESP BODY: {"error_message": "{\"code\": 406, \"description\": \"Version 1.21 was requested but the minor version is not supported by this service. The supported version range is: [1.1, 1.20].\", \"title\": \"Not Acceptable\"}"}
_http_log_response /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:366
2016-08-31T15:20:54.513042+00:00 err: 2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task [req-618ef27b-7261-41b5-9eb5-773cff188cac - - - - -] Error during ClusteredComputeManager.update_available_resource
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task Traceback (most recent call last):
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task task(self, context)
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6429, in update_available_resource
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task nodenames = set(self.driver.get_available_nodes())
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/nova/virt/ironic/driver.py", line 577, in get_available_nodes
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task self._refresh_cache()
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/nova/virt/ironic/driver.py", line 547, in _refresh_cache
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task for node in self._get_node_list(detail=True, limit=0):
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/nova/virt/ironic/driver.py", line 467, in _get_node_list
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task node_list = self.ironicclient.call("node.list", **kwargs)
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/nova/virt/ironic/client_wrapper.py", line 139, in call
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task return self._multi_getattr(client, method)(*args, **kwargs)
2016-08-31 15:2...

nova-compute logs contain the following:

2016-08-31T15:20:54.427570+00:00 debug:  2016-08-31 15:20:54.425 30148 DEBUG keystoneauth.session [req-618ef27b-7261-41b5-9eb5-773cff188cac - - - - -] REQ: curl -g -i -X GET http://10.109.25.5:6385/v1/nodes/detail -H "X-OpenStack-Ironic-API-Version: 1.21" -H "User-Agent: python-ironicclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}8044ce8b8891579ba962dd2155492783c054e963" _http_log_request /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:337
2016-08-31T15:20:54.513042+00:00 debug:  2016-08-31 15:20:54.505 30148 DEBUG keystoneauth.session [req-618ef27b-7261-41b5-9eb5-773cff188cac - - - - -] RESP: [406] X-Openstack-Ironic-Api-Minimum-Version: 1.1 X-Openstack-Ironic-Api-Maximum-Version: 1.20 Content-Type: application/json Content-Length: 217 Date: Wed, 31 Aug 2016 15:20:54 GMT Connection: close 
RESP BODY: {"error_message": "{\"code\": 406, \"description\": \"Version 1.21 was requested but the minor version is not supported by this service. The supported version range is: [1.1, 1.20].\", \"title\": \"Not Acceptable\"}"}
 _http_log_response /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:366
2016-08-31T15:20:54.513042+00:00 err:  2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task [req-618ef27b-7261-41b5-9eb5-773cff188cac - - - - -] Error during ClusteredComputeManager.update_available_resource
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task Traceback (most recent call last):
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task     task(self, context)
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6429, in update_available_resource
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task     nodenames = set(self.driver.get_available_nodes())
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/dist-packages/nova/virt/ironic/driver.py", line 577, in get_available_nodes
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task     self._refresh_cache()
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/dist-packages/nova/virt/ironic/driver.py", line 547, in _refresh_cache
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task     for node in self._get_node_list(detail=True, limit=0):
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/dist-packages/nova/virt/ironic/driver.py", line 467, in _get_node_list
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task     node_list = self.ironicclient.call("node.list", **kwargs)
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/dist-packages/nova/virt/ironic/client_wrapper.py", line 139, in call
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task     return self._multi_getattr(client, method)(*args, **kwargs)
2016-08-31 15:20:54.505 30148 ERROR oslo_service.periodic_task   File "/usr/

ironic-package is not newton.

Vitaly Sedelnik (vsedelnik) on 2016-09-07

Changed in fuel:
status:	New → Confirmed
tags:	added: area-ironic

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.