Nova cannot reach messaging service

Bug #1366077 reported by Kirill Omelchenko
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Invalid
High
MOS Nova
5.0.x
Invalid
High
MOS Nova
5.1.x
Invalid
High
MOS Nova
6.0.x
Invalid
High
MOS Nova

Bug Description

http://jenkins-product.srt.mirantis.net:8080/view/0_master_swarm/job/master_fuelmain.system_test.centos.thread_5/155/testReport/%28root%29/ha_destroy_controllers/ha_destroy_controllers/

Configuration (HA, CentOS, nova-flat):
1. 3x Controllers
2. 2x Compute

Scenario (after successfull deployment):
1. Destroy first controller
2. Check pacemaker status
3. Revert environment
4. Destroy second controller
5. Check pacemaker status
6. Run OSTF

Expected:
OSTF passes successfuly

Actual:
OSTF has failed tests:
* Check that required services are running
Some nova services have not been started.. Please refer to OpenStack logs for more details.

....

* Launch instance, create snapshot, launch instance from snapshot
Failed to get to expected status. In error state. Please refer to OpenStack logs for more details.

My investigation revealed next faults and errors: http://paste.openstack.org/show/106587/

Restart of nova-{api,network,compute} services didn't make any changes.

Revision history for this message
Kirill Omelchenko (komelchenko) wrote :
Changed in fuel:
status: New → Confirmed
importance: Undecided → High
assignee: nobody → MOS Oslo (mos-oslo)
Revision history for this message
Igor Marnat (imarnat) wrote :

Kirill, did you execute the script which verifies consistency of RabbitMQ cluster after test failed? There is a script for this purpose, do you have it?

Revision history for this message
Kirill Omelchenko (komelchenko) wrote :

I didn't run the script, but I have got it.

Revision history for this message
Dmitry Mescheryakov (dmitrymex) wrote :

The bug is in OpenStack, hence moving it to MOS.

affects: fuel → mos
Changed in mos:
milestone: 5.1 → none
milestone: none → 5.1
Changed in mos:
assignee: MOS Oslo (mos-oslo) → MOS Nova (mos-nova)
tags: added: nova
tags: added: oslo.messaging
Ilya Pekelny (i159)
Changed in mos:
assignee: MOS Nova (mos-nova) → Ilya Pekelny (i159)
Revision history for this message
Roman Podoliaka (rpodolyaka) wrote :

Mikhail Durnosvistov will help Ilya to investigate this from Nova side.

Changed in mos:
status: Confirmed → In Progress
Revision history for this message
Kirill Omelchenko (komelchenko) wrote :

Still needs to be verified more, can be the faults of the testing scenario.

Changed in mos:
status: In Progress → Incomplete
Ilya Pekelny (i159)
Changed in mos:
assignee: Ilya Pekelny (i159) → nobody
Revision history for this message
Ilya Pekelny (i159) wrote :

The bag takes place in the test https://github.com/stackforge/fuel-main/blob/master/fuelweb_test/tests/tests_strength/test_failover.py#L127. OSTF starts without checking of nodes state. There is no bug in Nova or Messaging.

Changed in mos:
assignee: nobody → Fuel for Openstack (fuel)
Changed in mos:
assignee: Fuel for Openstack (fuel) → Fuel QA Team (fuel-qa)
Revision history for this message
Kirill Omelchenko (komelchenko) wrote :

After a failover the cluster is getting back to work to late, this delay causes tests to fail.

Changed in mos:
status: Incomplete → Confirmed
assignee: Fuel QA Team (fuel-qa) → nobody
Igor Marnat (imarnat)
Changed in mos:
assignee: nobody → MOS Nova (mos-nova)
Revision history for this message
Irina Povolotskaya (ipovolotskaya) wrote :

Should this be included into Release notes?
It there any possible workaround?

Revision history for this message
Roman Podoliaka (rpodolyaka) wrote :

I think, we should not. This seems to be a temporary issue.

Revision history for this message
Roman Podoliaka (rpodolyaka) wrote :
Revision history for this message
Roman Podoliaka (rpodolyaka) wrote :

Haven't seen this since September.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.