Assigning of floating ip timed out after destroy of node with rabbit master

Bug #1565830 reported by Andrey Sledzinskiy
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Confirmed
High
MOS Oslo

Bug Description

iso - 9.0-152

Steps:
1. Deploy next cluster - Neutron Vlan, default storages, 3 controller, 2 compute, 1 cinder
2. Shutdown controller node where rabbit master isn't running (node-2 in test)
3. Wait HA OSTF tests pass
4. Run sanity and smoke OSTF tests
5. Start destroyed controller
6. Wait HA OSTF tests passed
7. Run sanity and smoke OSTF tests
8. Destroy controller with master rabbit (node-1)
9. Wait HA OSTF tests pass
10. Run sanity and smoke tests

Expected result - all tests pass
Actual - Check network connectivity from instance via floating IP test failed on assigning floating ip with timeout
from neutron-server.log:
2016-04-04 06:36:31.100 10131 DEBUG oslo_messaging._drivers.amqpdriver [req-50ef0029-6b95-4263-9319-88fe8fff8d4e 169be0e4d1fb4b61871626b9e61cd763 1067c79510ae490c99c9ff3a8f09457d - - -] CAST unique_id: 023
fe90daefd4355b7650f15551f1b61 exchange 'neutron' topic 'l3_agent' _send /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:454
2016-04-04 06:36:31.101 10131 DEBUG oslo.messaging._drivers.impl_rabbit [req-50ef0029-6b95-4263-9319-88fe8fff8d4e 169be0e4d1fb4b61871626b9e61cd763 1067c79510ae490c99c9ff3a8f09457d - - -] Received recoverab
le error from kombu: on_error /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py:678
2016-04-04 06:36:31.101 10131 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2016-04-04 06:36:31.101 10131 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 436, in _ensured
2016-04-04 06:36:31.101 10131 ERROR oslo.messaging._drivers.impl_rabbit return fun(*args, **kwargs)
2016-04-04 06:36:31.101 10131 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 508, in __call__
2016-04-04 06:36:31.101 10131 ERROR oslo.messaging._drivers.impl_rabbit return fun(*args, channel=channels[0], **kwargs), channels[0]
2016-04-04 06:36:31.101 10131 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 727, in execute_method
2016-04-04 06:36:31.101 10131 ERROR oslo.messaging._drivers.impl_rabbit method()
2016-04-04 06:36:31.101 10131 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1090, in _publish
2016-04-04 06:36:31.101 10131 ERROR oslo.messaging._drivers.impl_rabbit routing_key=routing_key)
2016-04-04 06:36:31.101 10131 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 85, in __init__
2016-04-04 06:36:31.101 10131 ERROR oslo.messaging._drivers.impl_rabbit self.revive(self._channel)
2016-04-04 06:36:31.101 10131 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 222, in revive
2016-04-04 06:36:31.101 10131 ERROR oslo.messaging._drivers.impl_rabbit self.declare()
2016-04-04 06:36:31.101 10131 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 105, in declare
2016-04-04 06:36:31.101 10131 ERROR oslo.messaging._drivers.impl_rabbit self.exchange.declare()
2016-04-04 06:36:31.101 10131 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 174, in declare
2016-04-04 06:36:31.101 10131 ERROR oslo.messaging._drivers.impl_rabbit nowait=nowait, passive=passive,
2016-04-04 06:36:31.101 10131 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 615, in exchange_declare
2016-04-04 06:36:31.101 10131 ERROR oslo.messaging._drivers.impl_rabbit self._send_method((40, 10), args)
2016-04-04 06:36:31.101 10131 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 56, in _send_method
2016-04-04 06:36:31.101 10131 ERROR oslo.messaging._drivers.impl_rabbit self.channel_id, method_sig, args, content,
2016-04-04 06:36:31.101 10131 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 221, in write_method
2016-04-04 06:36:31.101 10131 ERROR oslo.messaging._drivers.impl_rabbit write_frame(1, channel, payload)
2016-04-04 06:36:31.101 10131 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/transport.py", line 182, in write_frame
2016-04-04 06:36:31.101 10131 ERROR oslo.messaging._drivers.impl_rabbit frame_type, channel, size, payload, 0xce,
2016-04-04 06:36:31.101 10131 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 385, in sendall
2016-04-04 06:36:31.101 10131 ERROR oslo.messaging._drivers.impl_rabbit tail = self.send(data, flags)
2016-04-04 06:36:31.101 10131 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 379, in send
2016-04-04 06:36:31.101 10131 ERROR oslo.messaging._drivers.impl_rabbit return self._send_loop(self.fd.send, data, flags)
2016-04-04 06:36:31.101 10131 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 366, in _send_loop
2016-04-04 06:36:31.101 10131 ERROR oslo.messaging._drivers.impl_rabbit return send_method(data, *args)
2016-04-04 06:36:31.101 10131 ERROR oslo.messaging._drivers.impl_rabbit error: [Errno 110] Connection timed out
2016-04-04 06:36:31.101 10131 ERROR oslo.messaging._drivers.impl_rabbit
2016-04-04 06:36:31.108 10131 ERROR oslo.messaging._drivers.impl_rabbit [req-50ef0029-6b95-4263-9319-88fe8fff8d4e 169be0e4d1fb4b61871626b9e61cd763 1067c79510ae490c99c9ff3a8f09457d - - -] AMQP server on 10.109.16.4:5673 is unreachable: [Errno 110] Connection timed out. Trying again in 5 seconds.

fuel version - http://paste.openstack.org/show/492882/

Tags: area-oslo
Revision history for this message
Andrey Sledzinskiy (asledzinskiy) wrote :
Dina Belova (dbelova)
Changed in mos:
status: New → Confirmed
importance: Undecided → High
tags: added: area-oslo
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.