Was able to replicate issue after a fresh install with ISO


{"build_id": "2014-09-18_06-04-08", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "31", "auth_required": true, "api": "1.0", "nailgun_sha": "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d", "production": "docker", "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd", "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13", "feature_groups": ["experimental"], "release": "5.1", "release_versions": {"2014.1.1-5.1": {"VERSION": {"build_id": "2014-09-18_06-04-08", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "31", "api": "1.0", "nailgun_sha": "eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d", "production": "docker", "fuelmain_sha": "8ef433e939425eabd1034c0b70e90bdf888b69fd", "astute_sha": "f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13", "feature_groups": ["experimental"], "release": "5.1", "fuellib_sha": "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"}}}, "fuellib_sha": "d9b16846e54f76c8ebe7764d2b5b8231d6b25079"}

Was able to replicate RabbitMQ <-> Nova API Timeouts without shutting off any nodes;

182>Sep 19 19:30:12 node-2 nova-api 2014-09-19 19:30:12.802 17124 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 192.168.0.2
<180>Sep 19 19:30:12 node-2 nova-api 2014-09-19 19:30:12.838 17124 WARNING keystoneclient.middleware.auth_token [-] Authorization failed for token
<180>Sep 19 19:30:12 node-2 nova-api 2014-09-19 19:30:12.839 17124 WARNING keystoneclient.middleware.auth_token [-] Authorization failed for token
<182>Sep 19 19:30:12 node-2 nova-api 2014-09-19 19:30:12.840 17124 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request
<182>Sep 19 19:30:12 node-2 nova-api 2014-09-19 19:30:12.842 17124 INFO nova.osapi_compute.wsgi.server [-] 192.168.0.2 "GET /v2/8986a0ce721840bd8cd692d542475940/servers/detail?all_tenants=True&host=node-12 HT
TP/1.1" status: 401 len: 194 time: 0.0441570
<179>Sep 19 19:31:57 node-2 nova-api 2014-09-19 19:31:57.543 17121 ERROR oslo.messaging._drivers.impl_rabbit [req-45a12408-2c2b-4438-8d55-2bda1ed2eb20 ] Failed to publish message to topic 'compute.node-40': [
Errno 32] Broken pipe
2014-09-19 19:31:57.543 17121 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2014-09-19 19:31:57.543 17121 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 648, in ensure
2014-09-19 19:31:57.543 17121 TRACE oslo.messaging._drivers.impl_rabbit     return method(*args, **kwargs)
2014-09-19 19:31:57.543 17121 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 753, in _publish
2014-09-19 19:31:57.543 17121 TRACE oslo.messaging._drivers.impl_rabbit     publisher = cls(self.conf, self.channel, topic, **kwargs)
2014-09-19 19:31:57.543 17121 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 396, in __init__
2014-09-19 19:31:57.543 17121 TRACE oslo.messaging._drivers.impl_rabbit     **options)
2014-09-19 19:31:57.543 17121 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 339, in __init__
2014-09-19 19:31:57.543 17121 TRACE oslo.messaging._drivers.impl_rabbit     self.reconnect(channel)
2014-09-19 19:31:57.543 17121 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 347, in reconnect
2014-09-19 19:31:57.543 17121 TRACE oslo.messaging._drivers.impl_rabbit     routing_key=self.routing_key)
2014-09-19 19:31:57.543 17121 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 84, in __init__
2014-09-19 19:31:57.543 17121 TRACE oslo.messaging._drivers.impl_rabbit     self.revive(self._channel)
2014-09-19 19:31:57.543 17121 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 218, in revive
2014-09-19 19:31:57.543 17121 TRACE oslo.messaging._drivers.impl_rabbit     self.declare()
2014-09-19 19:31:57.543 17121 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 104, in declare
2014-09-19 19:31:57.543 17121 TRACE oslo.messaging._drivers.impl_rabbit     self.exchange.declare()
2014-09-19 19:31:57.543 17121 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 166, in declare
2014-09-19 19:31:57.543 17121 TRACE oslo.messaging._drivers.impl_rabbit     nowait=nowait, passive=passive,
2014-09-19 19:31:57.543 17121 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 613, in exchange_declare
2014-09-19 19:31:57.543 17121 TRACE oslo.messaging._drivers.impl_rabbit     self._send_method((40, 10), args)
2014-09-19 19:31:57.543 17121 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 56, in _send_method
2014-09-19 19:31:57.543 17121 TRACE oslo.messaging._drivers.impl_rabbit     self.channel_id, method_sig, args, content,
2014-09-19 19:31:57.543 17121 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 221, in write_method
2014-09-19 19:31:57.543 17121 TRACE oslo.messaging._drivers.impl_rabbit     write_frame(1, channel, payload)
2014-09-19 19:31:57.543 17121 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/transport.py", line 177, in write_frame
2014-09-19 19:31:57.543 17121 TRACE oslo.messaging._drivers.impl_rabbit     frame_type, channel, size, payload, 0xce,
2014-09-19 19:31:57.543 17121 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 307, in sendall
2014-09-19 19:31:57.543 17121 TRACE oslo.messaging._drivers.impl_rabbit     tail = self.send(data, flags)
2014-09-19 19:31:57.543 17121 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 293, in send
2014-09-19 19:31:57.543 17121 TRACE oslo.messaging._drivers.impl_rabbit     total_sent += fd.send(data[total_sent:], flags)
2014-09-19 19:31:57.543 17121 TRACE oslo.messaging._drivers.impl_rabbit error: [Errno 32] Broken pipe
<182>Sep 19 19:31:57 node-2 nova-api 2014-09-19 19:31:57.547 17121 INFO oslo.messaging._drivers.impl_rabbit [req-45a12408-2c2b-4438-8d55-2bda1ed2eb20 ] Reconnecting to AMQP server on 192.168.0.6:5673
<182>Sep 19 19:31:57 node-2 nova-api 2014-09-19 19:31:57.547 17121 INFO oslo.messaging._drivers.impl_rabbit [req-45a12408-2c2b-4438-8d55-2bda1ed2eb20 ] Delaying reconnect for 1.0 seconds...
<182>Sep 19 19:31:58 node-2 nova-api 2014-09-19 19:31:58.567 17121 INFO oslo.messaging._drivers.impl_rabbit [req-45a12408-2c2b-4438-8d55-2bda1ed2eb20 ] Connected to AMQP server on 192.168.0.6:5673
<179>Sep 19 19:31:58 node-2 nova-api 2014-09-19 19:31:58.571 17121 ERROR oslo.messaging._drivers.impl_rabbit [req-45a12408-2c2b-4438-8d55-2bda1ed2eb20 ] Failed to consume message from queue: Socket closed
2014-09-19 19:31:58.571 17121 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2014-09-19 19:31:58.571 17121 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 648, in ensure
2014-09-19 19:31:58.571 17121 TRACE oslo.messaging._drivers.impl_rabbit     return method(*args, **kwargs)
2014-09-19 19:31:58.571 17121 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 737, in _consume
2014-09-19 19:31:58.571 17121 TRACE oslo.messaging._drivers.impl_rabbit     return self.connection.drain_events(timeout=timeout)
2014-09-19 19:31:58.571 17121 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 279, in drain_events
2014-09-19 19:31:58.571 17121 TRACE oslo.messaging._drivers.impl_rabbit     return self.transport.drain_events(self.connection, **kwargs)
2014-09-19 19:31:58.571 17121 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in drain_events
2014-09-19 19:31:58.571 17121 TRACE oslo.messaging._drivers.impl_rabbit     return connection.drain_events(**kwargs)
2014-09-19 19:31:58.571 17121 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 299, in drain_events
2014-09-19 19:31:58.571 17121 TRACE oslo.messaging._drivers.impl_rabbit     chanmap, None, timeout=timeout,
2014-09-19 19:31:58.571 17121 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 362, in _wait_multiple
2014-09-19 19:31:58.571 17121 TRACE oslo.messaging._drivers.impl_rabbit     channel, method_sig, args, content = read_timeout(timeout)
2014-09-19 19:31:58.571 17121 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 333, in read_timeout
2014-09-19 19:31:58.571 17121 TRACE oslo.messaging._drivers.impl_rabbit     return self.method_reader.read_method()
2014-09-19 19:31:58.571 17121 TRACE oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 189, in read_method
2014-09-19 19:31:58.571 17121 TRACE oslo.messaging._drivers.impl_rabbit     raise m
2014-09-19 19:31:58.571 17121 TRACE oslo.messaging._drivers.impl_rabbit IOError: Socket closed
2014-09-19 19:31:58.571 17121 TRACE oslo.messaging._drivers.impl_rabbit
<182>Sep 19 19:31:58 node-2 nova-api 2014-09-19 19:31:58.573 17121 INFO oslo.messaging._drivers.impl_rabbit [req-45a12408-2c2b-4438-8d55-2bda1ed2eb20 ] Reconnecting to AMQP server on 192.168.0.5:5673
<182>Sep 19 19:31:58 node-2 nova-api 2014-09-19 19:31:58.573 17121 INFO oslo.messaging._drivers.impl_rabbit [req-45a12408-2c2b-4438-8d55-2bda1ed2eb20 ] Delaying reconnect for 1.0 seconds...
<182>Sep 19 19:31:59 node-2 nova-api 2014-09-19 19:31:59.612 17121 INFO oslo.messaging._drivers.impl_rabbit [req-45a12408-2c2b-4438-8d55-2bda1ed2eb20 ] Connected to AMQP server on 192.168.0.5:5673
<182>Sep 19 19:32:33 node-2 nova-api 2014-09-19 19:32:33.717 17124 INFO nova.osapi_compute.wsgi.server [req-bf23572a-7318-4c34-a910-1945b963313b None] 192.168.0.2 "GET /v2/ec3c73a6d22a48b4845d257abb4c8356/servers/483d2bf6-ba85-4176-95fe-0ab8e98b38a0 HTTP/1.1" status: 200 len: 2043 time: 0.1149740
<182>Sep 19 19:32:33 node-2 nova-api 2014-09-19 19:32:33.774 17124 INFO nova.osapi_compute.wsgi.server [req-c272cf45-7770-4a16-9be7-68a3c483bd9a None] 192.168.0.2 "GET /v2/ec3c73a6d22a48b4845d257abb4c8356/servers/483d2bf6-ba85-4176-95fe-0ab8e98b38a0/os-volume_attachments HTTP/1.1" status: 200 len: 197 time: 0.0549340
<182>Sep 19 19:32:33 node-2 nova-api 2014-09-19 19:32:33.784 17124 INFO nova.osapi_compute.wsgi.server [req-c748972f-f8e1-410a-a65f-fb0c9b563454 None] 192.168.0.2 "GET /v2/ec3c73a6d22a48b4845d257abb4c8356/flavors/d819e6d0-1044-4dcc-8763-e24a13b01bf9 HTTP/1.1" status: 200 len: 711 time: 0.0076358
<179>Sep 19 19:32:33 node-2 nova-api 2014-09-19 19:32:33.929 17124 ERROR oslo.messaging._drivers.impl_rabbit [req-47113ea5-4140-401e-a0ad-e5fda6ad02d5 ] Failed to publish message to topic<179>Sep 19 19:32:58 node-2 nova-api 2014-09-19 19:32:58.698 17121 ERROR nova.api.openstack [req-45a12408-2c2b-4438-8d55-2bda1ed2eb20 None] Caught error: Timed out waiting for a reply to message ID c286441399
ab40f48ec986c8b25a6468
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack Traceback (most recent call last):
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 125, in __call__
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack     return req.get_response(self.application)
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack     application, catch_exc_info=False)
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack     return resp(environ, start_response)
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 582, in __call__
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack     return self.app(env, start_response)
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack     return resp(environ, start_response)
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack     return resp(environ, start_response)
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack     response = self.app(environ, start_response)
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack     return resp(environ, start_response)
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 917, in __call__
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack     content_type, body, accept)
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 983, in _process_stack
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack     action_result = self.dispatch(meth, request, action_args)
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 1070, in dispatch
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack     return method(req=request, **action_args)
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/consoles.py", line 70, in get_spice_console
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack     console_type)
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 201, in wrapped
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack     return func(self, context, target, *args, **kwargs)
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 182, in wrapped
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack     return function(self, context, instance, *args, **kwargs)
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 2656, in get_spice_console
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack     instance=instance, console_type=console_type)
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/rpcapi.py", line 486, in get_spice_console
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack     instance=instance, console_type=console_type)
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 150, in call
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack     wait_for_reply=True, timeout=timeout)
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack     timeout=timeout)
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 430, in send
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack     return self._send(target, ctxt, message, wait_for_reply, timeout)
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 421, in _send
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack     result = self._waiter.wait(msg_id, timeout)
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 267, in wait
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack     reply, ending = self._poll_connection(msg_id, timeout)
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 217, in _poll_connection
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack     % msg_id)
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack MessagingTimeout: Timed out waiting for a reply to message ID c286441399ab40f48ec986c8b25a6468
2014-09-19 19:32:58.698 17121 TRACE nova.api.openstack  'compute.node-40': [Errno 32] Broken pipe
<182>Sep 19 19:32:58 node-2 nova-api 2014-09-19 19:32:58.714 17121 INFO nova.api.openstack [req-45a12408-2c2b-4438-8d55-2bda1ed2eb20 None] http://23.109.32.2:8774/v2/ec3c73a6d22a48b4845d257abb4c8356/servers/483d2bf6-ba85-4176-95fe-0ab8e98b38a0/action returned with HTTP 500
<182>Sep 19 19:32:58 node-2 nova-api 2014-09-19 19:32:58.721 17121 INFO nova.osapi_compute.wsgi.server [req-45a12408-2c2b-4438-8d55-2bda1ed2eb20 None] Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 405, in handle_one_response
    write(''.join(towrite))
  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 349, in write
    _writelines(towrite)
  File "/usr/lib/python2.7/socket.py", line 334, in writelines
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
  File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 307, in sendall
    tail = self.send(data, flags)
  File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 293, in send
    total_sent += fd.send(data[total_sent:], flags)
error: [Errno 104] Connection reset by peer