@Bogdan Looks like 00:06:05 was the first hints at issues, possibly when the flapping started (check ceilometer-agent-notification.log)
2014-09-24T00:06:05.493953+01:00 err: 2014-09-23 23:06:05.480 21070 ERROR ceilometer.openstack.common.rpc.common [-] Failed to consume message from queue: (0, 0): (320) CONNECTION_FORCED - broker forced conn ection closure with reason 'shutdown' 2014-09-23 23:06:05.480 21070 TRACE ceilometer.openstack.common.rpc.common Traceback (most recent call last): 2014-09-23 23:06:05.480 21070 TRACE ceilometer.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 591, in ensure 2014-09-23 23:06:05.480 21070 TRACE ceilometer.openstack.common.rpc.common return method(*args, **kwargs) 2014-09-23 23:06:05.480 21070 TRACE ceilometer.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 671, in _consume 2014-09-23 23:06:05.480 21070 TRACE ceilometer.openstack.common.rpc.common return self.connection.drain_events(timeout=timeout) 2014-09-23 23:06:05.480 21070 TRACE ceilometer.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 279, in drain_events 2014-09-23 23:06:05.480 21070 TRACE ceilometer.openstack.common.rpc.common return self.transport.drain_events(self.connection, **kwargs) 2014-09-23 23:06:05.480 21070 TRACE ceilometer.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in drain_events 2014-09-23 23:06:05.480 21070 TRACE ceilometer.openstack.common.rpc.common return connection.drain_events(**kwargs) 2014-09-23 23:06:05.480 21070 TRACE ceilometer.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 320, in drain_events 2014-09-23 23:06:05.480 21070 TRACE ceilometer.openstack.common.rpc.common return amqp_method(channel, args) 2014-09-23 23:06:05.480 21070 TRACE ceilometer.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 526, in _close 2014-09-23 23:06:05.480 21070 TRACE ceilometer.openstack.common.rpc.common (class_id, method_id), ConnectionError) 2014-09-23 23:06:05.480 21070 TRACE ceilometer.openstack.common.rpc.common ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown' 2014-09-24T00:06:05.493953+01:00 info: 2014-09-23 23:06:05.484 21070 INFO ceilometer.openstack.common.rpc.common [-] Reconnecting to AMQP server on 127.0.0.1:5673 2014-09-24T00:06:05.495063+01:00 info: 2014-09-23 23:06:05.484 21070 INFO ceilometer.openstack.common.rpc.common [-] Delaying reconnect for 5.0 seconds... 2014-09-24T00:06:10.511682+01:00 err: 2014-09-23 23:06:10.501 21070 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on 127.0.0.1:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
@Bogdan Looks like 00:06:05 was the first hints at issues, possibly when the flapping started (check ceilometer- agent-notificat ion.log)
2014-09- 24T00:06: 05.493953+ 01:00 err: 2014-09-23 23:06:05.480 21070 ERROR ceilometer. openstack. common. rpc.common [-] Failed to consume message from queue: (0, 0): (320) CONNECTION_FORCED - broker forced conn openstack. common. rpc.common Traceback (most recent call last): openstack. common. rpc.common File "/usr/lib/ python2. 7/dist- packages/ ceilometer/ openstack/ common/ rpc/impl_ kombu.py" , line 591, in ensure openstack. common. rpc.common return method(*args, **kwargs) openstack. common. rpc.common File "/usr/lib/ python2. 7/dist- packages/ ceilometer/ openstack/ common/ rpc/impl_ kombu.py" , line 671, in _consume openstack. common. rpc.common return self.connection .drain_ events( timeout= timeout) openstack. common. rpc.common File "/usr/lib/ python2. 7/dist- packages/ kombu/connectio n.py", line 279, in drain_events openstack. common. rpc.common return self.transport. drain_events( self.connection , **kwargs) openstack. common. rpc.common File "/usr/lib/ python2. 7/dist- packages/ kombu/transport /pyamqp. py", line 91, in drain_events openstack. common. rpc.common return connection. drain_events( **kwargs) openstack. common. rpc.common File "/usr/lib/ python2. 7/dist- packages/ amqp/connection .py", line 320, in drain_events openstack. common. rpc.common return amqp_method( channel, args) openstack. common. rpc.common File "/usr/lib/ python2. 7/dist- packages/ amqp/connection .py", line 526, in _close openstack. common. rpc.common (class_id, method_id), ConnectionError) openstack. common. rpc.common ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown' 24T00:06: 05.493953+ 01:00 info: 2014-09-23 23:06:05.484 21070 INFO ceilometer. openstack. common. rpc.common [-] Reconnecting to AMQP server on 127.0.0.1:5673 24T00:06: 05.495063+ 01:00 info: 2014-09-23 23:06:05.484 21070 INFO ceilometer. openstack. common. rpc.common [-] Delaying reconnect for 5.0 seconds... 24T00:06: 10.511682+ 01:00 err: 2014-09-23 23:06:10.501 21070 ERROR ceilometer. openstack. common. rpc.common [-] AMQP server on 127.0.0.1:5673 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
ection closure with reason 'shutdown'
2014-09-23 23:06:05.480 21070 TRACE ceilometer.
2014-09-23 23:06:05.480 21070 TRACE ceilometer.
2014-09-23 23:06:05.480 21070 TRACE ceilometer.
2014-09-23 23:06:05.480 21070 TRACE ceilometer.
2014-09-23 23:06:05.480 21070 TRACE ceilometer.
2014-09-23 23:06:05.480 21070 TRACE ceilometer.
2014-09-23 23:06:05.480 21070 TRACE ceilometer.
2014-09-23 23:06:05.480 21070 TRACE ceilometer.
2014-09-23 23:06:05.480 21070 TRACE ceilometer.
2014-09-23 23:06:05.480 21070 TRACE ceilometer.
2014-09-23 23:06:05.480 21070 TRACE ceilometer.
2014-09-23 23:06:05.480 21070 TRACE ceilometer.
2014-09-23 23:06:05.480 21070 TRACE ceilometer.
2014-09-23 23:06:05.480 21070 TRACE ceilometer.
2014-09-
2014-09-
2014-09-