From log analysis, ceilometer-collector is not working properly on primary controller. It uses wrong credentials to access rabbitmq.
<15>May 5 20:24:03 node-1 ceilometer-collector: 2016-05-05 20:24:03.069 24960 DEBUG oslo.messaging._drivers.impl_rabbit [-] Received recoverable error from kombu: on_error /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py:650 2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/utils/__init__.py", line 243, in retry_over_time 2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit return fun(*args, **kwargs) 2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 237, in connect 2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit return self.connection 2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 741, in connection 2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit self._connection = self._establish_connection() 2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 696, in _establish_connection 2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit conn = self.transport.establish_connection() 2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 116, in establish_connection 2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit conn = self.Connection(**opts) 2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 180, in __init__ 2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit (10, 30), # tune 2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 67, in wait 2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit self.channel_id, allowed_methods) 2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 240, in _wait_method 2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit self.method_reader.read_method() 2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 189, in read_method 2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit raise m 2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit IOError: Socket closed 2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit <11>May 5 20:24:03 node-1 ceilometer-collector: 2016-05-05 20:24:03.073 24960 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server 192.168.0.7:5673 closed the connection. Check login credentials: Socket closed
root@node-1:~# grep rabbit /etc/ceilometer/ceilometer.conf | grep -v -e ^# rpc_backend= rabbit [oslo_messaging_rabbit] rabbit_hosts= 192.168.0.7:5673, 192.168.0.5:5673, 192.168.0.6:5673 rabbit_use_ssl= False rabbit_userid= nova rabbit_password= BPcxNenb77eB8uoAjYm5N3Hp rabbit_virtual_host= / rabbit_ha_queues= True
=ERROR REPORT==== 5-May-2016::20:25:47 === closing AMQP connection <0.18161.0> (192.168.0.7:57631 -> 192.168.0.7:5673): {handshake_error,starting,0, {amqp_error,access_refused, "AMQPLAIN login refused: user 'guest' - invalid credentials", 'connection.start_ok'}}
From log analysis, collector is not working properly on primary controller.
ceilometer-
It uses wrong credentials to access rabbitmq.
<15>May 5 20:24:03 node-1 ceilometer- collector: 2016-05-05 20:24:03.069 24960 DEBUG oslo.messaging. _drivers. impl_rabbit [-] Received recoverable error from kombu: on_error /usr/lib/ python2. 7/dist- packages/ oslo_messaging/ _drivers/ impl_rabbit. py:650 _drivers. impl_rabbit Traceback (most recent call last): _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ kombu/utils/ __init_ _.py", line 243, in retry_over_time _drivers. impl_rabbit return fun(*args, **kwargs) _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ kombu/connectio n.py", line 237, in connect _drivers. impl_rabbit return self.connection _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ kombu/connectio n.py", line 741, in connection _drivers. impl_rabbit self._connection = self._establish _connection( ) _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ kombu/connectio n.py", line 696, in _establish_ connection _drivers. impl_rabbit conn = self.transport. establish_ connection( ) _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ kombu/transport /pyamqp. py", line 116, in establish_ connection _drivers. impl_rabbit conn = self.Connection (**opts) _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ amqp/connection .py", line 180, in __init__ _drivers. impl_rabbit (10, 30), # tune _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ amqp/abstract_ channel. py", line 67, in wait _drivers. impl_rabbit self.channel_id, allowed_methods) _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ amqp/connection .py", line 240, in _wait_method _drivers. impl_rabbit self.method_ reader. read_method( ) _drivers. impl_rabbit File "/usr/lib/ python2. 7/dist- packages/ amqp/method_ framing. py", line 189, in read_method _drivers. impl_rabbit raise m _drivers. impl_rabbit IOError: Socket closed _drivers. impl_rabbit collector: 2016-05-05 20:24:03.073 24960 ERROR oslo.messaging. _drivers. impl_rabbit [-] AMQP server 192.168.0.7:5673 closed the connection. Check login credentials: Socket closed
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging.
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging.
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging.
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging.
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging.
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging.
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging.
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging.
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging.
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging.
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging.
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging.
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging.
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging.
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging.
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging.
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging.
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging.
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging.
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging.
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging.
<11>May 5 20:24:03 node-1 ceilometer-
root@node-1:~# grep rabbit /etc/ceilometer /ceilometer. conf | grep -v -e ^# _rabbit] AjYm5N3Hp virtual_ host= /
rpc_backend= rabbit
[oslo_messaging
rabbit_hosts= 192.168.0.7:5673, 192.168.0.5:5673, 192.168.0.6:5673
rabbit_use_ssl= False
rabbit_userid= nova
rabbit_password= BPcxNenb77eB8uo
rabbit_
rabbit_ha_queues= True
=ERROR REPORT==== 5-May-2016: :20:25: 47 === error,starting, 0,
{amqp_ error,access_ refused,
"AMQPLAIN login refused: user 'guest' - invalid credentials",
'connection. start_ok' }}
closing AMQP connection <0.18161.0> (192.168.0.7:57631 -> 192.168.0.7:5673):
{handshake_