Comment 2 for bug 1576691

Revision history for this message
Oleksandr Martsyniuk (omartsyniuk) wrote :

From log analysis,
ceilometer-collector is not working properly on primary controller.
It uses wrong credentials to access rabbitmq.

<15>May 5 20:24:03 node-1 ceilometer-collector: 2016-05-05 20:24:03.069 24960 DEBUG oslo.messaging._drivers.impl_rabbit [-] Received recoverable error from kombu: on_error /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py:650
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/utils/__init__.py", line 243, in retry_over_time
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit return fun(*args, **kwargs)
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 237, in connect
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit return self.connection
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 741, in connection
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit self._connection = self._establish_connection()
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 696, in _establish_connection
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit conn = self.transport.establish_connection()
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 116, in establish_connection
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit conn = self.Connection(**opts)
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 180, in __init__
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit (10, 30), # tune
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 67, in wait
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit self.channel_id, allowed_methods)
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 240, in _wait_method
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit self.method_reader.read_method()
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 189, in read_method
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit raise m
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit IOError: Socket closed
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit
<11>May 5 20:24:03 node-1 ceilometer-collector: 2016-05-05 20:24:03.073 24960 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server 192.168.0.7:5673 closed the connection. Check login credentials: Socket closed

root@node-1:~# grep rabbit /etc/ceilometer/ceilometer.conf | grep -v -e ^#
rpc_backend= rabbit
[oslo_messaging_rabbit]
rabbit_hosts= 192.168.0.7:5673, 192.168.0.5:5673, 192.168.0.6:5673
rabbit_use_ssl= False
rabbit_userid= nova
rabbit_password= BPcxNenb77eB8uoAjYm5N3Hp
rabbit_virtual_host= /
rabbit_ha_queues= True

=ERROR REPORT==== 5-May-2016::20:25:47 ===
closing AMQP connection <0.18161.0> (192.168.0.7:57631 -> 192.168.0.7:5673):
{handshake_error,starting,0,
                 {amqp_error,access_refused,
                             "AMQPLAIN login refused: user 'guest' - invalid credentials",
                             'connection.start_ok'}}