Fuel Contrail plugin

Some platform OSTF tests failed with Contrail+Ceilometer

Bug #1576691 reported by okosse on 2016-04-29

This bug report is a duplicate of: Bug #1583526: [Contrail 4.0.0] OSTF failled for ceilometer. Edit Remove

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Fuel Contrail plugin	In Progress	Undecided	Partner Centric Engineering	Fuel Contrail plugin 4.0.0

Bug Description

Step for reproduce:
    1. Create an environment with "Neutron with tunneling segmentation" as a network configuration and CEPH storage
    2. Enable and configure Contrail plugin
    3. Add a node with "controller" role
    4. Add a node with "controller" + "MongoDB" multirole
    5. Add a node with "controller"+ "ceph-OSD" multiroles
    6. Add a node with "compute" + "ceph-OSD" + "cinder" multiroles
    7. Add a node with "compute" + "ceph-OSD" multiroles
    8. Add a node with "MongoDB" role
    9. Add a node with "contrail-config", "contrail-control" and "contrail-db" roles
    10. Deploy cluster with plugin
    11. Run OSTF tests
Expected results: All steps must be completed successfully, without any errors.
Actual results: Failed 5 OSTF tests. Names of failed tests:
  - Typical stack actions: create, delete, show details, etc. (failure) Stack was not created properly. Please refer to OpenStack logs for more details.
  - Advanced stack actions: suspend, resume and check (failure) Stack was not created properly. Please refer to OpenStack logs for more details.
  - Check stack autoscaling (failure) Stack was not created properly. Please refer to OpenStack logs for more details.
  - Check stack rollback (failure) Stack creation was not started. Please refer to OpenStack logs for more details.
  - Update stack actions: inplace, replace and update whole template (failure) Stack was not created properly. Please refer to OpenStack logs for more details.

BUILD=14
fuel-8.0-570.iso
CONTRAIL_PLUGIN_PACK_UB_PATH="/storage/contrail/4.0.0/contrail-install-packages_3.0.2.0-25-14.04-liberty_all.deb"

Tags:

Revision history for this message

okosse (okosse) wrote on 2016-04-29:

fuel-snapshot-2016-04-29_13-11-44.tar.xz Edit (131.7 MiB, application/octet-stream)

Oleksandr Martsyniuk (omartsyniuk) on 2016-05-05

Changed in fuel-plugin-contrail:
status:	New → In Progress

Revision history for this message

Oleksandr Martsyniuk (omartsyniuk) wrote on 2016-05-05:

Download full text (3.9 KiB)

From log analysis,
ceilometer-collector is not working properly on primary controller.
It uses wrong credentials to access rabbitmq.

<15>May 5 20:24:03 node-1 ceilometer-collector: 2016-05-05 20:24:03.069 24960 DEBUG oslo.messaging._drivers.impl_rabbit [-] Received recoverable error from kombu: on_error /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py:650
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/utils/__init__.py", line 243, in retry_over_time
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit return fun(*args, **kwargs)
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 237, in connect
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit return self.connection
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 741, in connection
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit self._connection = self._establish_connection()
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 696, in _establish_connection
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit conn = self.transport.establish_connection()
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 116, in establish_connection
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit conn = self.Connection(**opts)
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 180, in __init__
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit (10, 30), # tune
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 67, in wait
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit self.channel_id, allowed_methods)
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 240, in _wait_method
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit self.method_reader.read_method()
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 189, in read_method
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit raise m
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit IOError: Socket closed
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit
<11>May 5 20:24:03 node-1 ceilometer-collector: 2016-05-05 20:24:03.073 24960 ERROR oslo.messaging._drivers.impl_rabbit [-] A...

From log analysis,
ceilometer-collector is not working properly on primary controller.
It uses wrong credentials to access rabbitmq.

<15>May  5 20:24:03 node-1 ceilometer-collector: 2016-05-05 20:24:03.069 24960 DEBUG oslo.messaging._drivers.impl_rabbit [-] Received recoverable error from kombu: on_error /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py:650
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/utils/__init__.py", line 243, in retry_over_time
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit     return fun(*args, **kwargs)
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 237, in connect
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit     return self.connection
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 741, in connection
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit     self._connection = self._establish_connection()
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 696, in _establish_connection
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit     conn = self.transport.establish_connection()
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 116, in establish_connection
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit     conn = self.Connection(**opts)
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 180, in __init__
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit     (10, 30),  # tune
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 67, in wait
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit     self.channel_id, allowed_methods)
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 240, in _wait_method
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit     self.method_reader.read_method()
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit   File "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 189, in read_method
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit     raise m
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit IOError: Socket closed
2016-05-05 20:24:03.069 24960 ERROR oslo.messaging._drivers.impl_rabbit 
<11>May  5 20:24:03 node-1 ceilometer-collector: 2016-05-05 20:24:03.073 24960 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server 192.168.0.7:5673 closed the connection. Check login credentials: Socket closed

root@node-1:~# grep rabbit /etc/ceilometer/ceilometer.conf | grep -v -e ^#
rpc_backend= rabbit
[oslo_messaging_rabbit]
rabbit_hosts= 192.168.0.7:5673, 192.168.0.5:5673, 192.168.0.6:5673
rabbit_use_ssl= False
rabbit_userid= nova
rabbit_password= BPcxNenb77eB8uoAjYm5N3Hp
rabbit_virtual_host= /
rabbit_ha_queues= True

=ERROR REPORT==== 5-May-2016::20:25:47 ===
closing AMQP connection <0.18161.0> (192.168.0.7:57631 -> 192.168.0.7:5673):
{handshake_error,starting,0,
                 {amqp_error,access_refused,
                             "AMQPLAIN login refused: user 'guest' - invalid credentials",
                             'connection.start_ok'}}

Report a bug

This report contains Public information

Everyone can see this information.

Duplicate of bug #1583526 Remove

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

fuel-snapshot-2016-04-29_13-11-44.tar.xz Edit

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.