Fuel for OpenStack

Ceilometer default metrics fail to create because of Authorization Failure

Bug #1264536 reported by Anastasiia Naboikina
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
Medium
Tatyanka
Fuel for OpenStack 4.1

Bug Description

Steps to reproduce:

1. Create new cluster with the following configuration:
    - Ubuntu Simple;
    - nova network VLAN with tagged networks;
    - Cinder LVM for storage;
    - ceilometer and savanna as additional services;
2. Add the following nodes:
    - 1 controller;
    - 1 compute;
    - 1 cinder;
3. Deploy cluster.
4. Check cluster was deployed successfully and run the OSTF tests.

Expected result:
All OSTF tests pass.

Actual result:
OSTF test "Ceilometer create, update, check, delete alarm." fails. There are no metrics created, which was verified by executing command "ceilometer meter-list" By investigating ceilometer logs the following errors were found:

Errors from ceilometer-alarm-notifier.log:

2013-12-27 12:54:30.412 10044 ERROR ceilometer.openstack.common.rpc.common [-] Failed to consume message from queue: 320: (CONNECTION_FORCED - broker forced connection closure with reason 'shutdown', (0, 0), None)
2013-12-27 12:54:30.412 10044 TRACE ceilometer.openstack.common.rpc.common Traceback (most recent call last):
2013-12-27 12:54:30.412 10044 TRACE ceilometer.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 577, in ensure
2013-12-27 12:54:30.412 10044 TRACE ceilometer.openstack.common.rpc.common return method(*args, **kwargs)
2013-12-27 12:54:30.412 10044 TRACE ceilometer.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 657, in _consume
2013-12-27 12:54:30.412 10044 TRACE ceilometer.openstack.common.rpc.common return self.connection.drain_events(timeout=timeout)
2013-12-27 12:54:30.412 10044 TRACE ceilometer.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in drain_events
2013-12-27 12:54:30.412 10044 TRACE ceilometer.openstack.common.rpc.common return self.transport.drain_events(self.connection, **kwargs)
2013-12-27 12:54:30.412 10044 TRACE ceilometer.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in drain_events
2013-12-27 12:54:30.412 10044 TRACE ceilometer.openstack.common.rpc.common return connection.drain_events(**kwargs)
2013-12-27 12:54:30.412 10044 TRACE ceilometer.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 286, in drain_events
2013-12-27 12:54:30.412 10044 TRACE ceilometer.openstack.common.rpc.common return amqp_method(channel, args)
2013-12-27 12:54:30.412 10044 TRACE ceilometer.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 491, in _close
2013-12-27 12:54:30.412 10044 TRACE ceilometer.openstack.common.rpc.common raise ConnectionError(reply_code, reply_text, (class_id, method_id))
2013-12-27 12:54:30.412 10044 TRACE ceilometer.openstack.common.rpc.common ConnectionError: 320: (CONNECTION_FORCED - broker forced connection closure with reason 'shutdown', (0, 0), None)
2013-12-27 12:54:30.412 10044 TRACE ceilometer.openstack.common.rpc.common

Errors from ceilometer-alarm-evaluator.log:
2013-12-27 12:56:35.616 10012 ERROR ceilometer.alarm.service [-] alarm evaluation cycle failed
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service Traceback (most recent call last):
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometer/alarm/service.py", line 95, in _evaluate_assigned_alarms
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service alarms = self._assigned_alarms()
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometer/alarm/service.py", line 137, in _assigned_alarms
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service return self._client.alarms.list(q=[{'field': 'enabled',
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometer/alarm/service.py", line 90, in _client
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service self.api_client = ceiloclient.get_client(2, **creds)
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometerclient/client.py", line 77, in get_client
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service _ksclient = _get_ksclient(**ks_kwargs)
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometerclient/client.py", line 33, in _get_ksclient
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service insecure=kwargs.get('insecure'))
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/client.py", line 139, in __init__
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service self.authenticate()
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/keystoneclient/httpclient.py", line 467, in authenticate
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service resp, body = self.get_raw_token_from_identity_service(**kwargs)
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/client.py", line 166, in get_raw_token_from_identity_service
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service "%s" % e)
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service AuthorizationFailure: Authorization Failed: An unexpected error prevented the server from fulfilling your request. (ProgrammingError) (1146, "Table 'keystone.domain' doesn't exist") 'SELECT domain.id AS domain_id, domain.name AS domain_name, domain.enabled AS domain_enabled, domain.extra AS domain_extra \nFROM domain \nWHERE domain.id = %s' ('default',) (HTTP 500)
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service
2013-12-27 12:57:35.618 10012 ERROR ceilometer.alarm.service [-] alarm evaluation cycle failed
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service Traceback (most recent call last):
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometer/alarm/service.py", line 95, in _evaluate_assigned_alarms
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service alarms = self._assigned_alarms()
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometer/alarm/service.py", line 137, in _assigned_alarms
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service return self._client.alarms.list(q=[{'field': 'enabled',
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometer/alarm/service.py", line 90, in _client
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service self.api_client = ceiloclient.get_client(2, **creds)
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometerclient/client.py", line 77, in get_client
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service _ksclient = _get_ksclient(**ks_kwargs)
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometerclient/client.py", line 33, in _get_ksclient
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service insecure=kwargs.get('insecure'))
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/client.py", line 139, in __init__
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service self.authenticate()
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/keystoneclient/httpclient.py", line 467, in authenticate
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service resp, body = self.get_raw_token_from_identity_service(**kwargs)
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/client.py", line 160, in get_raw_token_from_identity_service
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service token=token)
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/client.py", line 191, in _base_authN
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service resp, body = self.request(url, 'POST', body=params, headers=headers)
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/keystoneclient/httpclient.py", line 609, in request
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service **request_kwargs)
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/keystoneclient/httpclient.py", line 123, in request
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service raise exceptions.from_response(resp, method, url)
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service Unauthorized: The request you have made requires authentication. (HTTP 401)
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service

Tags: ceilometer
Revision history for this message
Anastasiia Naboikina (anaboikina) wrote :
Tatyanka (tatyana-leontovich)
Changed in fuel:
importance: Critical → Medium
milestone: none → 4.1
Revision history for this message
Anastasia Palkina (apalkina) wrote :
Download full text (4.0 KiB)

Reproduced on
"build_id": "2014-01-27_01-17-41",
"ostf_sha": "338ddf840c229918d1df8c6597588b853d02de4c",
"build_number": "63",
"nailgun_sha": "a77e67aa73bf7e58d1078e467d7644855549879a",
"fuelmain_sha": "93cb99fa035cea35459ef9062a1abce66a60f79e",
"astute_sha": "f967bf7d643c35e9c3bf8d8ad6cd6a13caa6e4ac",
"release": "4.1",
"fuellib_sha": "b118aefcc71b1a2719e3af9a91d659c1956ee610"

1. Create new environment (Ubuntu, simple mode)
2. Chose VLAN segmentation
3. Choose installing Ceph for images and Ceilometer
4. Add controller, compute, cinder and 2 ceph nodes
5. Untag Management network and move it to other interface
6. Start deployment. It was successful
7. Start OSTF tests
8. Ceilometer tests has failed on step with creating metrics

Errors from ceilometer-alarm-notifier.log:

2014-01-27 08:34:11.134 10760 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on localhost:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 9 seconds.
2014-01-27 08:34:20.148 10760 ERROR ceilometer.openstack.common.rpc.common [-] AMQP server on localhost:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 11 seconds.
2014-01-27 08:37:26.674 10760 ERROR ceilometer.openstack.common.rpc.common [-] Failed to consume message from queue: 320: (CONNECTION_FORCED - broker forced connection closure with reason 'shutdown', (0, 0), None)
2014-01-27 08:37:26.674 10760 TRACE ceilometer.openstack.common.rpc.common Traceback (most recent call last):

Errors from ceilometer-alarm-evaluator.log:

2014-01-27 08:34:54.843 10728 ERROR ceilometer.alarm.service [-] alarm evaluation cycle failed
2014-01-27 08:34:54.843 10728 TRACE ceilometer.alarm.service Traceback (most recent call last):
2014-01-27 08:34:54.843 10728 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometer/alarm/service.py", line 95, in _evaluate_assigned_alarms
2014-01-27 08:34:54.843 10728 TRACE ceilometer.alarm.service alarms = self._assigned_alarms()
2014-01-27 08:34:54.843 10728 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometer/alarm/service.py", line 137, in _assigned_alarms
2014-01-27 08:34:54.843 10728 TRACE ceilometer.alarm.service return self._client.alarms.list(q=[{'field': 'enabled',
2014-01-27 08:34:54.843 10728 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometer/alarm/service.py", line 90, in _client
2014-01-27 08:34:54.843 10728 TRACE ceilometer.alarm.service self.api_client = ceiloclient.get_client(2, **creds)
2014-01-27 08:34:54.843 10728 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometerclient/client.py", line 77, in get_client
2014-01-27 08:34:54.843 10728 TRACE ceilometer.alarm.service _ksclient = _get_ksclient(**ks_kwargs)
2014-01-27 08:34:54.843 10728 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometerclient/client.py", line 33, in _get_ksclient
2014-01-27 08:34:54.843 10728 TRACE ceilometer.alarm.service insecure=kwargs.get('insecure'))
2014-01-27 08:34:54.843 10728 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/client.py", line 139, in __init__
2...

Read more...

Changed in fuel:
status: New → Confirmed
Revision history for this message
Anastasia Palkina (apalkina) wrote :
Tatyanka (tatyana-leontovich)
Changed in fuel:
assignee: nobody → Tatyana (tatyana-leontovich)
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

verified on 128 iso with qemu - not reproduce

Changed in fuel:
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.