Mirantis OpenStack

Ceilometer statistics request logs Gateway Timeout (504) error and alarm evaluation cycle failed

Bug #1594947 reported by Rudy McComb
This bug report is a duplicate of:  Bug #1597679: ceilometer statistics timeout. Edit Remove
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Confirmed
Medium
MOS Ceilometer
Mirantis OpenStack 9.2
7.0.x
Confirmed
Medium
Sergii Rizvan
Mirantis OpenStack 7.0-updates
9.x
Confirmed
Medium
MOS Ceilometer
Mirantis OpenStack 9.2

Bug Description

While trying to get stats from Ceilometer the following request times out with 504 error:

root@(MVP-1 controller) ed-sl-d12:[~]:$ ceilometer statistics -m cpu_util -q 'timestamp>2016-01-01T00:00:01;timestamp<=2016-06-01T10:00:01' --period 60
Gateway Timeout (HTTP 504)

#ceilometer-alarm-evaluator.log
2016-06-16 16:06:52.458 45917 ERROR ceilometer.alarm.service [-] alarm evaluation cycle failed
2016-06-16 16:06:52.458 45917 TRACE ceilometer.alarm.service Traceback (most recent call last):
2016-06-16 16:06:52.458 45917 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometer/alarm/service.py", line 93, in _evaluate_assigned_alarms
2016-06-16 16:06:52.458 45917 TRACE ceilometer.alarm.service alarms = self._assigned_alarms()
2016-06-16 16:06:52.458 45917 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometer/alarm/service.py", line 174, in _assigned_alarms
2016-06-16 16:06:52.458 45917 TRACE ceilometer.alarm.service 'value': True}])
2016-06-16 16:06:52.458 45917 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometerclient/v2/alarms.py", line 77, in list
2016-06-16 16:06:52.458 45917 TRACE ceilometer.alarm.service return self._list(options.build_url(self._path(), q))
2016-06-16 16:06:52.458 45917 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometerclient/common/base.py", line 68, in _list
2016-06-16 16:06:52.458 45917 TRACE ceilometer.alarm.service resp = self.api.get(url)
2016-06-16 16:06:52.458 45917 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometerclient/openstack/common/apiclient/client.py", line 355, in get
2016-06-16 16:06:52.458 45917 TRACE ceilometer.alarm.service return self.client_request("GET", url, **kwargs)
2016-06-16 16:06:52.458 45917 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometerclient/openstack/common/apiclient/client.py", line 345, in client_request
2016-06-16 16:06:52.458 45917 TRACE ceilometer.alarm.service self, method, url, **kwargs)
2016-06-16 16:06:52.458 45917 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometerclient/openstack/common/apiclient/client.py", line 261, in client_request
2016-06-16 16:06:52.458 45917 TRACE ceilometer.alarm.service method, self.concat_url(endpoint, url), **kwargs)
2016-06-16 16:06:52.458 45917 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometerclient/openstack/common/apiclient/client.py", line 201, in request
2016-06-16 16:06:52.458 45917 TRACE ceilometer.alarm.service raise exceptions.from_response(resp, method, url)
2016-06-16 16:06:52.458 45917 TRACE ceilometer.alarm.service GatewayTimeout: Gateway Timeout (HTTP 504)
2016-06-16 16:06:52.458 45917 TRACE ceilometer.alarm.service
2016-06-16 16:06:52.460 45917 WARNING ceilometer.openstack.common.loopingcall [-] task u'<bound method SingletonAlarmService._evaluate_assigned_alarms of <ceilometer.alarm.service.SingletonAlarmService object at 0x7f020205f9d0>>' run outlasted interval by 0.01 sec
2016-06-16 16:06:52.503 45917 INFO ceilometer.alarm.service [-] initiating evaluation cycle on 0 alarms

Expected output
root@node-1:~# ceilometer statistics -m cpu_util -q 'timestamp>2016-01-01T00:00:01;timestamp<=2016-07-01T10:00:01' --period 60
+--------+---------------------+---------------------+----------------+----------------+----------------+----------------+-------+----------+---------------------+---------------------+
| Period | Period Start | Period End | Max | Min | Avg | Sum | Count | Duration | Duration Start | Duration End |
+--------+---------------------+---------------------+----------------+----------------+----------------+----------------+-------+----------+---------------------+---------------------+
| 60 | 2016-06-24T16:29:01 | 2016-06-24T16:30:01 | 92.1333333333 | 92.1333333333 | 92.1333333333 | 92.1333333333 | 1 | 0.0 | 2016-06-24T16:29:05 | 2016-06-24T16:29:05 |
| 60 | 2016-06-24T16:30:01 | 2016-06-24T16:31:01 | 1.01666666667 | 1.01666666667 | 1.01666666667 | 1.01666666667 | 1 | 0.0 | 2016-06-24T16:30:05 | 2016-06-24T16:30:05 |
| 60 | 2016-06-24T16:31:01 | 2016-06-24T16:32:01 | 0.533333333333 | 0.533333333333 | 0.533333333333 | 0.533333333333 | 1 | 0.0 | 2016-06-24T16:31:05 | 2016-06-24T16:31:05 |
| 60 | 2016-06-24T16:32:01 | 2016-06-24T16:33:01 | 0.583333333333 | 0.583333333333 | 0.583333333333 | 0.583333333333 | 1 | 0.0 | 2016-06-24T16:32:05 | 2016-06-24T16:32:05 |
| 60 | 2016-06-24T16:33:01 | 2016-06-24T16:34:01 | 0.533333333333 | 0.533333333333 | 0.533333333333 | 0.533333333333 | 1 | 0.0 | 2016-06-24T16:33:05 | 2016-06-24T16:33:05 |
+--------+---------------------+---------------------+----------------+----------------+----------------+----------------+-------+----------+---------------------+---------------------+

MOS 7.0
Ubuntu 14.04
python-ceilometerclient 1.0.14

See original description
Revision history for this message
Bug Checker Bot (bug-checker) wrote : Autochecker

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

version

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

tags: added: need-info
Revision history for this message
Igor Degtiarov (idegtiarov) wrote :

This not seems like a bug.

Long period statistics could be a very long operation in database and depends on amount of data that was stored during required period of time. MongoDB will operate with timestamp index because data should be sorted in write order and this factor make MongoDB choose timestamp index as priority one. That means that all other filter parameter will be used in search request while scanning all data stored between start and end timestamps in statistics request.
So if we have a lot of data in database such find process will be rather long and as result timeout error.

Sergey Kolekonov (skolekonov)
Changed in mos:
assignee: nobody → MOS Ceilometer (mos-ceilometer)
Revision history for this message
Dina Belova (dbelova) wrote :

Based on Igor's comment above - Rudy, may you please provide us with information about what amount of data in fact is extracted from the DB and using what period of time?

tags: added: area-ceilometer
Changed in mos:
status: New → Incomplete
assignee: MOS Ceilometer (mos-ceilometer) → Rudy McComb (rmccomb)
Revision history for this message
Rudy McComb (rmccomb) wrote :
Download full text (4.0 KiB)

Dina currently no information is being extracted due to the timeout. Even a 1 day query results in a timeout.

These are dedicated Mongo nodes with the following specs

Servers: 3 nodes hp dl360 gen9 sff8; 2x e5-2670; 2x 16384mb
MongoDB space on all 3 nodes:
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/mongo-mongodb 4.2T 104G 3.9T 3% /var/lib/mongo

root@(MVP-1 controller) ed-sl-d12:[~]:$ date && ceilometer statistics -m memory -q 'timestamp>2016-06-15T00:00:01;timestamp<=2016-06-16T10:00:01' --period 60
Tue Jun 21 09:51:09 UTC 2016
Gateway Timeout (HTTP 504)
root@(MVP-1 controller) ed-sl-d12:[~]:$ date && ceilometer statistics -m memory -q 'timestamp>2016-06-15T00:00:01;timestamp<=2016-06-16T10:00:01' --period 60
Tue Jun 21 09:52:18 UTC 2016
Gateway Timeout (HTTP 504)
root@(MVP-1 controller) ed-sl-d12:[~]:$ date && ceilometer statistics -m memory -q 'timestamp>2016-06-15T00:00:01;timestamp<=2016-06-16T10:00:01' --period 60
Tue Jun 21 09:53:24 UTC 2016
Gateway Timeout (HTTP 504)
root@(MVP-1 controller) ed-sl-d12:[~]:$ date && ceilometer statistics -m memory -q 'timestamp>2016-06-15T00:00:01;timestamp<=2016-06-16T10:00:01' --period 60
Tue Jun 21 09:55:41 UTC 2016
+--------+--------------+------------+-----+-----+-----+-----+-------+----------+----------------+--------------+
| Period | Period Start | Period End | Max | Min | Avg | Sum | Count | Duration | Duration Start | Duration End |
 +--------+--------------+------------+-----+-----+-----+-----+-------+----------+----------------+--------------+
+--------+--------------+------------+-----+-----+-----+-----+-------+----------+----------------+--------------+

RabbitMQ:
root@(MVP-1 controller) ed-sl-d12:[~]:$ rabbitmqctl list_queues name consumers messages | grep -vw "0$"
Listing queues ...
cinder-scheduler_fanout_1e18a07eb1fd4008aceb06d75aedf940 0 93762
cinder-scheduler_fanout_4f3dee682dc74fefbd3ae968a2192b1e 0 93761
cinder-scheduler_fanout_f4397e9726c843099be32b79e179e57f 0 93761
cinder-volume_fanout_3c4adeaeb09a4e0d8fcdd5715daaffe6 0 3
cinder-volume_fanout_59d362662cae4d26844c8c4c8b575b50 0 3
cinder-volume_fanout_aeb015f5695240a4b62122c01f602871 0 3
q-agent-notifier-port-delete_fanout_1d78a9cf2cca4beaac5470c7c3f91c71 0 6
q-agent-notifier-port-delete_fanout_46ddbe382b9e4909a409193f879f88d3 0 6
q-agent-notifier-port-delete_fanout_73e3f2b821be46d19498d7e880b552fd 0 6
q-agent-notifier-port-delete_fanout_77a2537764f042a3a5c6a5a191c27287 0 6
q-agent-notifier-port-delete_fanout_85fbcd1fc5cb4f54820eb7a790bc490e 0 6
q-agent-notifier-port-delete_fanout_fd2ae797cb6d42abbc452c36ecf95273 0 6
q-agent-notifier-port-delete_fanout_fde2aefcef714168b520b02ed001dd76 0 6
q-agent-notifier-port-update_fanout_3a40ea2c00b6416cac0bec49d2f296a2 0 27
q-agent-notifier-port-update_fanout_59ab0b5c25034b44a96da18d9dc4600a 0 27
q-agent-notifier-port-update_fanout_647b18851a4040f69f07ae2711bb4e53 0 27
q-agent-notifier-port-update_fanout_b770d0c5aa6d4a20bb17df1fb240f2e5 0 27
q-agent-notifier-port-update_fanout_c8c712547e574b27b367677d783df8f9 0 27
q-agent-notifier-port-update_fanout_e...

Read more...

Rudy McComb (rmccomb)
description: updated
Vitaly Sedelnik (vsedelnik)
Changed in mos:
status: Incomplete → Confirmed
assignee: Rudy McComb (rmccomb) → MOS Ceilometer (mos-ceilometer)
importance: Undecided → Medium
milestone: none → 9.1
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/ceilometer (9.0/mitaka)

Fix proposed to branch: 9.0/mitaka
Change author: Igor Degtiarov <email address hidden>
Review: https://review.fuel-infra.org/23859

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/ceilometer (9.0/mitaka)

Reviewed: https://review.fuel-infra.org/23859
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: 33fd3ac67dba9decdc89ca094a376cfefaf07eb4
Author: Igor Degtiarov <email address hidden>
Date: Wed Aug 3 09:52:26 2016

[MongoDB] add compound index for statistic request

MongoDB cannot use index intersection if at least one of indexes use with
boundary query. It ceilometer statistic we always have query with interval
for timestamp field. That is why only compound index could improve
performance of search stage in aggregation pipeline.

Change-Id: Id452f22e9b9ea1198b3ccee0ba5cd292b45bdc03
Closes-Bug: #1594947
Closes-Bug: #1597679

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/ceilometer (mcp/newton)

Fix proposed to branch: mcp/newton
Change author: Igor Degtiarov <email address hidden>
Review: https://review.fuel-infra.org/33376

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/ceilometer (11.0/ocata)

Fix proposed to branch: 11.0/ocata
Change author: Igor Degtiarov <email address hidden>
Review: https://review.fuel-infra.org/33755

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/ceilometer (mcp/ocata)

Fix proposed to branch: mcp/ocata
Change author: Igor Degtiarov <email address hidden>
Review: https://review.fuel-infra.org/34480

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on openstack/ceilometer (mcp/newton)

Change abandoned by Ilya Tyaptin <email address hidden> on branch: mcp/newton
Review: https://review.fuel-infra.org/33376

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on openstack/ceilometer (mcp/ocata)

Change abandoned by Ilya Tyaptin <email address hidden> on branch: mcp/ocata
Review: https://review.fuel-infra.org/34480

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on openstack/ceilometer (11.0/ocata)

Change abandoned by Roman Podoliaka <email address hidden> on branch: 11.0/ocata
Review: https://review.fuel-infra.org/33755
Reason: we don't use 11.0/ocata anymore - mcp/ocata is the correct branch name

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.