Reduce rabbit's stats collector mem usage for a high load systems

Bug #1510835 reported by Bogdan Dobrelya
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Bogdan Dobrelya
6.1.x
Won't Fix
High
Denis Puchkin
7.0.x
Fix Released
High
Denis Puchkin

Bug Description

The stats collector falls behind under high load on the messaging system.
The suggested optimization is to increase the stats polling and disable rates processing in management UI.
rabbit.collect_statistics_interval=30000
rabbitmq_management.rates_mode=none

See also
https://youtu.be/UAUbkqMKd8s?t=1380
https://www.rabbitmq.com/management.html

tags: added: rabbitmq scale
Changed in fuel:
importance: Undecided → High
assignee: nobody → Bogdan Dobrelya (bogdando)
milestone: none → 8.0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/239896

Changed in fuel:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/239896
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=f0ce9c4e4ae34859895ec444abb552a504633fe5
Submitter: Jenkins
Branch: master

commit f0ce9c4e4ae34859895ec444abb552a504633fe5
Author: Bogdan Dobrelya <email address hidden>
Date: Wed Oct 28 11:25:41 2015 +0100

    Adjust rmq stats collector settings

    The stats collector falls behind under
    high load on the messaging system.
    The suggested optimization is to increase
    the stats polling and disable rates
    processing in management UI:
    rabbit.collect_statistics_interval=30000
    rabbitmq_management.rates_mode=none.
    The new hiera parameter rabbit_config_management_variables
    may be used to override the rates_mode to other values.

    Related upstream PR
    https://github.com/puppetlabs/puppetlabs-rabbitmq/pull/407

    DocImpact
    Closes-bug: #1510835

    Change-Id: I9a3179a4f002064c9b6d2621a49bb0aa9aa707c7
    Signed-off-by: Bogdan Dobrelya <email address hidden>

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote :

Yes, let's please backport this.

Dmitry Pyzhov (dpyzhov)
tags: added: area-library
Revision history for this message
Alexey Lebedeff (alebedev-a) wrote :

Do we need to enable the management plugin at all? I've heard that it is used only for monitoring through its aliveness check. Could we use some custom monitoring script instead? This script should do the same checks that ocf script does and also should perform aliveness check by using AMQP client to publish/consume test message. And given this https://github.com/rabbitmq/rabbitmq-server/issues/23 , it will even do better monitoring.

So by disabling management plugin we can get rid of this resource waste completely. Especially given that no newer versions of rabbitmq with statsdb improvements will be used in 6.1.x and 7.0.x.

Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

We use the plugin for dump/restore definitions

Revision history for this message
Egor Kotko (ykotko) wrote :

ISO #506

Changed in fuel:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/7.0)

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/286534

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/6.1)

Fix proposed to branch: stable/6.1
Review: https://review.openstack.org/286543

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (stable/6.1)

Change abandoned by Fuel DevOps Robot (<email address hidden>) on branch: stable/6.1
Review: https://review.openstack.org/286543
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/7.0)

Reviewed: https://review.openstack.org/286534
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=82093037d2971ace6eae8ba4af46271002ba4b48
Submitter: Jenkins
Branch: stable/7.0

commit 82093037d2971ace6eae8ba4af46271002ba4b48
Author: Bogdan Dobrelya <email address hidden>
Date: Wed Oct 28 11:25:41 2015 +0100

    Adjust rmq stats collector settings

    The stats collector falls behind under
    high load on the messaging system.
    The suggested optimization is to increase
    the stats polling and disable rates
    processing in management UI:
    rabbit.collect_statistics_interval=30000
    rabbitmq_management.rates_mode=none.
    The new hiera parameter rabbit_config_management_variables
    may be used to override the rates_mode to other values.

    Related upstream PR
    https://github.com/puppetlabs/puppetlabs-rabbitmq/pull/407

    Conflicts:
     tests/noop/spec/hosts/rabbitmq/rabbitmq_spec.rb
    DocImpact

    Closes-bug: #1510835

    Change-Id: I9a3179a4f002064c9b6d2621a49bb0aa9aa707c7
    Signed-off-by: Bogdan Dobrelya <email address hidden>

Revision history for this message
Sergii Rizvan (srizvan) wrote :

Closed for 6.1 as Won't fix because we don't provide support for this version anymore.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.