Fuel for OpenStack

Shuffle amqp nodes in Openstack configs

Bug #1320184 reported by Bartosz Kupidura
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Bartosz Kupidura
Fuel for OpenStack 5.1
4.1.x
Fix Released
High
Vladimir Kuklin
Fuel for OpenStack 4.1.1
5.0.x
Won't Fix
High
Bartosz Kupidura
Fuel for OpenStack 5.0.1

Bug Description

Now each Openstack node (compute, cinder, ...) connect to #1 controller,
after failure it reconnects to #2, after that to #3 controller.

In this case, ALL AMQP traffic is served by #1.

We can shuffle 'rabbit_hosts' on each 'not controller' node.

Tags: ha
Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

https://review.openstack.org/#/c/93883/

Changed in fuel:
milestone: none → 5.0
assignee: nobody → Bartosz Kupidura (zynzel)
importance: Undecided → High
status: New → In Progress
tags: added: ha
Sergii Golovatiuk (sgolovatiuk)
Changed in fuel:
milestone: 5.0 → 5.1
milestone: 5.1 → 5.0
milestone: 5.0 → 5.1
milestone: 5.1 → 5.0
Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

Basically, the idea of the patch was to provide
1) an idempotent and deterministic way to shuffle rabbit nodes for non controller nodes in order to increase RabbitMQ cluster performance (multiple rabbit nodes should serve the network connections instead if the single one)
 and
2) exclude the node IP address for the given controllers from their rabbit hosts (controllers use localhost as well).
Note: Deterministic way means consequent 'puppet apply' runs must ensure the configuration of rabbit_hosts left intact once it has been shuffled at the deployment stage.

Revision history for this message
Vladimir Kuklin (vkuklin) wrote :

patch is not yet ready and severity of this bug is not so high. postponing.

Revision history for this message
Mike Scherbakov (mihgen) wrote :

Postponing to 5.1, as:
a) it is not critical for 5.0 HA story, and adds improvements
b) actually it adds more non-determinism, and should be analyzed from this point of view
c) additional research why ubuntu Fuel CI puts -1 is needed before merge
d) answer to comment from Ihar Hrachyshka is needed

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/94632
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=71b112d8442c5a60b9ebe95b99b7b2bf9eb34b2a
Submitter: Jenkins
Branch: master

commit 71b112d8442c5a60b9ebe95b99b7b2bf9eb34b2a
Author: Sergii Golovatiuk <email address hidden>
Date: Wed May 21 14:34:49 2014 +0000

    Replace concat.rb with latest stdlib equivalent

    Sync concat puppet function to fix some bugs

    Related-Bug: #1320184
    Related-Bug: #1321284

    Change-Id: I49a614122469b37b1c4eb2fbc29b0877a4427fbd

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/93883
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=695aa4118afcd514d1b6abffe2e83679fd6a1cac
Submitter: Jenkins
Branch: master

commit 695aa4118afcd514d1b6abffe2e83679fd6a1cac
Author: Bartosz Kupidura <email address hidden>
Date: Fri May 16 11:50:22 2014 +0200

    Shuffle AMQP nodes for all OpenStack services.

    - Assign 127.0.0.1 as first server in HA
    - Rotate servers using RR to have better balance
    - delete.rb from upstream to fix bug with deletion

    Related-Bug: 1320184
    Change-Id: I3762dedfa06d593f4347e246b4302852261f82cb

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-library (stable/4.1)

Related fix proposed to branch: stable/4.1
Review: https://review.openstack.org/96858

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-library (stable/4.1)

Reviewed: https://review.openstack.org/96858
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=18d81e17cfd13c135bb433a11dcc7d2c69f2c84d
Submitter: Jenkins
Branch: stable/4.1

commit 18d81e17cfd13c135bb433a11dcc7d2c69f2c84d
Author: Sergii Golovatiuk <email address hidden>
Date: Wed May 21 14:34:49 2014 +0000

    Replace concat.rb with latest stdlib equivalent

    Sync concat puppet function to fix some bugs

    Related-Bug: #1320184
    Related-Bug: #1321284

    Change-Id: I49a614122469b37b1c4eb2fbc29b0877a4427fbd

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-library (stable/4.1)

Related fix proposed to branch: stable/4.1
Review: https://review.openstack.org/97222

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-library (stable/4.1)

Reviewed: https://review.openstack.org/97222
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=b0e1792e532a62f98620e8ade84f5b18c206bff1
Submitter: Jenkins
Branch: stable/4.1

commit b0e1792e532a62f98620e8ade84f5b18c206bff1
Author: Bartosz Kupidura <email address hidden>
Date: Fri May 16 11:50:22 2014 +0200

    Shuffle AMQP nodes for all OpenStack services.

    - Assign 127.0.0.1 as first server in HA
    - Rotate servers using RR to have better balance
    - delete.rb from upstream to fix bug with deletion

    Related-Bug: 1320184
    Change-Id: I3762dedfa06d593f4347e246b4302852261f82cb
    (cherry picked from commit 695aa4118afcd514d1b6abffe2e83679fd6a1cac)

Revision history for this message
Meg McRoberts (dreidellhasa) wrote :

Documented as fixed in 4.1.1

Revision history for this message
Dima Shulyak (dshulyak) wrote :

fyi https://github.com/openstack/oslo.messaging/commit/53b9d741a8f5a2fa6cbaea9e23a3679a6af235a4

there is qpid and rabbit implementation for ha queues shuffle, it is in master, but not released yet

Vladimir Kuklin (vkuklin)
no longer affects: fuel/5.1.x
Nastya Urlapova (aurlapova)
Changed in fuel:
milestone: 5.0.1 → 5.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.