OpenStack-Ansible

Neutron Metadata Agents backlog value too low and differs from Neutron default

  1. Series mitaka
  2. Bug #1591297
Bug #1591297 reported by Ian Cordasco
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack-Ansible
Fix Released
Undecided
Ian Cordasco
Kilo
Fix Committed
Undecided
Ian Cordasco
Liberty
Fix Committed
Undecided
Ian Cordasco
Mitaka
Fix Committed
Undecided
Ian Cordasco
Trunk
Fix Released
Undecided
Ian Cordasco

Bug Description

Neutron Metadata Agents works roughly like this: The service runs on a Unix Domain Socket. It runs as many threads as cpu_count/2. Each thread has a maximum number of connections that it can accept. The service attempts to talk to Neutron API via RPC but (only on) Kilo and Liberty will fall back to HTTP if necessary.

By default (since Kilo) Neutron has that number set to 4096. Since kilo in OpenStack Ansible the number has been 128.

At high throughput on a large OpenStack Ansible cloud, keeping this value arbitrarily low (1/32nd of it's default value) causes RPC connections from Neutron Metadata Agent to fail and fallback to HTTP. Setting this value to even half of the upstream default (on only 2 of the 5 metadata agents containers) causes this fallback to stop happening.

OpenStack Ansible Master: https://github.com/openstack/openstack-ansible-os_neutron/blob/9898d7188341fe4a340a9f4f299620f205f026cb/defaults/main.yml#L303
Neutron Master: https://github.com/openstack/neutron/blob/dd4f1253c951d78a5b497680dfb31317ba469a58/neutron/agent/metadata/config.py#L112

OpenStack Ansible Mitaka: https://github.com/openstack/openstack-ansible-os_neutron/blob/dc7fd56d233e5ece1940b9344b3fee059f6597b9/defaults/main.yml#L287
Neutron Mitaka: https://github.com/openstack/neutron/blob/13438bad4136c6f5ff4f37c43cb151ec77a8ffd3/neutron/agent/metadata/config.py#L113

OpenStack Ansible Liberty: https://github.com/openstack/openstack-ansible/blob/ac596657132e4fbb05ac581e802e55c68a58da57/playbooks/roles/os_neutron/defaults/main.yml#L159
Neutron Liberty: https://github.com/openstack/neutron/blob/5067c7b49a5701d01c7277699ca5a93885913065/neutron/agent/metadata/config.py#L133

OpenStack Ansible Kilo: https://github.com/openstack/openstack-ansible/blob/98f2b2fd255952f4438f58de03a2f225d0664f2c/playbooks/roles/os_neutron/defaults/main.yml#L233
Neutron Kilo: https://github.com/openstack/neutron/blob/363eeb06104662ee38aeed04af043899379f6ab8/neutron/agent/metadata/config.py#L129

See original description
Ian Cordasco (icordasc)
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-ansible-os_neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/328441

Changed in openstack-ansible:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-ansible-os_neutron (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/328442

Revision history for this message
Bjoern (bjoern-t) wrote :

Liberty please too

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to openstack-ansible-os_neutron (master)

Reviewed: https://review.openstack.org/328441
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_neutron/commit/?id=357e5432fe70f237e6ef7d84b7224634fac4fd25
Submitter: Jenkins
Branch: master

commit 357e5432fe70f237e6ef7d84b7224634fac4fd25
Author: Ian Cordasco <email address hidden>
Date: Fri Jun 10 13:07:53 2016 -0500

    Use default metadata_backlog value from Neutron

    Change-Id: If8e24446e2b6df3efc330d197265871e8e98253c
    Closes-bug: 1591297

Changed in openstack-ansible:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-ansible (liberty)

Fix proposed to branch: liberty
Review: https://review.openstack.org/328502

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-ansible (kilo)

Fix proposed to branch: kilo
Review: https://review.openstack.org/328503

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to openstack-ansible-os_neutron (stable/mitaka)

Reviewed: https://review.openstack.org/328442
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_neutron/commit/?id=58abc3243274680c572b31f5b662f9caf56efd4d
Submitter: Jenkins
Branch: stable/mitaka

commit 58abc3243274680c572b31f5b662f9caf56efd4d
Author: Ian Cordasco <email address hidden>
Date: Fri Jun 10 13:07:53 2016 -0500

    Use default metadata_backlog value from Neutron

    Change-Id: If8e24446e2b6df3efc330d197265871e8e98253c
    Closes-bug: 1591297
    (cherry picked from commit 357e5432fe70f237e6ef7d84b7224634fac4fd25)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to openstack-ansible (kilo)

Reviewed: https://review.openstack.org/328503
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=23cf197223dc5b60bc9223b8180db46db43e2897
Submitter: Jenkins
Branch: kilo

commit 23cf197223dc5b60bc9223b8180db46db43e2897
Author: Ian Cordasco <email address hidden>
Date: Fri Jun 10 15:05:08 2016 -0500

    Use default metadata_backlog value from Neutron

    (cherry picked from commit f7ff05e01396a594c735e1fed82669a3e5db7b45)
    Change-Id: If8e24446e2b6df3efc330d197265871e8e98253c
    Closes-bug: 1591297

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to openstack-ansible (liberty)

Reviewed: https://review.openstack.org/328502
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=f7ff05e01396a594c735e1fed82669a3e5db7b45
Submitter: Jenkins
Branch: liberty

commit f7ff05e01396a594c735e1fed82669a3e5db7b45
Author: Ian Cordasco <email address hidden>
Date: Fri Jun 10 15:05:08 2016 -0500

    Use default metadata_backlog value from Neutron

    (cherry picked from the os-keystone role 58abc3243274680c572b31f5b662f9caf56efd4d)
    Change-Id: If8e24446e2b6df3efc330d197265871e8e98253c
    Closes-bug: 1591297

Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote : Fix included in openstack/openstack-ansible 12.0.15

This issue was fixed in the openstack/openstack-ansible 12.0.15 release.

Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/openstack-ansible-os_neutron 13.1.4

This issue was fixed in the openstack/openstack-ansible-os_neutron 13.1.4 release.

Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/openstack-ansible 11.2.17

This issue was fixed in the openstack/openstack-ansible 11.2.17 release.

Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/openstack-ansible-os_neutron 14.0.0.0b2

This issue was fixed in the openstack/openstack-ansible-os_neutron 14.0.0.0b2 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.