OpenStack Compute (nova)

Slow metadata API performance with security groups that have a lot of rules

  1. Series rocky
  2. Bug #1851430
Bug #1851430 reported by Matt Riedemann
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Doug Wiegley
Pike
Fix Released
Low
Lee Yarwood
Queens
Fix Released
Low
Matt Riedemann
Rocky
Fix Committed
Medium
Matt Riedemann
Stein
Fix Committed
Medium
Matt Riedemann
Train
Fix Committed
Medium
Matt Riedemann

Bug Description

This was reported here without a bug:

https://review.opendev.org/#/c/656084/

The EC2 metadata API response includes a 'security-groups' key that is a list of security group names attached to the instance.

The problem is for each security group attached to the instance, if the group has a lot of rules on it, it can be expensive to query (join) that information from neutron, especially if we don't care about the rules.

By default, listing security groups includes the rules in the response:

https://docs.openstack.org/api-ref/network/v2/index.html?expanded=list-security-groups-detail#list-security-groups

For the purpose of the EC2 metadata API, we should just query security groups for their names.

Matt Riedemann (mriedem)
Changed in nova:
importance: Undecided → Medium
status: New → Confirmed
Matt Riedemann (mriedem)
Changed in nova:
assignee: nobody → Doug Wiegley (dougwig)
OpenStack Infra (hudson-openstack)
Changed in nova:
assignee: Doug Wiegley (dougwig) → Matt Riedemann (mriedem)
status: Confirmed → In Progress
Matt Riedemann (mriedem)
Changed in nova:
assignee: Matt Riedemann (mriedem) → Doug Wiegley (dougwig)
summary: - slow metadata performance with security groups that have a lot of rules
+ Slow metadata API performance with security groups that have a lot of
+ rules
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.opendev.org/656084
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=eaf16fdde59a14fb38df669b21a911a0c2d2576f
Submitter: Zuul
Branch: master

commit eaf16fdde59a14fb38df669b21a911a0c2d2576f
Author: Doug Wiegley <email address hidden>
Date: Tue Nov 5 17:29:11 2019 -0500

    Improve metadata server performance with large security groups

    Don't include the rules in the SG fetch in the metadata server, since
    we don't need them there, and with >1000 rules, it starts to get
    really slow, especially in Pike and later.

    Closes-Bug: #1851430

    Co-Authored-By: Doug Wiegley <email address hidden>
    Co-Authored-By: Matt Riedemann <email address hidden>

    Change-Id: I7de14456d04370c842b4c35597dca3a628a826a2

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/694409

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/train)

Reviewed: https://review.opendev.org/694409
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=418af2d865809cfa907678f883dae07f4f31baa2
Submitter: Zuul
Branch: stable/train

commit 418af2d865809cfa907678f883dae07f4f31baa2
Author: Doug Wiegley <email address hidden>
Date: Tue Nov 5 17:29:11 2019 -0500

    Improve metadata server performance with large security groups

    Don't include the rules in the SG fetch in the metadata server, since
    we don't need them there, and with >1000 rules, it starts to get
    really slow, especially in Pike and later.

    Closes-Bug: #1851430

    Co-Authored-By: Doug Wiegley <email address hidden>
    Co-Authored-By: Matt Riedemann <email address hidden>

    Change-Id: I7de14456d04370c842b4c35597dca3a628a826a2
    (cherry picked from commit eaf16fdde59a14fb38df669b21a911a0c2d2576f)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/695925

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/697517

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/697518

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.opendev.org/697523

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/stein)

Reviewed: https://review.opendev.org/695925
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=fec95a2e4f763e15193504483383f918feb3e636
Submitter: Zuul
Branch: stable/stein

commit fec95a2e4f763e15193504483383f918feb3e636
Author: Doug Wiegley <email address hidden>
Date: Tue Nov 5 17:29:11 2019 -0500

    Improve metadata server performance with large security groups

    Don't include the rules in the SG fetch in the metadata server, since
    we don't need them there, and with >1000 rules, it starts to get
    really slow, especially in Pike and later.

    Closes-Bug: #1851430

    Co-Authored-By: Doug Wiegley <email address hidden>
    Co-Authored-By: Matt Riedemann <email address hidden>

    NOTE(mriedem): The test had to be modified in this backport because
    the mock call() behavior is different since Train and I'm not
    entirely sure why but I'm guessing it's due to change
    I4484e63c97bd1cdde3d88855eabe7545784f365e which was added in Train.
    To get around this, I just modified the test to use a simple
    predictable side_effect for _convert_to_nova_security_group_format.

    Change-Id: I7de14456d04370c842b4c35597dca3a628a826a2
    (cherry picked from commit eaf16fdde59a14fb38df669b21a911a0c2d2576f)
    (cherry picked from commit 418af2d865809cfa907678f883dae07f4f31baa2)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/rocky)

Reviewed: https://review.opendev.org/697517
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=38b2f68a17533e839819e654825613aefd4effd4
Submitter: Zuul
Branch: stable/rocky

commit 38b2f68a17533e839819e654825613aefd4effd4
Author: Doug Wiegley <email address hidden>
Date: Tue Nov 5 17:29:11 2019 -0500

    Improve metadata server performance with large security groups

    Don't include the rules in the SG fetch in the metadata server, since
    we don't need them there, and with >1000 rules, it starts to get
    really slow, especially in Pike and later.

    Closes-Bug: #1851430

    Co-Authored-By: Doug Wiegley <email address hidden>
    Co-Authored-By: Matt Riedemann <email address hidden>

    Change-Id: I7de14456d04370c842b4c35597dca3a628a826a2
    (cherry picked from commit eaf16fdde59a14fb38df669b21a911a0c2d2576f)
    (cherry picked from commit 418af2d865809cfa907678f883dae07f4f31baa2)
    (cherry picked from commit fec95a2e4f763e15193504483383f918feb3e636)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/queens)

Reviewed: https://review.opendev.org/697518
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=00d438adb325610a04af9f8f18cdb1c622df5418
Submitter: Zuul
Branch: stable/queens

commit 00d438adb325610a04af9f8f18cdb1c622df5418
Author: Doug Wiegley <email address hidden>
Date: Tue Nov 5 17:29:11 2019 -0500

    Improve metadata server performance with large security groups

    Don't include the rules in the SG fetch in the metadata server, since
    we don't need them there, and with >1000 rules, it starts to get
    really slow, especially in Pike and later.

    Closes-Bug: #1851430

    Co-Authored-By: Doug Wiegley <email address hidden>
    Co-Authored-By: Matt Riedemann <email address hidden>

    Change-Id: I7de14456d04370c842b4c35597dca3a628a826a2
    (cherry picked from commit eaf16fdde59a14fb38df669b21a911a0c2d2576f)
    (cherry picked from commit 418af2d865809cfa907678f883dae07f4f31baa2)
    (cherry picked from commit fec95a2e4f763e15193504483383f918feb3e636)
    (cherry picked from commit 38b2f68a17533e839819e654825613aefd4effd4)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 20.1.0

This issue was fixed in the openstack/nova 20.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 19.1.0

This issue was fixed in the openstack/nova 19.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 18.3.0

This issue was fixed in the openstack/nova 18.3.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/pike)

Reviewed: https://review.opendev.org/697523
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=13969dc32a4dedfe4ee0fbebec4b5491cf9f984b
Submitter: Zuul
Branch: stable/pike

commit 13969dc32a4dedfe4ee0fbebec4b5491cf9f984b
Author: Doug Wiegley <email address hidden>
Date: Tue Nov 5 17:29:11 2019 -0500

    Improve metadata server performance with large security groups

    Don't include the rules in the SG fetch in the metadata server, since
    we don't need them there, and with >1000 rules, it starts to get
    really slow, especially in Pike and later.

    Closes-Bug: #1851430

    Co-Authored-By: Doug Wiegley <email address hidden>
    Co-Authored-By: Matt Riedemann <email address hidden>

    Conflicts:
          nova/tests/unit/network/security_group/test_neutron_driver.py

    NOTE(mriedem): The conflict is due to not having change
    I31c9ea8628c6f3985f8e9118d9687bbfb8789b68 in Pike.

    Change-Id: I7de14456d04370c842b4c35597dca3a628a826a2
    (cherry picked from commit eaf16fdde59a14fb38df669b21a911a0c2d2576f)
    (cherry picked from commit 418af2d865809cfa907678f883dae07f4f31baa2)
    (cherry picked from commit fec95a2e4f763e15193504483383f918feb3e636)
    (cherry picked from commit 38b2f68a17533e839819e654825613aefd4effd4)
    (cherry picked from commit 00d438adb325610a04af9f8f18cdb1c622df5418)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova pike-eol

This issue was fixed in the openstack/nova pike-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova queens-eol

This issue was fixed in the openstack/nova queens-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.