[ovn] overlapping security group rules break neutron-ovn-db-sync-util

Bug #1961112 reported by Daniel Speichert
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu Cloud Archive
Fix Released
Undecided
Unassigned
Ussuri
Triaged
High
Unassigned
neutron
Fix Released
Critical
Jake Yip
neutron (Ubuntu)
Fix Released
Undecided
Unassigned
Focal
Triaged
High
Unassigned

Bug Description

Neutron (Xena) is happy to accept equivalent rules with overlapping remote CIDR prefix as long as the notation is different, e.g. 10.0.0.0/8 and 10.0.0.1/8.

However, OVN is smarter, normalizes the prefix and figures out that they both are 10.0.0.0/8.

This does not have any fatal effects in a running OVN deployment (creating and using such rules does not even trigger a warning) but upon running neutron-ovn-db-sync-util, it crashes and won't perform a sync. This is a blocker for upgrades (and other scenarios).

Security group's rules:

$ openstack security group rule list overlap-sgr
+--------------------------------------+-------------+-----------+------------+------------+-----------+-----------------------+----------------------+
| ID | IP Protocol | Ethertype | IP Range | Port Range | Direction | Remote Security Group | Remote Address Group |
+--------------------------------------+-------------+-----------+------------+------------+-----------+-----------------------+----------------------+
| 3c41fa80-1d23-49c9-9ec1-adf581e07e24 | tcp | IPv4 | 10.0.0.1/8 | | ingress | None | None |
| 639d263e-6873-47cb-b2c4-17fc824252db | None | IPv4 | 0.0.0.0/0 | | egress | None | None |
| 96e99039-cbc0-48fe-98fe-ef28d41b9d9b | tcp | IPv4 | 10.0.0.0/8 | | ingress | None | None |
| bf9160a3-fc9b-467e-85d5-c889811fd6ca | None | IPv6 | ::/0 | | egress | None | None |
+--------------------------------------+-------------+-----------+------------+------------+-----------+-----------------------+----------------------+

Log excerpt:
16/Feb/2022:20:55:40.568 527216 INFO neutron.cmd.ovn.neutron_ovn_db_sync_util [req-c595a893-db9b-484e-ae8a-bb7dbe8b31f3 - - - - -] Sync for Northbound db started with mode : repair
16/Feb/2022:20:55:42.105 527216 INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [req-c595a893-db9b-484e-ae8a-bb7dbe8b31f3 - - - - -] Starting OVNClientQosExtension
16/Feb/2022:20:55:42.380 527216 INFO neutron.db.ovn_revision_numbers_db [req-c595a893-db9b-484e-ae8a-bb7dbe8b31f3 - - - - -] Successfully bumped revision number for resource 49b3249a-7624-4711-b271-3e63c6a27658 (type: ports) to 17
16/Feb/2022:20:55:43.205 527216 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-c595a893-db9b-484e-ae8a-bb7dbe8b31f3 - - - - -] ACLs-to-be-added 1 ACLs-to-be-removed 0
16/Feb/2022:20:55:43.206 527216 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-c595a893-db9b-484e-ae8a-bb7dbe8b31f3 - - - - -] ACL found in Neutron but not in OVN DB for port group pg_e90b68f3_9f8d_4250_9b6a_7531e2249c99
16/Feb/2022:20:55:43.208 527216 ERROR ovsdbapp.backend.ovs_idl.transaction [req-c595a893-db9b-484e-ae8a-bb7dbe8b31f3 - - - - -] Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 131, in run
    txn.results.put(txn.do_commit())
  File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 93, in do_commit
    command.run_idl(txn)
  File "/usr/lib/python3/dist-packages/ovsdbapp/schema/ovn_northbound/commands.py", line 123, in run_idl
    raise RuntimeError("ACL (%s, %s, %s) already exists" % (
RuntimeError: ACL (to-lport, 1002, outport == @pg_e90b68f3_9f8d_4250_9b6a_7531e2249c99 && ip4 && ip4.src == 10.0.0.0/8 && tcp) already exists

===== Ubuntu SRU Details =====

[Impact]
See bug description.

[Test Case]
Deploy openstack with OVN. Create overlapping security group rules. Run neutron-ovn-db-sync-util and ensure it completes successfully.

[Where problems could occur]
If the logic driven by the may_exist parameter is not correct, the existing bug could still occur. Presumably this is not the case, but that is a theoritical potential for where problems could occur. All of these patches have already landed in the corresponding upstream branches.

tags: added: ovn
Changed in neutron:
importance: Undecided → Critical
Changed in neutron:
status: New → In Progress
Changed in neutron:
assignee: nobody → Jake Yip (waipengyip)
Revision history for this message
Slawek Kaplonski (slaweq) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/801707
Committed: https://opendev.org/openstack/neutron/commit/5a0a2b7847da067817640404f53e0807755e08d7
Submitter: "Zuul (22348)"
Branch: master

commit 5a0a2b7847da067817640404f53e0807755e08d7
Author: Jake Yip <email address hidden>
Date: Tue Jul 20 17:03:08 2021 +1000

    Allow ovn_db_sync to continue on duplicate normalised CIDR

    OVN now uses normalised CIDR when adding a security group rule[1]. It
    uses may_exist=True for adding ACL (secgroup rule), in case there are
    multiple CIDRs in neutron that normalises to the same.

    Do the same in ovn_db_sync, so that the sync don't fail hard on such
    duplicates.

    [1] https://review.opendev.org/c/openstack/neutron/+/736386/

    Change-Id: I9d9c21e460029e4a6a845520bfcc2889ad20429b
    Related-Bug: #1869129
    Closes-Bug: #1961112

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 20.0.0.0rc1

This issue was fixed in the openstack/neutron 20.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/833559

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/833560

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/833561

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/833562

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/833559
Committed: https://opendev.org/openstack/neutron/commit/bf308a12a1f87ead18f7c8478ada93dc292bfcba
Submitter: "Zuul (22348)"
Branch: stable/xena

commit bf308a12a1f87ead18f7c8478ada93dc292bfcba
Author: Jake Yip <email address hidden>
Date: Tue Jul 20 17:03:08 2021 +1000

    Allow ovn_db_sync to continue on duplicate normalised CIDR

    OVN now uses normalised CIDR when adding a security group rule[1]. It
    uses may_exist=True for adding ACL (secgroup rule), in case there are
    multiple CIDRs in neutron that normalises to the same.

    Do the same in ovn_db_sync, so that the sync don't fail hard on such
    duplicates.

    [1] https://review.opendev.org/c/openstack/neutron/+/736386/

    Change-Id: I9d9c21e460029e4a6a845520bfcc2889ad20429b
    Related-Bug: #1869129
    Closes-Bug: #1961112
    (cherry picked from commit 5a0a2b7847da067817640404f53e0807755e08d7)

tags: added: in-stable-xena
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/833560
Committed: https://opendev.org/openstack/neutron/commit/691a7ceeecb020e302e8d9a2ec3ceb67741144a8
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 691a7ceeecb020e302e8d9a2ec3ceb67741144a8
Author: Jake Yip <email address hidden>
Date: Tue Jul 20 17:03:08 2021 +1000

    Allow ovn_db_sync to continue on duplicate normalised CIDR

    OVN now uses normalised CIDR when adding a security group rule[1]. It
    uses may_exist=True for adding ACL (secgroup rule), in case there are
    multiple CIDRs in neutron that normalises to the same.

    Do the same in ovn_db_sync, so that the sync don't fail hard on such
    duplicates.

    [1] https://review.opendev.org/c/openstack/neutron/+/736386/

    Change-Id: I9d9c21e460029e4a6a845520bfcc2889ad20429b
    Related-Bug: #1869129
    Closes-Bug: #1961112
    (cherry picked from commit 5a0a2b7847da067817640404f53e0807755e08d7)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/833561
Committed: https://opendev.org/openstack/neutron/commit/10c8d590af7fc4e1355e38d455bbd4ecd4bb2a4c
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 10c8d590af7fc4e1355e38d455bbd4ecd4bb2a4c
Author: Jake Yip <email address hidden>
Date: Tue Jul 20 17:03:08 2021 +1000

    Allow ovn_db_sync to continue on duplicate normalised CIDR

    OVN now uses normalised CIDR when adding a security group rule[1]. It
    uses may_exist=True for adding ACL (secgroup rule), in case there are
    multiple CIDRs in neutron that normalises to the same.

    Do the same in ovn_db_sync, so that the sync don't fail hard on such
    duplicates.

    [1] https://review.opendev.org/c/openstack/neutron/+/736386/

    Change-Id: I9d9c21e460029e4a6a845520bfcc2889ad20429b
    Related-Bug: #1869129
    Closes-Bug: #1961112
    (cherry picked from commit 5a0a2b7847da067817640404f53e0807755e08d7)

tags: added: in-stable-victoria
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/833562
Committed: https://opendev.org/openstack/neutron/commit/f1fe5260c7f22415fef7e098a2b66e84f116c649
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit f1fe5260c7f22415fef7e098a2b66e84f116c649
Author: Jake Yip <email address hidden>
Date: Tue Jul 20 17:03:08 2021 +1000

    Allow ovn_db_sync to continue on duplicate normalised CIDR

    OVN now uses normalised CIDR when adding a security group rule[1]. It
    uses may_exist=True for adding ACL (secgroup rule), in case there are
    multiple CIDRs in neutron that normalises to the same.

    Do the same in ovn_db_sync, so that the sync don't fail hard on such
    duplicates.

    [1] https://review.opendev.org/c/openstack/neutron/+/736386/

    Change-Id: I9d9c21e460029e4a6a845520bfcc2889ad20429b
    Related-Bug: #1869129
    Closes-Bug: #1961112
    (cherry picked from commit 5a0a2b7847da067817640404f53e0807755e08d7)

tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 17.4.0

This issue was fixed in the openstack/neutron 17.4.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 18.3.0

This issue was fixed in the openstack/neutron 18.3.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 19.2.0

This issue was fixed in the openstack/neutron 19.2.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/networking-ovn train-eol

This issue was fixed in the openstack/networking-ovn train-eol release.

Changed in neutron (Ubuntu):
status: New → Fix Released
Changed in neutron (Ubuntu Focal):
status: New → Triaged
importance: Undecided → High
Changed in cloud-archive:
status: New → Fix Released
description: updated
description: updated
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.