Slow ovs flows processing in 9.2 with ovs firewall

Bug #1658711 reported by Alexander Ignatov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Confirmed
High
Inessa Vasilevskaya
9.x
Confirmed
High
Inessa Vasilevskaya

Bug Description

After some performance evaluations of ovs firewall in terms of existing limitations with large number of security group rules.
We found one potential performance problem. The scenario is:

1. Have a setup with ovs firewall enabled (In our case it is MOS 9.2, but I’m sure it’s reproducible with the devstack)
2. Create 1 security group with large number of sec group rules (in our case it was 4000) and use remote_group_id(with just default rules there) for each rule.
3. Then start booting VMs with this sec-group applied.
4. On each boot iteration measure the time when VMs gets pingable.

The issue: You will see that boot time increases almost non-linear. In our case when we spawned 10+ instances each next instance may go into error state because of timeouts.

This feature affects only 9.2+OVSFW enabled in case with only remote security groups.

Related bug in upstream: https://bugs.launchpad.net/neutron/+bug/1628819

Revision history for this message
Alexander Ignatov (aignatov) wrote :

Fix for this bug in fuel-infra https://review.fuel-infra.org/#/c/30019/

Changed in mos:
status: New → Confirmed
importance: Undecided → High
assignee: nobody → Inessa Vasilevskaya (ivasilevskaya)
milestone: none → 9.2
tags: added: area-neutron
Revision history for this message
Alexander Ignatov (aignatov) wrote :

We don't see such behavior for just security groups, we were able two spawn quickly >100 VMs on a single host with more than 60k+ flows in total.

Changed in mos:
milestone: 9.2 → 9.3
tags: added: release-notes
Revision history for this message
Inessa Vasilevskaya (ivasilevskaya) wrote :

I'd suggest we add this as known issue to release notes:

In case of openvswitch firewall booting vms with security group that has security group rules with remote_group_id defined may result in drastical performance degradation due to a solid number of ovs flows being generated. This happens because currently the number of ovs flows has quadratic growth if remote_group_id is specified during security group rule creation [https://bugs.launchpad.net/mos/+bug/1658711].
The workaround is to design security groups so that remote_group_id won't be set for security group rules.

Changed in mos:
milestone: 9.x-updates → 9.2
Revision history for this message
Maria Zlatkova (mzlatkova) wrote :
tags: added: release-notes-done
removed: release-notes
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.