DVR HA Router Update Error

Hello:

I deployed an environment with 3 HA controllers and 2 compute nodes. The L3 agent mode of the controllers was "dvr_snat" and the compute nodes "dvr".

I executed the following commands: https://paste.opendev.org/show/812228/

When the subnet was added to the router, the router routes of the instance in the controllers were:
[root@controller-0 ~]# ip netns exec $router ip r
10.10.0.0/26 dev qr-b66d5313-71 proto kernel scope link src 10.10.0.1
169.254.107.94/31 dev rfp-72922eaf-3 proto kernel scope link src 169.254.107.94

When the subnet was deleted:
[root@controller-0 ~]# ip netns exec $router ip r
169.254.107.94/31 dev rfp-72922eaf-3 proto kernel scope link src 169.254.107.94

The route for the subnet CIDR was correctly added and removed and no error was found in the L3 agent logs, thus I was not able to reproduce the error you reported. Did I miss something in my deployment?

BTW, I was using Wallaby.

Regards.

Hi Rodolfo, sorry for late response.

Thanks you for your test.

We have observed something interesting.

Yes, L3 agent mode of the controllers is "dvr_snat" and the compute nodes "dvr". We also upgraded to Wallaby but issue still persist.

Can you try when a vlan provider network attached as GW to the router? Provider Network subnet pool and Tenant Network subnet pool also resides on the same address scope to use bgp. Details: https://docs.openstack.org/neutron/wallaby/admin/config-bgp-dynamic-routing.html

You are right, in our case qrouter ns have the routes but does not have the ip rules ("$ ip rule")

Also, fip ns does not have routes at all!

Assume there exist 5 network node, max_l3_agents_per_router:3 and dvr ha router running on network node 01, 02 and 03. Router have a vlan provider network as GW. Provider Network and Tenant Network have subnets from two different subnet pools which in the same address scope.

When attaching a vxlan tenant network to router, then tenant network cidr route is updated in the fip netns for the network nodes which does not have the router (Network node 04 and 05) but not updated for network node 01,02 and 03 (which have the router) We have seen the "Starting router update for..." and "Finished a router update..." logs for all network node l3 agents. But somehow l3 agent ignore to add qrouter netns ip rules and fip netns ip routes for attached tenant network.

! The first network attachment to router after router creation may success but next attach operations have this situation.

Restarting the l3 agent fixes the all this issues!

Also when detaching the network, we also getting errors because l3 agent tries to remove routes which are not already added.

We are using dvr, address scopes and bgp. Only dns request of instances NAT'ted over controller nodes. If all of the dhcp agents resides on the network nodes which does not have that fip ns routes then our instances fail for dns queries.

Scenario

BGP Provider Network
  10.10.10.0/24

BGP Tenant Network
  10.10.12.0/24

# openstack address scope create --ip-version 4 test-scope
# openstack subnet pool create --pool-prefix 10.10.10.0/24 --address-scope test-scope test-provider-subnet-pool-01
# openstack subnet pool create --pool-prefix 10.10.12.0/24 --address-scope test-scope test-tenant-subnet-pool-01

# Create Provider Network and Subnet
# openstack network create \
--external \
--provider-network-type vlan \
--provider-physical-network physnet1 \
--provider-segment 118 test-provider-net-01

openstack subnet create test-provider-subnet-01 \
--network test-provider-net-01 \
--subnet-pool test-provider-subnet-pool-01 \
--prefix-length 24

===> We also changes the allocation pool and default gw after subnet creation. Because some of these IPs used by router. (For BGP)
    --allocation-pool start=10.10.10.20,end=10.10.10.253
    GW: 10.10.10.12

# Create vxlan tenant network and subnet
# openstack network create tenant-vxlan-net
# openstack subnet create tenant-vxlan-subnet-01 \
--network tenant-vxlan-net \
--subnet-pool test-tenant-subnet-pool-01 \
--prefix-length 26

# Create a router and set external gw as provider network. Also attach test-vxlan ...

Hi,

As a workaround we can add ip rule to qrouter ns and ip route to fip netns.

qrouter-netns# ip rule add pref 80000 from 10.216.12.112/28 table 16

fip-netns# ip route add 10.216.12.112/28 via 169.254.88.134 dev fpr-9ebc2eb1-c proto static

Then our instances can resolve domains and no more exception when detaching networks from router.

Hi again,

If not using address scopes, then ip rules added to qrouter netns. IP routes not added to fip netns but it is probably ok.

RedHat OSP 16.0 “Configuring distributed virtual routing (DVR)” documentation states that

- DVR is not supported in conjunction with L3 HA. If you use DVR with Red Hat OpenStack Platform 16.0 director, L3 HA is disabled. This means that routers are still scheduled on the Network nodes (and load-shared between the L3 agents), but if one agent fails, all routers hosted by this agent fail as well. This affects only SNAT traffic. The allow_automatic_l3agent_failover feature is recommended in such cases, so that if one network node fails, the routers are rescheduled to a different node.

Also end of the doc there exist “Migrating centralized routers to distributed routing” section which dictates to disable HA before enabling DVR.

Doc Link: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.0/html/networking_guide/sec-dvr

There exist an official openstack doc: Open vSwitch: High availability using DVR (https://docs.openstack.org/neutron/wallaby/admin/deploy-ovs-ha-dvr.html)

It is confusing for dvr+l3_ha is supported or not supported?

Also is it possible that fast-exit is not supported for dvr+l3_ha ? The commits which are linked to https://bugs.launchpad.net/neutron/+bug/1577488 [RFE]“Fast exit” show changes for "agent/l3/dvr_local_router.py" file.

When we are disabling the HA for DVR router then it works as expected! (ip rules added to qrouter netns, ip routes added to fip netns etc)