I have tried with the latest code pulled 2/4 and also with the commit at
https://review.openstack.org/#/c/275439/
I am trying to experiment with a topology when a packet may traverse through more than one router.
For debug purposes, since OpenStack does not allow chaining two routers directly, I have simplified the topology (created using OpenStack Horizon) to:
VM1 10.1.0.3 @ Network 1
VM2 10.2.0.4 @ Network 2
VM3 10.2.0.4 @ Network 3
Router 1 connects Network 1 & Network 2.
Router 2 connects Network 2 & Network 3.
I can successfully ping between the VM 1& VM2, but can't ping between VM2 and VM3 and the desirable VM1 & VM3.
Adding static routes did not resolve the problem too, it looks like once a network is connected to 2 routers, the packets will always pass to the gateway port towards one of the routers and will never be sent to the other router port.
Here are some dumps:
Only pinging from VM3 to VM2:
sudo ovs-dpctl dump-flows
recirc_id(0),in_port(7),eth(src=fa:16:3e:6e:72:de),eth_type(0x0800),ipv4(frag=no), packets:143, bytes:14014, used:3.912s, actions:ct(zone=6),recirc(0x7d)
recirc_id(0x7d),in_port(7),ct_state(+new-est-rel-inv+trk),eth(src=00:00:00:00:00:00/01:00:00:00:00:00,dst=fa:16:3e:86:be:88),eth_type(0x0800),ipv4(src=10.3.0.2/255.255.255.254,dst=10.2.0.4,ttl=64,frag=no), packets:143, bytes:14014, used:3.912s, actions:ct(commit,zone=6),ct(commit,zone=6)
Only pinging from VM2 to VM1:
recirc_id(0),in_port(6),eth(src=fa:16:3e:cb:bf:af),eth_type(0x0800),ipv4(frag=no), packets:14, bytes:1372, used:0.564s, actions:ct(zone=5),recirc(0x7e)
recirc_id(0x7e),in_port(6),ct_state(+new-est-rel-inv+trk),eth(src=00:00:00:00:00:00/01:00:00:00:00:00,dst=fa:16:3e:31:15:e2),eth_type(0x0800),ipv4(src=10.2.0.4,dst=10.3.0.2/255.255.255.254,ttl=64,frag=no), packets:14, bytes:1372, used:0.564s, actions:ct(commit,zone=5),ct(commit,zone=5)
Successful ping between VM1 & VM2:
recirc_id(0x81),in_port(6),ct_state(-new+est-rel-inv+trk),eth(src=fa:16:3e:cb:bf:af,dst=fa:16:3e:31:15:e2),eth_type(0x0800),ipv4(src=10.2.0.4,dst=10.1.0.3,ttl=64,frag=no), packets:162, bytes:15876, used:0.188s, actions:set(eth(src=fa:16:3e:47:ac:e9,dst=fa:16:3e:17:99:1d)),set(ipv4(src=10.2.0.4,dst=10.1.0.3,ttl=63)),ct(commit),ct(zone=4),recirc(0x82)
recirc_id(0),in_port(5),eth(src=fa:16:3e:17:99:1d),eth_type(0x0800),ipv4(frag=no), packets:163, bytes:15974, used:0.188s, actions:ct(zone=4),recirc(0x7f)
recirc_id(0),in_port(6),eth(src=fa:16:3e:cb:bf:af),eth_type(0x0800),ipv4(frag=no), packets:163, bytes:15974, used:0.188s, actions:ct(zone=5),recirc(0x81)
recirc_id(0x80),in_port(5),ct_state(-new+est-rel-inv+trk),eth(dst=fa:16:3e:cb:bf:af),eth_type(0x0800),ipv4(frag=no), packets:162, bytes:15876, used:0.188s, actions:6
recirc_id(0x7f),in_port(5),ct_state(-new+est-rel-inv+trk),eth(src=fa:16:3e:17:99:1d,dst=fa:16:3e:47:ac:e9),eth_type(0x0800),ipv4(src=10.1.0.2/255.255.255.254,dst=10.2.0.4,ttl=64,frag=no), packets:162, bytes:15876, used:0.188s, actions:set(eth(src=fa:16:3e:31:15:e2,dst=fa:16:3e:cb:bf:af)),set(ipv4(src=10.1.0.2/255.255.255.254,dst=10.2.0.4,ttl=63)),ct(commit),ct(zone=5),recirc(0x80)
recirc_id(0x82),in_port(6),ct_state(-new+est-rel-inv+trk),eth(dst=fa:16:3e:17:99:1d),eth_type(0x0800),ipv4(frag=no), packets:162, bytes:15876, used:0.188s, actions:5
Some OVN dumps:
sudo ovs-vsctl show
86b17641-06b1-4cde-b82d-2490450c512c
Bridge br-int
fail_mode: secure
Port br-int
Interface br-int
type: internal
Port "patch-lrp-ee1c2b34-b70d-491c-8db9-f29a7fe35131-to-ee1c2b34-b70d-491c-8db9-f29a7fe35131"
Interface "patch-lrp-ee1c2b34-b70d-491c-8db9-f29a7fe35131-to-ee1c2b34-b70d-491c-8db9-f29a7fe35131"
type: patch
options: {peer="patch-ee1c2b34-b70d-491c-8db9-f29a7fe35131-to-lrp-ee1c2b34-b70d-491c-8db9-f29a7fe35131"}
Port "tap0bad0160-ef"
Interface "tap0bad0160-ef"
type: internal
Port "patch-lrp-143ea76d-691f-4440-aadf-90e9e0a03fcd-to-143ea76d-691f-4440-aadf-90e9e0a03fcd"
Interface "patch-lrp-143ea76d-691f-4440-aadf-90e9e0a03fcd-to-143ea76d-691f-4440-aadf-90e9e0a03fcd"
type: patch
options: {peer="patch-143ea76d-691f-4440-aadf-90e9e0a03fcd-to-lrp-143ea76d-691f-4440-aadf-90e9e0a03fcd"}
Port "tap211d732f-8f"
Interface "tap211d732f-8f"
Port "tap005a356d-55"
Interface "tap005a356d-55"
type: internal
Port "patch-lrp-be975971-f023-4824-8a9a-f49deac4c48e-to-be975971-f023-4824-8a9a-f49deac4c48e"
Interface "patch-lrp-be975971-f023-4824-8a9a-f49deac4c48e-to-be975971-f023-4824-8a9a-f49deac4c48e"
type: patch
options: {peer="patch-be975971-f023-4824-8a9a-f49deac4c48e-to-lrp-be975971-f023-4824-8a9a-f49deac4c48e"}
Port "tap494da741-31"
Interface "tap494da741-31"
type: internal
Port "patch-376d7039-66dc-40e6-9ebb-ca75ba986f11-to-lrp-376d7039-66dc-40e6-9ebb-ca75ba986f11"
Interface "patch-376d7039-66dc-40e6-9ebb-ca75ba986f11-to-lrp-376d7039-66dc-40e6-9ebb-ca75ba986f11"
type: patch
options: {peer="patch-lrp-376d7039-66dc-40e6-9ebb-ca75ba986f11-to-376d7039-66dc-40e6-9ebb-ca75ba986f11"}
Port "tapc9d18a59-8b"
Interface "tapc9d18a59-8b"
Port "patch-be975971-f023-4824-8a9a-f49deac4c48e-to-lrp-be975971-f023-4824-8a9a-f49deac4c48e"
Interface "patch-be975971-f023-4824-8a9a-f49deac4c48e-to-lrp-be975971-f023-4824-8a9a-f49deac4c48e"
type: patch
options: {peer="patch-lrp-be975971-f023-4824-8a9a-f49deac4c48e-to-be975971-f023-4824-8a9a-f49deac4c48e"}
Port "patch-143ea76d-691f-4440-aadf-90e9e0a03fcd-to-lrp-143ea76d-691f-4440-aadf-90e9e0a03fcd"
Interface "patch-143ea76d-691f-4440-aadf-90e9e0a03fcd-to-lrp-143ea76d-691f-4440-aadf-90e9e0a03fcd"
type: patch
options: {peer="patch-lrp-143ea76d-691f-4440-aadf-90e9e0a03fcd-to-143ea76d-691f-4440-aadf-90e9e0a03fcd"}
Port "patch-lrp-376d7039-66dc-40e6-9ebb-ca75ba986f11-to-376d7039-66dc-40e6-9ebb-ca75ba986f11"
Interface "patch-lrp-376d7039-66dc-40e6-9ebb-ca75ba986f11-to-376d7039-66dc-40e6-9ebb-ca75ba986f11"
type: patch
options: {peer="patch-376d7039-66dc-40e6-9ebb-ca75ba986f11-to-lrp-376d7039-66dc-40e6-9ebb-ca75ba986f11"}
Port "patch-ee1c2b34-b70d-491c-8db9-f29a7fe35131-to-lrp-ee1c2b34-b70d-491c-8db9-f29a7fe35131"
Interface "patch-ee1c2b34-b70d-491c-8db9-f29a7fe35131-to-lrp-ee1c2b34-b70d-491c-8db9-f29a7fe35131"
type: patch
options: {peer="patch-lrp-ee1c2b34-b70d-491c-8db9-f29a7fe35131-to-ee1c2b34-b70d-491c-8db9-f29a7fe35131"}
Port "tap6dfb3d54-ea"
Interface "tap6dfb3d54-ea"
Please advice...
sudo ovs-ofctl -O OpenFlow13 dump-flows br-int 52931.929s, table=0, n_packets=18, n_bytes=2314, priority= 100,in_ port=2 actions= set_field: 0x1->reg5, set_field: 0x2->metadata, set_field: 0x1->reg6, resubmit( ,16) 52911.922s, table=0, n_packets=26, n_bytes=3569, priority= 100,in_ port=3 actions= set_field: 0x2->reg5, set_field: 0x3->metadata, set_field: 0x1->reg6, resubmit( ,16) 52871.156s, table=0, n_packets=57163, n_bytes=5607608, priority= 100,in_ port=6 actions= set_field: 0x2->metadata, set_field: 0x2->reg6, resubmit( ,16) 52871.156s, table=0, n_packets=97551, n_bytes=9558462, priority= 100,in_ port=5 actions= set_field: 0x5->metadata, set_field: 0x1->reg6, resubmit( ,16) 52859.599s, table=0, n_packets=17248, n_bytes=1689358, priority= 100,in_ port=7 actions= set_field: 0x3->metadata, set_field: 0x2->reg6, resubmit( ,16) 52859.599s, table=0, n_packets=59056, n_bytes=5793998, priority= 100,in_ port=8 actions= set_field: 0x5->metadata, set_field: 0x2->reg6, resubmit( ,16) 52753.744s, table=0, n_packets=99051, n_bytes=9622439, priority= 100,in_ port=13 actions= set_field: 0x4->reg5, set_field: 0x2->metadata, set_field: 0x3->reg6, resubmit( ,16) 100,in_ port=31 actions= set_field: 0x5->reg5, set_field: 0x3->metadata, set_field: 0x6->reg6, resubmit( ,16) 100,in_ port=34 actions= set_field: 0x3->metadata, set_field: 0x7->reg6, resubmit( ,16) 100,in_ port=35 actions= set_field: 0x6->metadata, set_field: 0x1->reg6, resubmit( ,16) 100,in_ port=36 actions= set_field: 0x3->reg5, set_field: 0x8->metadata, set_field: 0x1->reg6, resubmit( ,16) 100,in_ port=39 actions= set_field: 0x6->reg5, set_field: 0x8->metadata, set_field: 0x3->reg6, resubmit( ,16) 100,in_ port=41 actions= set_field: 0x8->metadata, set_field: 0x4->reg6, resubmit( ,16) 100,in_ port=42 actions= set_field: 0x6->metadata, set_field: 0x2->reg6, resubmit( ,16) 52932.057s, table=16, n_packets=0, n_bytes=0, priority= 100,metadata= 0x2,vlan_ tci=0x1000/ 0x1000 actions=drop 52911.961s, table=16, n_packets=0, n_bytes=0, priority= 100,metadata= 0x3,vlan_ tci=0x1000/ 0x1000 actions=drop 52871.158s, table=16, n_packets=0, n_bytes=0, priority= 100,metadata= 0x5,vlan_ tci=0x1000/ 0x1000 actions=drop 100,metadata= 0x6,vlan_ tci=0x1000/ 0x1000 actions=drop 100,metadata= 0x8,vlan_ tci=0x1000/ 0x1000 actions=drop 52932.057s, table=16, n_packets=0, n_bytes=0, priority=100,met...
OFPST_FLOW reply (OF1.3) (xid=0x2):
cookie=0x0, duration=
cookie=0x0, duration=
cookie=0x0, duration=
cookie=0x0, duration=
cookie=0x0, duration=
cookie=0x0, duration=
cookie=0x0, duration=
cookie=0x0, duration=2581.858s, table=0, n_packets=3200, n_bytes=310219, priority=
cookie=0x0, duration=1896.264s, table=0, n_packets=0, n_bytes=0, priority=
cookie=0x0, duration=1896.264s, table=0, n_packets=0, n_bytes=0, priority=
cookie=0x0, duration=1830.586s, table=0, n_packets=16, n_bytes=1903, priority=
cookie=0x0, duration=1695.314s, table=0, n_packets=189, n_bytes=18445, priority=
cookie=0x0, duration=869.925s, table=0, n_packets=0, n_bytes=0, priority=
cookie=0x0, duration=869.925s, table=0, n_packets=160, n_bytes=15680, priority=
cookie=0x0, duration=
cookie=0x0, duration=
cookie=0x0, duration=
cookie=0x0, duration=1896.271s, table=16, n_packets=0, n_bytes=0, priority=
cookie=0x0, duration=1830.720s, table=16, n_packets=0, n_bytes=0, priority=
cookie=0x0, duration=