neutron

Bug #1883321
Comment #2

Comment 2 for bug 1883321

Revision history for this message

Philipp Krivanec (pkrivanec) wrote on 2020-06-17:

Hello,

I am seeing more strange behavior on my test cluster.

In this scenario I have created two L2 networks; two VMs on one compute
node in the first L2 network; a third VM on another compute node in the
second L2 network.

Compute1
============
L2 Net 1:
----------
VMa
VMb

Compute3
============
L2 Net 2:
----------
VMc

Pinging from VMa to VMc works as expected, but I can not ping from VMb
to VMc at the same time.
VMa -> VMc OK
VMb -> VMc Broken

If the ping from VMb is started first it works, but then the ping from
VMa is broken.

As can be seen in the packet dump, traffic from VMb reaches VMc but the
return traffic is forwarded to VMa, which is incorrect.

The destination MAC address of the return traffic is correct, but
OpenvSwitch forwardeds the traffic to the wrong VM.

You can see in the dump-flows on compute1, there is no flow to output
9, only a flow to output 5.

I tested with agent_mode=dvr_no_external and with agent_mode=dvr with
identical results.

If I deploy a centralized router it works.

In my opinion this is the same bug.

Best Regards
Phil

tcpdump on VMs
==========================================

DUMP VMa
------------------
eth0:
link/ether fa:16:3e:e7:c3:cb
inet 10.10.10.185/24

DUMP VMb
------------------
eth0:
link/ether fa:16:3e:f9:c2:b7
inet 10.10.10.242/24

DUMP VMc
------------------
eth0:
link/ether fa:16:3e:b2:47:43
inet 10.10.20.57/24

debian@debian:~$ sudo tcpdump -netti eth0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
1592330137.026450 fa:16:3e:b2:47:43 > fa:16:3e:c0:43:39, ethertype IPv4 (0x0800), length 98: 10.10.20.57 > 10.10.10.242: ICMP echo reply, id 577, seq 38, length 64
1592330138.050433 fa:16:3e:c0:43:39 > fa:16:3e:b2:47:43, ethertype IPv4 (0x0800), length 98: 10.10.10.242 > 10.10.20.57: ICMP echo request, id 577, seq 39, length 64
1592330138.050461 fa:16:3e:b2:47:43 > fa:16:3e:c0:43:39, ethertype IPv4 (0x0800), length 98: 10.10.20.57 > 10.10.10.242: ICMP echo reply, id 577, seq 39, length 64
1592330139.074473 fa:16:3e:c0:43:39 > fa:16:3e:b2:47:43, ethertype IPv4 (0x0800), length 98: 10.10.10.242 > 10.10.20.57: ICMP echo request, id 577, seq 40, length 64

Flow dumps on comute node 1
==========================================

[root@compute1 ~]# ovs-appctl dpctl/show
system@ovs-system:
  lookups: hit:3335 missed:233 lost:0
  flows: 5
  masks: hit:13026 total:4 hit/pkt:3.65
  port 0: ovs-system (internal)
  port 1: br-ex (internal)
  port 2: enp3s0
  port 3: br-int (internal)
  port 4: br-tun (internal)
  port 5: tapf494600d-62
  port 6: vxlan_sys_4789 (vxlan: packet_type=ptap)
  port 7: qr-56f77029-3e (internal)
  port 8: qr-dfc3396a-66 (internal)
  port 9: tapbd3a7589-3f

[root@compute1 ~]# ovs-appctl dpctl/dump-flows
recirc_id(0),in_port(9),eth(src=fa:16:3e:f9:c2:b7,dst=fa:16:3e:5b:19:10),eth_type(0x0800),ipv4(frag=no), packets:493, bytes:48314, used:0.456s, actions:7
recirc_id(0),in_port(8),skb_mark(0x4000000),eth(src=fa:16:3e:c0:43:39),eth_type(0x0800),ipv4(tos=0/0x3,frag=no), packets:539, bytes:51618, used:0.456s, flags:P., actions:set(tunnel(tun_id=0x2,src=10.0.2.100,dst=10.0.2.102,ttl=64,tp_dst=4789,flags(df|key))),set(eth(src=fa:16:3f:9c:aa:5e)),set(skb_mark(0)),6
recirc_id(0),tunnel(tun_id=0x1,src=10.0.2.102,dst=10.0.2.100,flags(-df-csum+key)),in_port(6),eth(src=fa:16:3f:58:95:6f),eth_type(0x0800),ipv4(frag=no), packets:536, bytes:55028, used:0.455s, flags:P., actions:set(eth(src=fa:16:3e:5b:19:10)),5
recirc_id(0),in_port(2),eth(src=00:17:e0:1f:63:94,dst=00:17:e0:1f:63:94),eth_type(0x9000), packets:107, bytes:6420, used:0.318s, actions:drop

Flow dumps on comute node 3
==========================================

[root@compute3 ~]# ovs-appctl dpctl/show
system@ovs-system:
  lookups: hit:2090 missed:186 lost:0
  flows: 7
  masks: hit:6106 total:5 hit/pkt:2.68
  port 0: ovs-system (internal)
  port 1: br-ex (internal)
  port 2: enp3s0
  port 3: br-int (internal)
  port 4: br-tun (internal)
  port 5: qr-56f77029-3e (internal)
  port 6: qr-dfc3396a-66 (internal)
  port 7: vxlan_sys_4789 (vxlan: packet_type=ptap)
  port 8: tap447c86ea-d9

[root@compute3 ~]# ovs-appctl dpctl/dump-flows
recirc_id(0),in_port(5),skb_mark(0x4000000),eth(src=fa:16:3e:5b:19:10),eth_type(0x0800),ipv4(tos=0/0x3,frag=no), packets:561, bytes:57478, used:0.299s, flags:P., actions:set(tunnel(tun_id=0x1,src=10.0.2.102,dst=10.0.2.100,ttl=64,tp_dst=4789,flags(df|key))),set(eth(src=fa:16:3f:58:95:6f)),set(skb_mark(0)),7
recirc_id(0),tunnel(tun_id=0x2,src=10.0.2.100,dst=10.0.2.102,flags(-df-csum+key)),in_port(7),eth(src=fa:16:3f:9c:aa:5e),eth_type(0x0800),ipv4(frag=no), packets:564, bytes:54068, used:0.299s, flags:P., actions:set(eth(src=fa:16:3e:c0:43:39)),8
recirc_id(0),in_port(2),eth(src=00:17:e0:1f:63:93,dst=01:00:0c:cc:cc:cc),eth_type(0/0xffff), packets:0, bytes:0, used:never, actions:drop
recirc_id(0),in_port(2),eth(src=00:17:e0:1f:63:93,dst=00:17:e0:1f:63:93),eth_type(0x9000), packets:110, bytes:6600, used:5.755s, actions:drop
recirc_id(0),in_port(6),eth(src=fa:16:3e:c0:43:39,dst=fa:16:3e:b2:47:43),eth_type(0x0806), packets:0, bytes:0, used:never, actions:8
recirc_id(0),in_port(2),eth(src=fa:16:3e:14:30:b5,dst=33:33:00:00:00:02),eth_type(0x8100),vlan(vid=97),encap(eth_type(0x86dd),ipv6(frag=no)), packets:0, bytes:0, used:never, actions:1
recirc_id(0),in_port(8),eth(src=fa:16:3e:b2:47:43,dst=fa:16:3e:c0:43:39),eth_type(0x0800),ipv4(frag=no), packets:561, bytes:57478, used:0.299s, flags:P., actions:6
recirc_id(0),in_port(8),eth(src=fa:16:3e:b2:47:43,dst=fa:16:3e:c0:43:39),eth_type(0x0806),arp(sip=10.10.20.57), packets:0, bytes:0, used:never, actions:6