Gateway_less_Fwd: Ping to floating IP fails when source and destinations VNs configured with IP Fabric provider network

Ping to floating IP fails when source and destinations VNs configured with IP Fabric provider network

Build
------
R4.1.0.0 Build 42 Ubuntu 14.04.5 Mitaka

Steps
Configure 2 VNs (VN1: 10.10.10.0/24, VN2: 20.20.20.0/24)
Configure IP Fabric VN as provider network for both the VNs
Launch a VM in each VN across 2 compute nodes (VM1: 10.10.10.3/24 and VM2: 20.20.20.3/24 )
Configure floating IP pool on VN2 and create a floating IP (say 20.20.20.6/24)
Configure floating IP on VM1
Now, pinging floating IP (20.20.20.6) from VM2 (20.20.20.3) fails.

Packets are getting dropped on destination compute with “Invalid Source”

Please see the log below:

root@nodek11:~# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-lib 4.1.0.0-42 42
contrail-nodemgr 4.1.0.0-42 42
contrail-setup 4.1.0.0-42 42
contrail-utils 4.1.0.0-42 42
contrail-vrouter-agent 4.1.0.0-42 42
contrail-vrouter-common 4.1.0.0-42 42
contrail-vrouter-dkms 4.1.0.0-42 42
contrail-vrouter-init 4.1.0.0-42 42
contrail-vrouter-utils 4.1.0.0-42 42
nova-common 2:13.0.0-0ubuntu2~cloud0.1contrail1 42
nova-compute 2:13.0.0-0ubuntu2~cloud0.1contrail1 42
nova-compute-kvm 2:13.0.0-0ubuntu2~cloud0.1contrail1 42
nova-compute-libvirt 2:13.0.0-0ubuntu2~cloud0.1contrail1 42
python-contrail 4.1.0.0-42 42
python-contrail-vrouter-api 4.1.0.0-42 42
python-neutronclient 1:4.1.1-2~cloud0.2contrail 42
python-nova 2:13.0.0-0ubuntu2~cloud0.1contrail1 42
python-opencontrail-vrouter-netns 4.1.0.0-42 42
root@nodek11:~#
root@nodek11:~# contrail-status
== Contrail vRouter ==
supervisor-vrouter: active
contrail-vrouter-agent active
contrail-vrouter-nodemgr active

root@nodek11:~# vif --list
Vrouter Interface Table

Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror
       Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2
       D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged
       Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored
       Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload, Df=Drop New Flows, L=MAC Learning Enabled
       Proxy=MAC Requests Proxied Always, Er=Etree Root

vif0/0 OS: em1 (Speed 1000, Duplex 1)
            Type:Physical HWaddr:0c:c4:7a:32:0a:88 IPaddr:0.0.0.0
            Vrf:0 Flags:L3L2VpEr QOS:-1 Ref:7
            RX packets:50038 bytes:5267863 errors:1
            TX packets:11754 bytes:10194291 errors:0
            Drops:2728

vif0/1 OS: vhost0
            Type:Host HWaddr:0c:c4:7a:32:0a:88 IPaddr:10.204.216.231
            Vrf:0 Flags:PL3DEr QOS:-1 Ref:7
            RX packets:10964 bytes:10090898 errors:0
            TX packets:47015 bytes:4918086 errors:0
            Drops:1

vif0/2 OS: pkt0
            Type:Agent HWaddr:00:00:5e:00:01:00 IPaddr:0.0.0.0
            Vrf:65535 Flags:L3Er QOS:-1 Ref:3
            RX packets:1586 bytes:201016 errors:0
            TX packets:29717 bytes:3135323 errors:0
            Drops:0

vif0/3 OS: tap0c2e23af-a7
            Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:10.10.10.3
            Vrf:0 Flags:PL3L2DProxyEr QOS:-1 Ref:5
            RX packets:2973 bytes:289065 errors:0
            TX packets:2175 bytes:213085 errors:0
            Drops:982

vif0/4350 OS: pkt3
            Type:Stats HWaddr:00:00:00:00:00:00 IPaddr:0.0.0.0
            Vrf:65535 Flags:L3L2 QOS:0 Ref:1
            RX packets:173 bytes:16954 errors:0
            TX packets:173 bytes:14532 errors:0
            Drops:0

vif0/4351 OS: pkt1
            Type:Stats HWaddr:00:00:00:00:00:00 IPaddr:0.0.0.0
            Vrf:65535 Flags:L3L2 QOS:0 Ref:1
            RX packets:660 bytes:55440 errors:0
            TX packets:660 bytes:55440 errors:0
            Drops:0

root@nodek11:~#
root@nodek11:~#
root@nodek11:~# tcpdump -n -i em1 host 20.20.20.6 -vvv -xxx
tcpdump: WARNING: em1: no IPv4 address assigned
tcpdump: listening on em1, link-type EN10MB (Ethernet), capture size 65535 bytes
15:39:53.464235 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
    20.20.20.5 > 20.20.20.6: ICMP echo request, id 5335, seq 1885, length 64
0x0000: 0cc4 7a32 0a88 0025 9093 d252 0800 4500
0x0010: 0054 0000 4000 3f01 eb76 1414 1405 1414
0x0020: 1406 0800 5aff 14d7 075d 70b2 b759 0000
0x0030: 0000 8fed 0a00 0000 0000 1011 1213 1415
0x0040: 1617 1819 1a1b 1c1d 1e1f 2021 2223 2425
0x0050: 2627 2829 2a2b 2c2d 2e2f 3031 3233 3435
0x0060: 3637
15:39:54.472250 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
    20.20.20.5 > 20.20.20.6: ICMP echo request, id 5335, seq 1886, length 64
0x0000: 0cc4 7a32 0a88 0025 9093 d252 0800 4500
0x0010: 0054 0000 4000 3f01 eb76 1414 1405 1414
0x0020: 1406 0800 fdde 14d7 075e 71b2 b759 0000
0x0030: 0000 eb0c 0b00 0000 0000 1011 1213 1415
0x0040: 1617 1819 1a1b 1c1d 1e1f 2021 2223 2425
0x0050: 2627 2829 2a2b 2c2d 2e2f 3031 3233 3435
0x0060: 3637
15:39:55.480150 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
    20.20.20.5 > 20.20.20.6: ICMP echo request, id 5335, seq 1887, length 64
0x0000: 0cc4 7a32 0a88 0025 9093 d252 0800 4500
0x0010: 0054 0000 4000 3f01 eb76 1414 1405 1414
0x0020: 1406 0800 c2be 14d7 075f 72b2 b759 0000
0x0030: 0000 252c 0b00 0000 0000 1011 1213 1415
0x0040: 1617 1819 1a1b 1c1d 1e1f 2021 2223 2425
0x0050: 2627 2829 2a2b 2c2d 2e2f 3031 3233 3435
0x0060: 3637
^C
3 packets captured
4 packets received by filter
0 packets dropped by kernel
root@nodek11:~#
root@nodek11:~#
root@nodek11:~#
root@nodek11:~# flow --match 20.20.20.6
Flow table(size 80609280, entries 629760)

Entries: Created 53 Added 55 Deleted 80 Changed 106 Processed 53 Used Overflow entries 0
(Created Flows/CPU: 4 4 2 2 4 0 3 5 0 2 0 0 1 1 0 0 2 2 5 2 0 10 2 2 0 0 0 0 0 0 0 0)(oflows 0)

Action:F=Forward, D=Drop N=NAT(S=SNAT, D=DNAT, Ps=SPAT, Pd=DPAT, L=Link Local Port)
 Other:K(nh)=Key_Nexthop, S(nh)=RPF_Nexthop
 Flags:E=Evicted, Ec=Evict Candidate, N=New Flow, M=Modified Dm=Delete Marked
TCP(r=reverse):S=SYN, F=FIN, R=RST, C=HalfClose, E=Established, D=Dead

Listing flows matching ([20.20.20.6]:*)

    Index Source:Port/Destination:Port Proto(V)
-----------------------------------------------------------------------------------
   274008<=>191048 20.20.20.5:5335 1 (0->4)
                         20.20.20.6:0
(Gen: 1, K(nh):16, Action:N(D), Flags:, QOS:-1, S(nh):23, Stats:1896/159264,
 SPort 60738, TTL 0, Sinfo 0.0.0.0)

root@nodek11:~# flow --match 20.20.20.5
Flow table(size 80609280, entries 629760)

Entries: Created 53 Added 55 Deleted 80 Changed 106 Processed 53 Used Overflow entries 0
(Created Flows/CPU: 4 4 2 2 4 0 3 5 0 2 0 0 1 1 0 0 2 2 5 2 0 10 2 2 0 0 0 0 0 0 0 0)(oflows 0)

Action:F=Forward, D=Drop N=NAT(S=SNAT, D=DNAT, Ps=SPAT, Pd=DPAT, L=Link Local Port)
 Other:K(nh)=Key_Nexthop, S(nh)=RPF_Nexthop
 Flags:E=Evicted, Ec=Evict Candidate, N=New Flow, M=Modified Dm=Delete Marked
TCP(r=reverse):S=SYN, F=FIN, R=RST, C=HalfClose, E=Established, D=Dead

Listing flows matching ([20.20.20.5]:*)

    Index Source:Port/Destination:Port Proto(V)
-----------------------------------------------------------------------------------
   191048<=>274008 10.10.10.3:5335 1 (4->2)
                         20.20.20.5:0
(Gen: 2, K(nh):16, Action:N(S), Flags:, QOS:-1, S(nh):16, Stats:0/0,
 SPort 64584, TTL 0, Sinfo 0.0.0.0)

   274008<=>191048 20.20.20.5:5335 1 (0->4)
                         20.20.20.6:0
(Gen: 1, K(nh):16, Action:N(D), Flags:, QOS:-1, S(nh):23, Stats:1900/159600,
 SPort 60738, TTL 0, Sinfo 0.0.0.0)

root@nodek11:~# dropstats
Invalid IF 0
Trap No IF 0
IF TX Discard 0
IF Drop 1
IF RX Discard 0

Flow Unusable 0
Flow No Memory 0
Flow Table Full 0
Flow NAT no rflow 0
Flow Action Drop 0
Flow Action Invalid 0
Flow Invalid Protocol 0
Flow Queue Limit Exceeded 0
New Flow Drops 0
Flow Unusable (Eviction) 0

Original Packet Trapped 0

Discards 0
TTL Exceeded 0
Mcast Clone Fail 0
Cloned Original 324

Invalid NH 0
Invalid Label 0
Invalid Protocol 0
Etree Leaf to Leaf 0
Bmac/ISID Mismatch 0
Rewrite Fail 0
Invalid Mcast Source 0
Packet Loop 0

Push Fails 0
Pull Fails 0
Duplicated 1
Head Alloc Fails 0
PCOW fails 0
Invalid Packets 0

Misc 0
Nowhere to go 0
Checksum errors 0
No Fmd 0
Invalid VNID 0
Fragment errors 0
Invalid Source 2621
Jumbo Mcast Pkt with DF Bit 0
No L2 Route 839
Memory Failures 0
Fragment Queueing Failures 0

root@nodek11:~# dropstats
Invalid IF 0
Trap No IF 0
IF TX Discard 0
IF Drop 1
IF RX Discard 0

Flow Unusable 0
Flow No Memory 0
Flow Table Full 0
Flow NAT no rflow 0
Flow Action Drop 0
Flow Action Invalid 0
Flow Invalid Protocol 0
Flow Queue Limit Exceeded 0
New Flow Drops 0
Flow Unusable (Eviction) 0

Original Packet Trapped 0

Discards 0
TTL Exceeded 0
Mcast Clone Fail 0
Cloned Original 324

Invalid NH 0
Invalid Label 0
Invalid Protocol 0
Etree Leaf to Leaf 0
Bmac/ISID Mismatch 0
Rewrite Fail 0
Invalid Mcast Source 0
Packet Loop 0

Push Fails 0
Pull Fails 0
Duplicated 1
Head Alloc Fails 0
PCOW fails 0
Invalid Packets 0

Misc 0
Nowhere to go 0
Checksum errors 0
No Fmd 0
Invalid VNID 0
Fragment errors 0
Invalid Source 2622
Jumbo Mcast Pkt with DF Bit 0
No L2 Route 839
Memory Failures 0
Fragment Queueing Failures 0

root@nodek11:~# dropstats
Invalid IF 0
Trap No IF 0
IF TX Discard 0
IF Drop 1
IF RX Discard 0

Flow Unusable 0
Flow No Memory 0
Flow Table Full 0
Flow NAT no rflow 0
Flow Action Drop 0
Flow Action Invalid 0
Flow Invalid Protocol 0
Flow Queue Limit Exceeded 0
New Flow Drops 0
Flow Unusable (Eviction) 0

Original Packet Trapped 0

Discards 0
TTL Exceeded 0
Mcast Clone Fail 0
Cloned Original 324

Invalid NH 0
Invalid Label 0
Invalid Protocol 0
Etree Leaf to Leaf 0
Bmac/ISID Mismatch 0
Rewrite Fail 0
Invalid Mcast Source 0
Packet Loop 0

Push Fails 0
Pull Fails 0
Duplicated 1
Head Alloc Fails 0
PCOW fails 0
Invalid Packets 0

Misc 0
Nowhere to go 0
Checksum errors 0
No Fmd 0
Invalid VNID 0
Fragment errors 0
Invalid Source 2624
Jumbo Mcast Pkt with DF Bit 0
No L2 Route 840
Memory Failures 0
Fragment Queueing Failures 0

root@nodek11:~#