ECMP load-balancing using floating ip is not working correctly

Bug #1687879 reported by Vedamurthy Joshi
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R3.2
Fix Committed
High
Hari Prasad Killi
R4.0
Fix Committed
High
Hari Prasad Killi
Trunk
Fix Committed
High
Hari Prasad Killi

Bug Description

R4.0 3060
kubernetes setup

A service has a floating ip 10.111.84.52 with 5 backend pods
Two are in one compute node(CN1) and 3 are on the other compute node(CN2)

Client pod is on CN1 doing a http request to the floating IP

It is seen that load balancing is happening only on 3 out of the 5 pods.
On the source compute, load-balancing is fine, but across the tunnel, traffic always goes to the first ecmp index

Tags: vrouter dt
Revision history for this message
Vedamurthy Joshi (vedujoshi) wrote :

This issue was still seen during 4.0 build 10 test run

It was seen that this time, traffic need not fail by going to the first ecmp index only though, but could go to other indices also.

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.0

Review in progress for https://review.opencontrail.org/32221
Submitter: Praveen K V (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/32223
Submitter: Hari Prasad Killi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/32221
Committed: http://github.com/Juniper/contrail-controller/commit/a9dec8d5c9c3eac71c4668af0ecd6cfacdee192c
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit a9dec8d5c9c3eac71c4668af0ecd6cfacdee192c
Author: Praveen K V <email address hidden>
Date: Fri May 26 12:12:30 2017 +0530

Add vhost ip-address to ECMP hash

In case of flows from one compute to another, the 5-tuple used to
compute hash is same on both compute nodes. At source, hash is computed
on hash of 5-tuple and a remote compute is selected based on hash.
When flow reaches destination, hash value for flow is computed on same key
and results in same value. At destination, the entropy for hash is
significantly reduced (since only flows with specific hash pattern reach
destination). This results in either skewed distribution or in some
cases flows not selecting some interfaces

Example:
Consider source has 2 member composite-nh (destined to dest-1 and dest-2).
All flows reaching dest-1 will have even-hash value and all flows reaching
dest-2 will have odd hash-value. On dest-1, the flows will only go to
even members on local composite-nh and on dest-2, the flows will only go
to odd members of local composite-nh.

Fix:
Use vhost-ip on compute node for hash computation. This will result in
diffrent hash value on source and destination. This results in better
distribution at destination.

Change-Id: I65e17a58fc778a1f2ce3546d03d69f8827c64d2a
Closes-Bug: #1687879

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/32223
Committed: http://github.com/Juniper/contrail-controller/commit/c1f80a42ad5b4d23f0a0b044f371a457788be814
Submitter: Zuul (<email address hidden>)
Branch: master

commit c1f80a42ad5b4d23f0a0b044f371a457788be814
Author: Praveen K V <email address hidden>
Date: Fri May 26 12:12:30 2017 +0530

Add vhost ip-address to ECMP hash

In case of flows from one compute to another, the 5-tuple used to
compute hash is same on both compute nodes. At source, hash is computed
on hash of 5-tuple and a remote compute is selected based on hash.
When flow reaches destination, hash value for flow is computed on same key
and results in same value. At destination, the entropy for hash is
significantly reduced (since only flows with specific hash pattern reach
destination). This results in either skewed distribution or in some
cases flows not selecting some interfaces

Example:
Consider source has 2 member composite-nh (destined to dest-1 and dest-2).
All flows reaching dest-1 will have even-hash value and all flows reaching
dest-2 will have odd hash-value. On dest-1, the flows will only go to
even members on local composite-nh and on dest-2, the flows will only go
to odd members of local composite-nh.

Fix:
Use vhost-ip on compute node for hash computation. This will result in
diffrent hash value on source and destination. This results in better
distribution at destination.

Change-Id: I65e17a58fc778a1f2ce3546d03d69f8827c64d2a
Closes-Bug: #1687879

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.2

Review in progress for https://review.opencontrail.org/32473
Submitter: Hari Prasad Killi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/32473
Committed: http://github.com/Juniper/contrail-controller/commit/7813ea9ef99c5dea11aecbc8ea451a3f54b13ed8
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit 7813ea9ef99c5dea11aecbc8ea451a3f54b13ed8
Author: Praveen K V <email address hidden>
Date: Fri May 26 12:12:30 2017 +0530

Add vhost ip-address to ECMP hash

In case of flows from one compute to another, the 5-tuple used to
compute hash is same on both compute nodes. At source, hash is computed
on hash of 5-tuple and a remote compute is selected based on hash.
When flow reaches destination, hash value for flow is computed on same key
and results in same value. At destination, the entropy for hash is
significantly reduced (since only flows with specific hash pattern reach
destination). This results in either skewed distribution or in some
cases flows not selecting some interfaces

Example:
Consider source has 2 member composite-nh (destined to dest-1 and dest-2).
All flows reaching dest-1 will have even-hash value and all flows reaching
dest-2 will have odd hash-value. On dest-1, the flows will only go to
even members on local composite-nh and on dest-2, the flows will only go
to odd members of local composite-nh.

Fix:
Use vhost-ip on compute node for hash computation. This will result in
diffrent hash value on source and destination. This results in better
distribution at destination.

Conflicts:
 src/vnsw/agent/pkt/test/SConscript
 src/vnsw/agent/pkt/test/test_pkt_parse.cc

Change-Id: I65e17a58fc778a1f2ce3546d03d69f8827c64d2a
Closes-Bug: #1687879

tags: added: dt
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.