Juniper Openstack

Intra-VN flow creation gets affected when a policy is bound/unbound from the VN

Bug #1572270 reported by Vedamurthy Joshi
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R3.0
Fix Committed
High
Praveen
Juniper Openstack r3.0.2.0
Trunk
Fix Committed
High
Praveen

Bug Description

R3.0.2.0 Build 26 Ubuntu 14.04 Kilo

10.1.1.4 is constantly creating tcp connections to 10.1.1.10 using hping3 inside a VN net1 10.1.1.0/24
A policy with below rule is applied to the VN net1.
During that time, it is seen that tcp rtt times reach upto about 400ms (as against <10ms normally)
hping3 also reported packet losses, but i wasnt able to map it easily with drops from the dropstats cmd

Similar issue was seen when the policy is removed from the VN .

pass log protocol any network net1 ports any <> network frontend1 (default-domain:Vedu) ports any

hping3 cmd : hping3 -p 22 -S 10.1.1.10 -i u1000 | grep -v "rtt=1\." | grep -v "rtt=0\." | grep -v "rtt=2\."
Ex hping3 output snippet :

len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=13089 win=14600 rtt=426.3 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=13090 win=14600 rtt=425.4 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=13091 win=14600 rtt=424.4 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=13092 win=14600 rtt=423.4 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=13093 win=14600 rtt=422.5 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=13094 win=14600 rtt=421.5 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=13095 win=14600 rtt=429.4 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=13096 win=14600 rtt=428.3 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=13097 win=14600 rtt=427.3 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=13098 win=14600 rtt=426.3 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=13099 win=14600 rtt=434.4 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=13100 win=14600 rtt=433.4 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=13101 win=14600 rtt=432.4 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=13102 win=14600 rtt=431.5 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=13103 win=14600 rtt=430.5 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=13104 win=14600 rtt=430.2 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=13105 win=14600 rtt=429.3 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=13106 win=14600 rtt=428.3 ms
len=44 ip=10.1.1.10 ttl=64 DF id=0 sport=22 flags=SA seq=13107 win=14600 rtt=427.4 ms
^C
--- 10.1.1.10 hping statistic ---
15529 packets transmitted, 14366 packets received, 8% packet loss
round-trip min/avg/max = 0.7/40.9/1365.9 ms

ubuntu@net1-vm2:~$

Tags: vrouter
Hari Prasad Killi (haripk)
Changed in juniperopenstack:
assignee: Hari Prasad Killi (haripk) → Praveen (praveen-karadakal)
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/19731
Submitter: Praveen K V (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Review in progress for https://review.opencontrail.org/19772
Submitter: Praveen K V (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/19731
Committed: http://github.org/Juniper/contrail-controller/commit/463f1cb921d958dc11792514b6f255412d95d6a4
Submitter: Zuul
Branch: R3.0

commit 463f1cb921d958dc11792514b6f255412d95d6a4
Author: Praveen K V <email address hidden>
Date: Fri Apr 29 15:58:11 2016 +0530

Unify flow-update handling

The current code handles flow-update resulting from add/change for
DBEntries (interface, acl, vn, nh) as different cases. However, there is
only a minor difference between handling based on different DBEntries.

This commit unifies the code to handle DBEntry add/change into two
cases,
- Flows need re-valuation. This is case where we only do
policy-lookup and identify new actions
- Flows needing re-computation. This is case where there is route
change and flow need complete re-valuation due to add/delete of
routes

Additionally, two flags are added in flow to ensure state-compression.

Change-Id: Ifeb696d2ab651f656fbafd45e5e66a52adb65ade
Partial-Bug: #1568126
Partial-Bug: #1572270
Partial-Bug: #1572471

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/19772
Submitter: Praveen K V (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/19772
Committed: http://github.org/Juniper/contrail-controller/commit/cf98c2115d76ccc482850265ed8cbb0fa6e41a48
Submitter: Zuul
Branch: R3.0

commit cf98c2115d76ccc482850265ed8cbb0fa6e41a48
Author: Praveen K V <email address hidden>
Date: Sat Apr 30 12:52:36 2016 +0530

Enable flow-tests

Split test_pkt_flow into different files,
- test_flow_add.cc
- test_flow_nat.cc
- test_flow_error.cc
- test_flow_policy.cc
- test_flow_update.cc

Add UT for flow-update handling

Change-Id: I3abec2ed0e314dd584230dc3631c016b6ba34202
Partial-Bug: #1568126
Partial-Bug: #1572270
Partial-Bug: #1572471

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/19837
Submitter: Praveen K V (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/19837
Committed: http://github.org/Juniper/contrail-controller/commit/7dd85a94fc8070d2d702c1ad7d616d935419a3fb
Submitter: Zuul
Branch: R3.0

commit 7dd85a94fc8070d2d702c1ad7d616d935419a3fb
Author: Praveen K V <email address hidden>
Date: Mon May 2 14:56:14 2016 +0530

Fix wrong token used for flow-updates

- Flow-Update operations were using add-token instead of update-tokens
- Minor clean-ups

Change-Id: I0f36d549c240d2417a1b5914faab836119cd4870
Partial-Bug: #1568126
Partial-Bug: #1572270
Partial-Bug: #1572471

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/19937
Submitter: Praveen K V (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/19937
Committed: http://github.org/Juniper/contrail-controller/commit/333d3e4454ae0e851b93b7aed2687905b22c14cb
Submitter: Zuul
Branch: R3.0

commit 333d3e4454ae0e851b93b7aed2687905b22c14cb
Author: Praveen K V <email address hidden>
Date: Thu May 5 12:49:46 2016 +0530

Define new class for FlowEvent queues

1. Define new class for FlowEvent queues. The new class makes it easier to
add common functionality needed for all flow-event queues
2. Added functionality to log message if time taken for single run of a
queue exceeds a threshold
3. Fix crash in UpdateStats. When FlowEvent is enqueued to work-queue,
it can be processed and released before we hit UpdateStats. Ensure
that UpdateStats is invoked before enqueuing FlowStats

Change-Id: I7fa217b1b200cb01202e7731d792318ea88f3f91
Closes-Bug: #1578660
Partial-Bug: #1568126
Partial-Bug: #1572270
Partial-Bug: #1572471

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/20087
Submitter: Praveen K V (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Review in progress for https://review.opencontrail.org/20100
Submitter: Praveen K V (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/20087
Committed: http://github.org/Juniper/contrail-controller/commit/145db6eebcf72fa07df0ce7c04992bcd2a72acbc
Submitter: Zuul
Branch: R3.0

commit 145db6eebcf72fa07df0ce7c04992bcd2a72acbc
Author: Praveen K V <email address hidden>
Date: Mon May 9 23:12:57 2016 +0530

Support state-compression for flow delete events

Move the PendingAction state-compression logic to FlowEvent handler
module.
Rename REVALUATE_FLOW to RECOPMUTE_FLOW
Change FLOW_MESSAGE event to have FlowEntry as member
Add UT for state-compression of delete events

Change-Id: I5480c02c27385e1fa8c63820da1a09b24b7898bb
Closes-Bug: #1578660
Partial-Bug: #1568126
Partial-Bug: #1572270
Partial-Bug: #1572471

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/20100
Committed: http://github.org/Juniper/contrail-controller/commit/c27138827fc77fc995d5ba24de205c1514deccfd
Submitter: Zuul
Branch: R3.0

commit c27138827fc77fc995d5ba24de205c1514deccfd
Author: Praveen K V <email address hidden>
Date: Wed May 11 16:26:24 2016 +0530

Make flow tokens tunable

Add new token for ksync event queues

Change-Id: I5ed836f054f8ae305fa08756dbe481df9e455061
Partial-Bug: #1568126
Partial-Bug: #1572270
Partial-Bug: #1572471

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/20226
Submitter: Praveen K V (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Review in progress for https://review.opencontrail.org/20227
Submitter: Praveen K V (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Review in progress for https://review.opencontrail.org/20228
Submitter: Praveen K V (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Review in progress for https://review.opencontrail.org/20229
Submitter: Praveen K V (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Review in progress for https://review.opencontrail.org/20230
Submitter: Praveen K V (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Review in progress for https://review.opencontrail.org/20231
Submitter: Praveen K V (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/20226
Committed: http://github.org/Juniper/contrail-controller/commit/eb7e5a11b3334d1d58ca8b0cd8103a31d6b49bea
Submitter: Zuul
Branch: master

commit eb7e5a11b3334d1d58ca8b0cd8103a31d6b49bea
Author: Praveen K V <email address hidden>
Date: Fri Apr 29 15:58:11 2016 +0530

Unify flow-update handling

The current code handles flow-update resulting from add/change for
DBEntries (interface, acl, vn, nh) as different cases. However, there is
only a minor difference between handling based on different DBEntries.

This commit unifies the code to handle DBEntry add/change into two
cases,
- Flows need re-valuation. This is case where we only do
policy-lookup and identify new actions
- Flows needing re-computation. This is case where there is route
change and flow need complete re-valuation due to add/delete of
routes

Additionally, two flags are added in flow to ensure state-compression.

Conflicts:
 src/vnsw/agent/pkt/flow_entry.cc

Change-Id: Ifeb696d2ab651f656fbafd45e5e66a52adb65ade
Partial-Bug: #1568126
Partial-Bug: #1572270
Partial-Bug: #1572471

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/20227
Committed: http://github.org/Juniper/contrail-controller/commit/df1c747717e6e0417ab5b98fa34497d0f7ed3ec1
Submitter: Zuul
Branch: master

commit df1c747717e6e0417ab5b98fa34497d0f7ed3ec1
Author: Praveen K V <email address hidden>
Date: Sat Apr 30 12:52:36 2016 +0530

Enable flow-tests

Split test_pkt_flow into different files,
- test_flow_add.cc
- test_flow_nat.cc
- test_flow_error.cc
- test_flow_policy.cc
- test_flow_update.cc

Add UT for flow-update handling

Change-Id: I3abec2ed0e314dd584230dc3631c016b6ba34202
Partial-Bug: #1568126
Partial-Bug: #1572270
Partial-Bug: #1572471

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/20228
Committed: http://github.org/Juniper/contrail-controller/commit/feaf4cf85070e5daa37fd4b2aac5504dd49c9dcd
Submitter: Zuul
Branch: master

commit feaf4cf85070e5daa37fd4b2aac5504dd49c9dcd
Author: Praveen K V <email address hidden>
Date: Mon May 2 14:56:14 2016 +0530

Fix wrong token used for flow-updates

- Flow-Update operations were using add-token instead of update-tokens
- Minor clean-ups

Change-Id: I0f36d549c240d2417a1b5914faab836119cd4870
Partial-Bug: #1568126
Partial-Bug: #1572270
Partial-Bug: #1572471

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/20229
Committed: http://github.org/Juniper/contrail-controller/commit/89a2b24d7fd3276b1408ac3d5c52f8569be89e9d
Submitter: Zuul
Branch: master

commit 89a2b24d7fd3276b1408ac3d5c52f8569be89e9d
Author: Praveen K V <email address hidden>
Date: Thu May 5 12:49:46 2016 +0530

Define new class for FlowEvent queues

1. Define new class for FlowEvent queues. The new class makes it easier to
add common functionality needed for all flow-event queues
2. Added functionality to log message if time taken for single run of a
queue exceeds a threshold
3. Fix crash in UpdateStats. When FlowEvent is enqueued to work-queue,
it can be processed and released before we hit UpdateStats. Ensure
that UpdateStats is invoked before enqueuing FlowStats

Conflicts:
 src/vnsw/agent/pkt/flow_proto.h

Change-Id: I7fa217b1b200cb01202e7731d792318ea88f3f91
Closes-Bug: #1578660
Partial-Bug: #1568126
Partial-Bug: #1572270
Partial-Bug: #1572471

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/20230
Committed: http://github.org/Juniper/contrail-controller/commit/951cc987316570db3ba90019219182a7cb8e5fd9
Submitter: Zuul
Branch: master

commit 951cc987316570db3ba90019219182a7cb8e5fd9
Author: Praveen K V <email address hidden>
Date: Mon May 9 23:12:57 2016 +0530

Support state-compression for flow delete events

Move the PendingAction state-compression logic to FlowEvent handler
module.
Rename REVALUATE_FLOW to RECOPMUTE_FLOW
Change FLOW_MESSAGE event to have FlowEntry as member
Add UT for state-compression of delete events

Conflicts:
 src/vnsw/agent/pkt/flow_proto.h
 src/vnsw/agent/vrouter/flow_stats/flow_stats_collector.cc

Change-Id: I5480c02c27385e1fa8c63820da1a09b24b7898bb
Closes-Bug: #1578660
Partial-Bug: #1568126
Partial-Bug: #1572270
Partial-Bug: #1572471

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/20231
Committed: http://github.org/Juniper/contrail-controller/commit/33869fc03d881f343322a7308a5b8a336d5104da
Submitter: Zuul
Branch: master

commit 33869fc03d881f343322a7308a5b8a336d5104da
Author: Praveen K V <email address hidden>
Date: Wed May 11 16:26:24 2016 +0530

Make flow tokens tunable

Add new token for ksync event queues

Change-Id: I5ed836f054f8ae305fa08756dbe481df9e455061
Partial-Bug: #1568126
Partial-Bug: #1572270
Partial-Bug: #1572471

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/20468
Submitter: Praveen K V (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/20468
Committed: http://github.org/Juniper/contrail-controller/commit/61c4c3044aeb21a3199c2980873474cc95941695
Submitter: Zuul
Branch: R3.0

commit 61c4c3044aeb21a3199c2980873474cc95941695
Author: Praveen K V <email address hidden>
Date: Fri May 20 23:49:35 2016 +0530

Limit number of entries visited per task for ageing

Problem:
Flow ageing currently fires a timer every 50msec. As the flow-table size
increases, the number of entries to visit per timer also increases. This can
lead to large latencies on firing of a timer. It is desirable to limit the
latency

Solution:
When timer is fired, ageing algorithm compute number of entries to visit.
Instead of scanning flow table in context of timer, it starts a task and
visits 256 entries for every run of task. Task continuation is used if there
are more entries pending on completion of task.

Change-Id: Icb10bec94d2923442c453c2b4071ea7ab4d9d702
Partial-Bug: #1568126
Partial-Bug: #1572270
Partial-Bug: #1572471

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/20976
Submitter: Praveen K V (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/20976
Committed: http://github.org/Juniper/contrail-controller/commit/8acd4ead47d98e3f91f35881c20c422330b2044b
Submitter: Zuul
Branch: master

commit 8acd4ead47d98e3f91f35881c20c422330b2044b
Author: Praveen K V <email address hidden>
Date: Fri May 20 23:49:35 2016 +0530

Limit number of entries visited per task for ageing

Problem:
Flow ageing currently fires a timer every 50msec. As the flow-table size
increases, the number of entries to visit per timer also increases. This can
lead to large latencies on firing of a timer. It is desirable to limit the
latency

Solution:
When timer is fired, ageing algorithm compute number of entries to visit.
Instead of scanning flow table in context of timer, it starts a task and
visits 256 entries for every run of task. Task continuation is used if there
are more entries pending on completion of task.

Change-Id: Icb10bec94d2923442c453c2b4071ea7ab4d9d702
Partial-Bug: #1568126
Partial-Bug: #1572270
Partial-Bug: #1572471

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.