some flows stuck in vrouter forever

Bug #1569967 reported by Vedamurthy Joshi
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R3.0
Fix Committed
High
Prabhjot Singh Sethi
Trunk
Fix Committed
High
Prabhjot Singh Sethi

Bug Description

R3.0.2.0 Build 26 Ubuntu 14.04 Kilo

Between 10.1.1.3 and 10.1.1.4, was sending scapy traffic as below.
After few mins, it was stopped.

send(fuzz(IP(dst='10.1.1.3'))/fuzz(TCP())/Raw(RandString(size=100)), loop=1)

Later, it was seen that few flows were stuck in vrouter forever and not aged out

Many more flows were not aged out in agent as well

On nodel7, below flow without a reverse flow was stuck forever
==========

    Index Source:Port/Destination:Port Proto(V)
-----------------------------------------------------------------------------------
   817364<=>4193380 10.1.1.3:14283 6 (1)
                         10.1.1.4:2926
(Gen: 6, K(nh):16, Action:F, Flags:, TCP:, S(nh):22, Stats:0/0, SPort 57301)

On nodel4, similar thing :

    Index Source:Port/Destination:Port Proto(V)
-----------------------------------------------------------------------------------
  2137948<=>3118804 10.1.1.3:2165 6 (1)
                         10.1.1.4:2174
(Gen: 5, K(nh):16, Action:F, Flags:, TCP:, S(nh):16, Stats:0/0, SPort 49530)

  2884188<=>1342742 10.1.1.4:42181 6 (1)
                         10.1.1.3:2137
(Gen: 2, K(nh):16, Action:F, Flags:, TCP:, S(nh):22, Stats:0/0, SPort 61719)

-----------------

gcore of agent is in nodel7:/root/gcore.13606.contrail-vrouter-agent

Tags: vrouter
summary: - some flows stuck in agent and vrouter forever
+ some flows stuck in vrouter forever
Revision history for this message
Vedamurthy Joshi (vedujoshi) wrote :

Even without fuzzed scapy traffic, long term hping port-scan tests also seem to be resulting in this :

  4186620<=>3691752 10.1.1.64:22 6 (1)
                         10.1.1.17:6202
(Gen: 156, K(nh):63, Action:F, Flags:, TCP:Sr, S(nh):63, Stats:0/0, SPort 55361)

  4187784<=>2991652 10.1.1.76:22 6 (1)
                         10.1.1.59:6091
(Gen: 126, K(nh):78, Action:D(NoDstRt), Flags:, TCP:Sr, S(nh):2, Stats:0/0, SPort 50050)

  4189732<=>4004520 10.1.1.76:22 6 (1)
                         10.1.1.61:3706
(Gen: 113, K(nh):58, Action:F, Flags:, TCP:Sr, S(nh):51, Stats:0/0, SPort 62374)

  4190812<=>1732228 10.1.1.76:22 6 (1)
                         10.1.1.59:4999
(Gen: 114, K(nh):78, Action:F, Flags:, TCP:Sr, S(nh):51, Stats:0/0, SPort 54255)

  4192020<=>2778592 10.1.1.76:22 6 (1)
                         10.1.1.61:4751
(Gen: 124, K(nh):58, Action:D(NoDstRt), Flags:, TCP:Sr, S(nh):2, Stats:0/0, SPort 54154)

  4192500<=>3842844 10.1.1.79:22 6 (1)
                         10.1.1.60:4311
(Gen: 108, K(nh):16, Action:F, Flags:, TCP:Sr, S(nh):51, Stats:0/0, SPort 61081)

  4196424<=>3064736 10.1.1.59:22 6 (1)
                         10.1.1.20:5430
(Gen: 67, K(nh):78, Action:F, Flags:, TCP:Sr, S(nh):78, Stats:0/0, SPort 56493)

  4199000<=>2231804 10.1.1.94:22 6 (1)
                         10.1.1.62:5942
(Gen: 125, K(nh):83, Action:F, Flags:, TCP:Sr, S(nh):53, Stats:0/0, SPort 64109)

Changed in juniperopenstack:
milestone: r3.0.1.0 → none
Revision history for this message
Prabhjot Singh Sethi (prabhjot) wrote :

issue happened where delete message for reverse flow was not sent if flow was deleted before getting an Ack with index from vrouter.

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/19437
Submitter: Prabhjot Singh Sethi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/19438
Submitter: Prabhjot Singh Sethi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/19438
Committed: http://github.org/Juniper/contrail-controller/commit/26e0c9e060ff12b86cf48b179d60091d87fc11be
Submitter: Zuul
Branch: R3.0

commit 26e0c9e060ff12b86cf48b179d60091d87fc11be
Author: Prabhjot Singh Sethi <email address hidden>
Date: Tue Apr 19 15:48:48 2016 +0530

Fix to delete flow from vrouter appropriately

Issue:
------
If a delete is triggered for reverse flow before getting
flow handle allocated, it was trying to acquire index with
index and gen_id info from flow entry instead of KSyncEntry
Since the flow is is already marked deleted or reused, its
index and gen id will not match with the one allocated to
KSyncEntry, this results in false evaluation as if the flow
is already Evicted, resulting in skipping Delete Message to
Vrouter

Fix:
----
Acquire Index using the KSyncEntry hash id and gen id, while
Ksync entry is already deleted.
Added Event Log for message skip for evicted flow

Closes-Bug: 1569967
Change-Id: Ib3f761d05e3d522905b012184dc8528bb0d5eba3
(cherry picked from commit c6bc5d93bc28dc20ac3e8cfb75405fd58f74247e)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/19437
Committed: http://github.org/Juniper/contrail-controller/commit/c6bc5d93bc28dc20ac3e8cfb75405fd58f74247e
Submitter: Zuul
Branch: master

commit c6bc5d93bc28dc20ac3e8cfb75405fd58f74247e
Author: Prabhjot Singh Sethi <email address hidden>
Date: Tue Apr 19 15:48:48 2016 +0530

Fix to delete flow from vrouter appropriately

Issue:
------
If a delete is triggered for reverse flow before getting
flow handle allocated, it was trying to acquire index with
index and gen_id info from flow entry instead of KSyncEntry
Since the flow is is already marked deleted or reused, its
index and gen id will not match with the one allocated to
KSyncEntry, this results in false evaluation as if the flow
is already Evicted, resulting in skipping Delete Message to
Vrouter

Fix:
----
Acquire Index using the KSyncEntry hash id and gen id, while
Ksync entry is already deleted.
Added Event Log for message skip for evicted flow

Closes-Bug: 1569967
Change-Id: Ib3f761d05e3d522905b012184dc8528bb0d5eba3

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.