Juniper Openstack

when a large burst of udp flows are generated, some flows are getting into Hold state

Bug #1539403 reported by Vedamurthy Joshi
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R3.0
Fix Committed
High
Praveen
Juniper Openstack r3.0.1.0
Trunk
Fix Committed
High
Praveen
Juniper Openstack r3.1.0.0-fcs

Bug Description

R3.0 Build 2704 Ubuntu 14.04 Kilo multi-node setup

Using the flow_test_pktgen.sh , generated 20K udp packets in 1 second.

export PPS=20000; bash -x flow_test_pktgen.sh 10.1.1.4 10.1.1.4 10.1.1.3 10000 30000 20000 02:76:aa:8c:4c:29

About 100-300 flows are getting into permanent Hold state.
Reproducible every time
Ex :

     7208<=>463328 10.1.1.3:16290 17 (1)
                         10.1.1.4:9
    (K(nh):14, Action:F, Flags:, S(nh):20, Statistics:0/0 UdpSrcPort 65062

     7209 10.1.1.4:9 17 (1)
                         10.1.1.3:25206
    (K(nh):14, Action:H, Flags:, S(nh):0, Statistics:1/60 UdpSrcPort 0

Revision history for this message
Vedamurthy Joshi (vedujoshi) wrote :

gcore is in http://10.204.216.50/Docs/bugs/1539403

Revision history for this message
Vedamurthy Joshi (vedujoshi) wrote :

     1340 10.1.1.4:9 17 (1)
                         10.1.1.3:11171
    (K(nh):21, Action:H, Flags:, S(nh):0, Statistics:1/60 UdpSrcPort 0

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/16736
Submitter: Praveen K V (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/16736
Committed: http://github.org/Juniper/contrail-controller/commit/bd365c40cfebef20313fefae9289cc8e235bd499
Submitter: Zuul
Branch: master

commit bd365c40cfebef20313fefae9289cc8e235bd499
Author: Praveen K V <email address hidden>
Date: Mon Feb 1 11:12:47 2016 +0530

Initialize vrouter_evicted_ flag on allocate

1. Initailie vrouter_evicted_ flag on allocated. The un-initialized
value was resulting in flows not being writtedn to vrouter
2. Update Flow debug stats to include audit and vrouter-error counters

Change-Id: I0e2d5ce80893d9d75db8b1e53bbb4affa4de508d
Closes-Bug: #1539403

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/16960
Submitter: Praveen K V (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/18374
Submitter: Praveen K V (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/18375
Submitter: Praveen K V (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/18375
Committed: http://github.org/Juniper/contrail-controller/commit/7ee026edafc7920f051c3194303b78a0b435c241
Submitter: Zuul
Branch: master

commit 7ee026edafc7920f051c3194303b78a0b435c241
Author: Praveen K V <email address hidden>
Date: Mon Mar 14 11:18:14 2016 +0530

Fix KSYNC_FREE event handling in INDEX_UNASSIGN state

A flow in INDEX_UNASSIGN state means, flow is reused and is waiting for
old-ksync to be freed.

Due to bug in handling KSYNC_FREE event, the flow is not added if it
already has flow-handle assigned.

Modify the KSYNC_FREE event handling as follows,
- If flow-handle is already known
- acquire the index
- write to KSync
- If flow is deleted, do an immediate delete
- If flow-handle is not known
- If flow is deleted, move to INIT state as nothing to be done
- Invoke KSync write to allocate index

Change-Id: Iedf1252b9816fdd79e1fb5e4f78de7091b2c2424
Closes-Bug: #1539403

Changed in juniperopenstack:
milestone: r3.0-fcs → r3.1.0.0-fcs
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/18374
Committed: http://github.org/Juniper/contrail-controller/commit/3d84ac7b0918f07173b816cd1c7dbef60d6b99ac
Submitter: Zuul
Branch: R3.0

commit 3d84ac7b0918f07173b816cd1c7dbef60d6b99ac
Author: Praveen K V <email address hidden>
Date: Mon Mar 14 11:18:14 2016 +0530

Fix KSYNC_FREE event handling in INDEX_UNASSIGN state

A flow in INDEX_UNASSIGN state means, flow is reused and is waiting for
old-ksync to be freed.

Due to bug in handling KSYNC_FREE event, the flow is not added if it
already has flow-handle assigned.

Modify the KSYNC_FREE event handling as follows,
- If flow-handle is already known
- acquire the index
- write to KSync
- If flow is deleted, do an immediate delete
- If flow-handle is not known
- If flow is deleted, move to INIT state as nothing to be done
- Invoke KSync write to allocate index

Change-Id: Iedf1252b9816fdd79e1fb5e4f78de7091b2c2424
Closes-Bug: #1539403

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.