Juniper Openstack

Agent crash due to parallel access

Bug #1726709 reported by Hari Prasad Killi
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R3.0
Fix Committed
High
Hari Prasad Killi
Juniper Openstack r3.0.4.0
R3.0.3.x
Fix Committed
High
Hari Prasad Killi
Juniper Openstack r3.0.3.6
R3.1
Fix Committed
High
Hari Prasad Killi
Juniper Openstack r3.1.4.0
R3.2
Fix Committed
High
Hari Prasad Killi
Juniper Openstack r3.2.7.0
R3.2.3.x
Fix Committed
High
Hari Prasad Killi
Juniper Openstack r3.2.3.3
R4.0
Fix Committed
High
Hari Prasad Killi
Juniper Openstack r4.0.3.0
R4.1
Fix Committed
High
Hari Prasad Killi
Juniper Openstack r4.1.0.0-fcs "r4.1"
Trunk
Fix Committed
High
Hari Prasad Killi
Juniper Openstack r5.0.0

Bug Description

(gdb) bt
#0 next_node (node=<synthetic pointer>) at /usr/include/boost/intrusive/detail/tree_algorithms.hpp:446
#1 next_node (p=<synthetic pointer>) at /usr/include/boost/intrusive/rbtree_algorithms.hpp:352
#2 operator++ (this=<synthetic pointer>) at /usr/include/boost/intrusive/detail/tree_node.hpp:135
#3 KSyncObject::BackRefReEval (this=this@entry=0x7fc0a001ccc8, key=key@entry=0x7fc0a0542de8) at controller/src/ksync/ksync_object.cc:1497
#4 0x0000000000ed256b in KSyncObject::NotifyEvent (this=0x7fc0a001ccc8, entry=0x7fc0a0542de8, event=<optimized out>) at controller/src/ksync/ksync_object.cc:1402
#5 0x0000000000ed272d in KSyncObject::SafeNotifyEvent (this=0x7fc0a001ccc8, entry=0x7fc0a0542de8, event=KSyncEntry::ADD_CHANGE_REQ) at controller/src/ksync/ksync_object.cc:277
#6 0x0000000000ed2777 in KSyncObject::Change (this=<optimized out>, entry=<optimized out>) at controller/src/ksync/ksync_object.cc:228
#7 0x0000000000dbc0af in KSyncFlowIndexManager::Update (this=0x7fc0a00165f8, flow=0x7fc0a1efb388) at controller/src/vnsw/agent/vrouter/ksync/ksync_flow_index_manager.cc:76
#8 0x0000000000cebfdc in FlowTable::AddInternal (this=this@entry=0x7fc0a066d368, flow_req=flow_req@entry=0x7fc0a24f5db8, flow=flow@entry=0x7fc0a1efb388,
    rflow_req=rflow_req@entry=0x7fc0a0cc2098, rflow=<optimized out>, rflow@entry=0x7fc0a105bca8, fwd_flow_update=fwd_flow_update@entry=true,
    rev_flow_update=rev_flow_update@entry=true) at controller/src/vnsw/agent/pkt/flow_table.cc:300
#9 0x0000000000cec448 in FlowTable::Update (this=0x7fc0a066d368, flow=flow@entry=0x7fc0a24f5db8, rflow=0x7fc0a0cc2098) at controller/src/vnsw/agent/pkt/flow_table.cc:184
#10 0x0000000000d02e54 in FlowProto::UpdateFlow (this=<optimized out>, flow=flow@entry=0x7fc0a24f5db8) at controller/src/vnsw/agent/pkt/flow_proto.cc:305
#11 0x0000000000d132f2 in PktFlowInfo::Add (this=this@entry=0x7fc0b5a968b0, pkt=<optimized out>, in=in@entry=0x7fc0b5a96830, out=out@entry=0x7fc0b5a96870)
    at controller/src/vnsw/agent/pkt/pkt_flow_info.cc:1750
#12 0x0000000000d25ae5 in FlowHandler::Run (this=0x7fc0a4375e18) at controller/src/vnsw/agent/pkt/flow_handler.cc:150
#13 0x0000000000d1ef7d in Proto::RunProtoHandler (this=this@entry=0x7fc0a066c918, handler=handler@entry=0x7fc0a4375e18) at controller/src/vnsw/agent/pkt/proto.cc:51
#14 0x0000000000d031a1 in FlowProto::FlowEventHandler (this=0x7fc0a066c918, req=<optimized out>, table=<optimized out>) at controller/src/vnsw/agent/pkt/flow_proto.cc:429
#15 0x0000000000d22e5d in FlowEventQueueBase::Handler (this=0x7fc0a2617628, event=0x7fc0b09bd498) at controller/src/vnsw/agent/pkt/flow_event.cc:105
#16 0x0000000000d081d2 in operator() (a0=0x7fc0b09bd498, this=0x7fc0b5a96b30) at /usr/include/boost/function/function_template.hpp:767
#17 QueueTaskRunner<FlowEvent*, WorkQueue<FlowEvent*> >::RunQueue (this=0x7fc0b03f5458) at controller/src/base/queue_task.h:92
#18 0x0000000001305d6f in TaskImpl::execute (this=0x7fc0b7455b40) at controller/src/base/task.cc:262

(gdb) p back_ref_tree_
$1 = boost::intrusive::set<KSyncBackReference> with 0 elements

back_ref_tree_ is updated while iteration is in progress.

Hari Prasad Killi (haripk)
Changed in juniperopenstack:
assignee: nobody → Hari Prasad Killi (haripk)
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.2

Review in progress for https://review.opencontrail.org/36760
Submitter: Hari Prasad Killi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.1

Review in progress for https://review.opencontrail.org/36761
Submitter: Hari Prasad Killi (<email address hidden>)

Jim Reilly (jpreilly)
information type: Public → Private
tags: added: att-aic-contrail
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.1

Review in progress for https://review.opencontrail.org/36768
Submitter: Hari Prasad Killi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/36760
Committed: http://github.com/Juniper/contrail-controller/commit/ef13c6b5ec0250df8bf7be0241a240c799e6b86f
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit ef13c6b5ec0250df8bf7be0241a240c799e6b86f
Author: Hari Prasad Killi <email address hidden>
Date: Tue Oct 24 20:35:17 2017 +0530

Do not re-evaluate backrefs for ksync flow entries

The backref entries are updated in DB and ksync tasks, which are mutually
exclusive. The flow event task was accessing the same and it doesnt actually
require. Avoiding backref reevaluation.

Change-Id: I8cb9b92196626ea219f7f9e1055e2579c62c7c86
closes-bug: #1726709

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.0

Review in progress for https://review.opencontrail.org/36813
Submitter: Hari Prasad Killi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/36822
Submitter: Hari Prasad Killi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/36768
Committed: http://github.com/Juniper/contrail-controller/commit/849be7c017bf13b7c054d0cc5b436a6a87c3b0ed
Submitter: Zuul (<email address hidden>)
Branch: R3.1

commit 849be7c017bf13b7c054d0cc5b436a6a87c3b0ed
Author: Hari Prasad Killi <email address hidden>
Date: Tue Oct 24 20:35:17 2017 +0530

Do not re-evaluate backrefs for ksync flow entries

The backref entries are updated in DB and ksync tasks, which are mutually
exclusive. The flow event task was accessing the same and it doesnt actually
require. Avoiding backref reevaluation.

Change-Id: I8cb9b92196626ea219f7f9e1055e2579c62c7c86
closes-bug: #1726709

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/36761
Committed: http://github.com/Juniper/contrail-controller/commit/ae7bb5713543b07d0e2d0aa0d1c9b8d2d99fe4b3
Submitter: Zuul (<email address hidden>)
Branch: R4.1

commit ae7bb5713543b07d0e2d0aa0d1c9b8d2d99fe4b3
Author: Hari Prasad Killi <email address hidden>
Date: Tue Oct 24 20:35:17 2017 +0530

Do not re-evaluate backrefs for ksync flow entries

The backref entries are updated in DB and ksync tasks, which are mutually
exclusive. The flow event task was accessing the same and it doesnt actually
require. Avoiding backref reevaluation.

Change-Id: I8cb9b92196626ea219f7f9e1055e2579c62c7c86
closes-bug: #1726709

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/36822
Committed: http://github.com/Juniper/contrail-controller/commit/46c9d4683f5a5d75a3271c395980a8dc2e64dba6
Submitter: Zuul (<email address hidden>)
Branch: master

commit 46c9d4683f5a5d75a3271c395980a8dc2e64dba6
Author: Hari Prasad Killi <email address hidden>
Date: Tue Oct 24 20:35:17 2017 +0530

Do not re-evaluate backrefs for ksync flow entries

The backref entries are updated in DB and ksync tasks, which are mutually
exclusive. The flow event task was accessing the same and it doesnt actually
require. Avoiding backref reevaluation.

Change-Id: I8cb9b92196626ea219f7f9e1055e2579c62c7c86
closes-bug: #1726709

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/36813
Committed: http://github.com/Juniper/contrail-controller/commit/7e1054b88f43183ae484a52574996adc83556f2a
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit 7e1054b88f43183ae484a52574996adc83556f2a
Author: Hari Prasad Killi <email address hidden>
Date: Tue Oct 24 20:35:17 2017 +0530

Do not re-evaluate backrefs for ksync flow entries

The backref entries are updated in DB and ksync tasks, which are mutually
exclusive. The flow event task was accessing the same and it doesnt actually
require. Avoiding backref reevaluation.

Change-Id: I8cb9b92196626ea219f7f9e1055e2579c62c7c86
closes-bug: #1726709

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/37773
Submitter: Hari Prasad Killi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0.3.x

Review in progress for https://review.opencontrail.org/37774
Submitter: Hari Prasad Killi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/37774
Committed: http://github.com/Juniper/contrail-controller/commit/745426628a50197b76a057f7012e35d18d50d233
Submitter: Zuul (<email address hidden>)
Branch: R3.0.3.x

commit 745426628a50197b76a057f7012e35d18d50d233
Author: Hari Prasad Killi <email address hidden>
Date: Tue Oct 24 20:35:17 2017 +0530

Do not re-evaluate backrefs for ksync flow entries

The backref entries are updated in DB and ksync tasks, which are mutually
exclusive. The flow event task was accessing the same and it doesnt actually
require. Avoiding backref reevaluation.

Change-Id: I8cb9b92196626ea219f7f9e1055e2579c62c7c86
closes-bug: #1726709
(cherry picked from commit 849be7c017bf13b7c054d0cc5b436a6a87c3b0ed)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/37773
Committed: http://github.com/Juniper/contrail-controller/commit/73a4537cacc504d79727709764bbd93d448de429
Submitter: Zuul (<email address hidden>)
Branch: R3.0

commit 73a4537cacc504d79727709764bbd93d448de429
Author: Hari Prasad Killi <email address hidden>
Date: Tue Oct 24 20:35:17 2017 +0530

Do not re-evaluate backrefs for ksync flow entries

The backref entries are updated in DB and ksync tasks, which are mutually
exclusive. The flow event task was accessing the same and it doesnt actually
require. Avoiding backref reevaluation.

Change-Id: I8cb9b92196626ea219f7f9e1055e2579c62c7c86
closes-bug: #1726709
(cherry picked from commit 849be7c017bf13b7c054d0cc5b436a6a87c3b0ed)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.2.3.x

Review in progress for https://review.opencontrail.org/38193
Submitter: Hari Prasad Killi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/38193
Committed: http://github.com/Juniper/contrail-controller/commit/158da92f29af86e5345d79a51e864271dbb9594f
Submitter: Zuul (<email address hidden>)
Branch: R3.2.3.x

commit 158da92f29af86e5345d79a51e864271dbb9594f
Author: Hari Prasad Killi <email address hidden>
Date: Tue Oct 24 20:35:17 2017 +0530

Do not re-evaluate backrefs for ksync flow entries

The backref entries are updated in DB and ksync tasks, which are mutually
exclusive. The flow event task was accessing the same and it doesnt actually
require. Avoiding backref reevaluation.

Change-Id: I8cb9b92196626ea219f7f9e1055e2579c62c7c86
closes-bug: #1726709
(cherry picked from commit ef13c6b5ec0250df8bf7be0241a240c799e6b86f)

Édouard Thuleau (ethuleau)
information type: Private → Public
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.