Juniper Openstack

vrouter crash in FlowStatsManager::GetFlowStatsCollector

Bug #1571588 reported by vageesan
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R3.0
Fix Committed
Critical
Ashok Singh
Juniper Openstack r3.0.2.0
Trunk
Fix Committed
Critical
Ashok Singh
Juniper Openstack r3.1.0.0-fcs

Bug Description

vrouter crashed with following backtrace in solution test run.

3.0.2.0-26~kilo

log is in 10.84.5.112:/cs-shared/bugs/<bug-id>/

[New LWP 10020]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-vrouter-agent'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000000000bb3500 in std::_Rb_tree<FlowAgingTableKey const, std::pair<FlowAgingTableKey const, boost::shared_ptr<FlowStatsCollector> >, std::_Select1st<std::pair<FlowAgingTableKey const, boost::shared_ptr<FlowStatsCollector> > >, std::less<FlowAgingTableKey const>, std::allocator<std::pair<FlowAgingTableKey const, boost::shared_ptr<FlowStatsCollector> > > >::find(FlowAgingTableKey const&) const ()
#0 0x0000000000bb3500 in std::_Rb_tree<FlowAgingTableKey const, std::pair<FlowAgingTableKey const, boost::shared_ptr<FlowStatsCollector> >, std::_Select1st<std::pair<FlowAgingTableKey const, boost::shared_ptr<FlowStatsCollector> > >, std::less<FlowAgingTableKey const>, std::allocator<std::pair<FlowAgingTableKey const, boost::shared_ptr<FlowStatsCollector> > > >::find(FlowAgingTableKey const&) const ()
#1 0x0000000000bb12e1 in FlowStatsManager::GetFlowStatsCollector(FlowEntry const*) const ()
#2 0x0000000000bb1395 in FlowStatsManager::AddEvent(boost::intrusive_ptr<FlowEntry>&) ()
#3 0x0000000000c17689 in FlowMgmtManager::LogHandler(boost::shared_ptr<FlowMgmtRequest>) ()
#4 0x0000000000c1b9dc in boost::detail::function::function_obj_invoker1<boost::_bi::bind_t<bool, boost::_mfi::mf1<bool, FlowMgmtManager, boost::shared_ptr<FlowMgmtRequest> >, boost::_bi::list2<boost::_bi::value<FlowMgmtManager*>, boost::arg<1> > >, bool, boost::shared_ptr<FlowMgmtRequest> >::invoke(boost::detail::function::function_buffer&, boost::shared_ptr<FlowMgmtRequest>) ()
#5 0x0000000000c21f44 in QueueTaskRunner<boost::shared_ptr<FlowMgmtRequest>, WorkQueue<boost::shared_ptr<FlowMgmtRequest> > >::RunQueue() ()
#6 0x0000000001186b3c in TaskImpl::execute() ()
#7 0x00007fb5044c8b3a in ?? () from /usr/lib/libtbb.so.2
#8 0x00007fb5044c4816 in ?? () from /usr/lib/libtbb.so.2
#9 0x00007fb5044c3f4b in ?? () from /usr/lib/libtbb.so.2
#10 0x00007fb5044c00ff in ?? () from /usr/lib/libtbb.so.2
#11 0x00007fb5044c02f9 in ?? () from /usr/lib/libtbb.so.2
#12 0x00007fb5046e4182 in start_thread (arg=0x7fb4f7fff700) at pthread_create.c:312
#13 0x00007fb5039bd47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Revision history for this message
vageesan (vageesant) wrote :

vrouter in tor agent is frequently crashing with this back trace.

tags: added: blocker
Revision history for this message
vageesan (vageesant) wrote :

vrouter in tor-node* is crashing frequently

vageesan (vageesant)
Changed in juniperopenstack:
milestone: r3.0-fcs → none
Hari Prasad Killi (haripk)
Changed in juniperopenstack:
assignee: Hari Prasad Killi (haripk) → Ashok Singh (ashoksr)
Jeba Paulaiyan (jebap)
information type: Proprietary → Public
Revision history for this message
Ashok Singh (ashoksr) wrote :

Crash is on TSN agent because of NULL FlowStatsManager

(gdb) fr 4
#4 0x0000000000c17689 in FlowMgmtManager::LogHandler (this=0x7fb4dbf87ab0, req=...)
    at controller/src/vnsw/agent/pkt/flow_mgmt.cc:439
439 controller/src/vnsw/agent/pkt/flow_mgmt.cc: No such file or directory.
(gdb) p Agent::singleton_->flow_stats_manager_
$2 = (FlowStatsManager *) 0x0
(gdb) p Agent::singleton_->tsn_enabled_
$3 = true

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/19633
Submitter: Ashok Singh (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/19634
Submitter: Ashok Singh (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/19633
Committed: http://github.org/Juniper/contrail-controller/commit/7ad9a9a1a4a17164168ba9a7f049a66acbd33f51
Submitter: Zuul
Branch: R3.0

commit 7ad9a9a1a4a17164168ba9a7f049a66acbd33f51
Author: ashoksingh <email address hidden>
Date: Tue Apr 26 17:32:07 2016 +0530

Flow messages received in TSN mode are made as short flows.

Issue
———
In TSN mode, agent does not create FlowStatsManager/FlowStatsCollector.
TSN agent crashed because it received a flow message request and during
processing of this request, we were accessing FlowStatsManager which was
NULL.

Fix
——
Handle Flow message requests in TSN mode. Create flows as short-flows in
TSN mode. To delete these flows we now create FlowStatsManager/FlowStatsCollector
even in TSN mode. We still don't add/delete FlowStatsCollector based on user
config in TSN mode. Also export of flows are not done in TSN mode.

Change-Id: I13c14ba7b0036f0a55d73e70387195be2f0855c9
Closes-Bug: #1571588 1571888

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/19634
Committed: http://github.org/Juniper/contrail-controller/commit/942c54320a3db8f63e61fd8870fbc3b9d0396f46
Submitter: Zuul
Branch: master

commit 942c54320a3db8f63e61fd8870fbc3b9d0396f46
Author: ashoksingh <email address hidden>
Date: Tue Apr 26 18:47:53 2016 +0530

Flow messages received in TSN mode are made as short flows.

Issue
———
In TSN mode, agent does not create FlowStatsManager/FlowStatsCollector.
TSN agent crashed because it received a flow message request and during
processing of this request, we were accessing FlowStatsManager which was
NULL.

Fix
——
Handle Flow message requests in TSN mode. Create flows as short-flows in
TSN mode. To delete these flows we now create FlowStatsManager/FlowStatsCollector
even in TSN mode. We still don't add/delete FlowStatsCollector based on user
config in TSN mode. Also export of flows are not done in TSN mode.

Change-Id: I4533102851acb16948d9ff1ecce328124c473d42
Closes-Bug: #1571588 1571888

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.