contrail-vrouter-agent crashed at smallbin double linked list corrupted

Bug #1570809 reported by Sandip Dey
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R3.0
Fix Committed
Critical
Prabhjot Singh Sethi
Trunk
Fix Committed
Critical
Prabhjot Singh Sethi

Bug Description

Logs saved at :http://10.204.216.50/Docs/bugs/<bug-id>

BT
===
#0 0x00007fb142577cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
Traceback (most recent call last):
  File "/usr/share/gdb/auto-load/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19-gdb.py", line 63, in <module>
    from libstdcxx.v6.printers import register_libstdcxx_printers
ImportError: No module named 'libstdcxx'
(gdb) bt
#0 0x00007fb142577cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007fb14257b0d8 in __GI_abort () at abort.c:89
#2 0x00007fb1425b4394 in __libc_message (do_abort=do_abort@entry=1, fmt=fmt@entry=0x7fb1426c2b28 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
#3 0x00007fb1425bf0f7 in malloc_printerr (action=<optimized out>, str=0x7fb1426c2ef0 "malloc(): smallbin double linked list corrupted", ptr=<optimized out>) at malloc.c:4996
#4 0x00007fb1425c1e04 in _int_malloc (av=0x7fb12c000020, bytes=61) at malloc.c:3359
#5 0x00007fb1425c37b0 in __GI___libc_malloc (bytes=61) at malloc.c:2891
#6 0x00007fb142e80dad in operator new(unsigned long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#7 0x00007fb142edc209 in std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8 0x00007fb142edc396 in std::string::_M_mutate(unsigned long, unsigned long, unsigned long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#9 0x00007fb142edc93e in std::string::_M_replace_safe(unsigned long, unsigned long, char const*, unsigned long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x0000000000c1a1a2 in FlowEntry::Reset() ()
#11 0x0000000000c24b7d in FlowEntryFreeList::Free(FlowEntry*) ()
#12 0x0000000000c18430 in intrusive_ptr_release(FlowEntry*) ()
#13 0x0000000000c2f2cd in FlowEvent::~FlowEvent() ()
#14 0x0000000000c3c421 in FlowProto::FlowEventHandler(FlowEvent*, FlowTable*) ()
#15 0x0000000000c3e8ff in QueueTaskRunner<FlowEvent*, WorkQueue<FlowEvent*> >::Run() ()
#16 0x0000000001193eec in TaskImpl::execute() ()
#17 0x00007fb143146b3a in ?? () from /usr/lib/libtbb.so.2
#18 0x00007fb143142816 in ?? () from /usr/lib/libtbb.so.2
#19 0x00007fb143141f4b in ?? () from /usr/lib/libtbb.so.2
#20 0x00007fb14313e0ff in ?? () from /usr/lib/libtbb.so.2
#21 0x00007fb14313e2f9 in ?? () from /usr/lib/libtbb.so.2
#22 0x00007fb143362182 in start_thread (arg=0x7fb13b7fd700) at pthread_create.c:312

Jeba Paulaiyan (jebap)
Changed in juniperopenstack:
milestone: none → r3.1.0.0-fcs
information type: Proprietary → Public
Changed in juniperopenstack:
importance: Undecided → Critical
tags: added: blocker
Revision history for this message
Hari Prasad Killi (haripk) wrote :
Changed in juniperopenstack:
assignee: Hari Prasad Killi (haripk) → Prabhjot Singh Sethi (prabhjot)
Revision history for this message
Prabhjot Singh Sethi (prabhjot) wrote :

free list in flow table can be accessed in parallel from two threads, which can result in two flow entries using same memory address
and can cause freed memory write causing heap corruption

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/19483
Submitter: Prabhjot Singh Sethi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/19484
Submitter: Prabhjot Singh Sethi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/19483
Committed: http://github.org/Juniper/contrail-controller/commit/e742d5db1aee8125b1203ac95cb7649ce8678681
Submitter: Zuul
Branch: master

commit e742d5db1aee8125b1203ac95cb7649ce8678681
Author: Prabhjot Singh Sethi <email address hidden>
Date: Wed Apr 20 14:04:05 2016 +0530

Fix parallel access to free list in flow table

Issue:
------
ConcurrencyCheck was not checking whether the instance
for the task is correct or not, which sometimes results
in accessing free list in flow table from a different
task instance to free and push back the flow entry.

Fix:
----
ConcurrenyCheck to validate against task instance as well

Closes-Bug: 1569708
Closes-Bug: 1570809
Change-Id: I94a96ffebdcf4b4350561e7321c83a7f3e782c10

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/19484
Committed: http://github.org/Juniper/contrail-controller/commit/bffba2c739359e5aab746d8385b6ebc0a171913f
Submitter: Zuul
Branch: R3.0

commit bffba2c739359e5aab746d8385b6ebc0a171913f
Author: Prabhjot Singh Sethi <email address hidden>
Date: Wed Apr 20 14:04:05 2016 +0530

Fix parallel access to free list in flow table

Issue:
------
ConcurrencyCheck was not checking whether the instance
for the task is correct or not, which sometimes results
in accessing free list in flow table from a different
task instance to free and push back the flow entry.

Fix:
----
ConcurrenyCheck to validate against task instance as well

Closes-Bug: 1569708
Closes-Bug: 1570809
Change-Id: I94a96ffebdcf4b4350561e7321c83a7f3e782c10
(cherry picked from commit e742d5db1aee8125b1203ac95cb7649ce8678681)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.