Juniper Openstack

contrail-dns crash on scaling more than 1000 dns records

Bug #1573316 reported by manishkn
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R3.0
Fix Committed
Medium
Hari Prasad Killi
Juniper Openstack r3.0.2.0
Trunk
Fix Committed
Medium
Hari Prasad Killi
Juniper Openstack r3.1.0.0-fcs

Bug Description

Contrail DNS crashed on system with more than 1000 vdns record spread across multiple projects.
This crash is seen two times and I guess it happened on stop/start contrail-dns service when 1000+ records are already available.

Exact steps to re-create is yet to be identified

attaching the core in core file is stored in /cs-shared/bugs/

]$gdb /cs-shared/github-build/R3.0/26/ubuntu-14-04/juno/store/sandbox/build/production/dns/contrail-dns -c core.contrail-dns.26557.sdkvse4.1461099962
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /cs-shared/github-build/R3.0/26/ubuntu-14-04/juno/store/sandbox/build/production/dns/contrail-dns...done.
[New LWP 26557]

warning: .dynamic section for "/usr/lib/x86_64-linux-gnu/libstdc++.so.6" is not at the expected address (wrong library or version mismatch?)
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-dns'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f0fad1e1cc9 in __GI___sigtimedwait (set=0x67bd, info=0x67bd, timeout=0x6)
    at ../nptl/sysdeps/unix/sysv/linux/../../../../../sysdeps/unix/sysv/linux/sigtimedwait.c:74
74 ../nptl/sysdeps/unix/sysv/linux/../../../../../sysdeps/unix/sysv/linux/sigtimedwait.c: No such file or directory.
(gdb) bt
#0 0x00007f0fad1e1cc9 in __GI___sigtimedwait (set=0x67bd, info=0x67bd, timeout=0x6)
    at ../nptl/sysdeps/unix/sysv/linux/../../../../../sysdeps/unix/sysv/linux/sigtimedwait.c:74
#1 0x00000000ffffffff in ?? ()
#2 0x00007f0fad1dc4e3 in _nl_load_domain (domain_file=0xaab0a8,
    domainbinding=<error reading variable: Cannot access memory at address 0xffffffffffffff59>)
    at loadmsgcat.c:858
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb)

Tags: blocker vdns
manishkn (manishkn)
Changed in juniperopenstack:
milestone: none → r3.1.0.0-fcs
no longer affects: juniperopenstack/trunk
Jeba Paulaiyan (jebap)
information type: Proprietary → Public
tags: added: blocker
Revision history for this message
Hari Prasad Killi (haripk) wrote :

Manish, Couldnt find the core file at the specified place. Please copy it.

Revision history for this message
Hari Prasad Killi (haripk) wrote :

proper bt:

#0 0x00007f0fad1e1cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007f0fad1e50d8 in __GI_abort () at abort.c:89
#2 0x00007f0fad1dab86 in __assert_fail_base (fmt=0x7f0fad32b830 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0xaab053 "_ == __null",
    file=file@entry=0xaab0a8 "olver.cc", line=line@entry=49,
    function=function@entry=0xaab1a0 <BindResolver::BindResolver(boost::asio::io_service&, std::vector<BindResolver::DnsServer, std::allocator<BindResolver::DnsServer> > const&, unsigned short, boost::function<void (unsigned char*)>)::__PRETTY_FUNCTION__+32> "::asio::io_service&, const std::vector<BindResolver::DnsServer>&, uint16_t, BindResolver::Callback)")
    at assert.c:92
#3 0x00007f0fad1dac32 in __GI___assert_fail (assertion=0xaab053 "_ == __null", file=0xaab0a8 "olver.cc", line=49,
    function=0xaab1a0 <BindResolver::BindResolver(boost::asio::io_service&, std::vector<BindResolver::DnsServer, std::allocator<BindResolver::DnsServer> > const&, unsigned short, boost::function<void (unsigned char*)>)::__PRETTY_FUNCTION__+32> "::asio::io_service&, const std::vector<BindResolver::DnsServer>&, uint16_t, BindResolver::Callback)")
    at assert.c:101
#4 0x00000000007a2713 in BindResolver::BindResolver(boost::asio::io_service&, std::vector<BindResolver::DnsServer, std::allocator<BindResolver::DnsServer> > const&, unsigned short, boost::function<void (unsigned char*)>) (this=0x143f7f0, io=..., dns_servers=..., client_port=<optimized out>, cb=...) at controller/src/dns/bind/bind_resolver.cc:49
#5 0x00000000007a281f in BindResolver::Init(boost::asio::io_service&, std::vector<BindResolver::DnsServer, std::allocator<BindResolver::DnsServer> > const&, unsigned short, boost::function<void (unsigned char*)>) (io=..., dns_servers=..., client_port=<optimized out>, cb=...) at controller/src/dns/bind/bind_resolver.cc:15
#6 0x00000000007804c7 in DnsManager::DnsManager (this=0x7fff59507db0) at controller/src/dns/mgr/dns_mgr.cc:26
#7 0x00000000005f429a in main (argc=<optimized out>, argv=<optimized out>) at controller/src/dns/main.cc:189

Revision history for this message
Hari Prasad Killi (haripk) wrote :

contrail-dns is using binding to UDP port 8092 during init and this port is already taken by some other process, causing this assert. Could possibly bind to any other port if this port is not available.

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/20000
Submitter: Hari Prasad Killi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/20000
Committed: http://github.org/Juniper/contrail-controller/commit/26861661f9779135480eff669e27a8c1c2b662cf
Submitter: Zuul
Branch: R3.0

commit 26861661f9779135480eff669e27a8c1c2b662cf
Author: Hari <email address hidden>
Date: Mon May 9 14:17:22 2016 +0530

Use different client port if the reserved UDP port is in use.

In case the reserved client port is in use, use different port
instead of assert.

Change-Id: Ie99e3e4a29bd4a809086ae4505ce8498bf40f79a
closes-bug: #1573316

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/20037
Submitter: Hari Prasad Killi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/20037
Committed: http://github.org/Juniper/contrail-controller/commit/ec60446bb48395eb65492282231bbb70a6fff85f
Submitter: Zuul
Branch: master

commit ec60446bb48395eb65492282231bbb70a6fff85f
Author: Hari <email address hidden>
Date: Mon May 9 14:17:22 2016 +0530

Use different client port if the reserved UDP port is in use.

In case the reserved client port is in use, use different port
instead of assert.

Change-Id: Ie99e3e4a29bd4a809086ae4505ce8498bf40f79a
closes-bug: #1573316
(cherry picked from commit 26861661f9779135480eff669e27a8c1c2b662cf)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.22.x

Review in progress for https://review.opencontrail.org/20155
Submitter: Hari Prasad Killi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.20

Review in progress for https://review.opencontrail.org/20156
Submitter: Hari Prasad Killi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/20155
Committed: http://github.org/Juniper/contrail-controller/commit/ea76687ae57d03851d5b6c372356a1b091b32239
Submitter: Zuul
Branch: R2.22.x

commit ea76687ae57d03851d5b6c372356a1b091b32239
Author: Hari <email address hidden>
Date: Mon May 9 14:17:22 2016 +0530

Use different client port if the reserved UDP port is in use.

In case the reserved client port is in use, use different port
instead of assert.

Change-Id: Ie99e3e4a29bd4a809086ae4505ce8498bf40f79a
closes-bug: #1573316
(cherry picked from commit 26861661f9779135480eff669e27a8c1c2b662cf)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.22.x

Review in progress for https://review.opencontrail.org/20198
Submitter: Hari Prasad Killi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/20198
Committed: http://github.org/Juniper/contrail-controller/commit/141166913a38bb086dbe4445b10a71e53927e989
Submitter: Zuul
Branch: R2.22.x

commit 141166913a38bb086dbe4445b10a71e53927e989
Author: Hari <email address hidden>
Date: Fri May 13 21:09:52 2016 +0530

Undoing the change

Change-Id: I5496f444696723bbef48b96c27be8951ab8f7262
related-bug: #1573316

Revision history for this message
Nagabhushana R (bhushana) wrote :

Not applicable for any R2.2x branches

no longer affects: juniperopenstack/r2.20
no longer affects: juniperopenstack/r2.22.x
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.