Bug #1710804 “contrail-collector crashes with basic_string::_S_c...” : Bugs : Juniper Openstack

Prashanth Nageshappa (nprashanth) on 2017-08-15

Changed in juniperopenstack:
assignee:	nobody → Prashanth Nageshappa (nprashanth)

Revision history for this message

Prashanth Nageshappa (nprashanth) wrote on 2017-08-15:

#1

Even though I run the contrail-collector process is run after setting ulimit -c unlimited, I still see core file with 0 size

information type:

Proprietary → Public

Megh Bhatt (meghb) on 2017-08-22

Changed in juniperopenstack:
assignee:	Prashanth Nageshappa (nprashanth) → Megh Bhatt (meghb)
milestone:	none → r4.0.1.0

Revision history for this message

Megh Bhatt (meghb) wrote on 2017-08-22:

#2

Download full text (7.2 KiB)

(gdb)
#0 0x00007ffff4784c37 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ffff4788028 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007ffff477dbf6 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007ffff477dca2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x0000000000465785 in TaskImpl::execute (this=0x7fffeddd5940)
    at controller/src/base/task.cc:306
#5 0x00007ffff5d0fb3a in ?? () from /usr/lib/libtbb.so.2
#6 0x00007ffff5d0b816 in ?? () from /usr/lib/libtbb.so.2
#7 0x00007ffff5d0af4b in ?? () from /usr/lib/libtbb.so.2
#8 0x00007ffff5d070ff in ?? () from /usr/lib/libtbb.so.2
#9 0x00007ffff5d072f9 in ?? () from /usr/lib/libtbb.so.2
#10 0x00007ffff5f2b184 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#11 0x00007ffff484837d in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0 0x00007ffff4784c37 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ffff4788028 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007ffff477dbf6 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007ffff477dca2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x0000000000465785 in TaskImpl::execute (this=0x7fffeddd5940)
    at controller/src/base/task.cc:306
#5 0x00007ffff5d0fb3a in ?? () from /usr/lib/libtbb.so.2
#6 0x00007ffff5d0b816 in ?? () from /usr/lib/libtbb.so.2
#7 0x00007ffff5d0af4b in ?? () from /usr/lib/libtbb.so.2
#8 0x00007ffff5d070ff in ?? () from /usr/lib/libtbb.so.2
#9 0x00007ffff5d072f9 in ?? () from /usr/lib/libtbb.so.2
#10 0x00007ffff5f2b184 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#11 0x00007ffff484837d in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) f 4
#4 0x0000000000465785 in TaskImpl::execute (this=0x7fffeddd5940)
    at controller/src/base/task.cc:306
306 controller/src/base/task.cc: No such file or directory.
(gdb) i what
Undefined info command: "what". Try "help info".
(gdb) p what
$1 = "basic_string::_S_construct null not valid"
(gdb) info locals
what = "basic_string::_S_construct null not valid"
e = @0x7ffed84e7520: <incomplete type>
running = @0x7fffc812fb80: 0x117ed10
__PRETTY_FUNCTION__ = "virtual tbb::task* TaskImpl::execute()"
(gdb) set print object on
(gdb) p running
$2 = (const tbb::interface6::enumerable_thread_specific<Task*, tbb::cache_aligned_allocator<Task*>, (tbb::ets_key_usage_type)1>::reference) @0x7fffc812fb80: 0x117ed10
(gdb) p *running
$3 = (Timer::TimerTask) {<Task> = {_vptr.Task = 0xa38e10 <vtable for Timer::TimerTask+16>,
    static kTaskInstanceAny = -1, task_id_ = 4, task_instance_ = -1,
    task_impl_ = 0x7fffeddd5940, state_ = Task::RUN, tbb_state_ = Task::TBB_EXEC,
    seqno_ = 2211546, task_recycle_ = false, task_cancel_ = false, enqueue_time_ = 0,
    schedule_time_ = 0, execute_delay_ = 0, schedule_delay_ = 0,
    waitq_hook_ = {<boost::intrusive::detail::generic_hook<boost::intrusive::get_list_node_algo<void*>, boost::intrusive::member_tag, (boost::intrusive::link_mode_type)1, 0>> = {<boost::intrusive::detail::no_default_definer> = {<No data fields>}, <boost::intrusive::list_node<void*>> = {next_ = 0x0, prev_ = 0x0}, <No data fields>}, <No data fields>}}, timer_ = {
...

(gdb)
#0  0x00007ffff4784c37 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ffff4788028 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007ffff477dbf6 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007ffff477dca2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x0000000000465785 in TaskImpl::execute (this=0x7fffeddd5940)
    at controller/src/base/task.cc:306
#5  0x00007ffff5d0fb3a in ?? () from /usr/lib/libtbb.so.2
#6  0x00007ffff5d0b816 in ?? () from /usr/lib/libtbb.so.2
#7  0x00007ffff5d0af4b in ?? () from /usr/lib/libtbb.so.2
#8  0x00007ffff5d070ff in ?? () from /usr/lib/libtbb.so.2
#9  0x00007ffff5d072f9 in ?? () from /usr/lib/libtbb.so.2
#10 0x00007ffff5f2b184 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#11 0x00007ffff484837d in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0  0x00007ffff4784c37 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ffff4788028 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007ffff477dbf6 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007ffff477dca2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x0000000000465785 in TaskImpl::execute (this=0x7fffeddd5940)
    at controller/src/base/task.cc:306
#5  0x00007ffff5d0fb3a in ?? () from /usr/lib/libtbb.so.2
#6  0x00007ffff5d0b816 in ?? () from /usr/lib/libtbb.so.2
#7  0x00007ffff5d0af4b in ?? () from /usr/lib/libtbb.so.2
#8  0x00007ffff5d070ff in ?? () from /usr/lib/libtbb.so.2
#9  0x00007ffff5d072f9 in ?? () from /usr/lib/libtbb.so.2
#10 0x00007ffff5f2b184 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#11 0x00007ffff484837d in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) f 4
#4  0x0000000000465785 in TaskImpl::execute (this=0x7fffeddd5940)
    at controller/src/base/task.cc:306
306     controller/src/base/task.cc: No such file or directory.
(gdb) i what
Undefined info command: "what".  Try "help info".
(gdb) p what
$1 = "basic_string::_S_construct null not valid"
(gdb) info locals
what = "basic_string::_S_construct null not valid"
e = @0x7ffed84e7520: <incomplete type>
running = @0x7fffc812fb80: 0x117ed10
__PRETTY_FUNCTION__ = "virtual tbb::task* TaskImpl::execute()"
(gdb) set print object on
(gdb) p running
$2 = (const tbb::interface6::enumerable_thread_specific<Task*, tbb::cache_aligned_allocator<Task*>, (tbb::ets_key_usage_type)1>::reference) @0x7fffc812fb80: 0x117ed10
(gdb) p *running
$3 = (Timer::TimerTask) {<Task> = {_vptr.Task = 0xa38e10 <vtable for Timer::TimerTask+16>,
    static kTaskInstanceAny = -1, task_id_ = 4, task_instance_ = -1,
    task_impl_ = 0x7fffeddd5940, state_ = Task::RUN, tbb_state_ = Task::TBB_EXEC,
    seqno_ = 2211546, task_recycle_ = false, task_cancel_ = false, enqueue_time_ = 0,
    schedule_time_ = 0, execute_delay_ = 0, schedule_delay_ = 0,
    waitq_hook_ = {<boost::intrusive::detail::generic_hook<boost::intrusive::get_list_node_algo<void*>, boost::intrusive::member_tag, (boost::intrusive::link_mode_type)1, 0>> = {<boost::intrusive::detail::no_default_definer> = {<No data fields>}, <boost::intrusive::list_node<void*>> = {next_ = 0x0, prev_ = 0x0}, <No data fields>}, <No data fields>}}, timer_ = {
    px = 0x10ede60}, ec_ = {m_val = 0, m_cat = 0x7ffff65a60d0}}
(gdb) p running->task_impl_
$4 = (tbb::task *) 0x7fffeddd5940
(gdb) p *running->task_impl_
$5 = <incomplete type>
(gdb) p TaskScheduler::instance_
There is no field named instance_
(gdb) p TaskScheduler::singleton_
$6 = {px = 0x1089f70}
(gdb) p TaskScheduler::singleton_.px
$7 = (TaskScheduler *) 0x1089f70
(gdb) p *TaskScheduler::singleton_.px
$8 = {static kVectorGrowSize = 16, static singleton_ = {px = 0x1089f70}, use_spawn_ = false,
  stop_entry_ = 0x10a6180,
  task_scheduler_ = {<tbb::internal::no_copy> = {<tbb::internal::no_assign> = {<No data fields>}, <No data fields>}, my_scheduler = 0x7fffeddd3800, static automatic = -1,
    static deferred = -2}, mutex_ = {static is_rw_mutex = false,
    static is_recursive_mutex = false, static is_fair_mutex = false, impl = {__data = {
        __lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0,
        __elision = 0, __list = {__prev = 0x0, __next = 0x0}},
      __size = '\000' <repeats 39 times>, __align = 0}}, running_ = true, seqno_ = 2211548,
  task_group_db_ = std::vector of length 16, capacity 16 = {0x0, 0x0, 0x1176b60, 0x0,
    0x116d7d0, 0x10d7730, 0x10d7f10, 0x10d7330, 0x0, 0x0, 0x1172140, 0x0, 0x11715b0,
    0x116c010, 0x116cc30, 0x1175ae0},
  id_map_mutex_ = {<tbb::internal::no_copy> = {<tbb::internal::no_assign> = {<No data fields>}, <No data fields>},
    reader_head = {<tbb::internal::atomic_impl_with_arithmetic<tbb::interface5::reader_writer_lock::scoped_lock_read*, long, tbb::interface5::reader_writer_lock::scoped_lock_read>> = {<tbb::internal::atomic_impl<tbb::interface5::reader_writer_lock::scoped_lock_read*>> = {
          my_storage = {my_value = 0x0}}, <No data fields>}, <No data fields>},
    writer_head = {<tbb::internal::atomic_impl_with_arithmetic<tbb::interface5::reader_writer_lock::scoped_lock*, long, tbb::interface5::reader_writer_lock::scoped_lock>> = {<tbb::internal::atomic_impl<tbb::interface5::reader_writer_lock::scoped_lock*>> = {my_storage = {
            my_value = 0x0}}, <No data fields>}, <No data fields>},
    writer_tail = {<tbb::internal::atomic_impl_with_arithmetic<tbb::interface5::reader_writer_lock::scoped_lock*, long, tbb::interface5::reader_writer_lock::scoped_lock>> = {<tbb::internal::atomic_impl<tbb::interface5::reader_writer_lock::scoped_lock*>> = {my_storage = {
            my_value = 0x0}}, <No data fields>}, <No data fields>}, my_current_writer = {
---Type <return> to continue, or q <return> to quit---
      my_id = 0},
    rdr_count_and_flags = {<tbb::internal::atomic_impl_with_arithmetic<unsigned long, unsigned long, char>> = {<tbb::internal::atomic_impl<unsigned long>> = {my_storage = {
            my_value = 0}}, <No data fields>}, <No data fields>}},
  id_map_ = std::map with 15 elements = {["CqlIfImpl::Task"] = 1, ["Kafka Timer"] = 4,
    ["collector:DbIf"] = 3, ["io::ReaderTask"] = 6, ["io::udp::ReaderTask"] = 9,
    ["sandesh::LifetimeMgr"] = 7, ["sandesh::RecvQueue"] = 11,
    ["sandesh::SandeshClientReader"] = 14, ["sandesh::SandeshClientSM"] = 12,
    ["sandesh::SandeshClientSession"] = 13, ["sandesh::SandeshStateMachine"] = 5,
    ["tcpForwarder poller"] = 10, ["vizd::Stats"] = 15, ["vizd::syslog"] = 8,
    ["vnc-api http client"] = 2}, id_max_ = 15,
  log_fn_ = {<boost::function5<void, char const*, unsigned int, Task const*, char const*, unsigned int>> = {<boost::function_base> = {vtable = 0x0, functor = {obj_ptr = 0x0, type = {
            type = 0x0, const_qualified = false, volatile_qualified = false}, func_ptr = 0x0,
          bound_memfunc_ptr = {memfunc_ptr = NULL, obj_ptr = 0x0}, obj_ref = {obj_ptr = 0x0,
            is_const_qualified = false, is_volatile_qualified = false}, data = 0 '\000'}},
      static args = <optimized out>, static arity = <optimized out>}, <No data fields>},
  hw_thread_count_ = 48, track_run_time_ = false, measure_delay_ = false,
  schedule_delay_ = 0, execute_delay_ = 0, enqueue_count_ = 2211548, done_count_ = 2211543,
  cancel_count_ = 1, evm_ = 0x0, static ThreadAmpFactor_ = 1, tbb_awake_task_ = 0x0,
  task_monitor_ = 0x0}
(gdb)

Revision history for this message

Megh Bhatt (meghb) wrote on 2017-08-22:

#3

Download full text (4.6 KiB)

regress@ci-nh03-nb:~$ dpkg -l | grep contrail
ii contrail-analytics 4.0.0.0-20 amd64 OpenContrail analytics
ii contrail-config 4.0.0.0-20 amd64 OpenContrail configuration management
ii contrail-config-openstack 4.0.0.0-20 amd64 OpenContrail configuration OpenStack module
ii contrail-database-common 4.0.0.0-20 all Contrail database composite debian package
ii contrail-docs 4.0.0.0-20 amd64 Documentation for OpenContrail
ii contrail-f5 4.0.0.0-20 all Big IP interface for f5 load balancer
ii contrail-fabric-utils 4.0.0.0-20 all Contrail Fabric Utilities for cluster management
ii contrail-install-packages 4.0.0.0-20~kilo all Contrail Installer Packages - Container of debian packages
ii contrail-lib 4.0.0.0-20 amd64 OpenContrail libraries
ii contrail-nodemgr 4.0.0.0-20 amd64 OpenContrail nodemgr implementation
ii contrail-openstack-analytics 4.0.0.0-20 all Contrail Openstack Analytics composite debian package
ii contrail-openstack-config 4.0.0.0-20 all Contrail Openstack Config composite debian package
ii contrail-openstack-database 4.0.0.0-20 all Contrail Openstack database composite debian package
ii contrail-openstack-webui 4.0.0.0-20 all Contrail Openstack WebUI composite debian package
ii contrail-setup 4.0.0.0-20 all Contrail Setup package with scripts for provisioning
ii contrail-utils 4.0.0.0-20 amd64 OpenContrail tools and utilities
ii contrail-web-controller 4.0.0.0-20 amd64 Contrail Systems Web UI Feature
ii contrail-web-core 4.0.0.0-20 amd64 Contrail Systems Web UI
ii kafka 2.11-0.9.0.1-0contrail0 amd64 <insert up to 60 chars description>
ii neutron-plugin-contrail 4.0.0.0-20 all OpenStack virtual network service - Opencontrail
ii nodejs 0.10.35-1contrail1 amd64 Node.js event-based server-side javascript engine
ii python-bitarray 0.8.0-2contrail1 amd64 Python module for efficient boolean array handling
ii python-cassandra-driver 3.0.0-0contrail3 amd64 Python driver for Cassandra
ii python-consistent-hash 1.0-0contrail1 amd64 ...

regress@ci-nh03-nb:~$ dpkg -l | grep contrail
ii  contrail-analytics                   4.0.0.0-20                            amd64        OpenContrail analytics
ii  contrail-config                      4.0.0.0-20                            amd64        OpenContrail configuration management
ii  contrail-config-openstack            4.0.0.0-20                            amd64        OpenContrail configuration OpenStack module
ii  contrail-database-common             4.0.0.0-20                            all          Contrail database composite debian package
ii  contrail-docs                        4.0.0.0-20                            amd64        Documentation for OpenContrail
ii  contrail-f5                          4.0.0.0-20                            all          Big IP interface for f5 load balancer
ii  contrail-fabric-utils                4.0.0.0-20                            all          Contrail Fabric Utilities for cluster management
ii  contrail-install-packages            4.0.0.0-20~kilo                       all          Contrail Installer Packages - Container of debian packages
ii  contrail-lib                         4.0.0.0-20                            amd64        OpenContrail libraries
ii  contrail-nodemgr                     4.0.0.0-20                            amd64        OpenContrail nodemgr implementation
ii  contrail-openstack-analytics         4.0.0.0-20                            all          Contrail Openstack Analytics composite debian package
ii  contrail-openstack-config            4.0.0.0-20                            all          Contrail Openstack Config composite debian package
ii  contrail-openstack-database          4.0.0.0-20                            all          Contrail Openstack database composite debian package
ii  contrail-openstack-webui             4.0.0.0-20                            all          Contrail Openstack WebUI composite debian package
ii  contrail-setup                       4.0.0.0-20                            all          Contrail Setup package with scripts for provisioning
ii  contrail-utils                       4.0.0.0-20                            amd64        OpenContrail tools and utilities
ii  contrail-web-controller              4.0.0.0-20                            amd64        Contrail Systems Web UI Feature
ii  contrail-web-core                    4.0.0.0-20                            amd64        Contrail Systems Web UI
ii  kafka                                2.11-0.9.0.1-0contrail0               amd64        <insert up to 60 chars description>
ii  neutron-plugin-contrail              4.0.0.0-20                            all          OpenStack virtual network service - Opencontrail
ii  nodejs                               0.10.35-1contrail1                    amd64        Node.js event-based server-side javascript engine
ii  python-bitarray                      0.8.0-2contrail1                      amd64        Python module for efficient boolean array handling
ii  python-cassandra-driver              3.0.0-0contrail3                      amd64        Python driver for Cassandra
ii  python-consistent-hash               1.0-0contrail1                        amd64        <insert up to 60 chars description>
ii  python-contrail                      4.0.0.0-20                            amd64        OpenContrail python-libs
ii  python-geventhttpclient              1.0a-1contrail0                       amd64        High performance, concurrent HTTP client using gevent.
ii  python-kafka                         1.0.1-0contrail1                      amd64        Python client for the Apache Kafka distributed stream processing system
ii  python-kazoo                         1.3.1-1contrail3                      all          higher level API to Apache Zookeeper (Python 2)
ii  python-lxml                          3.3.1-1contrail1                      amd64        pythonic binding for the libxml2 and libxslt libraries
ii  python-ncclient                      0.4.1-1contrail1                      all          Python library for NETCONF clients
ii  python-neutronclient                 1:2.3.11-0ubuntu1~cloud0.3contrail    all          client - Neutron is a virtual network service for Openstack
ii  python-pycassa                       1.11.0-1contrail2                     all          Client library for Apache Cassandra
ii  python-sseclient                     0.0.11-0contrail0                     amd64        <insert up to 60 chars description>
ii  python-xmltodict                     0.9.0-1contrail1                      all          Makes working with XML feel like you are working with JSON
regress@ci-nh03-nb:~$

Changed in juniperopenstack:
importance:	Undecided → Medium
milestone:	r4.0.1.0 → r4.1.0.0-fcs

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2017-08-22: [Review update] R4.0

#4

Review in progress for https://review.opencontrail.org/34826
Submitter: Megh Bhatt (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2017-08-22: [Review update] master

#6

Review in progress for https://review.opencontrail.org/34827
Submitter: Megh Bhatt (<email address hidden>)

	Status	Importance	Assigned to	Milestone
Juniper Openstack	Status tracked in Trunk
R4.0	Fix Committed	Medium	Megh Bhatt	Juniper Openstack r4.0.1.0 "r4.0.1.0"
Trunk	Fix Committed	Medium	Megh Bhatt	Juniper Openstack r4.1.0.0-fcs "r4.1"

Juniper Openstack

contrail-collector crashes with basic_string::_S_construct null not valid

Bug Description

Other bug subscribers

Remote bug watches