Juniper Openstack

collector cores in TCP Forwarder

Bug #1719990 reported by Anish Mehta on 2017-09-27

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
Juniper Openstack	Status tracked in Trunk
R4.0	Fix Committed	High	Prashanth Nageshappa	Juniper Openstack r4.0.2.0
R4.1	Fix Committed	High	Prashanth Nageshappa	Juniper Openstack r4.1.0.0-fcs "r4.1"
Trunk	Fix Committed	High	Prashanth Nageshappa	Juniper Openstack r5.0.0

Bug Description

These collector cores are seen in CSP V3.1FRS deployment.

The contrail analytics version was 4.0.0-15.

root@canvm(analytics):/# gdb /var/crashes/vizd /var/crashes/core.contrail-collec.2880.canvm.1506359705
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.3) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /var/crashes/vizd...done.

warning: core file may not match specified executable file.
[New LWP 3317]
[New LWP 3330]
[New LWP 3318]
[New LWP 2890]
[New LWP 2888]
[New LWP 3329]
[New LWP 3304]
[New LWP 3331]
[New LWP 2884]
[New LWP 2880]
[New LWP 3303]
[New LWP 3314]
[New LWP 3312]
[New LWP 2886]
[New LWP 3328]
[New LWP 3319]
[New LWP 3313]
[New LWP 3316]
[New LWP 3302]
[New LWP 3315]
[New LWP 2887]
[New LWP 2885]
[New LWP 2889]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-collector'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000000000673d0c in structured_syslog::StructuredSyslogTcpForwarder::Connected (this=0x2ad9980fcbb0) at controller/src/analytics/structured_syslog_server.cc:905
905 controller/src/analytics/structured_syslog_server.cc: No such file or directory.
(gdb) bt
#0 0x0000000000673d0c in structured_syslog::StructuredSyslogTcpForwarder::Connected (this=0x2ad9980fcbb0) at controller/src/analytics/structured_syslog_server.cc:905
#1 0x00000000006813cd in structured_syslog::StructuredSyslogForwarder::PollTcpForwarder (this=0xfa3b40) at controller/src/analytics/structured_syslog_server.cc:975
#2 0x000000000048a5e9 in operator() (this=<optimized out>) at /usr/include/boost/function/function_template.hpp:767
#3 Timer::TimerTask::Run (this=0xfa8410) at controller/src/base/timer.cc:44
#4 0x0000000000478757 in TaskImpl::execute (this=0x2ad95a1ddc40) at controller/src/base/task.cc:279
#5 0x00002ad951f09b3a in ?? () from /usr/lib/libtbb.so.2
#6 0x00002ad951f05816 in ?? () from /usr/lib/libtbb.so.2
#7 0x00002ad951f04f4b in ?? () from /usr/lib/libtbb.so.2
#8 0x00002ad951f010ff in ?? () from /usr/lib/libtbb.so.2
#9 0x00002ad951f012f9 in ?? () from /usr/lib/libtbb.so.2
#10 0x00002ad951cd3184 in start_thread (arg=0x2ad97a209700) at pthread_create.c:312
#11 0x00002ad9533f037d in eventfd (count=-1744249640, flags=0) at ../sysdeps/unix/sysv/linux/eventfd.c:42
#12 0x0000000000000000 in ?? ()
(gdb)

The other one is:

value has been optimized out
(gdb) bt
#0 0x00002b7120000258 in ?? ()
#1 0x00000000007476af in TcpServer::ConnectHandler (this=<optimized out>, server=..., session=..., error=...) at controller/src/io/tcp_server.cc:406
#2 0x000000000074beea in operator() (a3=..., a2=..., a1=..., p=<optimized out>, this=0x7ffed3135bf0) at /usr/include/boost/bind/mem_fn_template.hpp:393
#3 operator()<boost::_mfi::mf3<void, TcpServer, boost::intrusive_ptr<TcpServer>, boost::intrusive_ptr<TcpSession>, const boost::system::error_code&>, boost::_bi::list1<const boost::system::error_code&> > (a=<synthetic pointer>, f=..., this=0x7ffed3135c00) at /usr/include/boost/bind/bind.hpp:457
#4 operator()<boost::system::error_code> (a1=..., this=0x7ffed3135bf0) at /usr/include/boost/bind/bind_template.hpp:47
#5 operator() (this=0x7ffed3135bf0) at /usr/include/boost/asio/detail/bind_handler.hpp:47
#6 asio_handler_invoke<boost::asio::detail::binder1<boost::_bi::bind_t<void, boost::_mfi::mf3<void, TcpServer, boost::intrusive_ptr<TcpServer>, boost::intrusive_ptr<TcpSession>, boost::system::error_code const&>, boost::_bi::list4<boost::_bi::value<TcpServer*>, boost::_bi::value<boost::intrusive_ptr<TcpServer> >, boost::_bi::value<boost::intrusive_ptr<TcpSession> >, boost::arg<1> (*)()> >, boost::system::error_code> > (function=<error reading variable: access outside bounds of object referenced via synthetic pointer>) at /usr/include/boost/asio/handler_invoke_hook.hpp:64
#7 invoke<boost::asio::detail::binder1<boost::_bi::bind_t<void, boost::_mfi::mf3<void, TcpServer, boost::intrusive_ptr<TcpServer>, boost::intrusive_ptr<TcpSession>, boost::system::error_code const&>, boost::_bi::list4<boost::_bi::value<TcpServer*>, boost::_bi::value<boost::intrusive_ptr<TcpServer> >, boost::_bi::value<boost::intrusive_ptr<TcpSession> >, boost::arg<1> (*)()> >, boost::system::error_code>, boost::_bi::bind_t<void, boost::_mfi::mf3<void, TcpServer, boost::intrusive_ptr<TcpServer>, boost::intrusive_ptr<TcpSession>, boost::system::error_code const&>, boost::_bi::list4<boost::_bi::value<TcpServer*>, boost::_bi::value<boost::intrusive_ptr<TcpServer> >, boost::_bi::value<boost::intrusive_ptr<TcpSession> >, boost::arg<1> (*)()> > > (context=..., function=...) at /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#8 asio_handler_invoke<boost::asio::detail::binder1<boost::_bi::bind_t<void, boost::_mfi::mf3<void, TcpServer, boost::intrusive_ptr<TcpServer>, boost::intrusive_ptr<TcpSession>, boost::system::error_code const&>, boost::_bi::list4<boost::_bi::value<TcpServer*>, boost::_bi::value<boost::intrusive_ptr<TcpServer> >, boost::_bi::value<boost::intrusive_ptr<TcpSession> >, boost::arg<1> (*)()> >, boost::system::error_code>, boost::_bi::bind_t<void, boost::_mfi::mf3<void, TcpServer, boost::intrusive_ptr<TcpServer>, boost::intrusive_ptr<TcpSession>, boost::system::error_code const&>, boost::_bi::list4<boost::_bi::value<TcpServer*>, boost::_bi::value<boost::intrusive_ptr<TcpServer> >, boost::_bi::value<boost::intrusive_ptr<TcpSession> >, boost::arg<1> (*)()> >, boost::system::error_code> (this_handler=0x7ffed3135bb0, function=...) at /usr/include/boost/asio/detail/bind_handler.hpp:88
#9 invoke<boost::asio::detail::binder1<boost::_bi::bind_t<void, boost::_mfi::mf3<void, TcpServer, boost::intrusive_ptr<TcpServer>, boost::intrusive_ptr<TcpSession>, boost::system::error_code const&>, boost::_bi::list4<boost::_bi::value<TcpServer*>, boost::_bi::value<boost::intrusive_ptr<TcpServer> >, boost::_bi::value<boost::intrusive_ptr<TcpSession> >, boost::arg<1> (*)()> >, boost::system::error_code>, boost::asio::detail::binder1<boost::_bi::bind_t<void, boost::_mfi::mf3<void, TcpServer, boost::intrusive_ptr<TcpServer>, boost::intrusive_ptr<TcpSession>, boost::system::error_code const&>, boost::_bi::list4<boost::_bi::value<TcpServer*>, boost::_bi::value<boost::intrusive_ptr<TcpServer> >, boost::_bi::value<boost::intrusive_ptr<TcpSession> >, boost::arg<1> (*)()> >, boost::system::error_code> > (context=..., function=...) at /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#10 boost::asio::detail::reactive_socket_connect_op<boost::asio::ip::tcp, boost::_bi::bind_t<void, boost::_mfi::mf3<void, TcpServer, boost::intrusive_ptr<TcpServer>, boost::intrusive_ptr<TcpSession>, boost::system::error_code const&>, boost::_bi::list4<boost::_bi::value<TcpServer*>, boost::_bi::value<boost::intrusive_ptr<TcpServer> >, boost::_bi::value<boost::intrusive_ptr<TcpSession> >, boost::arg<1> (*)()> > >::do_complete (owner=0x18fc4d0, base=<optimized out>) at /usr/include/boost/asio/detail/reactive_socket_connect_op.hpp:100
#11 0x000000000074b7ff in complete (bytes_transferred=0, ec=..., owner=..., this=<optimized out>) at /usr/include/boost/asio/detail/task_io_service_operation.hpp:37
#12 boost::asio::detail::epoll_reactor::descriptor_state::do_complete (owner=0x18fc4d0, base=0x2b715038d890, ec=..., bytes_transferred=<optimized out>) at /usr/include/boost/asio/detail/impl/epoll_reactor.ipp:651
#13 0x000000000072dea4 in complete (bytes_transferred=4, ec=..., owner=..., this=0x2b715038d890) at /usr/include/boost/asio/detail/task_io_service_operation.hpp:37
#14 boost::asio::detail::task_io_service::do_run_one (this=0x18fc4d0, lock=..., this_thread=..., ec=...) at /usr/include/boost/asio/detail/impl/task_io_service.ipp:384
#15 0x000000000072e061 in boost::asio::detail::task_io_service::run (this=0x18fc4d0, ec=...) at /usr/include/boost/asio/detail/impl/task_io_service.ipp:153
#16 0x000000000072c221 in run (this=0x18ba040, ec=...) at /usr/include/boost/asio/impl/io_service.ipp:66
#17 EventManager::Run (this=0x18ba040) at controller/src/io/event_manager.cc:35
#18 0x00000000004334cb in main (argc=<optimized out>, argv=<optimized out>) at controller/src/analytics/main.cc:448
(gdb)

Tags:

Anish Mehta (amehta00) on 2017-09-27

Changed in juniperopenstack:
importance:	Undecided → High

Revision history for this message

Anish Mehta (amehta00) wrote on 2017-09-27:

Please access the cores from the 172.30.200.176 systems, are per access details provided by Hartmut.

Megh Bhatt (meghb) on 2017-09-28

information type:

Proprietary → Public

Revision history for this message

Prashanth Nageshappa (nprashanth) wrote on 2017-09-29:

Download full text (26.0 KiB)

After moving to 4.0.1 FRS, 3 crashes were seen one SIGABRT is in _quicksort() and 2 SIGSEGV are in TcpServer::ConnectHandler()

root@canvm(analytics):/etc/contrailctl/sangupta# gdb vizd /var/crashes/core.contrail-collec.14026.canvm.1506609811

After moving to 4.0.1 FRS, 3 crashes were seen one SIGABRT is in _quicksort() and 2 SIGSEGV are in TcpServer::ConnectHandler()

root@canvm(analytics):/etc/contrailctl/sangupta# gdb vizd /var/crashes/core.contrail-collec.14026.canvm.1506609811

[New LWP 14049]
[New LWP 14047]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by usr/bin/contrail-collector'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000000000077996d in TcpServer::ConnectHandler (this=0x2b72701afd10, server=..., session=..., error=...) at controller/src/io/tcp_server.cc:410
410     controller/src/io/tcp_server.cc: No such file or directory.
(gdb) bt
#0  0x000000000077996d in TcpServer::ConnectHandler (this=0x2b72701afd10, server=..., session=..., error=...) at controller/src/io/tcp_server.cc:410
#1  0x000000000077e19a in operator() (a3=..., a2=..., a1=..., p=<optimized out>, this=0x7fff8b82e070) at /usr/include/boost/bind/mem_fn_template.hpp:393
#2  operator()<boost::_mfi::mf3<void, TcpServer, boost::intrusive_ptr<TcpServer>, boost::intrusive_ptr<TcpSession>, const boost::system::error_code&>, boost::_bi::list1<const boost::system::error_code&> > (
    a=<synthetic pointer>, f=..., this=0x7fff8b82e080) at /usr/include/boost/bind/bind.hpp:457
#3  operator()<boost::system::error_code> (a1=..., this=0x7fff8b82e070) at /usr/include/boost/bind/bind_template.hpp:47
#4  operator() (this=0x7fff8b82e070) at /usr/include/boost/asio/detail/bind_handler.hpp:47
#5  asio_handler_invoke<boost::asio::detail::binder1<boost::_bi::bind_t<void, boost::_mfi::mf3<void, TcpServer, boost::intrusive_ptr<TcpServer>, boost::intrusive_ptr<TcpSession>, boost::system::error_code const&>, boost::_bi::list4<boost::_bi::value<TcpServer*>, boost::_bi::value<boost::intrusive_ptr<TcpServer> >, boost::_bi::value<boost::intrusive_ptr<TcpSession> >, boost::arg<1> (*)()> >, boost::system::error_code> > (function=<error reading variable: access outside bounds of object referenced via synthetic pointer>) at /usr/include/boost/asio/handler_invoke_hook.hpp:64
#6  invoke<boost::asio::detail::binder1<boost::_bi::bind_t<void, boost::_mfi::mf3<void, TcpServer, boost::intrusive_ptr<TcpServer>, boost::intrusive_ptr<TcpSession>, boost::system::error_code const&>, boost::_bi::list4<boost::_bi::value<TcpServer*>, boost::_bi::value<boost::intrusive_ptr<TcpServer> >, boost::_bi::value<boost::intrusive_ptr<TcpSession> >, boost::arg<1> (*)()> >, boost::system::error_code>, boost::_bi::bind_t<void, boost::_mfi::mf3<void, TcpServer, boost::intrusive_ptr<TcpServer>, boost::intrusive_ptr<TcpSession>, boost::system::error_code const&>, boost::_bi::list4<boost::_bi::value<TcpServer*>, boost::_bi::value<boost::intrusive_ptr<TcpServer> >, boost::_bi::value<boost::intrusive_ptr<TcpSession> >, boost::arg<1> (*)()> > > (context=..., function=...)
    at /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#7  asio_handler_invoke<boost::asio::detail::binder1<boost::_bi::bind_t<void, boost::_mfi::mf3<void, TcpServer, boost::intrusive_ptr<TcpServer>, boost::intrusive_ptr<TcpSession>, boost::system::error_code const&>, boost::_bi::list4<boost::_bi::value<TcpServer*>, boost::_bi::value<boost::intrusive_ptr<TcpServer> >, boost::_bi::value<boost::intrusive_ptr<TcpSession> >, boost::arg<1> (*)()> >, boost::system::error_code>, boost::_bi::bind_t<void, boost::_mfi::mf3<void, TcpServer, boost::intrusive_ptr<TcpServer>, boost::intrusive_ptr<TcpSession>, boost::system::error_code const&>, boost::_bi::list4<boost::_bi::value<TcpServer*>, boost::_bi::value<boost::intrusive_ptr<TcpServer> >, boost::_bi::value<boost::intrusive_ptr<TcpSession> >, boost::arg<1> (*)()> >, boost::system::error_code> (this_handler=0x7fff8b82e030,
    function=...) at /usr/include/boost/asio/detail/bind_handler.hpp:88
#8  invoke<boost::asio::detail::binder1<boost::_bi::bind_t<void, boost::_mfi::mf3<void, TcpServer, boost::intrusive_ptr<TcpServer>, boost::intrusive_ptr<TcpSession>, boost::system::error_code const&>, boost::_bi::list4<boost::_bi::value<TcpServer*>, boost::_bi::value<boost::intrusive_ptr<TcpServer> >, boost::_bi::value<boost::intrusive_ptr<TcpSession> >, boost::arg<1> (*)()> >, boost::system::error_code>, boost::asio::detail::binder1<boost::_bi::bind_t<void, boost::_mfi::mf3<void, TcpServer, boost::intrusive_ptr<TcpServer>, boost::intrusive_ptr<TcpSession>, boost::system::error_code const&>, boost::_bi::list4<boost::_bi::value<TcpServer*>, boost::_bi::value<boost::intrusive_ptr<TcpServer> >, boost::_bi::value<boost::intrusive_ptr<TcpSession> >, boost::arg<1> (*)()> >, boost::system::error_code> > (context=...,
    function=...) at /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#9  boost::asio::detail::reactive_socket_connect_op<boost::asio::ip::tcp, boost::_bi::bind_t<void, boost::_mfi::mf3<void, TcpServer, boost::intrusive_ptr<TcpServer>, boost::intrusive_ptr<TcpSession>, boost::system::error_code const&>, boost::_bi::list4<boost::_bi::value<TcpServer*>, boost::_bi::value<boost::intrusive_ptr<TcpServer> >, boost::_bi::value<boost::intrusive_ptr<TcpSession> >, boost::arg<1> (*)()> > >::do_complete (owner=0x1b3dd00, base=<optimized out>) at /usr/include/boost/asio/detail/reactive_socket_connect_op.hpp:100
#10 0x000000000077daaf in complete (bytes_transferred=0, ec=..., owner=..., this=<optimized out>) at /usr/include/boost/asio/detail/task_io_service_operation.hpp:37
#11 boost::asio::detail::epoll_reactor::descriptor_state::do_complete (owner=0x1b3dd00, base=0x2b72940266e0, ec=..., bytes_transferred=<optimized out>)
    at /usr/include/boost/asio/detail/impl/epoll_reactor.ipp:651
#12 0x0000000000761104 in complete (bytes_transferred=4, ec=..., owner=..., this=0x2b72940266e0) at /usr/include/boost/asio/detail/task_io_service_operation.hpp:37
#13 boost::asio::detail::task_io_service::do_run_one (this=0x1b3dd00, lock=..., this_thread=..., ec=...) at /usr/include/boost/asio/detail/impl/task_io_service.ipp:384
#14 0x00000000007612c1 in boost::asio::detail::task_io_service::run (this=0x1b3dd00, ec=...) at /usr/include/boost/asio/detail/impl/task_io_service.ipp:153
#15 0x000000000075f481 in run (this=0x1ae2830, ec=...) at /usr/include/boost/asio/impl/io_service.ipp:66
#16 EventManager::Run (this=0x1ae2830) at controller/src/io/event_manager.cc:35
#17 0x0000000000434387 in main (argc=<optimized out>, argv=<optimized out>) at controller/src/analytics/main.cc:460
(gdb)

root@canvm(analytics):/etc/contrailctl/sangupta# gdb vizd /var/crashes/core.contrail-collec.5172.canvm.1506606690
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by usr/bin/contrail-collector'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00002b5d21cf2c37 in _quicksort (pbase=0x0, total_elems=<optimized out>, size=32, cmp=0x2b5d55401530, arg=0x2b5d55401340) at qsort.c:125
125     qsort.c: No such file or directory.
(gdb)
#0  0x00002b5d21cf2c37 in _quicksort (pbase=0x0, total_elems=<optimized out>, size=32, cmp=0x2b5d55401530, arg=0x2b5d55401340) at qsort.c:125
#1  0x00002b5d55401770 in ?? ()
#2  0x0000000000746074 in checked_delete<SandeshTrace const> (x=<optimized out>) at /usr/include/boost/checked_delete.hpp:34
#3  delete_clone<SandeshTrace> (r=<optimized out>) at /usr/include/boost/ptr_container/clone_allocator.hpp:56
#4  deallocate_clone<SandeshTrace> (r=<optimized out>) at /usr/include/boost/ptr_container/clone_allocator.hpp:74
#5  deallocate_clone (x=<optimized out>) at /usr/include/boost/ptr_container/detail/reversible_ptr_container.hpp:126
#6  operator()<SandeshTrace> (this=<synthetic pointer>, p=<optimized out>) at /usr/include/boost/ptr_container/detail/reversible_ptr_container.hpp:57
#7  ~static_move_ptr (this=<synthetic pointer>, __in_chrg=<optimized out>) at /usr/include/boost/ptr_container/detail/static_move_ptr.hpp:85
#8  push_back (ptr=0x746e33 <cass::cql::cql_trace::~cql_trace()+51>, this=0x1068010) at /usr/include/boost/ptr_container/ptr_circular_buffer.hpp:293
#9  TraceWrite (trace_entry=0x746e33 <cass::cql::cql_trace::~cql_trace()+51>, this=0x1068000) at controller/src/base/trace.h:61
#10 cass::cql::cql_trace::TraceMsg (trace_buf=..., file="controller/src/database/cassandra/cql/cql_if.cc", line=line@entry=1407,
    message="ExecuteQueryAsync:controller/src/database/cassandra/cql/cql_if.cc:1407: AsyncQuery: INSERT INTO MessageTable (key, \"Source\", \"Namespace\", \"ModuleId\", \"InstanceId\", \"NodeType\", \"MessageTS\", \"Category\","...) at build/production/database/cassandra/cql/cql_types.cpp:1229
#11 0x0000000000739a17 in ExecuteQueryAsync (cb=<error reading variable: access outside bounds of object referenced via synthetic pointer>, consistency=CASS_CONSISTENCY_LOCAL_ONE,
    query=0x2b5d604cc598 "INSERT INTO MessageTable (key, \"Source\", \"Namespace\", \"ModuleId\", \"InstanceId\", \"NodeType\", \"MessageTS\", \"Category\", \"Level\", \"Messagetype\", \"SequenceNum\", \"VersionSig\", \"Type\", \"Xmlmessage\") VALUES ("..., session=0x1068000, cci=0x101a220) at controller/src/database/cassandra/cql/cql_if.cc:1407
#12 cass::cql::CqlIfImpl::InsertIntoTableInternal(std::auto_ptr<GenDb::ColList>, CassConsistency_, bool, boost::function<void (GenDb::DbOpResult::type, std::auto_ptr<GenDb::ColList>)>) (
    this=this@entry=0x106f350, v_columns=..., consistency=consistency@entry=CASS_CONSISTENCY_LOCAL_ONE, sync=sync@entry=false, cb=...) at controller/src/database/cassandra/cql/cql_if.cc:2192
#13 0x0000000000739e59 in cass::cql::CqlIfImpl::InsertIntoTableAsync(std::auto_ptr<GenDb::ColList>, CassConsistency_, boost::function<void (GenDb::DbOpResult::type, std::auto_ptr<GenDb::ColList>)>) (
    this=0x106f350, v_columns=..., consistency=consistency@entry=CASS_CONSISTENCY_LOCAL_ONE, cb=...) at controller/src/database/cassandra/cql/cql_if.cc:1922
#14 0x000000000073a1c9 in cass::cql::CqlIf::Db_AddColumn(std::auto_ptr<GenDb::ColList>, GenDb::DbConsistency::type, boost::function<void (GenDb::DbOpResult::type)>) (this=0x106f210, cl=...,
    dconsistency=<optimized out>, cb=...) at controller/src/database/cassandra/cql/cql_if.cc:2463
#15 0x00000000006df9de in DbHandler::InsertIntoDb(std::auto_ptr<GenDb::ColList>, GenDb::DbConsistency::type, boost::function<void (GenDb::DbOpResult::type)>) (this=this@entry=0x106f600, col_list=...,
    dconsistency=dconsistency@entry=GenDb::DbConsistency::LOCAL_ONE, db_cb=...) at controller/src/analytics/db_handler.cc:664
#16 0x00000000006e5c7a in DbHandler::MessageTableOnlyInsert(VizMsg const*, boost::function<void (GenDb::DbOpResult::type)>) (this=this@entry=0x106f600, vmsgp=vmsgp@entry=0x2b5d55402480, db_cb=...)
    at controller/src/analytics/db_handler.cc:811
#17 0x00000000006ea896 in DbHandler::MessageTableInsert(VizMsg const*, boost::function<void (GenDb::DbOpResult::type)>) (this=this@entry=0x106f600, vmsgp=vmsgp@entry=0x2b5d55402480, db_cb=...)
    at controller/src/analytics/db_handler.cc:864
#18 0x0000000000658760 in Ruleeng::rule_execute(VizMsg const*, bool, DbHandler*, boost::function<void (GenDb::DbOpResult::type)>) (this=0x1082cd0, vmsgp=0x2b5d55402480, uveproc=<optimized out>,
    db=0x106f600, db_cb=...) at controller/src/analytics/ruleeng.cc:872
#19 0x000000000064f918 in operator() (a4=<error reading variable: access outside bounds of object referenced via synthetic pointer>, a3=0x106f600, a2=true, a1=0x2b5d55402480, p=0x1082cd0,
    this=0x2b5d55402238) at /usr/include/boost/bind/mem_fn_template.hpp:506
#20 operator()<bool, boost::_mfi::mf4<bool, Ruleeng, const VizMsg*, bool, DbHandler*, boost::function<void(GenDb::DbOpResult::type)> >, boost::_bi::list4<const VizMsg*&, bool&, DbHandler*&, boost::function<void(GenDb::DbOpResult::type)>&> > (a=<synthetic pointer>, f=..., this=0x2b5d55402248) at /usr/include/boost/bind/bind.hpp:515
#21 operator()<const VizMsg*, bool, DbHandler*, boost::function<void(GenDb::DbOpResult::type)> > (a4=..., a3=<synthetic pointer>, a2=<synthetic pointer>, a1=<synthetic pointer>, this=0x2b5d55402238)
    at /usr/include/boost/bind/bind_template.hpp:145
#22 boost::detail::function::function_obj_invoker4<boost::_bi::bind_t<bool, boost::_mfi::mf4<bool, Ruleeng, VizMsg const*, bool, DbHandler*, boost::function<void (GenDb::DbOpResult::type)> >, boost::_bi::list5<boost::_bi::value<Ruleeng*>, boost::arg<1>, boost::arg<2>, boost::arg<3>, boost::arg<4> > >, bool, VizMsg const*, bool, DbHandler*, boost::function<void (GenDb::DbOpResult::type)> >::invoke(boost::detail::function::function_buffer&, VizMsg const*, bool, DbHandler*, boost::function<void (GenDb::DbOpResult::type)>) (function_obj_ptr=..., a0=0x2b5d55402480, a1=<optimized out>, a2=0x106f600, a3=...)
    at /usr/include/boost/function/function_template.hpp:132
#23 0x000000000066fc56 in operator() (a3=<error reading variable: access outside bounds of object referenced via synthetic pointer>, a2=0x106f600, a1=true, a0=0x2b5d55402480, this=0x2b5d55402230)
    at /usr/include/boost/function/function_template.hpp:767
#24 SandeshGenerator::ProcessRules (this=<optimized out>, vmsg=0x2b5d55402480, rsc=<optimized out>) at controller/src/analytics/generator.cc:272
#25 0x0000000000660a9f in Collector::ReceiveSandeshMsg (this=0x10a4eb0, session=<optimized out>, msg=0x2b5d584aa900, rsc=<optimized out>) at controller/src/analytics/collector.cc:168
#26 0x0000000000849c40 in SandeshServerConnection::ProcessSandeshMessage (this=0x10917a0, msg=<optimized out>, resource=<optimized out>) at tools/sandesh/library/cpp/sandesh_connection.cc:193
#27 0x0000000000846e5c in react (event=..., this=0x2b5d58000950) at tools/sandesh/library/cpp/sandesh_state_machine.cc:371
#28 react<ssm::Established, boost::statechart::event_base, void const*> (eventType=<synthetic pointer>, evt=..., stt=...) at /usr/include/boost/statechart/custom_reaction.hpp:42
#29 local_react_impl<boost::mpl::list3<boost::statechart::custom_reaction<ssm::EvSandeshMessageRecv>, boost::statechart::in_state_reaction<ssm::EvTcpDeleteSession, SandeshStateMachine, &SandeshStateMachine::DeleteTcpSession>, boost::statechart::custom_reaction<ssm::EvResourceUpdate> >, boost::statechart::simple_state<ssm::Established, SandeshStateMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0> > (
    eventType=0xd08880 <boost::statechart::detail::id_holder<ssm::EvSandeshMessageRecv>::idProvider_>, evt=..., stt=...) at /usr/include/boost/statechart/simple_state.hpp:816
#30 local_react<boost::mpl::list3<boost::statechart::custom_reaction<ssm::EvSandeshMessageRecv>, boost::statechart::in_state_reaction<ssm::EvTcpDeleteSession, SandeshStateMachine, &SandeshStateMachine::DeleteTcpSession>, boost::statechart::custom_reaction<ssm::EvResourceUpdate> > > (eventType=0xd08880 <boost::statechart::detail::id_holder<ssm::EvSandeshMessageRecv>::idProvider_>, evt=..., this=0x2b5d58000950)
    at /usr/include/boost/statechart/simple_state.hpp:851
#31 local_react_impl<boost::mpl::list4<boost::statechart::custom_reaction<ssm::EvTcpClose>, boost::statechart::custom_reaction<ssm::EvSandeshMessageRecv>, boost::statechart::in_state_reaction<ssm::EvTcpDeleteSession, SandeshStateMachine, &SandeshStateMachine::DeleteTcpSession>, boost::statechart::custom_reaction<ssm::EvResourceUpdate> >, boost::statechart::simple_state<ssm::Established, SandeshStateMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0> > (eventType=<optimized out>, evt=..., stt=...) at /usr/include/boost/statechart/simple_state.hpp:820
#32 local_react<boost::mpl::list4<boost::statechart::custom_reaction<ssm::EvTcpClose>, boost::statechart::custom_reaction<ssm::EvSandeshMessageRecv>, boost::statechart::in_state_reaction<ssm::EvTcpDeleteSess
ion, SandeshStateMachine, &SandeshStateMachine::DeleteTcpSession>, boost::statechart::custom_reaction<ssm::EvResourceUpdate> > > (eventType=<optimized out>, evt=..., this=<optimized out>)
    at /usr/include/boost/statechart/simple_state.hpp:851
#33 local_react_impl<boost::mpl::list<boost::statechart::transition<ssm::EvStop, ssm::Idle, SandeshStateMachine, &SandeshStateMachine::OnIdle>, boost::statechart::custom_reaction<ssm::EvTcpClose>, boost::statechart::custom_reaction<ssm::EvSandeshMessageRecv>, boost::statechart::in_state_reaction<ssm::EvTcpDeleteSession, SandeshStateMachine, &SandeshStateMachine::DeleteTcpSession>, boost::statechart::custom_reaction<ssm::EvResourceUpdate> >, boost::statechart::simple_state<ssm::Established, SandeshStateMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0> > (
    eventType=0xd08880 <boost::statechart::detail::id_holder<ssm::EvSandeshMessageRecv>::idProvider_>, evt=..., stt=...) at /usr/include/boost/statechart/simple_state.hpp:820
#34 local_react<boost::mpl::list<boost::statechart::transition<ssm::EvStop, ssm::Idle, SandeshStateMachine, &SandeshStateMachine::OnIdle>, boost::statechart::custom_reaction<ssm::EvTcpClose>, boost::statechart::custom_reaction<ssm::EvSandeshMessageRecv>, boost::statechart::in_state_reaction<ssm::EvTcpDeleteSession, SandeshStateMachine, &SandeshStateMachine::DeleteTcpSession>, boost::statechart::custom_reaction<ssm::EvResourceUpdate> > > (eventType=0xd08880 <boost::statechart::detail::id_holder<ssm::EvSandeshMessageRecv>::idProvider_>, evt=..., this=0x2b5d58000950)
    at /usr/include/boost/statechart/simple_state.hpp:851
#35 boost::statechart::simple_state<ssm::Established, SandeshStateMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl (this=0x2b5d58000950, evt=...,
    eventType=0xd08880 <boost::statechart::detail::id_holder<ssm::EvSandeshMessageRecv>::idProvider_>) at /usr/include/boost/statechart/simple_state.hpp:489
#36 0x000000000084684b in operator() (this=<synthetic pointer>) at /usr/include/boost/statechart/state_machine.hpp:87
#37 operator()<boost::statechart::detail::send_function<boost::statechart::detail::state_base<std::allocator<void>, boost::statechart::detail::rtti_policy>, boost::statechart::event_base, void const*>, boost::statechart::state_machine<SandeshStateMachine, ssm::Idle>::exception_event_handler> (this=0x1123c58, action=...) at /usr/include/boost/statechart/null_exception_translator.hpp:33
#38 boost::statechart::state_machine<SandeshStateMachine, ssm::Idle, std::allocator<void>, boost::statechart::null_exception_translator>::send_event (this=0x1123c00, evt=...)
    at /usr/include/boost/statechart/state_machine.hpp:889
#39 0x000000000083f1b5 in process_event (evt=..., this=0x1123c00) at /usr/include/boost/statechart/state_machine.hpp:275
#40 SandeshStateMachine::DequeueEvent (this=0x1123c00, ec=...) at tools/sandesh/library/cpp/sandesh_state_machine.cc:800
#41 0x0000000000845ce5 in operator() (a0=<error reading variable: access outside bounds of object referenced via synthetic pointer>, this=0x2b5d55402a70)
    at /usr/include/boost/function/function_template.hpp:767
#42 QueueTaskRunner<SandeshStateMachine::EventContainer, WorkQueue<SandeshStateMachine::EventContainer> >::RunQueue (this=0x2b5d58540c40) at controller/src/base/queue_task.h:67
#43 0x0000000000479ba7 in TaskImpl::execute (this=0x2b5d2876be40) at controller/src/base/task.cc:279
#44 0x00002b5d208cfb3a in ?? () from /usr/lib/libtbb.so.2
#45 0x00002b5d208cb816 in ?? () from /usr/lib/libtbb.so.2
#46 0x00002b5d208caf4b in ?? () from /usr/lib/libtbb.so.2
#47 0x00002b5d208c70ff in ?? () from /usr/lib/libtbb.so.2
#48 0x00002b5d208c72f9 in ?? () from /usr/lib/libtbb.so.2
#49 0x00002b5d20699184 in start_thread (arg=0x2b5d55404700) at pthread_create.c:312
#50 0x00002b5d21db637d in eventfd (count=5172, flags=926298164) at ../sysdeps/unix/sysv/linux/eventfd.c:42
#51 0x0000000000000000 in ?? ()
(gdb)

root@canvm(analytics):/etc/contrailctl/sangupta# gdb vizd /var/crashes/core.contrail-collec.17112.canvm.1506612332
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by usr/bin/contrail-collector'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00002b35b0373358 in ?? ()
(gdb)
(gdb) bt
#0  0x00002b35b0373358 in ?? ()
#1  0x000000000077996f in TcpServer::ConnectHandler (this=<optimized out>, server=..., session=..., error=...) at controller/src/io/tcp_server.cc:410
#2  0x000000000077e19a in operator() (a3=..., a2=..., a1=..., p=<optimized out>, this=0x7ffcd9a6f590) at /usr/include/boost/bind/mem_fn_template.hpp:393
#3  operator()<boost::_mfi::mf3<void, TcpServer, boost::intrusive_ptr<TcpServer>, boost::intrusive_ptr<TcpSession>, const boost::system::error_code&>, boost::_bi::list1<const boost::system::error_code&> > (
    a=<synthetic pointer>, f=..., this=0x7ffcd9a6f5a0) at /usr/include/boost/bind/bind.hpp:457
#4  operator()<boost::system::error_code> (a1=..., this=0x7ffcd9a6f590) at /usr/include/boost/bind/bind_template.hpp:47
#5  operator() (this=0x7ffcd9a6f590) at /usr/include/boost/asio/detail/bind_handler.hpp:47
#6  asio_handler_invoke<boost::asio::detail::binder1<boost::_bi::bind_t<void, boost::_mfi::mf3<void, TcpServer, boost::intrusive_ptr<TcpServer>, boost::intrusive_ptr<TcpSession>, boost::system::error_code const&>, boost::_bi::list4<boost::_bi::value<TcpServer*>, boost::_bi::value<boost::intrusive_ptr<TcpServer> >, boost::_bi::value<boost::intrusive_ptr<TcpSession> >, boost::arg<1> (*)()> >, boost::system::error_code> > (function=<error reading variable: access outside bounds of object referenced via synthetic pointer>) at /usr/include/boost/asio/handler_invoke_hook.hpp:64
#7  invoke<boost::asio::detail::binder1<boost::_bi::bind_t<void, boost::_mfi::mf3<void, TcpServer, boost::intrusive_ptr<TcpServer>, boost::intrusive_ptr<TcpSession>, boost::system::error_code const&>, boost::_bi::list4<boost::_bi::value<TcpServer*>, boost::_bi::value<boost::intrusive_ptr<TcpServer> >, boost::_bi::value<boost::intrusive_ptr<TcpSession> >, boost::arg<1> (*)()> >, boost::system::error_code>, boost::_bi::bind_t<void, boost::_mfi::mf3<void, TcpServer, boost::intrusive_ptr<TcpServer>, boost::intrusive_ptr<TcpSession>, boost::system::error_code const&>, boost::_bi::list4<boost::_bi::value<TcpServer*>, boost::_bi::value<boost::intrusive_ptr<TcpServer> >, boost::_bi::value<boost::intrusive_ptr<TcpSession> >, boost::arg<1> (*)()> > > (context=..., function=...)
    at /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#8  asio_handler_invoke<boost::asio::detail::binder1<boost::_bi::bind_t<void, boost::_mfi::mf3<void, TcpServer, boost::intrusive_ptr<TcpServer>, boost::intrusive_ptr<TcpSession>, boost::system::error_code const&>, boost::_bi::list4<boost::_bi::value<TcpServer*>, boost::_bi::value<boost::intrusive_ptr<TcpServer> >, boost::_bi::value<boost::intrusive_ptr<TcpSession> >, boost::arg<1> (*)()> >, boost::system::error_code>, boost::_bi::bind_t<void, boost::_mfi::mf3<void, TcpServer, boost::intrusive_ptr<TcpServer>, boost::intrusive_ptr<TcpSession>, boost::system::error_code const&>, boost::_bi::list4<boost::_bi::value<TcpServer*>, boost::_bi::value<boost::intrusive_ptr<TcpServer> >, boost::_bi::value<boost::intrusive_ptr<TcpSession> >, boost::arg<1> (*)()> >, boost::system::error_code> (this_handler=0x7ffcd9a6f550,
    function=...) at /usr/include/boost/asio/detail/bind_handler.hpp:88
#9  invoke<boost::asio::detail::binder1<boost::_bi::bind_t<void, boost::_mfi::mf3<void, TcpServer, boost::intrusive_ptr<TcpServer>, boost::intrusive_ptr<TcpSession>, boost::system::error_code const&>, boost::_bi::list4<boost::_bi::value<TcpServer*>, boost::_bi::value<boost::intrusive_ptr<TcpServer> >, boost::_bi::value<boost::intrusive_ptr<TcpSession> >, boost::arg<1> (*)()> >, boost::system::error_code>, boost::asio::detail::binder1<boost::_bi::bind_t<void, boost::_mfi::mf3<void, TcpServer, boost::intrusive_ptr<TcpServer>, boost::intrusive_ptr<TcpSession>, boost::system::error_code const&>, boost::_bi::list4<boost::_bi::value<TcpServer*>, boost::_bi::value<boost::intrusive_ptr<TcpServer> >, boost::_bi::value<boost::intrusive_ptr<TcpSession> >, boost::arg<1> (*)()> >, boost::system::error_code> > (context=...,
    function=...) at /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#10 boost::asio::detail::reactive_socket_connect_op<boost::asio::ip::tcp, boost::_bi::bind_t<void, boost::_mfi::mf3<void, TcpServer, boost::intrusive_ptr<TcpServer>, boost::intrusive_ptr<TcpSession>, boost::system::error_code const&>, boost::_bi::list4<boost::_bi::value<TcpServer*>, boost::_bi::value<boost::intrusive_ptr<TcpServer> >, boost::_bi::value<boost::intrusive_ptr<TcpSession> >, boost::arg<1> (*)()> > >::do_complete (owner=0x2835d00, base=<optimized out>) at /usr/include/boost/asio/detail/reactive_socket_connect_op.hpp:100
#11 0x000000000077daaf in complete (bytes_transferred=0, ec=..., owner=..., this=<optimized out>) at /usr/include/boost/asio/detail/task_io_service_operation.hpp:37
#12 boost::asio::detail::epoll_reactor::descriptor_state::do_complete (owner=0x2835d00, base=0x28a4020, ec=..., bytes_transferred=<optimized out>)
    at /usr/include/boost/asio/detail/impl/epoll_reactor.ipp:651
#13 0x0000000000761104 in complete (bytes_transferred=4, ec=..., owner=..., this=0x28a4020) at /usr/include/boost/asio/detail/task_io_service_operation.hpp:37
#14 boost::asio::detail::task_io_service::do_run_one (this=0x2835d00, lock=..., this_thread=..., ec=...) at /usr/include/boost/asio/detail/impl/task_io_service.ipp:384
#15 0x00000000007612c1 in boost::asio::detail::task_io_service::run (this=0x2835d00, ec=...) at /usr/include/boost/asio/detail/impl/task_io_service.ipp:153
#16 0x000000000075f481 in run (this=0x27da830, ec=...) at /usr/include/boost/asio/impl/io_service.ipp:66
#17 EventManager::Run (this=0x27da830) at controller/src/io/event_manager.cc:35
#18 0x0000000000434387 in main (argc=<optimized out>, argv=<optimized out>) at controller/src/analytics/main.cc:460

Revision history for this message

Prashanth Nageshappa (nprashanth) wrote on 2017-09-29:

Download full text (4.2 KiB)

I see another crash in the 4.0.1FRS installation

root@canvm(analytics):/etc/contrailctl/sangupta# gdb vizd /var/crashes/core.contrail-collec.28561.canvm.1506649563
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-collector'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00002b87ce5d3c37 in __GI_raise (sig=sig@entry=6)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 0x00002b87ce5d3c37 in __GI_raise (sig=sig@entry=6)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00002b87ce5d7028 in __GI_abort () at abort.c:89
#2 0x00002b87ce6102a4 in __libc_message (do_abort=1,
    fmt=fmt@entry=0x2b87ce722310 "*** Error in `%s': %s: 0x%s ***\n")
    at ../sysdeps/posix/libc_fatal.c:175
#3 0x00002b87ce61be23 in malloc_printerr (ptr=0x2b88080d3cf0,
    str=0x2b87ce71e424 "corrupted size vs. prev_size", action=<optimized out>)
    at malloc.c:4998
#4 malloc_consolidate (av=av@entry=0x2b8808000020) at malloc.c:4167
#5 0x00002b87ce61d8b8 in _int_malloc (av=0x2b8808000020, bytes=20448) at malloc.c:3425
#6 0x00002b87ce61fae0 in __GI___libc_malloc (bytes=20448) at malloc.c:2893
#7 0x00002b87cdddbdad in operator new(unsigned long) ()
   from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8 0x00002b87cde37249 in std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#9 0x00002b87cde37e0b in std::string::_Rep::_M_clone(std::allocator<char> const&, unsigned long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00002b87cde37ea4 in std::string::reserve(unsigned long) ()
   from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#11 0x00000000008288e6 in ReplaceBuf (str="", this=0x22cfe40)
    at tools/sandesh/library/cpp/sandesh_session.cc:505
#12 SetBuf (str="", this=0x22cfe40) at tools/sandesh/library/cpp/sandesh_session.cc:496
#13 SandeshReader::ExtractMsg (this=0x22cfe40, buffer=..., result=0x2b87f57358d0,
    NewBuf=<optimized out>) at tools/sandesh/library/cpp/sandesh_session.cc:556
#14 0x000000000082896f in SandeshReader::OnRead (this=0x22cfe40, buffer=...)
---Type <return> to continue, or q <return> to quit---
    at tools/sandesh/library/cpp/sandesh_session.cc:585
#15 0x000000000076b3f0 in call<SslSession*, boost::asio::const_buffer> (u=<optimized out>,
    b1=<synthetic pointer>, this=<optimized out>)
    at /usr/include/boost/bind/mem_fn_template.hpp:156
#16 operator()<SslSession*> (u=<optimized out>, a1=..., this=<optimized out>)
    at /usr/include/boost/bind/mem_fn_template.hpp:171
#17 operator()<boost::_mfi::mf1<void, TcpSession, boost::asio::const_buffer>, boost::_bi::list1<boost::asio::const_buffer&> > (a=<synthetic pointer>, f=..., this=<optimized out>)
    at /usr/include/boost/bind/bind.hpp:313
#18 operator()<boost::asio::const_buffer> (a1=<synthetic pointer>, this=<optimized out>)
    at /usr/include/boost/bind/bind_template.hpp:32
#19 boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, TcpSession, boost::...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.