Juniper Openstack

Redhat-osp10-R4.0.1-build-25-Contrail-collector crash: OpServerProxy::OpServerImpl::processorCallbackProcess(redisAsyncContext const*, void*, void*)

Bug #1711250 reported by shajuvk
22
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R4.0
Fix Committed
High
Zhiqiang Cui
Juniper Openstack r4.0.1.0 "r4.0.1.0"
Trunk
Fix Committed
High
Zhiqiang Cui
Juniper Openstack r4.1.0.0-fcs "r4.1"

Bug Description

root@overcloud-contrailanalytics-2(analytics):/# contrail-status
== Contrail Analytics ==
supervisor-analytics: active
contrail-alarm-gen active
contrail-analytics-api active
contrail-analytics-nodemgr active
contrail-collector active
contrail-query-engine active
contrail-snmp-collector active
contrail-topology active

========Run time service failures=============
/var/crashes/core.contrail-collec.3181.overcloud-contrailanalytics-2.1502927778
 bt
====

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-collector'.
Program terminated with signal 6, Aborted.
#0 0x00002ab21f9a11d7 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install contrail-analytics-4.0.1.0-25.el7.x86_64
(gdb) bt
#0 0x00002ab21f9a11d7 in raise () from /lib64/libc.so.6
#1 0x00002ab21f9a28c8 in abort () from /lib64/libc.so.6
#2 0x00002ab21f99a146 in __assert_fail_base () from /lib64/libc.so.6
#3 0x00002ab21f99a1f2 in __assert_fail () from /lib64/libc.so.6
#4 0x000000000077550f in OpServerProxy::OpServerImpl::processorCallbackProcess(redisAsyncContext const*, void*, void*) ()
#5 0x000000000077b029 in boost::_mfi::mf3<void, OpServerProxy::OpServerImpl, redisAsyncContext const*, void*, void*>::operator()(OpServerProxy::OpServerImpl*, redisAsyncContext const*, void*, void*) const ()
#6 0x0000000000778da5 in void boost::_bi::list4<boost::_bi::value<OpServerProxy::OpServerImpl*>, boost::arg<1>, boost::arg<2>, boost::arg<3> >::operator()<boost::_mfi::mf3<void, OpServerProxy::OpServerImpl, redisAsyncContext const*, void*, void*>, boost::_bi::list3<redisAsyncContext const*&, void*&, void*&> >(boost::_bi::type<void>, boost::_mfi::mf3<void, OpServerProxy::OpServerImpl, redisAsyncContext const*, void*, void*>&, boost::_bi::list3<redisAsyncContext const*&, void*&, void*&>&, int) ()
#7 0x0000000000778905 in void boost::_bi::bind_t<void, boost::_mfi::mf3<void, OpServerProxy::OpServerImpl, redisAsyncContext const*, void*, void*>, boost::_bi::list4<boost::_bi::value<OpServerProxy::OpServerImpl*>, boost::arg<1>, boost::arg<2>, boost::arg<3> > >::operator()<redisAsyncContext const*, void*, void*>(redisAsyncContext const*&, void*&, void*&) ()
#8 0x0000000000778216 in boost::detail::function::void_function_obj_invoker3<boost::_bi::bind_t<void, boost::_mfi::mf3<void, OpServerProxy::OpServerImpl, redisAsyncContext const*, void*, void*>, boost::_bi::list4<boost::_bi::value<OpServerProxy::OpServerImpl*>, boost::arg<1>, boost::arg<2>, boost::arg<3> > >, void, redisAsyncContext const*, void*, void*>::invoke(boost::detail::function::function_buffer&, redisAsyncContext const*, void*, void*) ()
#9 0x00000000006ac6d1 in boost::function3<void, redisAsyncContext const*, void*, void*>::operator()(redisAsyncContext const*, void*, void*) const
    ()
#10 0x00000000006a9b65 in RedisAsyncConnection::RAC_AsyncCmdCallback(redisAsyncContext*, void*, void*) ()
#11 0x0000000000a1f510 in __redisRunCallback ()
#12 0x0000000000a1fc2f in redisProcessCallbacks ()
#13 0x0000000000a1fe2c in redisAsyncHandleRead ()
#14 0x0000000000a20863 in redisBoostClient::handle_read(boost::system::error_code) ()
#15 0x0000000000a21e38 in void boost::_mfi::mf1<void, redisBoostClient, boost::system::error_code>::call<boost::shared_ptr<redisBoostClient>, boost::system::error_code>(boost::shared_ptr<redisBoostClient>&, void const*, boost::system::error_code&) const ()
#16 0x0000000000a21daf in void boost::_mfi::mf1<void, redisBoostClient, boost::system::error_code>::operator()<boost::shared_ptr<redisBoostClient> >(boost::shared_ptr<redisBoostClient>&, boost::system::error_code) const ()
#17 0x0000000000a21d1a in void boost::_bi::list2<boost::_bi::value<boost::shared_ptr<redisBoostClient> >, boost::arg<1> (*)()>::operator()<boost::_mfi::mf1<void, redisBoostClient, boost::system::error_code>, boost::_bi::list2<boost::system::error_code const&, unsigned long const&> >(boost::_bi::type<void>, boost::_mfi::mf1<void, redisBoostClient, boost::system::error_code>&, boost::_bi::list2<boost::system::error_code const&, unsigned long const&>&, int) ()
#18 0x0000000000a21ca0 in void boost::_bi::bind_t<void, boost::_mfi::mf1<void, redisBoostClient, boost::system::error_code>, boost::_bi::list2<boost::_bi::value<boost::shared_ptr<redisBoostClient> >, boost::arg<1> (*)()> >::operator()<boost::system::error_code, unsigned long>(boost::system::error_code const&, unsigned long const&) ()
#19 0x0000000000a21c51 in boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf1<void, redisBoostClient, boost::system::error_code>, boost::_bi::list2<boost::_bi::value<boost::shared_ptr<redisBoostClient> >, boost::arg<1> (*)()> >, boost::system::error_code, unsigned long>::operator()() ()
#20 0x0000000000a21c23 in void boost::asio::asio_handler_invoke<boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf1<void, redisBoostClient, boost::system::error_code>, boost::_bi::list2<boost::_bi::value<boost::shared_ptr<redisBoostClient> >, boost::arg<1> (*)()> >, boost::system::error_code, unsigned long> >(boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf1<void, redisBoostClient, boost::system::error_code>, boost::_bi::list2<boost::_bi::value<boost::shared_ptr<redisBoostClient> >, boost::arg<1> (*)()> >, boost::system::error_code, unsigned long>, ...) ()
#21 0x0000000000a21b8c in void boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf1<void, redisBoostClient, boost::system::error_code>, boost::_bi::list2<boost::_bi::value<boost::shared_ptr<redisBoostClient> >, boost::arg<1> (*)()> >, boost::system::error_code, unsigned long>, boost::_bi::bind_t<void, boost::_mfi::mf1<void, redisBoostClient, boost::system::error_code>, boost::_bi::list2<boost::_bi::value<boost::shared_ptr<redisBoostClient> >, boost::arg<1> (*)()> > >(boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf1<void, redisBoostClient, boost::system::error_code>, boost::_bi::list2<boost::_bi::value<boost::shared_ptr<redisBoostClient> >, boost::arg<1> (*)()> >, boost::system::error_code, unsigned long>&, boost::_bi::bind_t<void, boost::_mfi::mf1<void, redisBoostClient, boost::system::error_code>, bo---Type <return> to continue, or q <return> to quit---
ost::_bi::list2<boost::_bi::value<boost::shared_ptr<redisBoostClient> >, boost::arg<1> (*)()> >&) ()
#22 0x0000000000a219f3 in boost::asio::detail::reactive_null_buffers_op<boost::_bi::bind_t<void, boost::_mfi::mf1<void, redisBoostClient, boost::system::error_code>, boost::_bi::list2<boost::_bi::value<boost::shared_ptr<redisBoostClient> >, boost::arg<1> (*)()> > >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) ()
#23 0x0000000000832b84 in boost::asio::detail::task_io_service_operation::complete(boost::asio::detail::task_io_service&, boost::system::error_code const&, unsigned long) ()
#24 0x0000000000833af4 in boost::asio::detail::task_io_service::do_run_one(boost::asio::detail::scoped_lock<boost::asio::detail::posix_mutex>&, boost::asio::detail::task_io_service::thread_info&, boost::asio::detail::op_queue<boost::asio::detail::task_io_service_operation>&, boost::system::error_code const&) ()
#25 0x0000000000833425 in boost::asio::detail::task_io_service::run(boost::system::error_code&) ()
#26 0x0000000000833fb1 in boost::asio::io_service::run(boost::system::error_code&) ()
#27 0x0000000000831992 in EventManager::Run() ()
#28 0x0000000000726073 in main ()
(gdb)

===

root@overcloud-contrailanalytics-2(analytics):/# contrail-version
Package Version Build-ID | Repo | RPM Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-analytics 4.0.1.0-25.el7 @contrail-install-repo
contrail-docs 4.0.1.0-25.el7 @contrail-install-repo
contrail-lib 4.0.1.0-25.el7 @contrail-install-repo
contrail-nodemgr 4.0.1.0-25.el7 @contrail-install-repo
contrail-openstack-analytics 4.0.1.0-25.el7 @contrail-install-repo
contrail-setup 4.0.1.0-25.el7 @contrail-install-repo
contrail-utils 4.0.1.0-25.el7 @contrail-install-repo
python-contrail 4.0.1.0-25.el7 @contrail-install-repo

Revision history for this message
shajuvk (shajuvk) wrote :

core location:

bash-4.1$ pwd
/cs-shared/bugs/1711250
bash-4.1$ ls
core.contrail-collec.3181.overcloud-contrailanalytics-2.1502927778

tags: added: analytics redhat
summary: - Redhat-R4.0.1-Contrail-collector error:
+ Redhat-R4.0.1-Contrail-collector crash:
OpServerProxy::OpServerImpl::processorCallbackProcess(redisAsyncContext
const*, void*, void*)
summary: - Redhat-R4.0.1-Contrail-collector crash:
+ Redhat-R4.0.1-build-25-Contrail-collector crash:
OpServerProxy::OpServerImpl::processorCallbackProcess(redisAsyncContext
const*, void*, void*)
summary: - Redhat-R4.0.1-build-25-Contrail-collector crash:
+ Redhat-osp10-R4.0.1-build-25-Contrail-collector crash:
OpServerProxy::OpServerImpl::processorCallbackProcess(redisAsyncContext
const*, void*, void*)
information type: Proprietary → Public
Revision history for this message
Zhiqiang Cui (zcui) wrote :

Need log for analyzing:
/var/log/contrail/*.*
/var/log/redis/*.*

Revision history for this message
shajuvk (shajuvk) wrote :

Hi Zhiqiang,

I have another system with similar crash. Could you please check ?

Steps to connect to system:

1. ssh root@10.87.67.42
2. su stack
3. ssh heat-admin@10.87.67.19
4. sudo su
5 docker exec -it contrail-analytics-container bash
6 check contrail-status

If you have any difficulty in login, please let me know.

Thanks,
Shaju

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.0

Review in progress for https://review.opencontrail.org/34861
Submitter: Zhiqiang Cui (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/34861
Committed: http://github.com/Juniper/contrail-controller/commit/9ab515fde8800b6cc3d7552520cebf3a4495f3fe
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit 9ab515fde8800b6cc3d7552520cebf3a4495f3fe
Author: zcui <email address hidden>
Date: Wed Aug 23 19:12:23 2017 -0700

Redis lua script return nil lead assert. Add more log for more
invetigation.

Change-Id: I10cd8d6b103dbd3198e4a149128836c796107fa8
Partial-Bug:1711250

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/34989
Submitter: Zhiqiang Cui (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/34989
Committed: http://github.com/Juniper/contrail-controller/commit/de09ed2056736756dbf8ce29488c67714be1a052
Submitter: Zuul (<email address hidden>)
Branch: master

commit de09ed2056736756dbf8ce29488c67714be1a052
Author: zcui <email address hidden>
Date: Wed Aug 23 19:12:23 2017 -0700

Redis lua script return nil lead assert. Add more log for more
invetigation.

Change-Id: I10cd8d6b103dbd3198e4a149128836c796107fa8
Partial-Bug:1711250
(cherry picked from commit 9ab515fde8800b6cc3d7552520cebf3a4495f3fe)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.0

Review in progress for https://review.opencontrail.org/35219
Submitter: Zhiqiang Cui (<email address hidden>)

Jeba Paulaiyan (jebap)
tags: added: sanity
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/35276
Submitter: Zhiqiang Cui (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/35219
Committed: http://github.com/Juniper/contrail-controller/commit/aaa620acbbf42f8a97f58063843300c60459923a
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit aaa620acbbf42f8a97f58063843300c60459923a
Author: zcui <email address hidden>
Date: Sat Sep 2 12:09:28 2017 -0700

Docker senario: At the start of docker, a rapid generator client up/down/up
action, this will trigger getseq/updateUVE/delreqUVE in a very short time.
Because updateUVE is ansync, but delreqUVE is sync, so lua receive del early
than update, update will lead return nil.
So we remove assert for nil in opserver proxy, since current log is enough
to catch real problem.

Change-Id: Iad8a3f9e74b5b965daccffddd193ee02a4b0fc5b
Closes-bug:1711250

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/35276
Committed: http://github.com/Juniper/contrail-controller/commit/4dd602bebb371f87f63d8bf9d4bcb286a9e8ccda
Submitter: Zuul (<email address hidden>)
Branch: master

commit 4dd602bebb371f87f63d8bf9d4bcb286a9e8ccda
Author: zcui <email address hidden>
Date: Sat Sep 2 12:09:28 2017 -0700

Docker senario: At the start of docker, a rapid generator client up/down/up
action, this will trigger getseq/updateUVE/delreqUVE in a very short time.
Because updateUVE is ansync, but delreqUVE is sync, so lua receive del early
than update, update will lead return nil.
So we remove assert for nil in opserver proxy, since current log is enough
to catch real problem.

Change-Id: Iad8a3f9e74b5b965daccffddd193ee02a4b0fc5b
Closes-bug:1711250

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.