RHOSP provision failed. Config database failed to come up

Bug #1743900 reported by shajuvk
26
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R4.0
Fix Committed
Critical
alexey-mr
R4.1
Fix Committed
Critical
alexey-mr
Trunk
Fix Committed
Critical
alexey-mr

Bug Description

INFO [main] 2018-01-17 20:24:03,938 RateBasedBackPressure.java:123 - Initialized back-pressure with high ratio: 0.9, factor: 5, flow: FAST, window size: 2000.
INFO [main] 2018-01-17 20:24:03,938 DatabaseDescriptor.java:710 - Back-pressure is disabled with strategy null.
INFO [main] 2018-01-17 20:33:54,467 GossipingPropertyFileSnitch.java:64 - Loaded cassandra-topology.properties for compatibility
Exception (java.lang.AbstractMethodError) encountered during startup: org.apache.cassandra.utils.JMXServerUtils$Exporter.exportObject(Ljava/rmi/Remote;ILjava/rmi/server/RMIClientSocketFactory;Ljava/rmi/server/RMIServerSocketFactory;Lsun/misc/ObjectInputFilter;)Ljava/rmi/Remote;
java.lang.AbstractMethodError: org.apache.cassandra.utils.JMXServerUtils$Exporter.exportObject(Ljava/rmi/Remote;ILjava/rmi/server/RMIClientSocketFactory;Ljava/rmi/server/RMIServerSocketFactory;Lsun/misc/ObjectInputFilter;)Ljava/rmi/Remote;
        at javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:150)
        at javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:135)
        at javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:405)
        at org.apache.cassandra.utils.JMXServerUtils.createJMXServer(JMXServerUtils.java:104)
        at org.apache.cassandra.service.CassandraDaemon.maybeInitJmx(CassandraDaemon.java:144)
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:189)
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:601)
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:735)
ERROR [main] 2018-01-17 20:33:54,512 CassandraDaemon.java:752 - Exception encountered during startup
java.lang.AbstractMethodError: org.apache.cassandra.utils.JMXServerUtils$Exporter.exportObject(Ljava/rmi/Remote;ILjava/rmi/server/RMIClientSocketFactory;Ljava/rmi/server/RMIServerSocketFactory;Lsun/misc/ObjectInputFilter;)Ljava/rmi/Remote;
        at javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:150) ~[na:1.8.0_161]
        at javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:135) ~[na:1.8.0_161]
        at javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:405) ~[na:1.8.0_161]
        at org.apache.cassandra.utils.JMXServerUtils.createJMXServer(JMXServerUtils.java:104) ~[apache-cassandra-3.10.jar:3.10]
        at org.apache.cassandra.service.CassandraDaemon.maybeInitJmx(CassandraDaemon.java:144) [apache-cassandra-3.10.jar:3.10]
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:189) [apache-cassandra-3.10.jar:3.10]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:601) [apache-cassandra-3.10.jar:3.10]
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:735) [apache-cassandra-3.10.jar:3.10]

Revision history for this message
shajuvk (shajuvk) wrote :

[root@overcloud-contrailcontroller-0 ~]# contrail-status
== Contrail Control ==
supervisor-control: active
contrail-control initializing (Database:Cassandra connection down)
contrail-control-nodemgr timeout
contrail-dns active
contrail-named active

== Contrail Config ==
supervisor-config: active
contrail-api:0 initializing (Database:Cassandra[] connection down)
contrail-config-nodemgr active
contrail-device-manager initializing (ApiServer:ApiServer[] connection down)
contrail-schema initializing (ApiServer:ApiServer[] connection down)
contrail-svc-monitor initializing (Database:Cassandra[] connection down)

== Contrail Web UI ==
supervisor-webui: active
contrail-webui active
contrail-webui-middleware active

== Contrail Database ==
contrail-database: active
[root@overcloud-contrailcontroller-0 ~]#

tags: added: sanityblocker
information type: Proprietary → Public
Revision history for this message
shajuvk (shajuvk) wrote :
Download full text (12.6 KiB)

  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
  deploy_stdout: |
    ...
    Notice: /Stage[main]/Contrail::Database::Provision_database/Exec[provision_database_node.py ]/returns: File "/opt/contrail/utils/provision_database_node.py", line 35, in __init__
    Notice: /Stage[main]/Contrail::Database::Provision_database/Exec[provision_database_node.py ]/returns: api_server_use_ssl=self._args.api_server_use_ssl)
    Notice: /Stage[main]/Contrail::Database::Provision_database/Exec[provision_database_node.py ]/returns: File "/opt/contrail/utils/vnc_admin_api.py", line 35, in __init__
    Notice: /Stage[main]/Contrail::Database::Provision_database/Exec[provision_database_node.py ]/returns: auth_host=self.auth_host)
    Notice: /Stage[main]/Contrail::Database::Provision_database/Exec[provision_database_node.py ]/returns: File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 454, in __init__
    Notice: /Stage[main]/Contrail::Database::Provision_database/Exec[provision_database_node.py ]/returns: retry_on_error=False)
    Notice: /Stage[main]/Contrail::Database::Provision_database/Exec[provision_database_node.py ]/returns: File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 935, in _request
    Notice: /Stage[main]/Contrail::Database::Provision_database/Exec[provision_database_node.py ]/returns: raise ConnectionError
    Notice: /Stage[main]/Contrail::Database::Provision_database/Exec[provision_database_node.py ]/returns: requests.exceptions.ConnectionError
    Notice: Finished catalog run in 360.28 seconds
    (truncated, view all with --long)
  deploy_stderr: |
    ...
    Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Array. There is further documentation for validate_legacy function in the README.
       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:19:in `deprecation')
    Warning: This method is deprecated, please use the stdlib validate_legacy function, with Pattern[]. There is further documentation for validate_legacy function in the README.
       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:19:in `deprecation')
    Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Numeric. There is further documentation for validate_legacy function in the README.
       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:19:in `deprecation')
    Warning: This method is deprecated, please use match expressions with Stdlib::Compat::String instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions.
       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:19:in `deprecation')
    Error: python /opt/contrail/utils/provision_database_node.py --host_name overcloud-contrailanalyticsdatabase-0 --host_ip 10.0.0.60 --api_server_ip 10.0.0.10 --api_server_port 8082 --admin_user admin --ad...

Revision history for this message
shajuvk (shajuvk) wrote :

2018-01-17 20:49:20Z [overcloud.AllNodesExtraConfig]: CREATE_IN_PROGRESS state changed
2018-01-17 20:49:21Z [overcloud.AllNodesExtraConfig]: CREATE_COMPLETE state changed
2018-01-17 20:49:22Z [overcloud.AllNodesDeploySteps]: CREATE_IN_PROGRESS state changed
2018-01-17 21:15:29Z [overcloud.AllNodesDeploySteps]: CREATE_FAILED Error: resources.AllNodesDeploySteps.resources.ContrailAnalyticsDeployment_Step5.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6
2018-01-17 21:15:29Z [overcloud]: CREATE_FAILED Resource CREATE failed: Error: resources.AllNodesDeploySteps.resources.ContrailAnalyticsDeployment_Step5.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6
[stack@undercloud ~]$

Revision history for this message
shajuvk (shajuvk) wrote :

[stack@undercloud ~]$ rpm -qa | grep contrail
python-gevent-1.1rc5-1contrail1.el7.x86_64
contrail-tripleo-heat-templates-4.1.1.0-88.el7.noarch
puppet-contrail-4.1.1.0-88.el7.noarch
contrail-tripleo-puppet-4.1.1.0-88.el7.noarch
[stack@undercloud ~]$
[stack@undercloud ~]$ sudo yum repolist
Loaded plugins: search-disabled-repos
repo id repo name status
contrail contrail rep 99
rhel-7-server-extras-rpms/x86_64 Red Hat Enterprise Linux 7 Server - Extras (RPMs) 712
rhel-7-server-nfv-rpms/x86_64 Red Hat Enterprise Linux for Real Time for NFV (RHEL 7 Server) (RPMs) 246
rhel-7-server-openstack-10-devtools-rpms/x86_64 Red Hat OpenStack Platform 10 Developer Tools for RHEL 7 (RPMs) 8
rhel-7-server-openstack-10-rpms/x86_64 Red Hat OpenStack Platform 10 for RHEL 7 (RPMs) 1,717
rhel-7-server-rh-common-rpms/x86_64 Red Hat Enterprise Linux 7 Server - RH Common (RPMs) 231
rhel-7-server-rpms/x86_64 Red Hat Enterprise Linux 7 Server (RPMs) 17,906
rhel-ha-for-rhel-7-server-rpms/x86_64 Red Hat Enterprise Linux High Availability (for RHEL 7 Server) (RPMs) 391
repolist: 21,310
[stack@undercloud ~]

Revision history for this message
shajuvk (shajuvk) wrote :

==========================
logs are at /cs-shared/bugs/1743900

====================

[root@overcloud-contrailcontroller-0 ~]# contrail-status
== Contrail Control ==
supervisor-control: active
contrail-control initializing (Database:Cassandra connection down)
contrail-control-nodemgr active
contrail-dns active
contrail-named active

== Contrail Config ==
supervisor-config: active
contrail-api:0 initializing (Database:Cassandra[] connection down)
contrail-config-nodemgr active
contrail-device-manager initializing (ApiServer:ApiServer[] connection down)
contrail-schema initializing (ApiServer:ApiServer[] connection down)
contrail-svc-monitor initializing (Database:Cassandra[] connection down)

== Contrail Web UI ==
supervisor-webui: active
contrail-webui active
contrail-webui-middleware active

== Contrail Database ==
contrail-database: active
[root@overcloud-contrailcontroller-0 ~]#

Revision history for this message
alexey-mr (alexey-morlang) wrote :

That looks to be cassandra+java issue with latest java: java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64

Last successful run was with java-1.8.0-openjdk-1.8.0.151-5.b12.el7_4.x86_64

Other people have the same:
https://<email address hidden>/msg54771.html

So, it looks cassandra people should fix it.

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/39036
Submitter: alexey-mr (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.1

Review in progress for https://review.opencontrail.org/39037
Submitter: alexey-mr (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.0

Review in progress for https://review.opencontrail.org/39038
Submitter: alexey-mr (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/39036
Committed: http://github.com/Juniper/puppet-contrail/commit/ecf700b52054ffb9d73768dc4ae31d87dc66fc98
Submitter: Zuul (<email address hidden>)
Branch: master

commit ecf700b52054ffb9d73768dc4ae31d87dc66fc98
Author: alexey-mr <email address hidden>
Date: Thu Jan 18 18:30:40 2018 +0300

Temporary fix java version because of Bug #1743900

This commit is to be reverted after 3.11 cassandra release

Change-Id: I22508499e79668bbc2090fa4a46936fc7c5db302
Closes-Bug: #1743900

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/39037
Committed: http://github.com/Juniper/puppet-contrail/commit/c56ff6d61d0490867b15f58445ec64f85fe735e5
Submitter: Zuul (<email address hidden>)
Branch: R4.1

commit c56ff6d61d0490867b15f58445ec64f85fe735e5
Author: alexey-mr <email address hidden>
Date: Thu Jan 18 18:30:40 2018 +0300

Temporary fix java version because of Bug #1743900

This commit is to be reverted after 3.11 cassandra release

Change-Id: I22508499e79668bbc2090fa4a46936fc7c5db302
Closes-Bug: #1743900

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/39038
Committed: http://github.com/Juniper/puppet-contrail/commit/6a15a12036a88155f606c1687fc98506f20981f2
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit 6a15a12036a88155f606c1687fc98506f20981f2
Author: alexey-mr <email address hidden>
Date: Thu Jan 18 18:30:40 2018 +0300

Temporary fix java version because of Bug #1743900

This commit is to be reverted after 3.11 cassandra release

Change-Id: I22508499e79668bbc2090fa4a46936fc7c5db302
Closes-Bug: #1743900

Revision history for this message
alexey-mr (alexey-morlang) wrote :
Revision history for this message
shajuvk (shajuvk) wrote :

yumdownloader java-1.8.0-openjdk-headless-1.8.0.151-5.b12.el7_4.x86_64

yumdownloader java-1.8.0-openjdk-1.8.0.151-5.b12.el7_4.x86_64

yum downgrade java-1.8.0-openjdk-1.8.0.151-5.b12.el7_4.x86_64.rpm java-1.8.0-openjdk-headless-1.8.0.151-5.b12.el7_4.x86_64.rpm

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.