RHOSP10-5.0-Build-161-Newton:Provisioning failed while executing provision_database.py

Bug #1741951 reported by shajuvk
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
Trunk
Fix Committed
Critical
alexey-mr

Bug Description

Provisioning of database on Newton-R5.0-Build-161 Failed.

Jan 8 02:59:13 localhost os-collect-config: returns: auth_host=self.auth_host)\u001b[0m\n\u001b[mNotice: /Stage[main]/Contrail::Database::Provision_database/Exec[provision_database_node.py ]/returns: File \"/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py\", line 460, in __init__\u001b[0m\n\u001b[mNotice: /Stage[main]/Contrail::Database::Provision_database/Exec[provision_database_node.py ]/returns: retry_on_error=False)\u001b[0m\n\u001b[mNotice: /Stage[main]/Contrail::Database::Provision_database/Exec[provision_database_node.py ]/returns: File \"/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py\", line 943, in _request\u001b[0m\n\u001b[mNotice: /Stage[main]/Contrail::Database::Provision_database/Exec[provision_database_node.py ]/returns: raise ConnectionError\u001b[0m\n\u001b[mNotice: /Stage[main]/Contrail::Database::Provision_database/Exec[provision_database_node.py ]/returns: requests.exceptions.ConnectionError\u001b[0m\n\u001b[mNotice: Finished catalog run in 346.94 seconds\u001b[0m\n", "deploy_stderr": "exception: connect failed\n\u001b[1;31mWarning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Bool. There is further documentation for validate_legacy function in the README.\n (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:19:in `deprecation')\u001b[0m\n\u001b[1;31mWarning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Absolute_Path. There is further documentation for validate_legacy function in the README.\n (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:19:in `deprecation')\u001b[0m\n\u001b[1;31mWarning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::String. There is further documentation for validate_legacy function in the README.\n (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:19:in `deprecation')\u001b[0m\n\u001b[1;31mWarning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::C
Jan 8 02:59:13 localhost os-collect-config: ompat::Array. There is further documentation for validate_legacy function in the README.\n (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:19:in `deprecation')\u001b[0m\n\u001b[1;31mWarning: This method is deprecated, please use the stdlib validate_legacy function, with Pattern[]. There is further documentation for validate_legacy function in the README.\n (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:19:in `deprecation')\u001b[0m\n\u001b[1;31mWarning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Numeric. There is further documentation for validate_legacy function in the README.\n (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:19:in `deprecation')\u001b[0m\n\u001b[1;31mWarning: This method is deprecated, please use match expressions with Stdlib::Compat::String instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions.\n (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:19:in `deprecation')\u001b[0m\n\u001b[1;31mError: python /opt/contrail/utils/provision_database_node.py --host_name overcloud-contrailanalyticsdatabase-0 --host_ip 10.0.0.60 --api_server_ip 10.0.0.10 --api_server_port 8082 --admin_user admin --admin_password CZxWF4F9PjHG6j67xDZHRT2wu --admin_tenant admin --openstack_ip 192.168.24.11 --oper add returned 1 instead of one of [0]\u001b[0m\n\u001b[1;31mError: /Stage[main]/Contrail::Database::Provision_database/Exec[provision_database_node.py ]/returns: change from notrun to 0 failed: python /opt/contrail/utils/provision_database_node.py --host_name overcloud-contrailanalyticsdatabase-0 --host_ip 10.0.0.60 --api_server_ip 10.0.0.10 --api_server_port 8082 --admin_user admin
Jan 8 02:59:13 localhost os-collect-config: --admin_password CZxWF4F9PjHG6j67xDZHRT2wu --admin_tenant admin --openstack_ip 192.168.24.11 --oper add returned 1 instead of one of [0]\u001b[0m\n", "deploy_status_code": 6}

=======

[root@overcloud-contrailanalyticsdatabase-0 ~]# yum repolist
Loaded plugins: product-id, search-disabled-repos, subscription-manager
repo id repo name status
!Contrail Contrail Repo 98
rhel-7-server-extras-rpms/x86_64 Red Hat Enterprise Linux 7 Server - Extras (RPMs) 712
rhel-7-server-openstack-10-devtools-rpms/7Server/x86_64 Red Hat OpenStack Platform 10 Developer Tools for RHEL 7 (RPMs) 8
rhel-7-server-openstack-10-rpms/7Server/x86_64 Red Hat OpenStack Platform 10 for RHEL 7 (RPMs) 1,715
rhel-7-server-rh-common-rpms/7Server/x86_64 Red Hat Enterprise Linux 7 Server - RH Common (RPMs) 231
rhel-7-server-rpms/7Server/x86_64 Red Hat Enterprise Linux 7 Server (RPMs) 17,881
rhel-ha-for-rhel-7-server-rpms/7Server/x86_64 Red Hat Enterprise Linux High Availability (for RHEL 7 Server) (RPMs) 391
repolist: 21,036
[root@overcloud-contrailanalyticsdatabase-0 ~]#
[root@overcloud-contrailanalyticsdatabase-0 ~]#
[root@overcloud-contrailanalyticsdatabase-0 ~]# rpm -qa | grep contrail
contrail-tripleo-puppet-5.0.0-161.el7.noarch
python-bitarray-0.8.0-0contrail.el7.x86_64
contrail-nodemgr-5.0.0-161.el7.x86_64
python-contrail-5.0.0-161.el7.x86_64
python-pycassa-1.10.0-0contrail.el7.noarch
supervisor-3.1.3-3.0contrail0.el7.noarch
contrail-database-5.0.0-161.el7.noarch
python-bottle-0.11.6-0contrail.el7.noarch
contrail-utils-5.0.0-161.el7.x86_64
kafka-2.11-0.9.0.1.0contrail0.el7.x86_64
contrail-database-common-5.0.0-161.el7.noarch
puppet-contrail-5.0.0-161.el7.noarch
contrail-setup-5.0.0-161.el7.noarch
zookeeper-3.4.8-0contrail1.el7.x86_64
contrail-lib-5.0.0-161.el7.x86_64
consistent_hash-1.0-0contrail0.el7.noarch
python-thrift-0.9.1-0contrail.el7.x86_64
python-meld3-0.6.10-1.0contrail0.el7.x86_64
cassandra-3.10-0contrail0.el7.centos.noarch
xmltodict-0.7.0-0contrail.el7.noarch
contrail-openstack-database-5.0.0-161.el7.noarch

shajuvk (shajuvk)
information type: Proprietary → Public
tags: added: sanityblocker
summary: - RHOSP10-5.0-Newton:Provisioning failed while executing
+ RHOSP10-5.0-Build-161-Newton:Provisioning failed while executing
provision_database.py
Revision history for this message
shajuvk (shajuvk) wrote :
Download full text (8.8 KiB)

[stack@undercloud ~]$ openstack stack failures list overcloud
overcloud.AllNodesDeploySteps.ContrailAnalyticsDatabaseDeployment_Step5.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: e3bf9849-0b36-49d1-a43b-2996ca2f7a0a
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
  deploy_stdout: |
    ...
    Notice: /Stage[main]/Contrail::Database::Provision_database/Exec[provision_database_node.py ]/returns: File "/opt/contrail/utils/provision_database_node.py", line 35, in __init__
    Notice: /Stage[main]/Contrail::Database::Provision_database/Exec[provision_database_node.py ]/returns: api_server_use_ssl=self._args.api_server_use_ssl)
    Notice: /Stage[main]/Contrail::Database::Provision_database/Exec[provision_database_node.py ]/returns: File "/opt/contrail/utils/vnc_admin_api.py", line 35, in __init__
    Notice: /Stage[main]/Contrail::Database::Provision_database/Exec[provision_database_node.py ]/returns: auth_host=self.auth_host)
    Notice: /Stage[main]/Contrail::Database::Provision_database/Exec[provision_database_node.py ]/returns: File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 460, in __init__
    Notice: /Stage[main]/Contrail::Database::Provision_database/Exec[provision_database_node.py ]/returns: retry_on_error=False)
    Notice: /Stage[main]/Contrail::Database::Provision_database/Exec[provision_database_node.py ]/returns: File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 943, in _request
    Notice: /Stage[main]/Contrail::Database::Provision_database/Exec[provision_database_node.py ]/returns: raise ConnectionError
    Notice: /Stage[main]/Contrail::Database::Provision_database/Exec[provision_database_node.py ]/returns: requests.exceptions.ConnectionError
    Notice: Finished catalog run in 346.94 seconds
    (truncated, view all with --long)
  deploy_stderr: |
    ...
    Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Array. There is further documentation for validate_legacy function in the README.
       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:19:in `deprecation')
    Warning: This method is deprecated, please use the stdlib validate_legacy function, with Pattern[]. There is further documentation for validate_legacy function in the README.
       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:19:in `deprecation')
    Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Numeric. There is further documentation for validate_legacy function in the README.
       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:19:in `deprecation')
    Warning: This method is deprecated, please use match expressions with Stdlib::Compat::String instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions.
       (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:19:in `deprecation')
    Error: python /opt/contrail/utils/p...

Read more...

Revision history for this message
shajuvk (shajuvk) wrote :
Download full text (5.2 KiB)

2018-01-08 07:52:58Z [overcloud.AllNodesDeploySteps.ContrailControllerDeployment_Step5.0]: CREATE_IN_PROGRESS state changed
2018-01-08 07:52:58Z [overcloud.AllNodesDeploySteps.ContrailTsnDeployment_Step5]: CREATE_IN_PROGRESS state changed
2018-01-08 07:52:59Z [overcloud.AllNodesDeploySteps.ObjectStorageDeployment_Step5]: CREATE_IN_PROGRESS state changed
2018-01-08 07:52:59Z [overcloud.AllNodesDeploySteps.ContrailAnalyticsDatabaseDeployment_Step5]: CREATE_IN_PROGRESS state changed
2018-01-08 07:52:59Z [overcloud.AllNodesDeploySteps.ContrailAnalyticsDeployment_Step5]: CREATE_IN_PROGRESS state changed
2018-01-08 07:52:59Z [overcloud.AllNodesDeploySteps.ContrailAnalyticsDatabaseDeployment_Step5]: CREATE_IN_PROGRESS Stack CREATE started
2018-01-08 07:52:59Z [overcloud.AllNodesDeploySteps.ContrailAnalyticsDatabaseDeployment_Step5.0]: CREATE_IN_PROGRESS state changed
2018-01-08 07:52:59Z [overcloud.AllNodesDeploySteps.CephStorageDeployment_Step5]: CREATE_IN_PROGRESS state changed
2018-01-08 07:52:59Z [overcloud.AllNodesDeploySteps.ContrailAnalyticsDeployment_Step5]: CREATE_IN_PROGRESS Stack CREATE started
2018-01-08 07:52:59Z [overcloud.AllNodesDeploySteps.ContrailAnalyticsDeployment_Step5.0]: CREATE_IN_PROGRESS state changed
2018-01-08 07:53:00Z [overcloud.AllNodesDeploySteps.ComputeDeployment_Step5.0]: CREATE_IN_PROGRESS state changed
2018-01-08 07:53:01Z [overcloud.AllNodesDeploySteps.CephStorageDeployment_Step5]: CREATE_COMPLETE state changed
2018-01-08 07:53:01Z [overcloud.AllNodesDeploySteps.BlockStorageDeployment_Step5]: CREATE_COMPLETE state changed
2018-01-08 07:53:01Z [overcloud.AllNodesDeploySteps.ContrailDpdkDeployment_Step5]: CREATE_COMPLETE state changed
2018-01-08 07:53:01Z [overcloud.AllNodesDeploySteps.ContrailTsnDeployment_Step5]: CREATE_COMPLETE state changed
2018-01-08 07:53:01Z [overcloud.AllNodesDeploySteps.ObjectStorageDeployment_Step5]: CREATE_COMPLETE state changed
2018-01-08 07:53:02Z [overcloud.AllNodesDeploySteps.ComputeDeployment_Step5.2]: CREATE_IN_PROGRESS state changed
2018-01-08 07:57:48Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5.0]: SIGNAL_IN_PROGRESS Signal: deployment 1ec7a6bf-4fbb-4033-8e90-320f9342515b succ
eeded
2018-01-08 07:57:49Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5.0]: CREATE_COMPLETE state changed
2018-01-08 07:57:49Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5]: CREATE_COMPLETE Stack CREATE completed successfully
2018-01-08 07:57:50Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step5]: CREATE_COMPLETE state changed
2018-01-08 07:59:15Z [overcloud.AllNodesDeploySteps.ContrailAnalyticsDatabaseDeployment_Step5.0]: SIGNAL_IN_PROGRESS Signal: deployment e3bf9849-0b36-49d1-a43b-29
96ca2f7a0a failed (6)
2018-01-08 07:59:16Z [overcloud.AllNodesDeploySteps.ContrailAnalyticsDatabaseDeployment_Step5.0]: CREATE_FAILED Error: resources[0]: Deployment to server failed:
deploy_status_code : Deployment exited with non-zero status code: 6
2018-01-08 07:59:16Z [overcloud.AllNodesDeploySteps.ContrailAnalyticsDatabaseDeployment_Step5]: CREATE_FAILED Resource CREATE failed: Error: resources[0]: Deploym
ent to server failed: deploy_status_code :...

Read more...

Revision history for this message
alexey-mr (alexey-morlang) wrote :

this command failed:
python /opt/contrail/utils/provision_database_node.py --host_name overcloud-contrailanalyticsdatabase-0 --host_ip 10.0.0.60 --api_server_ip 10.0.0.10 --api_server_port 8082 --admin_user admin --admin_password CZxWF4F9PjHG6j67xDZHRT2wu --admin_tenant admin --openstack_ip 192.168.24.11 --oper add

it may mean that contrail api is not functioning..

is this env alive? is it possible to check contrail api state?

Revision history for this message
shajuvk (shajuvk) wrote :

This is an issue with amqp version.

https://review.opencontrail.org/#/c/38766/

This review yet to merge in master

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.