Juniper Openstack

cc13-OSP13:neutron and openstack commands are failing with

Bug #1782251 reported by shajuvk
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R5.0
Fix Committed
High
alexey-mr
Juniper Openstack r5.0.1
Trunk
Fix Committed
High
alexey-mr
Juniper Openstack r5.1.1

Bug Description

[root@overcloudey3-ctrl-0 heat-admin]# source overcloudrc
(overcloud) [root@overcloudey3-ctrl-0 heat-admin]# neutron net-list
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
The server has either erred or is incapable of performing the requested operation.<br /><br />

Neutron server returns request_ids: ['req-9d3aa5d8-32db-4a9a-8cde-1d5b933017d6']

(overcloud) [stack@undercloud ~]$ cat overcloudrc
# Clear any old environment that may conflict.
for key in $( set | awk '{FS="="} /^OS_/ {print $1}' ); do unset $key ; done
export OS_NO_CACHE=True
export COMPUTE_API_VERSION=1.1
export OS_USERNAME=admin
export no_proxy=,10.87.66.20,192.0.2.57
export OS_USER_DOMAIN_NAME=Default
export OS_VOLUME_API_VERSION=3
export OS_CLOUDNAME=overcloud
export OS_AUTH_URL=https://10.87.66.20:13000/v3
export NOVA_VERSION=1.1
export OS_IMAGE_API_VERSION=2
export OS_PASSWORD=AyYNJXCgRr7QFCdaxB4mhj68Y
export OS_PROJECT_DOMAIN_NAME=Default
export OS_IDENTITY_API_VERSION=3
export OS_PROJECT_NAME=admin
export OS_AUTH_TYPE=password
export PYTHONWARNINGS="ignore:Certificate has no, ignore:A true SSLContext object is not available"

# Add OS_CLOUDNAME to PS1
if [ -z "${CLOUDPROMPT_ENABLED:-}" ]; then
    export PS1=${PS1:-""}
    export PS1=\${OS_CLOUDNAME:+"(\$OS_CLOUDNAME)"}\ $PS1
    export CLOUDPROMPT_ENABLED=1
fi

See original description
Tags: cc13 queens rhosp
shajuvk (shajuvk)
information type: Proprietary → Public
shajuvk (shajuvk)
description: updated
Revision history for this message
tom murray (tmurray-a) wrote :
Download full text (10.2 KiB)

Neutron server log shows:

2018-07-17 15:09:26.244 28 INFO neutron.wsgi [-] 172.16.81.58 "GET /v2.0/floatingips.json HTTP/1.1" status: 500 len: 380 time: 0.0040829
2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors [-] An error occurred during processing the request: GET /v2.0/floatingips.json HTTP/1.0^M
Accept: application/json^M
Accept-Encoding: gzip, deflate^M
Content-Type: text/plain^M
Host: 10.87.1.100:13696^M
User-Agent: python-neutronclient^M
X-Auth-Token: *****
X-Forwarded-Proto: https: AttributeError: 'HTTPConnectionPool' object has no attribute 'key_file'
2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors Traceback (most recent call last):
2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/oslo_middleware/catch_errors.py", line 40, in __call__
2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors response = req.get_response(self.application)
2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/webob/request.py", line 1316, in send
2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors application, catch_exc_info=False)
2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/webob/request.py", line 1280, in call_application
2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors app_iter = application(self.environ, start_response)
2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/webob/dec.py", line 131, in __call__
2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors resp = self.call_func(req, *args, **self.kwargs)
2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/webob/dec.py", line 196, in call_func
2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors return self.func(req, *args, **kwargs)
2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 334, in __call__
2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors response = self.process_request(req)
2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 633, in process_request
2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors resp = super(AuthProtocol, self).process_request(request)
2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 407, in process_request
2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors allow_expired=allow_expired)
2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 440, in _do_fetch_token
2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors data = self.fetch_token(token, **kwargs)
2018-07-17 15:09:26.249 28 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/site-packages/keyst...

Revision history for this message
tom murray (tmurray-a) wrote :

This problem is caused by the /usr/share/openstack-puppet/modules/tripleo/manifests/network/contrail/neutron_plugin.pp file, which is delivered by the puppet-tripleo-8.3.2-8.el7ost.noarch package of the rhel-7-server-openstack-13-rpms repository.

Workaround - before overcloud deploy:

To prevent this issue you can customize the overcloud image used to deploy the overcloud roles. This image has the puppet-tripleo package already installed so you can modify it to remove the setting of `certfile`. For example:

mkdir images; cd images
for i in /usr/share/rhosp-director-images/overcloud-full-latest-13.0.tar /usr/share/rhosp-director-images/ironic-python-agent-latest-13.0.tar; do tar -xvf $i; done
openstack overcloud image upload --image-path
virt-customize -a overcloud-full.qcow2 --run-command 'sed -i "/certfile/d" /usr/share/openstack-puppet/modules/tripleo/manifests/network/contrail/neutron_plugin.pp'
openstack overcloud image upload --image-path $PWD

Workaround - after overcloud deploy

If the overcloud has already been deployed and neutron is having difficulty processing requests, then the configuration can be adjusted and the neutron_api container can be restarted. For example,

# SSH to the host(s) with the neutron_api container
sudo sed -i "/certfile/d" /var/lib/config-data/puppet-generated/neutron/etc/neutron/plugins/opencontrail/ContrailPlugin.ini
sudo docker stop neutron_api
sudo docker start neutron_api

This issue is fixed in Contrail Cloud 13.

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/44854
Submitter: alexey-mr (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Review in progress for https://review.opencontrail.org/44855
Submitter: alexey-mr (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R5.0

Review in progress for https://review.opencontrail.org/44858
Submitter: alexey-mr (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Review in progress for https://review.opencontrail.org/44859
Submitter: alexey-mr (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/44862
Submitter: alexey-mr (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Review in progress for https://review.opencontrail.org/44862
Submitter: Mithun Mistry (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Review in progress for https://review.opencontrail.org/44862
Submitter: alexey-mr (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/44862
Committed: http://github.com/Juniper/contrail-project-config/commit/61e06b0731f8338083bd3a8b1070bd566e3f887a
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit 61e06b0731f8338083bd3a8b1070bd566e3f887a
Author: alexey-mr <email address hidden>
Date: Mon Jul 23 18:11:25 2018 +0300

Added Juniper/contrail-tripleo-puppet

Change-Id: I6bceb1ac356908a7ffa9d22ddd734adabd89d8a2
Partial-Bug: #1782251

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R5.0

Review in progress for https://review.opencontrail.org/44859
Submitter: alexey-mr (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/44855
Submitter: alexey-mr (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/44854
Committed: http://github.com/Juniper/contrail-vnc/commit/9cf379d877a68ff4efea4733715adc6e5cd272b5
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit 9cf379d877a68ff4efea4733715adc6e5cd272b5
Author: alexey-mr <email address hidden>
Date: Mon Jul 23 17:23:25 2018 +0300

Added contrail-tripleo-puppet

This module is to be used in OSP13
to configure neutron and heat plugins

Change-Id: I6ab6cbdaeee5a4dcf05eec14268ec85baed61e14
Partial-Bug: #1782251

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/44858
Committed: http://github.com/Juniper/contrail-vnc/commit/360aa526c9ed797c6da9c15b5e5e37fca0c5c144
Submitter: Zuul v3 CI (<email address hidden>)
Branch: R5.0

commit 360aa526c9ed797c6da9c15b5e5e37fca0c5c144
Author: alexey-mr <email address hidden>
Date: Mon Jul 23 17:33:06 2018 +0300

Added contrail-tripleo-puppet

This module is to be used in OSP13
to configure neutron and heat plugins

Change-Id: Id8a4fb445e7d03537fed3db382082cf03c98f2c9
Partial-Bug: #1782251

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/44855
Committed: http://github.com/Juniper/contrail-packages/commit/4347b2fe99d2741b61d3ce84d186b623e8eb1ceb
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit 4347b2fe99d2741b61d3ce84d186b623e8eb1ceb
Author: alexey-mr <email address hidden>
Date: Mon Jul 23 16:48:26 2018 +0300

Added contrail-tripleo-puppet spec

Depends-On: I6ab6cbdaeee5a4dcf05eec14268ec85baed61e14
Change-Id: I5b2094d601559df6e31c1c00869b7612d353d9d4
Partial-Bug: #1782251

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/44859
Committed: http://github.com/Juniper/contrail-packages/commit/599c9d3378587c145e8bdc0b56894af5d93d7ec9
Submitter: Zuul v3 CI (<email address hidden>)
Branch: R5.0

commit 599c9d3378587c145e8bdc0b56894af5d93d7ec9
Author: alexey-mr <email address hidden>
Date: Mon Jul 23 16:48:26 2018 +0300

Added contrail-tripleo-puppet spec

Depends-On: Id8a4fb445e7d03537fed3db382082cf03c98f2c9
Change-Id: I5b2094d601559df6e31c1c00869b7612d353d9d4
Partial-Bug: #1782251

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/45035
Submitter: alexey-mr (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Review in progress for https://review.opencontrail.org/45057
Submitter: alexey-mr (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R5.0

Review in progress for https://review.opencontrail.org/45060
Submitter: alexey-mr (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/45063
Submitter: alexey-mr (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Review in progress for https://review.opencontrail.org/45057
Submitter: alexey-mr (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R5.0

Review in progress for https://review.opencontrail.org/45060
Submitter: alexey-mr (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/45057
Committed: http://github.com/Juniper/contrail-container-builder/commit/824ee15686e1a6783de03fc37b749fd1bb0cf26a
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit 824ee15686e1a6783de03fc37b749fd1bb0cf26a
Author: alexey-mr <email address hidden>
Date: Mon Jul 30 12:54:59 2018 +0300

Added contrail-tripleo-puppet module for node-init

Change-Id: Ic2341e496d2216bc575060ffff7de7280a307e30
Partial-Bug: #1782251

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] stable/queens

Review in progress for https://review.opencontrail.org/45077
Submitter: alexey-mr (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Review in progress for https://review.opencontrail.org/45079
Submitter: alexey-mr (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/45035
Committed: http://github.com/Juniper/contrail-tripleo-puppet/commit/64ebd61d013745e6d58eb06b7a07dc125ce28172
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit 64ebd61d013745e6d58eb06b7a07dc125ce28172
Author: alexey-mr <email address hidden>
Date: Sat Jul 28 12:50:00 2018 +0300

added .gitreview

Change-Id: Ic3d0d1d40735d91b787f05681c5fd7c70e0880bb
Partial-Bug: #1782251

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/45063
Committed: http://github.com/Juniper/contrail-tripleo-puppet/commit/d1fc0d74714549ee506cfe96a3837b5c2052d582
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit d1fc0d74714549ee506cfe96a3837b5c2052d582
Author: alexey-mr <email address hidden>
Date: Tue Jul 24 19:55:19 2018 +0300

Reworked to work with Contrail5.0

It is now supports only 5.0 in OSP13 which is
containerized deployement that needs only
neutron and heat plugins to be configured via puppets.

Depends-On: Ic3d0d1d40735d91b787f05681c5fd7c70e0880bb
Change-Id: I0f689dc184fb07e5c837f6c4c1d66ef4c9346b2f
Partial-Bug: #1782251

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/45060
Committed: http://github.com/Juniper/contrail-container-builder/commit/224b78cd4a7b4af7e4fbb544402f3a08755fb8b5
Submitter: Zuul v3 CI (<email address hidden>)
Branch: R5.0

commit 224b78cd4a7b4af7e4fbb544402f3a08755fb8b5
Author: alexey-mr <email address hidden>
Date: Mon Jul 30 12:54:59 2018 +0300

Added contrail-tripleo-puppet module for node-init

Change-Id: Ic2341e496d2216bc575060ffff7de7280a307e30
Partial-Bug: #1782251

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/45077
Committed: http://github.com/Juniper/contrail-tripleo-puppet/commit/847f9797c2b3b5638fdb03c423ce547d33a76ce8
Submitter: Zuul v3 CI (<email address hidden>)
Branch: stable/queens

commit 847f9797c2b3b5638fdb03c423ce547d33a76ce8
Author: alexey-mr <email address hidden>
Date: Mon Jul 30 22:08:47 2018 +0300

Added .gitreview

Change-Id: I12aaf9d346005ec15ae06b6af7e1969ad6e1e80c
Partial-Bug: #1782251

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/45079
Committed: http://github.com/Juniper/contrail-tripleo-puppet/commit/10d8b803623db0fc7ab298f8ce5c29245dc64c33
Submitter: Zuul v3 CI (<email address hidden>)
Branch: stable/queens

commit 10d8b803623db0fc7ab298f8ce5c29245dc64c33
Author: alexey-mr <email address hidden>
Date: Tue Jul 24 19:55:19 2018 +0300

Reworked to work with Contrail5.0

It is now supports only 5.0 in OSP13 which is
containerized deployement that needs only
neutron and heat plugins to be configured via puppets.

Depends-On: I12aaf9d346005ec15ae06b6af7e1969ad6e1e80c
Change-Id: I0f689dc184fb07e5c837f6c4c1d66ef4c9346b2f
Partial-Bug: #1782251

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] stable/queens

Review in progress for https://review.opencontrail.org/45167
Submitter: alexey-mr (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/45167
Committed: http://github.com/Juniper/contrail-tripleo-puppet/commit/8985704c9830a30f6f2b976c0b979c914c19a8d4
Submitter: Zuul v3 CI (<email address hidden>)
Branch: stable/queens

commit 8985704c9830a30f6f2b976c0b979c914c19a8d4
Author: alexey-mr <email address hidden>
Date: Wed Aug 1 16:08:38 2018 +0300

Removed unused files

Change-Id: Ice1b0095f68f7776e8a98d48505237c09c2e8ed5
Partial-Bug: #1782251

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] stable/queens

Review in progress for https://review.opencontrail.org/45310
Submitter: alexey-mr (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/45310
Committed: http://github.com/Juniper/contrail-tripleo-heat-templates/commit/c75847be08ed8fe79b5d7249011afd8e64b46710
Submitter: Zuul v3 CI (<email address hidden>)
Branch: stable/queens

commit c75847be08ed8fe79b5d7249011afd8e64b46710
Author: alexey-mr <email address hidden>
Date: Mon Aug 6 12:25:44 2018 +0300

Added .gitreview

Change-Id: Ibe9da741f5fdc3a3dd8e2ec03b9424d0cdf3b637
Partial-Bug: #1782251

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] stable/queens

Review in progress for https://review.opencontrail.org/45316
Submitter: alexey-mr (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/45316
Committed: http://github.com/Juniper/contrail-tripleo-heat-templates/commit/075f1ca10c70d4c47e83f4d8afdad0c729e3177a
Submitter: Zuul v3 CI (<email address hidden>)
Branch: stable/queens

commit 075f1ca10c70d4c47e83f4d8afdad0c729e3177a
Author: alexey-mr <email address hidden>
Date: Mon Aug 6 14:05:19 2018 +0300

Neutron and heat init

At pre deploy step contrail-tripleo-puppet is installed as rpm on host
(rpm is copied on host by node-init container )

- Use auth internal endpoint for webui

Change-Id: I4790a6f1d86f4c96f62f7139b1feb9b9639d1ff0
Partial-Bug: #1782251

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] stable/queens

Review in progress for https://review.opencontrail.org/45323
Submitter: alexey-mr (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Review in progress for https://review.opencontrail.org/45323
Submitter: Michael Henkel (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/45323
Committed: http://github.com/Juniper/contrail-tripleo-heat-templates/commit/3bc7451fafeacc0cfe1c03f4da90410e6080eaeb
Submitter: Zuul v3 CI (<email address hidden>)
Branch: stable/queens

commit 3bc7451fafeacc0cfe1c03f4da90410e6080eaeb
Author: alexey-mr <email address hidden>
Date: Mon Aug 6 17:33:46 2018 +0300

Remove duplicating auth envs.

- auth env vars are provided via ContrailBase environment

Change-Id: Iacfc2f87c54e819d24c402a7604fc2d6e1cecce7
Partial-Bug: #1782251

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.