Juniper Openstack

Contrail Ocata :: R4.0.1.0 build 19 :: Neutron unable to list, create VN.

Bug #1713453 reported by Ritam Gangopadhyay on 2017-08-28

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
Juniper Openstack	Status tracked in Trunk
R4.0	Invalid	Critical	Ritam Gangopadhyay	Juniper Openstack r4.0.2.0
R4.1	Invalid	Critical	Ritam Gangopadhyay	Juniper Openstack r4.1.0.0-fcs "r4.1"
Trunk	Invalid	Critical	Ramprakash R	Juniper Openstack r5.0.0

Bug Description

Setup:-

nodei19 10.204.217.131 openstack
nodei17 10.204.217.129 controller, analytics, analyticsdb
nodec28 10.204.217.13 controller, analytics, analyticsdb
nodec10 10.204.217.176 controller, analytics, analyticsdb
nodeg37 10.204.217.77 lb
nodei20 10.204.217.132 compute
nodec33 10.204.217.168 compute

Neutron net-list fails with following error and traceback in neutron server log

(nova-api)[nova@nodei19 /]$ neutron net-list
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-247bc189-946c-4ead-ac12-35ce8d58e7d0']
(nova-api)[nova@nodei19 /]$

2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource [req-247bc189-946c-4ead-ac12-35ce8d58e7d0 a7d57a1f36fd4b98876a9b504a17a700 963338afd6b243daa10c5a1b37ba99fa - - -] index failed: No details.
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource Traceback (most recent call last):
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/resource.py", line 79, in resource
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource result = method(request=request, **args)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 95, in wrapped
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource setattr(e, '_RETRY_EXCEEDED', True)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource self.force_reraise()
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 91, in wrapped
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource return f(*args, **kwargs)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 151, in wrapper
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource ectxt.value = e.inner_exc
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource self.force_reraise()
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 139, in wrapper
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource return f(*args, **kwargs)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 131, in wrapped
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource traceback.format_exc())
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource self.force_reraise()
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 126, in wrapped
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource return f(*dup_args, **dup_kwargs)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 368, in index
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource return self._items(request, True, parent_id)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 308, in _items
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource obj_list = obj_getter(request.context, **kwargs)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron_plugin_contrail/plugins/opencontrail/contrail_plugin_base.py", line 247, in get_networks
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource fields)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron_plugin_contrail/plugins/opencontrail/contrail_plugin.py", line 332, in _list_resource
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource res_type, 'READALL')
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron_plugin_contrail/plugins/opencontrail/contrail_plugin.py", line 206, in _request_backend
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource response = self._relay_request(url_path, data=data)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron_plugin_contrail/plugins/opencontrail/contrail_plugin.py", line 199, in _relay_request
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource url, data=data, headers={'Content-type': 'application/json'})
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron_plugin_contrail/plugins/opencontrail/contrail_plugin.py", line 187, in _request_api_server_authn
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource response = self._request_api_server(url, data, headers=authn_headers)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron_plugin_contrail/plugins/opencontrail/contrail_plugin.py", line 150, in _request_api_server
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource response = requests.post(url, data=data, headers=headers)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/requests/api.py", line 110, in post
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource return request('post', url, data=data, json=json, **kwargs)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/requests/api.py", line 56, in request
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource return session.request(method=method, url=url, **kwargs)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 488, in request
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource resp = self.send(prep, **send_kwargs)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 609, in send
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource r = adapter.send(request, **kwargs)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 487, in send
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource raise ConnectionError(e, request=request)
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource ConnectionError: HTTPConnectionPool(host='192.168.100.12', port=8082): Max retries exceeded with url: /neutron/network (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f8e1d191490>: Failed to establish a new connection: [Errno 113] EHOSTUNREACH',))
2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource
2017-08-28 15:32:38.741 20 INFO neutron.wsgi [req-247bc189-946c-4ead-ac12-35ce8d58e7d0 a7d57a1f36fd4b98876a9b504a17a700 963338afd6b243daa10c5a1b37ba99fa - - -] 10.204.217.131 - - [28/Aug/2017 15:32:38] "GET /v2.0/networks.json HTTP/1.1" 500 368 3.545291

Tags:

Revision history for this message

Ritam Gangopadhyay (ritam) wrote on 2017-08-28:

Using openstack cli also hits the same neutron server traceback

(keystone)[root@nodei19 /]# openstack network list
HttpException: Internal Server Error, Request Failed: internal server error while processing your request.
(keystone)[root@nodei19 /]#

Rudra Rugge (rrugge) on 2017-08-28

Changed in juniperopenstack:
assignee:	nobody → Ramprakash R (ramprakash)

Revision history for this message

Ramprakash R (ramprakash) wrote on 2017-08-29:

open stack commands need to be executed from the base host. Based on Ritam's followup email, the commands worked from the host. Closing this based on this email snippet:

%<-----------------------------

Hi,

The issues seen prior to R4.0.1.0 build 21 on a setup with control data interface separation were:-

1. Multiple patches to complete provisioning.
2. Neutron unable to list/create networks https://bugs.launchpad.net/juniperopenstack/+bug/1713453
3. Openstack compute provisioning failure https://bugs.launchpad.net/juniperopenstack/+bug/1713559
4. Horizon not reachable https://bugs.launchpad.net/juniperopenstack/+bug/1713444

       With R4.0.1.0 build 21 and a setup with “NO” control data interface separation (i.e. only management interface) and the following 2 patches
                https://review.opencontrail.org/#/c/35015/1/kolla-ansible/ansible/roles/nova/defaults/main.yml
                https://review.opencontrail.org/#/c/35013/1/src/client/preconfig.py

1. 0 touch provisioning works, just need to execute provision_containers.sh with proper json file and sku option
2. Neutron net-list, openstack net list, and create delete VN works fine.
3. Horizon works but has a upstream bug - https://bugs.launchpad.net/bugs/1672741 and needs to be patched with https://ask.openstack.org/en/question/107956/kolla-missing-js-and-css-in-horizon/ for it to work.
4. VN – VM creation from horizon and cli works and intra VN connectivity is fine. Inter VN connectivity, was planning to check through policy.

------------------------------>%

Changed in juniperopenstack:
status:	New → Invalid

Ritam Gangopadhyay (ritam) on 2017-08-30

Changed in juniperopenstack:
status:	Invalid → New

Revision history for this message

Ritam Gangopadhyay (ritam) wrote on 2017-08-30:

This doesn't work when neutron tries to connect over the control-data interface. The bug was filled in context of management and control-data plane separation.

2017-08-28 15:32:38.733 20 ERROR neutron.api.v2.resource ConnectionError: HTTPConnectionPool(host='192.168.100.12', port=8082): Max retries exceeded with url: /neutron/network (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f8e1d191490>: Failed to establish a new connection: [Errno 113] EHOSTUNREACH',))

On single interface setup this works when neutron connects over the management interface, but the bug still holds good for setup with cntl-data interface.

Jeba Paulaiyan (jebap) on 2017-08-31

tags:

added: sanity

Revision history for this message

Ramprakash R (ramprakash) wrote on 2017-08-31:

Please check if this works with something similar to the below config (cluster.json):

{
"cluster" : [
    {
        "id": "seven_node_kolla",
        "parameters" : {
            "domain": "sm-domain.com",
            "subnet_mask": "255.255.255.0",
            "gateway": "192.168.121.1",
            "provision": {
                "contrail": {
                    "ha": {
                        "contrail_internal_vip": "192.168.10.24",
                        "contrail_external_vip": "192.168.1.24"
                    },
                    "kernel_upgrade": false
                },
                "openstack":{
                    "openstack_manage_amqp": true,
                    "region": "RegionOne",
                    "multi_tenancy": true,
                    "openstack_manage_amqp": true,
                    "enable_ceilometer": false,
                    "keystone" : {
                        "mysql_service_password": "c0ntrail123",
                        "admin_user": "admin",
                        "admin_token": "c0ntrail123",
                        "ip": "192.168.10.100",
                        "auth_port": 35357,
                        "auth_protocol": "http",
                        "version": "v2.0",
                        "admin_password": "c0ntrail123",
                        "admin_tenant": "admin"
                    },
                    "ha": {
                        "internal_vip": "192.168.10.100",
                        "external_vip": "192.168.1.100"
                    }
                },
                "contrail_4": {
                    "rabbitmq_config": {
                        "user": "openstack",
                        "password": "c0ntrail123"
                    },
                    "global_config": {
                        "external_rabbitmq_servers": "192.168.10.25"
                    }
                },
                "kolla_globals": {
                    "enable_nova_compute": "yes",
                    "keystone_admin_user": "admin",
                    "keystone_admin_url": "{{ admin_protocol }}://{{ kolla_internal_fqdn }}:{{ keystone_admin_port }}",
                    "keystone_internal_url": "{{ internal_protocol }}://{{ kolla_internal_fqdn }}:{{ keystone_public_port }}",
                    "keystone_public_url": "{{ public_protocol }}://{{ kolla_external_fqdn }}:{{ keystone_public_port }}",
                    "kolla_base_distro": "ubuntu",
                    "openstack_release": "4.0.0",

                    "kolla_internal_vip_address": "192.168.10.100",
                    "network_interface": "eth2",
                    "kolla_external_vip_address": "192.168.1.100",
                    "kolla_external_vip_interface": "eth1",

                    "haproxy_password": "c0ntrail123",
                    "keepalived_password": "c0ntrail123",
                    "neutron_external_interface": "eth0",
                    "neutron_plugin_agent": "opencontrail",
                    "enable_neutron_opencontrail": "yes",
                    "contrail_api_interface_address": "192.168.10.24"
                }
            }
       }
    }
]
}

Please check if this works with something similar to the below config (cluster.json):

Revision history for this message

Ramprakash R (ramprakash) wrote on 2017-08-31:

Moving it to "incomplete" for Ritam to verify. Please reopen if there is still issue with neutron connectivity.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.