KeyError in services/plugin.py during the network association

Bug #1598807 reported by sureshkvl
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
networking-bgpvpn
Fix Released
Undecided
Thomas Monguillon

Bug Description

During the network association, the following error is observed in the neutron screen
2016-07-04 10:47:49.207 TRACE neutron.api.v2.resource if router_id in bgpvpn['routers']]
2016-07-04 10:47:49.207 TRACE neutron.api.v2.resource KeyError: 'routers'

Setup : devstack(mitaka) + networking-bgpvpn(master) + opencontrail

The neutron CLI outputs are below

⌂119% 2d [cloud:~/devstack] stable/mitaka* 2s ± neutron bgpvpn-list
+--------------------------------------+---------+------+----------+
| id | name | type | networks |
+--------------------------------------+---------+------+----------+
| 7a4cabc9-e883-4397-a2c8-2fe12ba05c60 | demobgp | l3 | [] |
+--------------------------------------+---------+------+----------+
⌂133% 2d [cloud:~/devstack] stable/mitaka* 2s ±
2d [cloud:~/devstack] stable/mitaka* 2s 1 ± neutron net-list
+--------------------------------------+---------------------------------------------------------------+----------------------------------------------------------+
| id | name | subnets |
+--------------------------------------+---------------------------------------------------------------+----------------------------------------------------------+
| 254b9e9b-38f0-4038-838f-e766d04fe6ea | public | be21c5ea-3085-4415-a267-5fd5b082110c |
| | | eac36e08-2d32-43f1-85fb-bf10db5d328c |
| 69e768ea-74b7-4f7d-862a-8cfb3f8aa2b1 | snat-si- | f979e89f-074d-4cc5-b537-da3655457fb2 100.64.0.0/29 |
| | left_snat_20718216-8696-4da7-8667-8862cc3ec2dd_71cabe6c-4457 | |
| | -483b-ae9c-284c35f86602 | |
| 2fb165ec-12ce-447b-b2ab-6c31d0eb426a | private | adac69c0-fe89-4620-9e7a-ffe8f050a7f5 10.0.0.0/24 |
| | | e7d323c4-a6b4-458c-81e1-defff134508e fd9e:803e:d86c::/64 |
+--------------------------------------+---------------------------------------------------------------+----------------------------------------------------------+
2d [cloud:~/devstack] stable/mitaka* ±

2d [cloud:~/devstack] stable/mitaka* ± neutron bgpvpn-net-assoc-create demobgp --network 2fb165ec-12ce-447b-b2ab-6c31d0eb426a
Request Failed: internal server error while processing your request.
Neutron server returns request_ids: ['req-cf7722f1-5a26-4713-9aad-c4938db51592']

2d [cloud:~/devstack] stable/mitaka* ±
⌂94% 2d [cloud:~/devstack] stable/mitaka* ± neutron router-list
+--------------------------------------+---------+-----------------------------------------------------------------------------+
| id | name | external_gateway_info |
+--------------------------------------+---------+-----------------------------------------------------------------------------+
| 20718216-8696-4da7-8667-8862cc3ec2dd | router1 | {"network_id": "254b9e9b-38f0-4038-838f-e766d04fe6ea", "enable_snat": true} |
+--------------------------------------+---------+-----------------------------------------------------------------------------+

2d [cloud:~/devstack] stable/mitaka* ± neutron port-list
+--------------------------------------+-----------------------------------------------------+-------------------+------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+--------------------------------------+-----------------------------------------------------+-------------------+------------------------------------------------------+
| 722d5215-fc0d-4901-b826-9c3c1394a8df | 722d5215-fc0d-4901-b826-9c3c1394a8df | 02:72:2d:52:15:fc | {"subnet_id": "adac69c0-fe89-4620-9e7a- |
| | | | ffe8f050a7f5", "ip_address": "10.0.0.1"} |
| bdb96d05-1d88-4ea8-a445-d01fe8d98140 | default-domain__demo__snat_20718216-8696-4da7-8667- | 02:4e:97:95:29:74 | {"subnet_id": |
| | 8862cc3ec2dd_71cabe6c-4457-483b-ae9c- | | "be21c5ea-3085-4415-a267-5fd5b082110c", |
| | 284c35f86602__1__right__1 | | "ip_address": "172.24.4.3"} |
| b6f4dfe4-fd7a-4e93-a185-064430b56453 | default-domain__demo__snat_20718216-8696-4da7-8667- | 02:07:48:b3:e7:e0 | {"subnet_id": "f979e89f-074d- |
| | 8862cc3ec2dd_71cabe6c-4457-483b-ae9c- | | 4cc5-b537-da3655457fb2", "ip_address": "100.64.0.4"} |
| | 284c35f86602__1__left__2 | | |
| f5a04682-f4f9-43f5-b8cc-17bd7d3925b9 | default-domain__demo__snat_20718216-8696-4da7-8667- | 02:4e:97:95:29:74 | {"subnet_id": |
| | 8862cc3ec2dd_71cabe6c-4457-483b-ae9c- | | "be21c5ea-3085-4415-a267-5fd5b082110c", |
| | 284c35f86602__2__right__1 | | "ip_address": "172.24.4.3"} |
| b0f715ea-b485-4af1-a486-3e599f1bc6eb | default-domain__demo__snat_20718216-8696-4da7-8667- | 02:07:48:b3:e7:e0 | {"subnet_id": "f979e89f-074d- |
| | 8862cc3ec2dd_71cabe6c-4457-483b-ae9c- | | 4cc5-b537-da3655457fb2", "ip_address": "100.64.0.4"} |
| | 284c35f86602__2__left__2 | | |
| d79e7d45-84f9-43a1-b185-292e3e547311 | d79e7d45-84f9-43a1-b185-292e3e547311 | 02:d7:9e:7d:45:84 | {"subnet_id": "e7d323c4-a6b4-458c- |
| | | | 81e1-defff134508e", "ip_address": |
| | | | "fd9e:803e:d86c::1"} |
| a0eb9ec7-9bf6-45eb-8ad3-6a590d48a59c | a0eb9ec7-9bf6-45eb-8ad3-6a590d48a59c | 02:a0:eb:9e:c7:9b | {"subnet_id": "e7d323c4-a6b4-458c- |
| | | | 81e1-defff134508e", "ip_address": |
| | | | "fd9e:803e:d86c::3"} |
| | | | {"subnet_id": "adac69c0-fe89-4620-9e7a- |
| | | | ffe8f050a7f5", "ip_address": "10.0.0.3"} |
+--------------------------------------+-----------------------------------------------------+-------------------+------------------------------------------------------+

Neutron(q-svc) Logs
======================

2016-07-04 10:47:49.206 DEBUG networking_bgpvpn.neutron.services.service_drivers.opencontrail.opencontrail_client [req-cf7722f1-5a26-4713-9aad-c4938db51592 demo 05f99c51
12ca45baa89ffaf26fab9335] RES:
HTTP 11.0 200 OK
Date: Mon, 04 Jul 2016 10:47:49 GMT
Content-Length: 1502
Content-Type: application/json

{"value": [{"value": "e7d323c4-a6b4-458c-81e1-defff134508e", "key": "2fb165ec-12ce-447b-b2ab-6c31d0eb426a fd9e:803e:d86c::/64"}, {"value": "69e768ea-74b7-4f7d-862a-8cfb3
f8aa2b1 100.64.0.0/29", "key": "f979e89f-074d-4cc5-b537-da3655457fb2"}, {"value": "eac36e08-2d32-43f1-85fb-bf10db5d328c", "key": "254b9e9b-38f0-4038-838f-e766d04fe6ea 20
01:db8::/64"}, {"value": "{\"bgpvpn\": {\"export_targets\": [\"64512:1000\"], \"name\": \"demobgp\", \"route_distinguishers\": [], \"tenant_id\": \"05f99c5112ca45baa89ff
af26fab9335\", \"import_targets\": [\"64512:2000\"], \"route_targets\": [\"64512:1000\"], \"type\": \"l3\", \"id\": \"7a4cabc9-e883-4397-a2c8-2fe12ba05c60\"}}", "key": "
7a4cabc9-e883-4397-a2c8-2fe12ba05c60"}, {"value": "2fb165ec-12ce-447b-b2ab-6c31d0eb426a 10.0.0.0/24", "key": "adac69c0-fe89-4620-9e7a-ffe8f050a7f5"}, {"value": "254b9e9b
-38f0-4038-838f-e766d04fe6ea 172.24.4.0/24", "key": "be21c5ea-3085-4415-a267-5fd5b082110c"}, {"value": "2fb165ec-12ce-447b-b2ab-6c31d0eb426a fd9e:803e:d86c::/64", "key":
 "e7d323c4-a6b4-458c-81e1-defff134508e"}, {"value": "adac69c0-fe89-4620-9e7a-ffe8f050a7f5", "key": "2fb165ec-12ce-447b-b2ab-6c31d0eb426a 10.0.0.0/24"}, {"value": "be21c5
ea-3085-4415-a267-5fd5b082110c", "key": "254b9e9b-38f0-4038-838f-e766d04fe6ea 172.24.4.0/24"}, {"value": "f979e89f-074d-4cc5-b537-da3655457fb2", "key": "69e768ea-74b7-4f
7d-862a-8cfb3f8aa2b1 100.64.0.0/29"}, {"value": "254b9e9b-38f0-4038-838f-e766d04fe6ea 2001:db8::/64", "key": "eac36e08-2d32-43f1-85fb-bf10db5d328c"}]}
 from (pid=6097) _log_res /opt/stack/networking-bgpvpn/networking_bgpvpn/neutron/services/service_drivers/opencontrail/opencontrail_client.py:215
2016-07-04 10:47:49.207 ERROR neutron.api.v2.resource [req-cf7722f1-5a26-4713-9aad-c4938db51592 demo 05f99c5112ca45baa89ffaf26fab9335] create failed
2016-07-04 10:47:49.207 TRACE neutron.api.v2.resource Traceback (most recent call last):
2016-07-04 10:47:49.207 TRACE neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/resource.py", line 84, in resource
2016-07-04 10:47:49.207 TRACE neutron.api.v2.resource result = method(request=request, **args)
2016-07-04 10:47:49.207 TRACE neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 410, in create
2016-07-04 10:47:49.207 TRACE neutron.api.v2.resource return self._create(request, body, **kwargs)
2016-07-04 10:47:49.207 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 148, in wrapper
2016-07-04 10:47:49.207 TRACE neutron.api.v2.resource ectxt.value = e.inner_exc
2016-07-04 10:47:49.207 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-07-04 10:47:49.207 TRACE neutron.api.v2.resource self.force_reraise()
2016-07-04 10:47:49.207 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-07-04 10:47:49.207 TRACE neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2016-07-04 10:47:49.207 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper
2016-07-04 10:47:49.207 TRACE neutron.api.v2.resource return f(*args, **kwargs)
2016-07-04 10:47:49.207 TRACE neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 521, in _create
2016-07-04 10:47:49.207 TRACE neutron.api.v2.resource obj = do_create(body)
2016-07-04 10:47:49.207 TRACE neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 503, in do_create
2016-07-04 10:47:49.207 TRACE neutron.api.v2.resource request.context, reservation.reservation_id)
2016-07-04 10:47:49.207 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-07-04 10:47:49.207 TRACE neutron.api.v2.resource self.force_reraise()
2016-07-04 10:47:49.207 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-07-04 10:47:49.207 TRACE neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2016-07-04 10:47:49.207 TRACE neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 496, in do_create
2016-07-04 10:47:49.207 TRACE neutron.api.v2.resource return obj_creator(request.context, **kwargs)
2016-07-04 10:47:49.207 TRACE neutron.api.v2.resource File "/opt/stack/networking-bgpvpn/networking_bgpvpn/neutron/services/plugin.py", line 147, in create_bgpvpn_network_association
2016-07-04 10:47:49.207 TRACE neutron.api.v2.resource net = self._validate_network(context, net_assoc)
2016-07-04 10:47:49.207 TRACE neutron.api.v2.resource File "/opt/stack/networking-bgpvpn/networking_bgpvpn/neutron/services/plugin.py", line 68, in _validate_network
2016-07-04 10:47:49.207 TRACE neutron.api.v2.resource self._validate_network_has_router_assoc(context, network, plugin)
2016-07-04 10:47:49.207 TRACE neutron.api.v2.resource File "/opt/stack/networking-bgpvpn/networking_bgpvpn/neutron/services/plugin.py", line 80, in _validate_network_has_router_assoc
2016-07-04 10:47:49.207 TRACE neutron.api.v2.resource if router_id in bgpvpn['routers']]
2016-07-04 10:47:49.207 TRACE neutron.api.v2.resource KeyError: 'routers'
2016-07-04 10:47:49.207 TRACE neutron.api.v2.resource
2016-07-04 10:47:49.218 INFO neutron.wsgi [req-cf7722f1-5a26-4713-9aad-c4938db51592 demo 05f99c5112ca45baa89ffaf26fab9335] 192.168.122.96 - - [04/Jul/2016 10:47:49] "POST /v2.0/bgpvpn/bgpvpns/7a4cabc9-e883-4397-a2c8-2fe12ba05c60/network_associations.json HTTP/1.1" 500 383 0.119445

Changed in bgpvpn:
assignee: nobody → sureshkvl (sureshkumar-s)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to networking-bgpvpn (master)

Fix proposed to branch: master
Review: https://review.openstack.org/337139

Changed in bgpvpn:
status: New → In Progress
Revision history for this message
Thomas Monguillon (thomas-monguillon) wrote :

Hi Suresh,

Like Thomas Morin says in your changes, the best and proper way to fix the problem is to add an additionnal initialization of the dictionnary in the function make_bgpvpn_dict on the utils.py file. I've just tested with the fix and it works fine on my platform.

I will push this fix. Please feel free to test it and share with me if it's ok for you.

Revision history for this message
sureshkvl (sureshkumar-s) wrote :

Hi Thomas Monguillon,
Sure . I am happy to test the fix on my environment and will share the results .

Thanks
suresh.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/337578

Changed in bgpvpn:
assignee: sureshkvl (sureshkumar-s) → Thomas Monguillon (thomas-monguillon)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on networking-bgpvpn (master)

Change abandoned by Thomas Morin (<email address hidden>) on branch: master
Review: https://review.openstack.org/337139
Reason: Abandon in favor of I61e699690c86e523e0479a4cb34d3db5f6b9ed0c

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to networking-bgpvpn (master)

Reviewed: https://review.openstack.org/337578
Committed: https://git.openstack.org/cgit/openstack/networking-bgpvpn/commit/?id=7f17824ded359ff00f80988ff82270ecb594add7
Submitter: Jenkins
Branch: master

commit 7f17824ded359ff00f80988ff82270ecb594add7
Author: Thomas Monguillon <email address hidden>
Date: Tue Jul 5 11:28:32 2016 +0200

    Initialize the routers key in make_bgpvpn_dict

    This patch prevents to trigger an error with OpenContrail when a
    network association is declared on a network bounded to a router.

    Change-Id: I61e699690c86e523e0479a4cb34d3db5f6b9ed0c
    Closes-bug: 1596962
    Closes-bug: 1598807

Changed in bgpvpn:
status: In Progress → Fix Released
Revision history for this message
sureshkvl (sureshkumar-s) wrote :

Hi Thomas Monguillon,

I have tested the fix. it works fine.

Thanks
suresh.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to networking-bgpvpn (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/340734

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to networking-bgpvpn (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/340735

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to networking-bgpvpn (backport/kilo)

Fix proposed to branch: backport/kilo
Review: https://review.openstack.org/340736

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to networking-bgpvpn (backport/juno)

Fix proposed to branch: backport/juno
Review: https://review.openstack.org/340737

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to networking-bgpvpn (stable/mitaka)

Reviewed: https://review.openstack.org/340734
Committed: https://git.openstack.org/cgit/openstack/networking-bgpvpn/commit/?id=f975736dfa064fafa3a8da73b90af2832ca43eab
Submitter: Jenkins
Branch: stable/mitaka

commit f975736dfa064fafa3a8da73b90af2832ca43eab
Author: Thomas Monguillon <email address hidden>
Date: Tue Jul 5 11:28:32 2016 +0200

    Initialize the routers key in make_bgpvpn_dict

    This patch prevents to trigger an error with OpenContrail when a
    network association is declared on a network bounded to a router.

    Change-Id: I61e699690c86e523e0479a4cb34d3db5f6b9ed0c
    Closes-bug: 1596962
    Closes-bug: 1598807
    (cherry picked from commit 7f17824ded359ff00f80988ff82270ecb594add7)

tags: added: in-stable-mitaka
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to networking-bgpvpn (stable/liberty)

Reviewed: https://review.openstack.org/340735
Committed: https://git.openstack.org/cgit/openstack/networking-bgpvpn/commit/?id=246b4888c456f3ec8b2119d18032c6f691f3df78
Submitter: Jenkins
Branch: stable/liberty

commit 246b4888c456f3ec8b2119d18032c6f691f3df78
Author: Thomas Monguillon <email address hidden>
Date: Tue Jul 5 11:28:32 2016 +0200

    Initialize the routers key in make_bgpvpn_dict

    This patch prevents to trigger an error with OpenContrail when a
    network association is declared on a network bounded to a router.

    Change-Id: I61e699690c86e523e0479a4cb34d3db5f6b9ed0c
    Closes-bug: 1596962
    Closes-bug: 1598807
    (cherry picked from commit 7f17824ded359ff00f80988ff82270ecb594add7)

tags: added: in-stable-liberty
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on networking-bgpvpn (backport/juno)

Change abandoned by Thomas Morin (<email address hidden>) on branch: backport/juno
Review: https://review.openstack.org/340737
Reason: This backport branch will not be supported anymore.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on networking-bgpvpn (backport/kilo)

Change abandoned by Thomas Morin (<email address hidden>) on branch: backport/kilo
Review: https://review.openstack.org/340736
Reason: This backport branch will not be supported anymore.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/networking-bgpvpn 4.0.2

This issue was fixed in the openstack/networking-bgpvpn 4.0.2 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/networking-bgpvpn 5.0.0

This issue was fixed in the openstack/networking-bgpvpn 5.0.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.