devstack fail when create_k8s_router_fake_service in overcloud

Bug #1697279 reported by Kevin Zhao
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kuryr-kubernetes
Fix Released
High
Antoni Segura Puimedon

Bug Description

Run nested kuryr-kubernetes, in overcloud get error:
Error Log:
bernetes
2017-06-11 08:43:33.288 | ++^[[3242m/opt/stack/kuryr-kubernetes/devstack/plugin.sh:source:576 ^[[m^O is_service_enabled kuryr-kubernetes
2017-06-11 08:43:33.325 | ++^[[3242mfunctions-common:is_service_enabled:2191 ^[[m^O return 0
2017-06-11 08:43:33.332 | ++^[[3242m/opt/stack/kuryr-kubernetes/devstack/plugin.sh:source:577 ^[[m^O [[ stack == \s\t\a\c\k ]]
2017-06-11 08:43:33.339 | ++^[[3242m/opt/stack/kuryr-kubernetes/devstack/plugin.sh:source:577 ^[[m^O [[ test-config == \i\n\s\t\a\l\l ]]
2017-06-11 08:43:33.346 | ++^[[3242m/opt/stack/kuryr-kubernetes/devstack/plugin.sh:source:583 ^[[m^O [[ stack == \s\t\a\c\k ]]
2017-06-11 08:43:33.354 | ++^[[3242m/opt/stack/kuryr-kubernetes/devstack/plugin.sh:source:583 ^[[m^O [[ test-config == \p\o\s\t\-\c\o\n\f\i\g ]]
2017-06-11 08:43:33.362 | ++^[[3242m/opt/stack/kuryr-kubernetes/devstack/plugin.sh:source:588 ^[[m^O [[ stack == \s\t\a\c\k ]]
2017-06-11 08:43:33.339 | ++^[[3242m/opt/stack/kuryr-kubernetes/devstack/plugin.sh:source:577 ^[[m^O [[ test-config == \i\n\s\t\a\l\l ]]
2017-06-11 08:43:33.346 | ++^[[3242m/opt/stack/kuryr-kubernetes/devstack/plugin.sh:source:583 ^[[m^O [[ stack == \s\t\a\c\k ]]
2017-06-11 08:43:33.354 | ++^[[3242m/opt/stack/kuryr-kubernetes/devstack/plugin.sh:source:583 ^[[m^O [[ test-config == \p\o\s\t\-\c\o\n\f\i\g ]]
2017-06-11 08:43:33.362 | ++^[[3242m/opt/stack/kuryr-kubernetes/devstack/plugin.sh:source:588 ^[[m^O [[ stack == \s\t\a\c\k ]]
2017-06-11 08:43:33.369 | ++^[[3242m/opt/stack/kuryr-kubernetes/devstack/plugin.sh:source:588 ^[[m^O [[ test-config == \e\x\t\r\a ]]
2017-06-11 08:43:33.376 | ++^[[3242m/opt/stack/kuryr-kubernetes/devstack/plugin.sh:source:646 ^[[m^O [[ stack == \s\t\a\c\k ]]
2017-06-11 08:43:33.383 | ++^[[3242m/opt/stack/kuryr-kubernetes/devstack/plugin.sh:source:646 ^[[m^O [[ test-config == \t\e\s\t\-\c\o\n\f\i\g ]]
2017-06-11 08:43:33.397 | ++^[[3242m/opt/stack/kuryr-kubernetes/devstack/plugin.sh:source:647 ^[[m^O create_k8s_router_fake_service
2017-06-11 08:43:33.407 | ++^[[3242m/opt/stack/kuryr-kubernetes/devstack/plugin.sh:create_k8s_router_fake_service:113 ^[[m^O local service_cidr
2017-06-11 08:43:33.413 | ++^[[3242m/opt/stack/kuryr-kubernetes/devstack/plugin.sh:create_k8s_router_fake_service:114 ^[[m^O local router_ip
2017-06-11 08:43:33.429 | ++^[[3242m/opt/stack/kuryr-kubernetes/devstack/plugin.sh:create_k8s_router_fake_service:115 ^[[m^O local existing_svc_ip
2017-06-11 08:43:33.437 | ++^[[3242m/opt/stack/kuryr-kubernetes/devstack/plugin.sh:create_k8s_router_fake_service:116 ^[[m^O local fake_svc_name
2017-06-11 08:43:33.446 | ++^[[3242m/opt/stack/kuryr-kubernetes/devstack/plugin.sh:create_k8s_router_fake_service:118 ^[[m^O fake_svc_name=kuryr-svc-router
2017-06-11 08:43:33.453 | +++^[[3242m/opt/stack/kuryr-kubernetes/devstack/plugin.sh:create_k8s_router_fake_service:122 ^[[m^O openstack --os-cloud devstack-admin --os-region RegionOne subnet show k8s-service-subnet -c cidr -f value
2017-06-11 08:43:36.224 | ++^[[3242m/opt/stack/kuryr-kubernetes/devstack/plugin.sh:create_k8s_router_fake_service:122 ^[[m^O service_cidr=10.0.8.0/24
2017-06-11 08:43:36.242 | +++^[[3242m/opt/stack/kuryr-kubernetes/devstack/plugin.sh:create_k8s_router_fake_service:123 ^[[m^O _cidr_range 10.0.8.0/24
2017-06-11 08:43:36.247 | +++^[[3242m/opt/stack/kuryr-kubernetes/devstack/plugin.sh:create_k8s_router_fake_service:123 ^[[m^O cut -f2
2017-06-11 08:43:36.256 | +++^[[3242m/opt/stack/kuryr-kubernetes/devstack/plugin.sh:_cidr_range:104 ^[[m^O python - 10.0.8.0/24
2017-06-11 08:43:36.319 | ++^[[3242m/opt/stack/kuryr-kubernetes/devstack/plugin.sh:create_k8s_router_fake_service:123 ^[[m^O router_ip=10.0.8.254
2017-06-11 08:43:36.329 | +++^[[3242m/opt/stack/kuryr-kubernetes/devstack/plugin.sh:create_k8s_router_fake_service:124 ^[[m^O /usr/local/bin/kubectl get svc --namespace kube-system -o 'jsonpath={.items[?(@.metadata.name=="kuryr-svc-router")].spec.clusterIP}'
2017-06-11 08:43:36.651 | ++^[[3242m/opt/stack/kuryr-kubernetes/devstack/plugin.sh:create_k8s_router_fake_service:124 ^[[m^O existing_svc_ip=
2017-06-11 08:43:36.661 | ++^[[3242m/opt/stack/kuryr-kubernetes/devstack/plugin.sh:create_k8s_router_fake_service:126 ^[[m^O [[ '' == '' ]]
2017-06-11 08:43:36.675 | ++^[[3242m/opt/stack/kuryr-kubernetes/devstack/plugin.sh:create_k8s_router_fake_service:128 ^[[m^O /usr/local/bin/kubectl create -f -
2017-06-11 08:43:36.678 | ++^[[3242m/opt/stack/kuryr-kubernetes/devstack/plugin.sh:create_k8s_router_fake_service:128 ^[[m^O cat
2017-06-11 08:43:36.979 | The Service "kuryr-svc-router" is invalid: spec.clusterIP: Invalid value: "10.0.8.254": provided IP is not in the valid range
2017-06-11 08:43:36.994 | +^[[3242m/opt/stack/kuryr-kubernetes/devstack/plugin.sh:create_k8s_router_fake_service:1 ^[[m^O exit_trap

=======================================================================================================
The k8s-service-subnet is generated in undercloud OpenStack.
$ openstack network create net2
$ openstack subnet create --network net2 --subnet-range 10.0.8.0/24 k8s-service-subnet
$ openstack router add subnet router1 k8s-service-subnet
$ openstack subnet set k8s-service-subnet --dns-nameserver=8.8.8.8
$ openstack subnet show 9906e9cb-c719-4262-abd3-1152b90df4e0
+-------------------------+--------------------------------------+
| Field | Value |
+-------------------------+--------------------------------------+
| allocation_pools | 10.0.8.2-10.0.8.254 |
| cidr | 10.0.8.0/24 |
| created_at | 2017-06-11T08:39:42Z |
| description | |
| dns_nameservers | 10.170.4.29, 8.8.8.8 |
| enable_dhcp | True |
| gateway_ip | 10.0.8.1 |
| host_routes | |
| id | 9906e9cb-c719-4262-abd3-1152b90df4e0 |
| ip_version | 4 |
| ipv6_address_mode | None |
| ipv6_ra_mode | None |
| name | k8s-service-subnet |
| network_id | 68ca7df1-a160-4bcd-a579-8df29a6f8061 |
| project_id | 866aeac4c87c43eeb560aad877546230 |
| revision_number | 4 |
| segment_id | None |
| service_types | |
| subnetpool_id | None |
| updated_at | 2017-06-11T14:20:03Z |
| use_default_subnet_pool | None |
+-------------------------+--------------------------------------+

Revision history for this message
Irena Berezovsky (irenab) wrote :

Was not this issue fixed here: https://review.openstack.org/#/c/469788 ?

Revision history for this message
Kevin Zhao (kevin-zhao) wrote :

I guess not.
I have used the latest version of kuryr-kubernetes and this patch has been included

Changed in kuryr-kubernetes:
importance: Undecided → High
milestone: none → pike-3
assignee: nobody → Antoni Segura Puimedon (celebdor)
Changed in kuryr-kubernetes:
status: New → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kuryr-kubernetes (master)

Fix proposed to branch: master
Review: https://review.openstack.org/489642

Changed in kuryr-kubernetes:
status: Triaged → In Progress
Revision history for this message
Antoni Segura Puimedon (celebdor) wrote :

@kevin-zhao: Please, test with the patch I sent and remember that you can't have the first address of the service subnet for the router, since Kubernetes uses it for the kubernetes api service.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kuryr-kubernetes (master)

Reviewed: https://review.openstack.org/489642
Committed: https://git.openstack.org/cgit/openstack/kuryr-kubernetes/commit/?id=9c3053872305c3eaf88f8ae2cc05110d61044740
Submitter: Jenkins
Branch: master

commit 9c3053872305c3eaf88f8ae2cc05110d61044740
Author: Antoni Segura Puimedon <email address hidden>
Date: Tue Aug 1 17:45:24 2017 +0200

    devstack: Don't assume the router ip

    When using devstack for Pod-in-VM scenarios there's no telling where the
    dev created service subnet will have the router IP. As such, better than
    assuming it, we'll just check.

    Change-Id: Iabbd18cb780bb433b0558bd332c42aa324118f50
    Closes-Bug: #1697279
    Signed-off-by: Antoni Segura Puimedon <email address hidden>

Changed in kuryr-kubernetes:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kuryr-kubernetes 0.2.0

This issue was fixed in the openstack/kuryr-kubernetes 0.2.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.