Endpoint-less Services keep the lbaasspec handler endlessly retrying

Bug #1697942 reported by Antoni Segura Puimedon
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kuryr-kubernetes
Fix Released
High
Luis Tomas Bolivar

Bug Description

When you define a Service without a selector, Kubernetes does not create an endpoints object for it.

    kind: Service
    apiVersion: v1
    metadata:
      name: kuryr-svc-router
      namespace: kube-system
    spec:
      type: ClusterIP
      clusterIP: 172.30.255.254
      ports:
        - protocol: TCP
          port: 80

This ends up in the LBaaSSpecHandler failing to set up the lbaas spec on the endpoint object:

    Jun 14 13:09:49 gerrit-472763vm-0 python[2454]: 2017-06-14 13:09:49.382 2454 ERROR kuryr_kubernetes.handlers.logging Traceback (most recent call last):
    Jun 14 13:09:49 gerrit-472763vm-0 python[2454]: 2017-06-14 13:09:49.382 2454 ERROR kuryr_kubernetes.handlers.logging File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/handlers/logging.py", line 37, in __call__
    Jun 14 13:09:49 gerrit-472763vm-0 python[2454]: 2017-06-14 13:09:49.382 2454 ERROR kuryr_kubernetes.handlers.logging self._handler(event)
    Jun 14 13:09:49 gerrit-472763vm-0 python[2454]: 2017-06-14 13:09:49.382 2454 ERROR kuryr_kubernetes.handlers.logging File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/handlers/retry.py", line 66, in __call__
    Jun 14 13:09:49 gerrit-472763vm-0 python[2454]: 2017-06-14 13:09:49.382 2454 ERROR kuryr_kubernetes.handlers.logging ex.reraise = False
    Jun 14 13:09:49 gerrit-472763vm-0 python[2454]: 2017-06-14 13:09:49.382 2454 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
    Jun 14 13:09:49 gerrit-472763vm-0 python[2454]: 2017-06-14 13:09:49.382 2454 ERROR kuryr_kubernetes.handlers.logging self.force_reraise()
    Jun 14 13:09:49 gerrit-472763vm-0 python[2454]: 2017-06-14 13:09:49.382 2454 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
    Jun 14 13:09:49 gerrit-472763vm-0 python[2454]: 2017-06-14 13:09:49.382 2454 ERROR kuryr_kubernetes.handlers.logging six.reraise(self.type_, self.value, self.tb)
    Jun 14 13:09:49 gerrit-472763vm-0 python[2454]: 2017-06-14 13:09:49.382 2454 ERROR kuryr_kubernetes.handlers.logging File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/handlers/retry.py", line 61, in __call__
    Jun 14 13:09:49 gerrit-472763vm-0 python[2454]: 2017-06-14 13:09:49.382 2454 ERROR kuryr_kubernetes.handlers.logging self._handler(event)
    Jun 14 13:09:49 gerrit-472763vm-0 python[2454]: 2017-06-14 13:09:49.382 2454 ERROR kuryr_kubernetes.handlers.logging File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/handlers/k8s_base.py", line 63, in __call__
    Jun 14 13:09:49 gerrit-472763vm-0 python[2454]: 2017-06-14 13:09:49.382 2454 ERROR kuryr_kubernetes.handlers.logging self.on_present(obj)
    Jun 14 13:09:49 gerrit-472763vm-0 python[2454]: 2017-06-14 13:09:49.382 2454 ERROR kuryr_kubernetes.handlers.logging File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/controller/handlers/lbaas.py", line 49, in on_present
    Jun 14 13:09:49 gerrit-472763vm-0 python[2454]: 2017-06-14 13:09:49.382 2454 ERROR kuryr_kubernetes.handlers.logging self._set_lbaas_spec(service, lbaas_spec)
    Jun 14 13:09:49 gerrit-472763vm-0 python[2454]: 2017-06-14 13:09:49.382 2454 ERROR kuryr_kubernetes.handlers.logging File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/controller/handlers/lbaas.py", line 165, in _set_lbaas_spec
    Jun 14 13:09:49 gerrit-472763vm-0 python[2454]: 2017-06-14 13:09:49.382 2454 ERROR kuryr_kubernetes.handlers.logging raise k_exc.ResourceNotReady(ep_link)
    Jun 14 13:09:49 gerrit-472763vm-0 python[2454]: 2017-06-14 13:09:49.382 2454 ERROR kuryr_kubernetes.handlers.logging ResourceNotReady: Resource not ready: u'/api/v1/namespaces/kube-system/endpoints/kuryr-svc-router'
    Jun 14 13:09:49 gerrit-472763vm-0 python[2454]: 2017-06-14 13:09:49.382 2454 ERROR kuryr_kubernetes.handlers.logging
    Jun 14 13:18:24 gerrit-472763vm-0 python[2454]: 2017-06-14 13:18:24.855 2454 ERROR kuryr_kubernetes.handlers.logging [-] Failed to handle event {u'object': {u'status': {u'loadBalancer': {}}, u'kind': u'Service', u'spec': {u'clusterIP': u'172.30.255.254', u'type': u'ClusterIP', u'ports': [{u'targetPort': 80, u'protocol': u'TCP', u'port': 80}], u'sessionAffinity': u'None'}, u'apiVersion': u'v1', u'metadata': {u'name': u'kuryr-svc-router', u'namespace': u'kube-system', u'resourceVersion': u'42', u'creationTimestamp': u'2017-06-14T10:30:33Z', u'selfLink': u'/api/v1/namespaces/kube-system/services/kuryr-svc-router', u'uid': u'7f2ee8a3-50ec-11e7-9fdb-fa163e55148f'}}, u'type': u'ADDED'}: ResourceNotReady: Resource not ready: u'/api/v1/namespaces/kube-system/endpoints/kuryr-svc-router

We should probably treat selector-less services differently or maybe use Third Party Resources (TPR).

Changed in kuryr-kubernetes:
importance: Undecided → High
status: New → Triaged
milestone: none → pike-3
Changed in kuryr-kubernetes:
assignee: nobody → Luis Tomas Bolivar (ltomasbo)
Changed in kuryr-kubernetes:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kuryr-kubernetes (master)

Reviewed: https://review.openstack.org/483891
Committed: https://git.openstack.org/cgit/openstack/kuryr-kubernetes/commit/?id=f303f32c153e25bbb2c392f54baa917130c33915
Submitter: Jenkins
Branch: master

commit f303f32c153e25bbb2c392f54baa917130c33915
Author: Luis Tomas Bolivar <email address hidden>
Date: Fri Jul 14 09:12:55 2017 +0000

    Skips kubernetes service without selector

    When a service is defined without a selector, Kubernetes does not
    create an endpoint object for it. This patch includes a pre-checking
    at the LBaaSSpecHandler on_present to skip this service.

    Change-Id: Ic5a9610175c0bf35664781c23f674c8c890e85f9
    Closes-Bug: 1697942

Changed in kuryr-kubernetes:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kuryr-kubernetes 0.2.0

This issue was fixed in the openstack/kuryr-kubernetes 0.2.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.