[Build 3060] K8S : Ingress : When service getting removed and added back, haproxy config is not getting updated

Bug #1686331 reported by chhandak
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R4.0
Fix Committed
High
Yuvaraja Mariappan
Trunk
Fix Committed
High
Yuvaraja Mariappan

Bug Description

Description:
When the service behind the Ingress is getting deleted and readded back with the same name (automatically get created with different clusterIP), Load balancer haproxy config is not getting updated in agent. Haproxy backend still pointing to old service reference. Observing a traceback in contrail-kube-manager.log

Steps:
1. Create Service
2. Create Ingress with service
3. Delete the Service
4. Add the service back with same name.

Output
-------
1. Service is created
-------------------------
[root@5b7s18 ~]# kubectl get svc
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes 10.96.0.1 <none> 443/TCP 8h
my-service 10.102.238.154 <none> 80/TCP 7m

2. Ingress is created
------------------------
[root@5b7s18 ~]# kubectl get ing
NAME HOSTS ADDRESS PORTS AGE
test-ingress * 80 6m
[root@5b7s18 ~]# kubectl get ing -o=custom-columns=NAME:.metadata.name,CLUSTER-IP:.metadata.annotations.clusterIP,EXTERNAL-IP:.metadata.annotations.externalIP
NAME CLUSTER-IP EXTERNAL-IP
test-ingress 10.47.255.248 10.87.117.251

Haproxy config in Agent
-------------------------
global
        daemon
        user haproxy
        group haproxy
        log /dev/log local0
        log /dev/log local1 notice
        tune.ssl.default-dh-param 2048
        ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
        ulimit-n 200000
        maxconn 65000
        stats socket /var/lib/contrail/loadbalancer/haproxy/e86aa14f-2a52-11e7-a674-0cc47aa89e64/haproxy.sock mode 0666 level user

defaults
        log global
        retries 3
        option redispatch
        timeout connect 5000
        timeout client 300000
        timeout server 300000

frontend e3cd0c9c-d856-40ff-9813-7b2a77b61b2d
        option tcplog
        bind 10.47.255.248:80
        mode http
        option forwardfor
        acl 914e7571-b33c-4e24-8c0b-37e1f27cfd7a_path path /testpath
        use_backend 914e7571-b33c-4e24-8c0b-37e1f27cfd7a if 914e7571-b33c-4e24-8c0b-37e1f27cfd7a_path

backend 914e7571-b33c-4e24-8c0b-37e1f27cfd7a
        mode http
        balance roundrobin
        option forwardfor
        server 73716d04-a99f-4dbe-8845-7836b1c1902f 10.102.238.154:80 weight 1

3. Delete the service
----------------------

[root@5b7s18 ~]# kubectl delete svc
error: resource(s) were provided, but no name, label selector, or --all flag specified
[root@5b7s18 ~]# kubectl delete svc my-service
service "my-service" deleted
[root@5b7s18 ~]#
[root@5b7s18 ~]#

4. Adding the service back but got created with diffrent clusterIP
--------------------------------------------------------------------
[root@5b7s18 ~]# kubectl create -f svc.yaml
service "my-service" created
[root@5b7s18 ~]# kubectl get svc
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes 10.96.0.1 <none> 443/TCP 8h
my-service 10.100.196.88 <none> 80/TCP 4s

Haproxy config in agent
-------------------------
global
        daemon
        user haproxy
        group haproxy
        log /dev/log local0
        log /dev/log local1 notice
        tune.ssl.default-dh-param 2048
        ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
        ulimit-n 200000
        maxconn 65000
        stats socket /var/lib/contrail/loadbalancer/haproxy/e86aa14f-2a52-11e7-a674-0cc47aa89e64/haproxy.sock mode 0666 level user

defaults
        log global
        retries 3
        option redispatch
        timeout connect 5000
        timeout client 300000
        timeout server 300000

frontend e3cd0c9c-d856-40ff-9813-7b2a77b61b2d
        option tcplog
        bind 10.47.255.248:80
        mode http
        option forwardfor
        acl 914e7571-b33c-4e24-8c0b-37e1f27cfd7a_path path /testpath
        use_backend 914e7571-b33c-4e24-8c0b-37e1f27cfd7a if 914e7571-b33c-4e24-8c0b-37e1f27cfd7a_path

backend 914e7571-b33c-4e24-8c0b-37e1f27cfd7a
        mode http
        balance roundrobin
        option forwardfor
        server 73716d04-a99f-4dbe-8845-7836b1c1902f 10.102.238.154:80 weight 1>>> still poiting to old IP

Traceback
-----------
The above is a description of an error in a Python program. Here is
the original traceback:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/kube_manager/vnc/vnc_kubernetes.py", line 337, in vnc_process
    self.endpoints_mgr.process(event)
  File "/usr/lib/python2.7/dist-packages/kube_manager/vnc/vnc_endpoints.py", line 301, in process
    self.vnc_endpoint_add(uid, name, namespace, event)
  File "/usr/lib/python2.7/dist-packages/kube_manager/vnc/vnc_endpoints.py", line 250, in vnc_endpoint_add
    for port in ports:
TypeError: 'NoneType' object is not iterable

chhandak (chhandak)
Changed in juniperopenstack:
importance: Undecided → High
assignee: nobody → ymariappan (ymariappan)
milestone: none → r4.0
summary: - [Build 3060] K8S : Ingress : When service getting removed and added back
- haproxy config is not getting updated
+ [Build 3060] K8S : Ingress : When service getting removed and added
+ back, haproxy config is not getting updated
description: updated
information type: Proprietary → Public
Revision history for this message
chhandak (chhandak) wrote :

Logs copied in below location
------------------------------
-rwxrwxrwx 1 chhandak epbg 3209282 Apr 26 00:53 contrail-kube-manager.log
ubuntu-build04:/auto/cores/1686331> pwd
/auto/cores/1686331

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/30857
Submitter: Yuvaraja Mariappan

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/30857
Committed: http://github.com/Juniper/contrail-controller/commit/fb8d8dcbaf37aa158bfc22dda9ee05fbabc399be
Submitter: Zuul (<email address hidden>)
Branch: master

commit fb8d8dcbaf37aa158bfc22dda9ee05fbabc399be
Author: Yuvaraja Mariappan <email address hidden>
Date: Fri Apr 28 08:43:09 2017 -0700

Fixed exception issue due to end points delete in k8s

port information may not be there in the end point delete event
which causes the issue. Fix is added to handle this

Change-Id: I170963558f6753a0f592e5752d9d98b3ef06bc04
Partial-bug: #1686331

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/31083
Submitter: Yuvaraja Mariappan

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.0

Review in progress for https://review.opencontrail.org/31177
Submitter: Yuvaraja Mariappan

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/31083
Committed: http://github.com/Juniper/contrail-controller/commit/0d0414accebf0b6540e85d3c6fb822897105cb66
Submitter: Zuul (<email address hidden>)
Branch: master

commit 0d0414accebf0b6540e85d3c6fb822897105cb66
Author: Yuvaraja Mariappan <email address hidden>
Date: Sat May 6 14:47:44 2017 -0700

Fixed service deletion/addition issue in running ingress in k8s

when the service gets deleted/added, ingress was not updated
which is fixed

Change-Id: I9118aec3aafaf961dc01351ebd5361c7bf33effd
Closes-bug: #1686331

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/31177
Committed: http://github.com/Juniper/contrail-controller/commit/da3dafd4e4222efe1fe0291f38ad4dce6800dcb2
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit da3dafd4e4222efe1fe0291f38ad4dce6800dcb2
Author: Yuvaraja Mariappan <email address hidden>
Date: Sat May 6 14:47:44 2017 -0700

Fixed service deletion/addition issue in running ingress in k8s

when the service gets deleted/added, ingress was not updated
which is fixed

Change-Id: I9118aec3aafaf961dc01351ebd5361c7bf33effd
Closes-bug: #1686331

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.