Juniper Openstack

k8s_5.0:Ingress: haproxy doesn't run with tls enabled as the node ip is taken as kubernetes_api_serve ip

Bug #1770154 reported by Venkatesh Velpula
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R5.0
Fix Released
Critical
Andrey Pavlov
Juniper Openstack r5.0.1
Trunk
Fix Released
Critical
Andrey Pavlov
Juniper Openstack r5.1.0

Bug Description

Only with tls enabled we see this issue and this both with single and multi(control and data)interfaces .

orchestrator :k8s
provisioing : Ansible deployer
Build :ocata-master-89,ocata-5.0-50

Setup
=====
[root@nodec19 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
nodec19 NotReady master 2d v1.9.2
nodei16 Ready <none> 2d v1.9.2
nodei18 Ready <none> 2d v1.9.2

[root@nodei18 /]# cat /etc/contrail/contrail-lbaas-auth.conf
[KUBERNETES]
kubernetes_api_server=10.204.217.130<<<<<<<<<<<<<<<<<<<<<node ip
kubernetes_api_port=8080
kubernetes_api_secure_port=6443

logs for the vrouter agent(contrail-lbaas-haproxy-stdout.log)
================================================================
05/09/2018 14:00:17 MSG Updating haproxy for Loadbalancer-ID fadcadae-5362-11e8-8d04-002590c3afaa
05/09/2018 14:01:04 ERROR Error in getting secrets ctest-namespace-42607436 - ctest-secret-11881374
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/opencontrail_vrouter_netns/cert_mgr/kubernetes_cert.py", line 78, in get_tls_certificates
    data = json['data']
KeyError: 'data'
05/09/2018 14:01:04 ERROR <type 'exceptions.KeyError'>
05/09/2018 14:01:04 ERROR Mapping key not found.
05/09/2018 14:01:04 ERROR data
05/09/2018 14:01:04 ERROR Exception in Createing haproxy config for Loadbalancer-ID fadcadae-5362-11e8-8d04-002590c3afaa
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/opencontrail_vrouter_netns/haproxy_process.py", line 189, in start_update_haproxy
    haproxy_cfg_file = get_haproxy_config_file(cfg_file, dir_name)
  File "/usr/lib/python2.7/site-packages/opencontrail_vrouter_netns/haproxy_process.py", line 91, in get_haproxy_config_file
    provider, haproxy_config, lbaas_auth_conf, dir_name)
  File "/usr/lib/python2.7/site-packages/opencontrail_vrouter_netns/cert_mgr/cert_manager.py", line 28, in update_ssl_config
    update_ssl_config(haproxy_config, dest_dir)
  File "/usr/lib/python2.7/site-packages/opencontrail_vrouter_netns/cert_mgr/kubernetes_cert.py", line 107, in update_ssl_config
    pem_file_name = tls.create_pem_file(dest_dir)
AttributeError: 'NoneType' object has no attribute 'create_pem_file'
05/09/2018 14:01:04 ERROR <type 'exceptions.AttributeError'>
05/09/2018 14:01:04 ERROR Attribute not found.
05/09/2018 14:01:04 ERROR 'NoneType' object has no attribute 'create_pem_file'

Venkatesh Velpula (vvelpula)
Changed in juniperopenstack:
milestone: none → r5.0.1
Revision history for this message
Yuvaraja Mariappan (ymariappan) wrote :

kubernetes_api_server is populated with wrong ip in /etc/contrail/contrail-lbaas-auth.conf.
It seems contrail-ansible-deployer is not setting the KUBERNETES_API_SERVER value properly.

Thanks,
Yuvaraja

Revision history for this message
Yuvaraja Mariappan (ymariappan) wrote :
Revision history for this message
Andrey Pavlov (apavlov-e) wrote :

from the ansible code I see that derivation logic looks correct

- name: add bms container hosts
  add_host:
    private_ip: "{{ item.value.ip }}"

- name: fill master list
  set_fact:
    master_list: "{{ master_list + [ hostvars[item]['private_ip'] ] }}"
  when: roles[hostvars[item]['instance_name']].k8s_master is defined and controller_item in hostvars[item].ansible_all_ipv4_addresses
  with_items:
    - "{{ groups['container_hosts'] }}"

- name: set master
  set_fact:
    master: "{{ master_list[0] }}"
  when: master_list | length > 0

- name: "create contrail_configuration.KUBERNETES_API_SERVER"
  set_fact:
    contrail_configuration: "{{ contrail_configuration|default({})|combine({'KUBERNETES_API_SERVER': master}) }}"
  when: master is defined and roles[instance_name].kubemanager.KUBERNETES_API_SERVER is undefined

can you please show ansible output for tasks:
- name: show master list
- name: show master inventory list
- name: show master
- name: show master inventory

Revision history for this message
Venkatesh Velpula (vvelpula) wrote :

TASK [install_contrail : init master list] ***********************************************************************************************************************************
ok: [10.204.217.4]
ok: [10.204.217.130]
ok: [10.204.217.128]

TASK [install_contrail : init master inventory list] *************************************************************************************************************************
ok: [10.204.217.4]
ok: [10.204.217.130]
ok: [10.204.217.128]

TASK [install_contrail : set master] *****************************************************************************************************************************************
included: /root/contrail-ansible-deployer/playbooks/roles/install_contrail/tasks/set_master.yml for 10.204.217.4, 10.204.217.128, 10.204.217.130

TASK [install_contrail : fill master list] ***********************************************************************************************************************************
ok: [10.204.217.4] => (item=10.204.217.4)
skipping: [10.204.217.4] => (item=10.204.217.128)
ok: [10.204.217.128] => (item=10.204.217.4)
skipping: [10.204.217.128] => (item=10.204.217.128)
skipping: [10.204.217.4] => (item=10.204.217.130)
ok: [10.204.217.130] => (item=10.204.217.4)
skipping: [10.204.217.128] => (item=10.204.217.130)
skipping: [10.204.217.130] => (item=10.204.217.128)
skipping: [10.204.217.130] => (item=10.204.217.130)

TASK [install_contrail : show master list] ***********************************************************************************************************************************
ok: [10.204.217.4] => {
    "msg": [
        "10.204.217.4"
    ]
}
ok: [10.204.217.128] => {
    "msg": [
        "10.204.217.4"
    ]
}
ok: [10.204.217.130] => {
    "msg": [
        "10.204.217.4"
    ]
}

TASK [install_contrail : set master] *****************************************************************************************************************************************
ok: [10.204.217.4]
ok: [10.204.217.128]
ok: [10.204.217.130]

TASK [install_contrail : fill master inventory list] *************************************************************************************************************************
ok: [10.204.217.4] => (item=10.204.217.4)
skipping: [10.204.217.4] => (item=10.204.217.128)
skipping: [10.204.217.4] => (item=10.204.217.130)
ok: [10.204.217.128] => (item=10.204.217.4)
skipping: [10.204.217.128] => (item=10.204.217.128)
ok: [10.204.217.130] => (item=10.204.217.4)
skipping: [10.204.217.128] => (item=10.204.217.130)
skipping: [10.204.217.130] => (item=10.204.217.128)
skipping: [10.204.217.130] => (item=10.204.217.130)

TASK [install_contrail : show master inventory list] *************************************************************************************************************************
ok: [10.204.217.4] => {
    "msg": [
        "10.204.217.4"
    ]
}
ok: [10.204.217.128] => {
    "msg": [
        "10.204.217.4"
    ]
}
ok: [10.204.217.130] => {
    "msg": [
        "10.204.217.4"
    ]
}

Revision history for this message
Venkatesh Velpula (vvelpula) wrote :

TASK [install_contrail : show master] ****************************************************************************************************************************************
ok: [10.204.217.4] => {
    "msg": "10.204.217.4"
}
ok: [10.204.217.128] => {
    "msg": "10.204.217.4"
}
ok: [10.204.217.130] => {
    "msg": "10.204.217.4"
}

TASK [install_contrail : show master inventory] ******************************************************************************************************************************
ok: [10.204.217.4] => {
    "msg": "10.204.217.4"
}
ok: [10.204.217.128] => {
    "msg": "10.204.217.4"
}
ok: [10.204.217.130] => {
    "msg": "10.204.217.4"
}

Revision history for this message
Andrey Pavlov (apavlov-e) wrote :

It's very confused. IP-s always different - In original description, in instances.yml, in last message.

From last message I see that master has just one IP - is it IP of k8s_master role? if so - what's the bug about?

Revision history for this message
Venkatesh Velpula (vvelpula) wrote :

Hi Andrey ..
   please find the instance.yaml file below yes the master has just one ip and tat ip is of k8s_matsre role ...

global_configuration:
  REGISTRY_PRIVATE_INSECURE: True
  CONTAINER_REGISTRY: 10.204.217.152:5000
provider_config:
  bms:
    ssh_pwd: c0ntrail123
    ssh_user: root
    ssh_public_key: /root/.ssh/id_rsa.pub
    ssh_private_key: /root/.ssh/id_rsa
    domainsuffix: local
instances:
  nodec19:
    provider: bms
    ip: 10.204.217.4
    roles:
       config_database:
       config:
       control:
       analytics_database:
       analytics:
       webui:
       k8s_master:
       kubemanager:
  nodei16:
    provider: bms
    ip: 10.204.217.128
    roles:
       k8s_node:
       vrouter:
  nodei18:
    provider: bms
    ip: 10.204.217.130
    roles:
       k8s_node:
       vrouter:
contrail_configuration:
  CONTAINER_REGISTRY: 10.204.217.152:5000
  CONTRAIL_VERSION: ocata-master-91
  KUBERNETES_IP_FABRIC_SUBNETS: 10.204.219.48/29
  KUBERNETES_PUBLIC_FIP_POOL: {'project': 'default', 'domain': 'default-domain', 'name': '__fip_pool_public__', 'network': '__public__'}
  CLOUD_ORCHESTRATOR: kubernetes
  VROUTER_GATEWAY: 10.204.217.254

Revision history for this message
Andrey Pavlov (apavlov-e) wrote :

@Yuvaraja - please add more details if it's still the bug for you.

Revision history for this message
Yuvaraja Mariappan (ymariappan) wrote :

@andrey - pasted the output of the contrail-lbaas-auth.conf from all the k8s-node.
kube-api-server should be 10.84.7.34 instead of the node ip. I have attached the instances.yaml as well.

[root@b4s41 /]# cat /etc/contrail/contrail-lbaas-auth.conf
[BARBICAN]
admin_tenant_name = service
admin_user = barbican
admin_password = contrail123
auth_url = http://127.0.0.1:35357/v2.0
region = RegionOne
user_domain_name = Default
project_domain_name = Default
region_name = RegionOne
insecure = True
certfile =
keyfile =
cafile =
[KUBERNETES]
kubernetes_token=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJjb250cmFpbCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjb250cmFpbC1rdWJlbWFuYWdlci10b2tlbi1ncG03eiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJjb250cmFpbC1rdWJlbWFuYWdlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjBjMGRiZTI1LTYwOWYtMTFlOC1hMTgzLTAwMjU5MGE1NDU4MyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpjb250cmFpbDpjb250cmFpbC1rdWJlbWFuYWdlciJ9.q3tRamhVQNLfkQ_eQqQBWGOpzLlbOa6_LQ7DNk8U-jR7eikT8NJi0GqstoacXQ5E1eEF1JNAHkoY5MD5n10gQSNyBwxxAeTYYQNZk7n9j9-DjfnnHJEO78MkM3oL-YEO5fGf_1xMv8AAc19tiQfoqiQxP-Prs1CmPJ3HsmCmF-Lx8Y2dDRd9Odf8BNkyzQwjaM9Saou91c703rxQDVbb98_LTPmb8VLZh-9gc3WQIoy8Mnjpft-54BZM1f80XZYQZb49X9CQ1abYlsYo9eZbdRiE7JRZliad5rqAq0qLfYu64UZVWegkFlnhBSQi2A8GBIleub7gfK1wetcO6zr2SA
kubernetes_api_server=10.84.24.61
kubernetes_api_port=8080
kubernetes_api_secure_port=6443
[root@b4s41 /]#

[root@b4s42 /]# cat /etc/contrail/contrail-lbaas-auth.conf
[BARBICAN]
admin_tenant_name = service
admin_user = barbican
admin_password = contrail123
auth_url = http://127.0.0.1:35357/v2.0
region = RegionOne
user_domain_name = Default
project_domain_name = Default
region_name = RegionOne
insecure = True
certfile =
keyfile =
cafile =
[KUBERNETES]
kubernetes_token=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJA
kubernetes_api_server=10.84.24.62
kubernetes_api_port=8080
kubernetes_api_secure_port=6443
[root@b4s42 /]#

instances.yaml
--------------
global_configuration:
  CONTAINER_REGISTRY: ci-repo.englab.juniper.net:5000
  REGISTRY_PRIVATE_INSECURE: true
  CONTRAIL_VERSION: master-115
provider_config:
  bms:
    ssh_pwd: c0ntrail123
    ssh_user: root
    domainsuffix: local
instances:
  bms1:
    provider: bms
    ip: 10.84.7.34
    roles:
        config_database:
        config:
        control:
        analytics_database:
        analytics:
        webui:
        k8s_master:
        kubemanager:
  bms2:
    provider: bms
    ip: 10.84.24.61
    roles:
        vrouter:
        k8s_node:
  bms3:
    provider: bms
    ip: 10.84.24.62
    roles:
        vrouter:
        k8s_node:
contrail_configuration:
  CLOUD_ORCHESTRATOR: kubernetes

Thanks,
Yuavraja

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/43316
Submitter: Andrey Pavlov (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R5.0

Review in progress for https://review.opencontrail.org/43319
Submitter: Andrey Pavlov (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/43316
Submitter: Andrey Pavlov (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R5.0

Review in progress for https://review.opencontrail.org/43319
Submitter: Andrey Pavlov (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/43316
Committed: http://github.com/Juniper/contrail-ansible-deployer/commit/378340682587aab50106d5359387c7f1821bda41
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit 378340682587aab50106d5359387c7f1821bda41
Author: Andrey Pavlov <email address hidden>
Date: Sat May 26 12:15:07 2018 +0300

re-order definitions for k8s

Change-Id: I3b29a9c35bf44caee798a8edd26d99e489371786
Closes-Bug: #1770154

Revision history for this message
Venkatesh Velpula (vvelpula) wrote :

verified on ocata-master-117 build

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/43319
Committed: http://github.com/Juniper/contrail-ansible-deployer/commit/ed766cd7c9b8c824695ec0affd3315745a4d0c94
Submitter: Zuul v3 CI (<email address hidden>)
Branch: R5.0

commit ed766cd7c9b8c824695ec0affd3315745a4d0c94
Author: Andrey Pavlov <email address hidden>
Date: Sat May 26 12:15:07 2018 +0300

re-order definitions for k8s

Change-Id: I3b29a9c35bf44caee798a8edd26d99e489371786
Closes-Bug: #1770154

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.