[Kube Manager]: Kube manager container restarting

Bug #1777277 reported by Pulkit Tandon
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R5.0
Fix Released
Critical
Ignatious Johnson Christopher
Trunk
Fix Released
Critical
Ignatious Johnson Christopher

Bug Description

R5.0-101
Contrail+K8s provisioning.

Kube manager container stuck in restart loop.

Following are the container logs:

+ mv /etc/contrail/vnc_api_lib.ini.tmp /etc/contrail/vnc_api_lib.ini
+ exec /usr/bin/python /usr/bin/contrail-kube-manager -c /etc/contrail/contrail-kubernetes.conf
06/16/2018 09:15:24 AM [contrail-kube-manager]: SANDESH: CONNECT TO COLLECTOR: True
06/16/2018 09:15:24 AM [contrail-kube-manager]: SANDESH: Logging: LEVEL: [SYS_INFO] -> [SYS_DEBUG]
06/16/2018 09:15:24 AM [contrail-kube-manager]: SANDESH: Logging: FILE: [None] -> [/var/log/contrail/contrail-kube-manager.log]
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
NamespaceMonitor - Got ADDED Namespace contrail:19fe9131-70cb-11e8-9179-002590c476a0
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
NamespaceMonitor - Got ADDED Namespace default:16c36eb2-70cb-11e8-9179-002590c476a0
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
NamespaceMonitor - Got ADDED Namespace kube-public:16c4ef25-70cb-11e8-9179-002590c476a0
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
NamespaceMonitor - Got ADDED Namespace kube-system:1238c513-70cb-11e8-9179-002590c476a0
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
PodMonitor - Got ADDED Pod kube-system:kube-dns-6f4fd4bdf-wcxvj:1be0a05e-70cb-11e8-9179-002590c476a0
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
ServiceMonitor - Got ADDED Service default:kubernetes:16c3de06-70cb-11e8-9179-002590c476a0
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
ServiceMonitor - Got ADDED Service kube-system:kube-dns:18a8edc5-70cb-11e8-9179-002590c476a0
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
ServiceMonitor - Got ADDED Service kube-system:kubernetes-dashboard:45bcbbb8-70cb-11e8-9179-002590c476a0
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
EndPointMonitor - Got ADDED Endpoints default:kubernetes:16c469bf-70cb-11e8-9179-002590c476a0
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
EndPointMonitor - Got ADDED Endpoints kube-system:kube-dns:1bdcf97e-70cb-11e8-9179-002590c476a0
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
EndPointMonitor - Got ADDED Endpoints kube-system:kubernetes-dashboard:45bf0bef-70cb-11e8-9179-002590c476a0
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:838: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/security.html
  InsecureRequestWarning)
Traceback (most recent call last):
  File "/usr/bin/contrail-kube-manager", line 9, in <module>
    load_entry_point('kube-manager==0.1dev', 'console_scripts', 'contrail-kube-manager')()
  File "/usr/lib/python2.7/site-packages/kube_manager/kube_manager.py", line 262, in main
    vnc_kubernetes_config_dict)
  File "/usr/lib/python2.7/site-packages/cfgm_common/zkclient.py", line 505, in master_election
    self._election.run(func, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/kazoo/recipe/election.py", line 53, in run
    func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/kube_manager/kube_manager.py", line 194, in run_kube_manager
    is_master=True)
  File "/usr/lib/python2.7/site-packages/kube_manager/kube_manager.py", line 89, in __init__
    vnc_kubernetes_config_dict=vnc_kubernetes_config_dict)
  File "/usr/lib/python2.7/site-packages/kube_manager/vnc/vnc_kubernetes.py", line 60, in __init__
    self.vnc_lib = self._vnc_connect()
  File "/usr/lib/python2.7/site-packages/kube_manager/vnc/vnc_kubernetes.py", line 170, in _vnc_connect
    auth_token_url=self.args.auth_token_url)
  File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 508, in __init__
    _read_cfg(cfg_parser, 'global', 'curl_log', False))
  File "/usr/lib/python2.7/site-packages/vnc_api/vnc_api.py", line 78, in __init__
    self.log_file.basename())
AttributeError: 'str' object has no attribute 'basename'

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/43902
Submitter: Ignatious Johnson Christopher (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R5.0

Review in progress for https://review.opencontrail.org/43903
Submitter: Ignatious Johnson Christopher (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/43902
Submitter: Ignatious Johnson Christopher (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R5.0

Review in progress for https://review.opencontrail.org/43937
Submitter: Ignatious Johnson Christopher (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/43902
Submitter: Ignatious Johnson Christopher (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R5.0

Review in progress for https://review.opencontrail.org/43937
Submitter: Ignatious Johnson Christopher (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/43902
Committed: http://github.com/Juniper/contrail-api-client/commit/6e24246a7b3de83a0c7475ab94ad952d2ea17308
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit 6e24246a7b3de83a0c7475ab94ad952d2ea17308
Author: Ignatious Johnson Christopher <email address hidden>
Date: Mon Jun 18 00:59:48 2018 +0000

Using the proper basename method to find the

basename of the given log file.
Also try creating the log dir if not exists.

Added UT's

Change-Id: Ib67699b17d032321535f4dac453ccc225cf90757
Closes-Bug: 1777277

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/43937
Committed: http://github.com/Juniper/contrail-api-client/commit/6c8b72993d27afa59067ec56f484e74c02c2d32f
Submitter: Zuul v3 CI (<email address hidden>)
Branch: R5.0

commit 6c8b72993d27afa59067ec56f484e74c02c2d32f
Author: Ignatious Johnson Christopher <email address hidden>
Date: Mon Jun 18 00:59:48 2018 +0000

Using the proper basename method to find the

basename of the given log file.
Also try creating the log dir if not exists.

Added UT's

Change-Id: Ib67699b17d032321535f4dac453ccc225cf90757
Closes-Bug: 1777277

Revision history for this message
Pulkit Tandon (pulkitt) wrote :

Verified on R5.0-111.
Issue not observed. hence closing the bug

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.