redhat-osp10-mainline3073:contrail-status shows api failed

Bug #1689400 reported by shajuvk
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R4.0
Fix Committed
High
Nagendra Prasath
Trunk
Fix Committed
High
Nagendra Prasath

Bug Description

Even after restart service didn't come up.

controller, analytics and analyticsdb are in containers and redhat openstack on separate node

[root@a5s10 ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d67332db3872 68844dbde56a "/lib/systemd/syst..." About an hour ago Up About an hour contrail-analyticsdb
8970ef3a4f43 247625e51067 "/lib/systemd/syst..." About an hour ago Up About an hour contrail-analytics
fe9cc2180f85 d7587cd3943b "/lib/systemd/syst..." About an hour ago Up About an hour contrail-controller
[root@a5s10 ~]#
[root@a5s10 ~]#
[root@a5s10 ~]# docker exec -it d67332db3872 bash
root@a5s10(analyticsdb):/#
root@a5s10(analyticsdb):/# exit
exit
[root@a5s10 ~]# docker exec -it fe9cc2180f85 bash
root@a5s10(controller):/#
root@a5s10(controller):/# contrail-status
== Contrail Control ==
supervisor-control: active (disabled on boot)
contrail-control active
contrail-control-nodemgr initializing (NTP state unsynchronized.)
contrail-dns active
contrail-named active

== Contrail Config ==
supervisor-config: active (disabled on boot)
contrail-api:0 failed
contrail-config-nodemgr initializing (NTP state unsynchronized.)
contrail-device-manager active
contrail-schema active
contrail-svc-monitor active

== Contrail Web UI ==
supervisor-webui: active (disabled on boot)
contrail-webui active
contrail-webui-middleware active

== Contrail Database ==
contrail-database: active (disabled on boot)

supervisor-database: inactive (disabled on boot)
unix:///var/run/supervisord_database.sockno

root@a5s10(controller):/# contrail-version
Package Version Build-ID | Repo | RPM Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-config 4.0.0.0-3073.el7 @contrail-install-repo
contrail-config-openstack 4.0.0.0-3073.el7 @contrail-install-repo
contrail-control 4.0.0.0-3073.el7 @contrail-install-repo
contrail-database 4.0.0.0-3073.el7 @contrail-install-repo
contrail-database-common 4.0.0.0-3073.el7 @contrail-install-repo
contrail-dns 4.0.0.0-3073.el7 @contrail-install-repo
contrail-lib 4.0.0.0-3073.el7 @contrail-install-repo

root@a5s10(controller):/# systemctl status contrail-api
● contrail-api.service - "Contrail API"
   Loaded: loaded (/etc/systemd/system/contrail-api.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2017-05-08 21:38:19 UTC; 52min ago
  Process: 29455 ExecStop=/bin/kill -s TERM $MAINPID (code=exited, status=1/FAILURE)
 Main PID: 29464 (contrail-api)
   CGroup: /docker/fe9cc2180f853c1ba002f1713be09b29fca26cce2fe6420319a17451e277b226/system.slice/contrail-api.service
           └─29464 /usr/bin/python /usr/bin/contrail-api --conf_file /etc/contrail/contrail-api.conf --conf_file /etc/contrail/contrail-keystone-auth.c...
           ‣ 29464 /usr/bin/python /usr/bin/contrail-api --conf_file /etc/contrail/contrail-api.conf --conf_file /etc/contrail/contrail-keystone-auth.c...

May 08 22:29:41 a5s10 contrail-api[29464]: WARNING:keystonemiddleware.auth_token:Use of the auth_admin_prefix, auth_host, auth_port, auth_protocol, ide...
May 08 22:29:41 a5s10 contrail-api[29464]: 10.84.14.10 - - [2017-05-08 22:29:41] "GET /obj-perms HTTP/1.1" 200 8702 0.356463
May 08 22:30:06 a5s10 contrail-api[29464]: 10.84.14.10 - - [2017-05-08 22:30:06] "GET /global-system-configs?fields=user_defined_log_statistic... 0.113401
May 08 22:30:07 a5s10 contrail-api[29464]: 10.84.14.10 - - [2017-05-08 22:30:07] "GET / HTTP/1.1" 200 20357 0.001192
May 08 22:30:07 a5s10 contrail-api[29464]: 10.84.14.10 - - [2017-05-08 22:30:07] "GET /physical-routers?count=False&shared=False&detail=False ... 0.000647
May 08 22:30:07 a5s10 contrail-api[29464]: 10.84.14.10 - - [2017-05-08 22:30:07] "GET /physical-routers?count=False&shared=False&detail=False ... 0.099768
May 08 22:30:14 a5s10 contrail-api[29464]: WARNING:keystonemiddleware.auth_token:Use of the auth_admin_prefix, auth_host, auth_port, auth_protocol, ide...
May 08 22:30:14 a5s10 contrail-api[29464]: 10.84.14.10 - - [2017-05-08 22:30:14] "GET /obj-perms HTTP/1.1" 200 8702 0.342574
May 08 22:30:15 a5s10 contrail-api[29464]: WARNING:keystonemiddleware.auth_token:Use of the auth_admin_prefix, auth_host, auth_port, auth_protocol, ide...
May 08 22:30:15 a5s10 contrail-api[29464]: 10.84.14.10 - - [2017-05-08 22:30:15] "GET /obj-perms HTTP/1.1" 200 8702 0.344489
Hint: Some lines were ellipsized, use -l to show in full.
root@a5s10(controller):/# systemctl status cassandra
● cassandra.service - SYSV: Starts and stops Cassandra
   Loaded: loaded (/etc/rc.d/init.d/cassandra; bad; vendor preset: disabled)
   Active: active (exited) since Mon 2017-05-08 20:47:38 UTC; 1h 43min ago
     Docs: man:systemd-sysv-generator(8)

May 08 20:47:37 a5s10 systemd[1]: Starting SYSV: Starts and stops Cassandra...
May 08 20:47:37 a5s10 su[2563]: (to cassandra) root on none
May 08 20:47:38 a5s10 cassandra[2556]: Starting Cassandra: OK
May 08 20:47:38 a5s10 systemd[1]: Started SYSV: Starts and stops Cassandra.

shajuvk (shajuvk)
information type: Proprietary → Public
description: updated
Jeba Paulaiyan (jebap)
tags: added: blocker
Revision history for this message
Sachin Bansal (sbansal) wrote :

I looked at the setup. It seems the api server and cassandra are both running, but systemd still tried to launch them again and then it failed with addres 8084 already bound.

Revision history for this message
Ignatious Johnson Christopher (ijohnson-x) wrote :

I see that the contrail-api is running independently triggered by the /etc/systemd/system/contrail-api.service(systemd)

root@a5s10(controller):/var/log/contrail# ps -ef | grep 29464
root 6946 29614 0 13:17 ? 00:00:00 grep --color=auto 29464
contrail 29464 1 0 May08 ? 00:01:18 /usr/bin/python /usr/bin/contrail-api --conf_file /etc/contrail/contrail-api.conf --conf_file /etc/contrail/contrail-keystone-auth.conf --conf_file /etc/contrail/contrail-database.conf
root@a5s10(controller):/var/log/contrail#

In Redhat systems, all the contrail services are under supervisor,
root@a5s10(controller):/var/log/contrail# supervisorctl -s unix:///var/run/supervisord_config.sock status
contrail-api:0 EXITED May 08 09:12 PM
contrail-config-nodemgr RUNNING pid 4521, uptime 16:29:44
contrail-device-manager RUNNING pid 4523, uptime 16:29:44
contrail-schema RUNNING pid 4524, uptime 16:29:44
contrail-svc-monitor RUNNING pid 6263, uptime 16:29:15
root@a5s10(controller):/var/log/contrail#

I see that the contrail-api.service(systemd) file is brought in to the system by contrail-ansible-internal
https://github.com/Juniper/contrail-ansible-internal/blob/master/playbooks/roles/contrail/config/files/systemd/contrail-api.service
https://github.com/Juniper/contrail-ansible-internal/blob/master/playbooks/roles/contrail/config/tasks/setup.yml#L19

I would let Narendra to debug further.

Jeba Paulaiyan (jebap)
tags: removed: blocker
Revision history for this message
Nagendra Prasath (npchandran) wrote :
Revision history for this message
Nagendra Prasath (npchandran) wrote :

With https://github.com/Juniper/contrail-ansible-internal/commit/8ca70ec44742f29b66bba0339602d76c012485d0
contrail-api.service is being added through Ansible instead of packaging. Author has provided the details why he preferred that approach.

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.0

Review in progress for https://review.opencontrail.org/31466
Submitter: Nagendra Prasath (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/31467
Submitter: Nagendra Prasath (<email address hidden>)

Jeba Paulaiyan (jebap)
tags: added: blocker
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/31466
Committed: http://github.com/Juniper/contrail-ansible-internal/commit/4ffa59283cdf95162b3c99e0b21aee731e62190c
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit 4ffa59283cdf95162b3c99e0b21aee731e62190c
Author: Nagendra Maynattamai <email address hidden>
Date: Thu May 11 23:58:14 2017 -0700

contrail-api.service is brought in to all systemd platforms including
redhat unnecessarily during the commit 8ca70ec44742f29b66bba0339602d76c012485d0
Contrail redhat platform do not use systemd services yet so this case
is not applicable.

Change-Id: Ic1e6940904595511ea92b2fff3715daf8096b1b6
Closes-Bug: 1689400

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/31467
Committed: http://github.com/Juniper/contrail-ansible-internal/commit/47435aea2c8908779b6eccb353d6798f813ade39
Submitter: Zuul (<email address hidden>)
Branch: master

commit 47435aea2c8908779b6eccb353d6798f813ade39
Author: Nagendra Maynattamai <email address hidden>
Date: Thu May 11 23:58:14 2017 -0700

contrail-api.service is brought in to all systemd platforms including
redhat unnecessarily during the commit 8ca70ec44742f29b66bba0339602d76c012485d0
Contrail redhat platform do not use systemd services yet so this case
is not applicable.

Change-Id: Ic1e6940904595511ea92b2fff3715daf8096b1b6
Closes-Bug: 1689400

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.