Juniper Openstack

uves reports process status as running though it isnt

Bug #1694849 reported by Senthilnathan Murugappan
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R4.0
Fix Committed
High
Megh Bhatt
Juniper Openstack r4.0.1.0 "r4.0.1.0"
Trunk
Fix Committed
High
Megh Bhatt
Juniper Openstack r4.1.0.0-fcs "r4.1"

Bug Description

version: 4.0.0.0-16
OS: ubuntu-16.04

root@server4(controller):/# curl -s -uadmin:c0ntrail123 localhost:8181/analytics/uves/config-node/server4?flat | python -mjson.tool | grep -5 device-manager
                "core_file_list": [],
                "exit_count": 0,
                "last_exit_time": null,
                "last_start_time": "1496215012084262",
                "last_stop_time": null,
                "process_name": "contrail-device-manager",
                "process_state": "PROCESS_STATE_RUNNING",
                "start_count": 1,
                "stop_count": 0
            },

root@server4(controller):/# contrail-status | grep device
contrail-device-manager: inactive

root@server4(controller):/# ps ax | grep device
30380 ? S+ 0:00 grep --color=auto device
root@server4(controller):/#

Senthilnathan Murugappan (msenthil)
tags: added: sanity
Jeba Paulaiyan (jebap)
tags: added: blocker
Revision history for this message
Megh Bhatt (meghb) wrote :

It appears that systemd running inside the container is not sending unit changed messages on the system DBus and hence nodemgr is not able to update the process status.

Revision history for this message
Megh Bhatt (meghb) wrote :

Removing blocker since it affects only 16.04 docker images and only the ProcessStatus alarm.

tags: removed: blocker
tags: added: releasenote
Revision history for this message
Megh Bhatt (meghb) wrote :

Release Note:
On ubuntu 16.04 based docker containers sometimes the ProcessStatus alarm does not get activated even if a process is not running.

Workaround:
Restart the node manager process based on the process that is not running and respective container. For example, for the controller docker, either the contrail-control-nodemgr or contrail-config-nodemgr process needs to be restarted. On the analytics docker, contrail-analytics-nodemgr process needs to be restarted.

Soumil Kulkarni (soumilk)
information type: Proprietary → Public
Revision history for this message
Megh Bhatt (meghb) wrote :

Based on https://lwn.net/Articles/676831/ we should consider moving to using supervisord even on 16.04 containers

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/33614
Submitter: Megh Bhatt (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/33614
Committed: http://github.com/Juniper/contrail-controller/commit/6bdb195abf9eaa641085db56d6014b9df2f93f04
Submitter: Zuul (<email address hidden>)
Branch: master

commit 6bdb195abf9eaa641085db56d6014b9df2f93f04
Author: Megh Bhatt <email address hidden>
Date: Thu Jul 13 15:04:50 2017 -0700

Fix nodemgr reporting process state info on 16.04 container

On 16.04 docker we are using systemd inside the docker
as an init system to launch the required processes. However
sd_notify call does not work in the docker and hence
nodemgr does not receive unit change notifications from
systemd system dbus. Fix is to instead do a poll of the
unit status when running inside the docker.

Change-Id: Ie5de608dff74de5287bfa6b9f6e4661f9c30250c
Closes-Bug: #1694617
Closes-Bug: #1694849

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.0

Review in progress for https://review.opencontrail.org/33906
Submitter: Megh Bhatt (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/33906
Committed: http://github.com/Juniper/contrail-controller/commit/9a0d9b207a423d2bb197e120636a1aecde044414
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit 9a0d9b207a423d2bb197e120636a1aecde044414
Author: Megh Bhatt <email address hidden>
Date: Thu Jul 13 15:04:50 2017 -0700

Fix nodemgr reporting process state info on 16.04 container

On 16.04 docker we are using systemd inside the docker
as an init system to launch the required processes. However
sd_notify call does not work in the docker and hence
nodemgr does not receive unit change notifications from
systemd system dbus. Fix is to instead do a poll of the
unit status when running inside the docker.

Change-Id: Ie5de608dff74de5287bfa6b9f6e4661f9c30250c
Closes-Bug: #1694617
Closes-Bug: #1694849
(cherry picked from commit 6bdb195abf9eaa641085db56d6014b9df2f93f04)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.