nova-compute is down after failure during init_host()

I tentatively marked this as Medium. If there is a new reproduce, please feel free to raise to High.

No longer fixing Medium bugs in 8.0

New duplicate found - raising to High - https://bugs.launchpad.net/fuel/+bug/1530858

This looks interesting. According to stack traces of green/native threads attached above nova-compute is in a weird state: eventlet main loop is started and fully functional, nova-compute is connected to rabbitmq/libvirt (libvirt keep alives go into both direction periodically), while at the same time looks like nova-compute RPC server has not started properly, so it does not really wait for any new messages, nor executes periodic tasks of its own.

Based on the implementation (https://github.com/openstack/oslo.service/blob/master/oslo_service/service.py) we start a service in one green thread and then wait for the service correct termination (self.done conditional variable) from another green thread. So if the service thread fails - we'll never know (this is what we see in #7).

Do you mean case when we are starting many green threads inside service.start() procedure and then we are waiting on self.done. And during that some of green threads are crashed?

Marian, service.start() itself is run in a green thread and if it fails with exception, the service will still be alive from the ServiceLauncher point of view, so the launcher.wait() call will block, but nova-compute will not be able to do anything useful.

IMO, we need to fail (exit with some error code) early, if service.start() fails. The problem is that it's OK now for the start() call to end successfully in a thread - its task is to set up an RPC server and do other service initialisation steps - so this thread does not represent a running service and ends early on success.

Currently, I don't see a clean way to fail early with existing oslo.service API. What I was able to achieve is to call service.stop() immediately after service.start() call fails, which allows us to exit in service.wait(), although we still exit with a success error code (i.e. 0), not a failure one.

Fix proposed to branch: openstack-ci/fuel-8.0/liberty
Change author: Marian Horban <email address hidden>
Review: https://review.fuel-infra.org/16309

Fixed in Mitaka - https://review.openstack.org/#/c/265163/

So I've uploaded a change on review, which should fix this, but IMO it's too risky to introduce such a change to oslo.service such late in the release cycle: basically we change how every single service starts and the error code it exits with. We need to test this thoroughly.

This is how OpenStack services worked for ages. The last thing I want to do is to do a quick fix and break even more things. The workaround is easy - just restart the affected service. The situation itself must be rare and reproduced only in CI / slow environments.

Tentatively moving this to 8.0-updates.

Change abandoned by Roman Podoliaka <email address hidden> on branch: openstack-ci/fuel-8.0/liberty
Review: https://review.fuel-infra.org/16309

Verified on (3 controllers+2 computes, vlan, cinder lvm):

[root@nailgun ~]# shotgun2 short-report
cat /etc/fuel_build_id:
 154
cat /etc/fuel_build_number:
 154
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-misc-9.0.0-1.mos8259.noarch
 python-fuelclient-9.0.0-1.mos301.noarch
 fuel-9.0.0-1.mos6325.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8611.noarch
 fuelmenu-9.0.0-1.mos263.noarch
 fuel-ui-9.0.0-1.mos2641.noarch
 fuel-migrate-9.0.0-1.mos8259.noarch
 fuel-ostf-9.0.0-1.mos921.noarch
 shotgun-9.0.0-1.mos87.noarch
 python-packetary-9.0.0-1.mos130.noarch
 fuel-bootstrap-cli-9.0.0-1.mos272.noarch
 fuel-nailgun-9.0.0-1.mos8611.noarch
 fuel-setup-9.0.0-1.mos6325.noarch
 rubygem-astute-9.0.0-1.mos733.noarch
 fuel-mirror-9.0.0-1.mos130.noarch
 fuel-openstack-metadata-9.0.0-1.mos8611.noarch
 fuel-notify-9.0.0-1.mos8259.noarch
 fuel-release-9.0.0-1.mos6325.noarch
 nailgun-mcagents-9.0.0-1.mos733.noarch
 network-checker-9.0.0-1.mos72.x86_64
 fuel-utils-9.0.0-1.mos8259.noarch
 fuel-library9.0-9.0.0-1.mos8259.noarch
 fuel-agent-9.0.0-1.mos272.noarch
[root@nailgun ~]#

The verification was executed in accordance with Steps to reproduce and with adding raise to nova-compute's init_host() to emulate its crash

This is effectively fixed in https://review.fuel-infra.org/#/c/20853/ , otherwise it would require backporting a backwards incompatible change to oslo.service (https://review.openstack.org/#/c/265163/)

I suggest we proceed with the upstart dependency fix in 8.0-updates ^

Fixed in 8.0-updates by https://review.fuel-infra.org/#/c/20853/ and https://review.fuel-infra.org/#/c/21576/

Verified on MOS 8.0 + MU2 updates.