Murano

Error during murano-api restart: Calling waitall() from within one of the GreenPool's greenthreads will never terminate

Bug #1536666 reported by Anastasia Kuznetsova
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Murano
Fix Released
Medium
Stan Lagun
Murano mitaka-3 "m3"

Bug Description

Murano-liberty code:
after restarting of murano-api I've got following error:
2016-01-20 13:28:46.185 5650 DEBUG oslo_concurrency.lockutils [-] Acquired semaphore "singleton_lock" lock /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:198
2016-01-20 13:28:46.185 5650 DEBUG oslo_concurrency.lockutils [-] Releasing semaphore "singleton_lock" lock /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:211
2016-01-20 13:28:46.185 5650 DEBUG oslo_concurrency.lockutils [-] Acquired semaphore "singleton_lock" lock /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:198
2016-01-20 13:28:46.186 5650 DEBUG oslo_concurrency.lockutils [-] Releasing semaphore "singleton_lock" lock /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:211
2016-01-20 13:28:46.186 5650 INFO oslo_service.service [-] Caught SIGTERM, exiting
2016-01-20 13:28:46.186 5650 ERROR oslo_service.threadgroup [-] Error stopping thread.
2016-01-20 13:28:46.186 5650 ERROR oslo_service.threadgroup Traceback (most recent call last):
2016-01-20 13:28:46.186 5650 ERROR oslo_service.threadgroup File "/usr/lib/python2.7/dist-packages/oslo_service/threadgroup.py", line 112, in _stop_threads
2016-01-20 13:28:46.186 5650 ERROR oslo_service.threadgroup x.stop()
2016-01-20 13:28:46.186 5650 ERROR oslo_service.threadgroup File "/usr/lib/python2.7/dist-packages/oslo_service/threadgroup.py", line 53, in stop
2016-01-20 13:28:46.186 5650 ERROR oslo_service.threadgroup self.thread.kill()
2016-01-20 13:28:46.186 5650 ERROR oslo_service.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 240, in kill
2016-01-20 13:28:46.186 5650 ERROR oslo_service.threadgroup return kill(self, *throw_args)
2016-01-20 13:28:46.186 5650 ERROR oslo_service.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 294, in kill
2016-01-20 13:28:46.186 5650 ERROR oslo_service.threadgroup g.throw(*throw_args)
2016-01-20 13:28:46.186 5650 ERROR oslo_service.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
2016-01-20 13:28:46.186 5650 ERROR oslo_service.threadgroup result = function(*args, **kwargs)
2016-01-20 13:28:46.186 5650 ERROR oslo_service.threadgroup File "/usr/lib/python2.7/dist-packages/murano/common/wsgi.py", line 166, in _run
2016-01-20 13:28:46.186 5650 ERROR oslo_service.threadgroup log=loggers.WritableLogger(logger))
2016-01-20 13:28:46.186 5650 ERROR oslo_service.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 842, in server
2016-01-20 13:28:46.186 5650 ERROR oslo_service.threadgroup pool.waitall()
2016-01-20 13:28:46.186 5650 ERROR oslo_service.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/greenpool.py", line 117, in waitall
2016-01-20 13:28:46.186 5650 ERROR oslo_service.threadgroup "Calling waitall() from within one of the " \
2016-01-20 13:28:46.186 5650 ERROR oslo_service.threadgroup AssertionError: Calling waitall() from within one of the GreenPool's greenthreads will never terminate.
2016-01-20 13:28:46.186 5650 ERROR oslo_service.threadgroup
2016-01-20 13:28:46.227 5650 WARNING oslo_messaging.server [-] start/stop/wait must be called in the same thread
2016-01-20 13:28:46.227 5650 WARNING oslo_messaging.server [-] start/stop/wait must be called in the same thread
2016-01-20 13:28:46.228 5650 DEBUG oslo_concurrency.lockutils [-] Acquired semaphore "singleton_lock" lock /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:198
2016-01-20 13:28:46.228 5650 DEBUG oslo_concurrency.lockutils [-] Releasing semaphore "singleton_lock" lock /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:211

Ekaterina Chernova (efedorova)
Changed in murano:
importance: Undecided → High
milestone: none → mitaka-3
Anastasia Kuznetsova (akuznetsova)
Changed in murano:
assignee: nobody → Stan Lagun (slagun)
Ekaterina Chernova (efedorova)
Changed in murano:
importance: High → Medium
status: New → Confirmed
OpenStack Infra (hudson-openstack)
Changed in murano:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to murano (master)

Reviewed: https://review.openstack.org/270529
Committed: https://git.openstack.org/cgit/openstack/murano/commit/?id=862a301888fc51a785e98999111c7e0e61d5dd44
Submitter: Jenkins
Branch: master

commit 862a301888fc51a785e98999111c7e0e61d5dd44
Author: Stan Lagun <email address hidden>
Date: Thu Jan 21 02:27:18 2016 +0300

    API hanged on a termination signal

    When murano-api is running with several api worker threads sending it a
    termination signal (SIGTERM, SIGHUP or SIGINT) leads to waitall()
    being called inside a GreenPool's greenthread. The reason is that a
    wsgi server is started in a green thread from the same green pool that
    is passed to the server itself to be used for spawning client green
    threads. To avoid it, it is reasonable to use different pools for
    spawning a wsgi server and for its internal usage.

    Closes-Bug: #1536666
    Change-Id: Ib8c9609035a5585b0392811ebd9de669dbc0c986

Changed in murano:
status: In Progress → Fix Released
Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/murano 2.0.0.0b3

This issue was fixed in the openstack/murano 2.0.0.0b3 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.