Tempest errors running mistral-api as wsgi in apache

Bug #1671505 reported by Alfredo Moralejo
This bug affects 1 person
Affects Status Importance Assigned to Milestone

Bug Description

Running tempest tests on a mistral deployment from master (pike cycle) with mistral api in WSGI we get intermitent failures in different jobs with similar trace:

{1} mistral_tempest_tests.tests.api.v2.test_actions.ActionTestsV2.test_get_list_actions_equal_to_filter [500.009070s] ... FAILED

Captured traceback:
     Traceback (most recent call last):
       File "/usr/lib/python2.7/site-packages/mistral_tempest_tests/tests/api/v2/test_actions.py", line 134, in test_get_list_actions_equal_to_filter
         resp, body = self.client.create_action('action_v2.yaml')
       File "/usr/lib/python2.7/site-packages/mistral_tempest_tests/services/v2/mistral_client.py", line 123, in create_action
         resp, body = self.post_request('actions', yaml_file)
       File "/usr/lib/python2.7/site-packages/mistral_tempest_tests/services/v2/mistral_client.py", line 30, in post_request
         return self.post(url, base.get_resource(file_name), headers=headers)
       File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 276, in post
         return self.request('POST', url, extra_headers, headers, body, chunked)
       File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 650, in request
         body=body, chunked=chunked)
       File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 546, in _request
       File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 579, in raw_request
         body=body, chunked=chunked)
       File "/usr/lib/python2.7/site-packages/tempest/lib/common/http.py", line 60, in request
         *args, **new_kwargs)
       File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 73, in request
       File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 151, in request_encode_body
         return self.urlopen(method, url, **extra_kw)
       File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 248, in urlopen
         response = conn.urlopen(method, u.request_uri, **kw)
       File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 595, in urlopen
       File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 386, in _make_request
         httplib_response = conn.getresponse(buffering=True)
       File "/usr/lib64/python2.7/httplib.py", line 1089, in getresponse
       File "/usr/lib64/python2.7/httplib.py", line 444, in begin
         version, status, reason = self._read_status()
       File "/usr/lib64/python2.7/httplib.py", line 400, in _read_status
         line = self.fp.readline(_MAXLINE + 1)
       File "/usr/lib64/python2.7/socket.py", line 476, in readline
         data = self._sock.recv(self._rbufsize)
       File "/usr/lib64/python2.7/ssl.py", line 759, in recv
         return self.read(buflen)
       File "/usr/lib64/python2.7/ssl.py", line 653, in read
         v = self._sslobj.read(len or 1024)
       File "/usr/lib/python2.7/site-packages/fixtures/_fixtures/timeout.py", line 52, in signal_handler
         raise TimeoutException()

In all failures we find following error message in mistral apache wsgi log:

[Wed Mar 08 14:07:41.424793 2017] [:error] [pid 29897] Traceback (most recent call last):
[Wed Mar 08 14:07:41.424844 2017] [:error] [pid 29897] File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 457, in fire_timers
[Wed Mar 08 14:07:41.425250 2017] [:error] [pid 29897] timer()
[Wed Mar 08 14:07:41.425277 2017] [:error] [pid 29897] File "/usr/lib/python2.7/site-packages/eventlet/hubs/timer.py", line 58, in __call__
[Wed Mar 08 14:07:41.425381 2017] [:error] [pid 29897] cb(*args, **kw)
[Wed Mar 08 14:07:41.425401 2017] [:error] [pid 29897] File "/usr/lib/python2.7/site-packages/eventlet/semaphore.py", line 145, in _do_acquire
[Wed Mar 08 14:07:41.425584 2017] [:error] [pid 29897] waiter.switch()
[Wed Mar 08 14:07:41.425612 2017] [:error] [pid 29897] error: cannot switch to a different thread

Which i think it's related to the issue.

We are hitting this in RDO-CI jobs. You can find full logs for a failed job in:


Revision history for this message
Alfredo Moralejo (amoralej) wrote :

We haven't hit this error after 17-march so i think it has been fixed by some of the commits merged between 14-march (last hit) and 17-march (firs success).

Revision history for this message
Sharat Sharma (sharat-sharma) wrote :

This bug is no more valid as per the reporter.

Changed in mistral:
status: New → Invalid
Revision history for this message
Alfredo Moralejo (amoralej) wrote :

I'm reopening this. We haven't seen this in master branch as stated in previous message but i just found we are still hitting it in jobs running with ocata stable.

Changed in mistral:
status: Invalid → New
Revision history for this message
Brad P. Crochet (brad-9) wrote :

Is the rdo ci using a TZ other than UTC?

Revision history for this message
Brad P. Crochet (brad-9) wrote :

Never mind. I was thinking of a different error.

Brad P. Crochet (brad-9)
Changed in mistral:
assignee: nobody → Brad P. Crochet (brad-9)
Dougal Matthews (d0ugal)
Changed in mistral:
status: New → Confirmed
importance: Undecided → Medium
milestone: none → rocky-2
Dougal Matthews (d0ugal)
Changed in mistral:
milestone: rocky-2 → rocky-3
Dougal Matthews (d0ugal)
Changed in mistral:
milestone: rocky-3 → stein-1
Dougal Matthews (d0ugal)
Changed in mistral:
milestone: stein-1 → none
assignee: Brad P. Crochet (brad-9) → nobody
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.