deadlocks in nailgun tests

Bug #1319067 reported by Aleksey Kasatkin
This bug report is a duplicate of:  Bug #1308115: Deadlock in unit tests in Nailgun. Edit Remove
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Confirmed
High
Fuel Python (Deprecated)

Bug Description

These tests are failed time to time:

nailgun.test.integration.test_task_managers.TestTaskManagers.test_deletion_during_deployment
nailgun.test.integration.test_charset_issues.TestCharsetIssues.test_deletion_during_deployment

Fail reason is deadlock as it's shown in logs:

test_deletion_during_deployment (nailgun.test.integration.test_task_managers.TestTaskManagers) ... Exception in thread REMOVE_NODES:
Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/home/jenkins/workspace/stackforge-verify-fuel-web/nailgun/nailgun/task/fake.py", line 386, in run
    handling_db_errors(resp_method, **kwargs)
  File "/home/jenkins/workspace/stackforge-verify-fuel-web/nailgun/nailgun/task/fake.py", line 60, in handling_db_errors
    handling_db_errors(func, retries=retries, **kwargs)
  File "/home/jenkins/workspace/stackforge-verify-fuel-web/nailgun/nailgun/task/fake.py", line 46, in handling_db_errors
    func(**kwargs)
  File "/home/jenkins/workspace/stackforge-verify-fuel-web/nailgun/nailgun/rpc/receiver.py", line 115, in remove_cluster_resp
    cls.remove_nodes_resp(**kwargs)
  File "/home/jenkins/workspace/stackforge-verify-fuel-web/nailgun/nailgun/rpc/receiver.py", line 55, in remove_nodes_resp
    node_db = db().query(Node).get(node['uid'])
  File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/query.py", line 775, in get
    return self._load_on_ident(key)
  File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/query.py", line 2512, in _load_on_ident
    return q.one()
  File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/query.py", line 2184, in one
    ret = list(self)
  File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/query.py", line 2227, in __iter__
    return self._execute_and_instances(context)
  File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/query.py", line 2240, in _execute_and_instances
    close_with_result=True)
  File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/query.py", line 2231, in _connection_from_session
    **kw)
  File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/session.py", line 777, in connection
    close_with_result=close_with_result)
  File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/session.py", line 781, in _connection_for_bind
    return self.transaction._connection_for_bind(engine)
  File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/session.py", line 289, in _connection_for_bind
    self._assert_is_active()
  File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/session.py", line 213, in _assert_is_active
    % self._rollback_exception
InvalidRequestError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (TransactionRollbackError) deadlock detected
DETAIL: Process 23546 waits for ShareLock on transaction 24210489; blocked by process 23157.
Process 23157 waits for ShareLock on transaction 24210487; blocked by process 23546.
HINT: See server log for query details.
 'DELETE FROM clusters WHERE clusters.id = %(id)s' {'id': 274}
ERROR

test_deletion_during_deployment (nailgun.test.integration.test_charset_issues.TestCharsetIssues) ... Exception in thread REMOVE_NODES:
Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/home/jenkins/workspace/stackforge-verify-fuel-web/nailgun/nailgun/task/fake.py", line 386, in run
    handling_db_errors(resp_method, **kwargs)
  File "/home/jenkins/workspace/stackforge-verify-fuel-web/nailgun/nailgun/task/fake.py", line 60, in handling_db_errors
    handling_db_errors(func, retries=retries, **kwargs)
  File "/home/jenkins/workspace/stackforge-verify-fuel-web/nailgun/nailgun/task/fake.py", line 46, in handling_db_errors
    func(**kwargs)
  File "/home/jenkins/workspace/stackforge-verify-fuel-web/nailgun/nailgun/rpc/receiver.py", line 115, in remove_cluster_resp
    cls.remove_nodes_resp(**kwargs)
  File "/home/jenkins/workspace/stackforge-verify-fuel-web/nailgun/nailgun/rpc/receiver.py", line 55, in remove_nodes_resp
    node_db = db().query(Node).get(node['uid'])
  File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/query.py", line 775, in get
    return self._load_on_ident(key)
  File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/query.py", line 2512, in _load_on_ident
    return q.one()
  File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/query.py", line 2184, in one
    ret = list(self)
  File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/query.py", line 2227, in __iter__
    return self._execute_and_instances(context)
  File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/query.py", line 2240, in _execute_and_instances
    close_with_result=True)
  File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/query.py", line 2231, in _connection_from_session
    **kw)
  File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/session.py", line 777, in connection
    close_with_result=close_with_result)
  File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/session.py", line 781, in _connection_for_bind
    return self.transaction._connection_for_bind(engine)
  File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/session.py", line 289, in _connection_for_bind
    self._assert_is_active()
  File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/session.py", line 213, in _assert_is_active
    % self._rollback_exception
InvalidRequestError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (TransactionRollbackError) deadlock detected
DETAIL: Process 10497 waits for ShareLock on transaction 24280499; blocked by process 10482.
Process 10482 waits for ShareLock on transaction 24280497; blocked by process 10497.
HINT: See server log for query details.
 'DELETE FROM clusters WHERE clusters.id = %(id)s' {'id': 19}
ERROR

It's very annoying thing as tests fail in about 1/4 of CI test runs.

Revision history for this message
Aleksey Kasatkin (alekseyk-ru) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.