Mirantis OpenStack

Backport for neutron/+bug/1590298

Bug #1830087 reported by Maxim V. Yefimov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Fix Released
Medium
Valeriy Saharov
Mirantis OpenStack 9.2-mu-13

Bug Description

DB retry wrapper needs to look for savepoint errors.

Error on the neutron port-create execution:

019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource [req-e18cfc3b-f89b-4173-8884-2ec65629955b 9dc779e894cb4b1e90d432920e2342d3 9f65c8a3de5f4f5cbf5a30dd5b95c17d - - -] create failed
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource Traceback (most recent call last):
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/resource.py", line 84, in resource
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource result = method(request=request, **args)
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 410, in create
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource return self._create(request, body, **kwargs)
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 148, in wrapper
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource ectxt.value = e.inner_exc
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource self.force_reraise()
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource return f(*args, **kwargs)
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 525, in _create
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource obj)})
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 468, in notify
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource resource_registry.set_resources_dirty(request.context)
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/quota/resource_registry.py", line 70, in set_resources_dirty
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource res.mark_dirty(context)
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/quota/resource.py", line 196, in mark_dirty
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource {'tenant_id': tenant_id, 'resource': self.name})
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource self.gen.throw(type, value, traceback)
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 106, in autonested_transaction
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource yield tx
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 492, in __exit__
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource self.rollback()
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 419, in rollback
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource transaction._rollback_impl()
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 449, in _rollback_impl
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource t[1].rollback()
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1563, in rollback
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource self._do_rollback()
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1625, in _do_rollback
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource self._savepoint, self._parent)
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 717, in _rollback_to_savepoint_impl
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource self.engine.dialect.do_rollback_to_savepoint(self, name)
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 441, in do_rollback_to_savepoint
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource connection.execute(expression.RollbackToSavepointClause(name))
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource return meth(self, multiparams, params)
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource return connection._execute_clauseelement(self, multiparams, params)
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource compiled_sql, distilled_params
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource context)
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource util.raise_from_cause(newraise, exc_info)
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource reraise(type(exception), exception, tb=exc_tb, cause=cause)
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource context)
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource cursor.execute(statement, parameters)
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 219, in execute
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource self.errorhandler(self, exc, value)
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 38, in defaulterrorhandler
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource raise errorvalue
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource OperationalError: (_mysql_exceptions.OperationalError) (1305, 'SAVEPOINT sa_savepoint_1 does not exist') [SQL: u'ROLLBACK TO SAVEPOINT sa_savepoint_1']
2019-02-14 13:20:12.499 16590 ERROR neutron.api.v2.resource

MOS 9.0 Mitaka

Valeriy Saharov (vsakharov)
Changed in mos:
status: New → In Progress
importance: Undecided → Medium
assignee: nobody → Valeriy Saharov (vsakharov)
Denis Meltsaykin (dmeltsaykin)
Changed in mos:
milestone: 9.x-updates → 9.2-mu-13
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/neutron (9.0/mitaka)

Fix proposed to branch: 9.0/mitaka
Change author: Valeriy Saharov <email address hidden>
Review: https://review.fuel-infra.org/41328

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/neutron (9.0/mitaka)

Reviewed: https://review.fuel-infra.org/41328
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: 7fce38e31faf88d5d9a1d9e54d12d527f5720b3c
Author: Valeriy Saharov <email address hidden>
Date: Thu May 23 14:13:30 2019

Check for mysql SAVEPOINT error in retry decorator

This is backport fix from https://review.opendev.org/#/c/326927

Due to lost savepoints in mysql on deadlock errors, we can get
a non-existent savepoint error that is just masking a deadlock
error. This patch adjusts the retry decorator to check for these
savepoint errors as well. See the bug for more details about
the failure.

Change-Id: I1cb8425d0de4bf0b723a70e0ff19b99d43e68cf8
Closes-Bug: #1830087

Changed in mos:
status: In Progress → Fix Committed
Revision history for this message
Pavel Glazov (pglazovv) wrote :

Fix included in http://eu.mirror.fuel-infra.org/mos-repos/ubuntu/snapshots/9.0-2019-07-26-180320

Changed in mos:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.