Murano

Race condition while uploading packages with same tags in parallel

Bug #1498097 reported by Anastasia Kuznetsova on 2015-09-21

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Murano	Fix Released	Medium	Nikolay Starodubtsev	Murano mitaka-3 "m3"

Bug Description

During writing scenarios for testing performance of package creation / deletion / modification I periodically faced to a traceback:

2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog [req-1a4e849b-ea61-4059-8774-90016fd5d71e ecda7507f1f94ed88457052695babca3 7b146cf7a1af4ec88c42ee9e9a3d720c - - -] Package with specified full name is already registered
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog Traceback (most recent call last):
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/opt/stack/new/murano/murano/api/v1/catalog.py", line 243, in upload
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog package_meta, req.context.tenant)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/opt/stack/new/murano/murano/db/catalog/api.py", line 378, in package_upload
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog package.save(session)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/models.py", line 48, in save
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog session.flush()
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2004, in flush
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog self._flush(objects)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2122, in _flush
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog transaction.rollback(_capture_exception=True)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog compat.reraise(exc_type, exc_value, exc_tb)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2086, in _flush
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog flush_context.execute()
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 373, in execute
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog rec.execute(self)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 532, in execute
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog uow
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 174, in save_obj
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog mapper, table, insert)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 781, in _emit_insert_statements
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog execute(statement, params)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog return meth(self, multiparams, params)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog return connection._execute_clauseelement(self, multiparams, params)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog compiled_sql, distilled_params
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog context)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog util.raise_from_cause(newraise, exc_info)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog reraise(type(exception), exception, tb=exc_tb)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog context)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog cursor.execute(statement, parameters)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/pymysql/cursors.py", line 134, in execute
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog result = self._query(query)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/pymysql/cursors.py", line 282, in _query
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog conn.query(q)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 768, in query
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 929, in _read_query_result
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog result.read()
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 1125, in read
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog first_packet = self.connection._read_packet()
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 893, in _read_packet
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog packet.check_error()
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 369, in check_error
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog err.raise_mysql_exception(self._data)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/pymysql/err.py", line 120, in raise_mysql_exception
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog _check_mysql_exception(errinfo)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog File "/usr/local/lib/python2.7/dist-packages/pymysql/err.py", line 112, in _check_mysql_exception
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog raise errorclass(errno, errorvalue)
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, u"Duplicate entry 'App' for key 'name'") [SQL: u'INSERT INTO tag (created, updated, id, name) VALUES (%s, %s, %s, %s)'] [parameters: (datetime.datetime(2015, 9, 21, 13, 46, 48, 640836), datetime.datetime(2015, 9, 21, 13, 46, 48, 640843), '4475292c00044342a8643f9e894c3fde', 'App')]
2015-09-21 13:46:48.646 420 ERROR murano.api.v1.catalog

It happens because I tried to import a few packages with the same tag in parallel

Tags:

Serg Melikyan (smelikyan) on 2015-09-24

Changed in murano:
milestone:	none → mitaka-1

Ekaterina Chernova (efedorova) on 2015-09-25

summary:	- Incorrect algorithm for tags creation in the database + Race condition while uploading packages with same tags in parallel
Changed in murano:
importance:	Undecided → Medium
status:	New → Confirmed

Revision history for this message

Nikolay Starodubtsev (starodubcevna) wrote on 2015-11-09:

will be the fix with retrying good enough for this issue? I mean something like we did here: https://review.openstack.org/#/c/221827/

Changed in murano:
assignee:	nobody → Nikolay Starodubtsev (starodubcevna)

Revision history for this message

Anastasia Kuznetsova (akuznetsova) wrote on 2015-11-09:

Maybe it can be fixed in the transaction layer ?

Kirill Zaitsev (kzaitsev) on 2015-12-03

Changed in murano:
milestone:	mitaka-1 → mitaka-2

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-12-15: Fix proposed to murano (master)

Fix proposed to branch: master
Review: https://review.openstack.org/257858

Changed in murano:
status:	Confirmed → In Progress

Stan Lagun (slagun) on 2015-12-15

tags:

added: kilo-backport-potential liberty-backport-potential

OpenStack Infra (hudson-openstack) on 2015-12-30

Changed in murano:
assignee:	Nikolay Starodubtsev (starodubcevna) → Kirill Zaitsev (kzaitsev)

Revision history for this message

Kirill Zaitsev (kzaitsev) wrote on 2015-12-30:

I do not actually think that this one has the backport potential for kilo

Changed in murano:
assignee:	Kirill Zaitsev (kzaitsev) → Nikolay Starodubtsev (starodubcevna)

Kirill Zaitsev (kzaitsev) on 2016-01-21

Changed in murano:
milestone:	mitaka-2 → mitaka-3

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-02-02: Fix merged to murano (master)

Reviewed: https://review.openstack.org/257858
Committed: https://git.openstack.org/cgit/openstack/murano/commit/?id=8865876b1c56e2a33f3f3f3a19ee2ecd9261ffa9
Submitter: Jenkins
Branch: master

commit 8865876b1c56e2a33f3f3f3a19ee2ecd9261ffa9
Author: Nikolay Starodubtsev <email address hidden>
Date: Tue Dec 15 15:00:03 2015 +0300

Update _get_tags function to prevent race condition

Add `session.begin(nested=True` inside get tags to prevent duplicate
tags creation if murano executes in amount of threads.

Change-Id: I10968e2f2f758a5904943d273a9302a5cfb1a8c9
Closes-Bug: #1498097

Changed in murano:
status:	In Progress → Fix Released

Revision history for this message

Doug Hellmann (doug-hellmann) wrote on 2016-03-03: Fix included in openstack/murano 2.0.0.0b3

This issue was fixed in the openstack/murano 2.0.0.0b3 development milestone.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.