OpenStack DBaaS (Trove)

MySQL restore should wait for database to shutdown before killing the process

Bug #1414186 reported by Petr Malik
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack DBaaS (Trove)
Fix Released
Medium
Petr Malik
OpenStack DBaaS (Trove) 2015.1.0 "kilo"

Bug Description

MySQL restore wait for shutdown before killing

Problem:
MySQL restory strategy in function '_spawn_with_init_file' of '/trove/trove/guestagent/strategies/restore/mysql_impl.py' issues a shutdown command to the mysql process immediatelly followed by 'killall'. The kill call is however necessary only if the soft shutdown fails.

This currently does not cause issues with MySQL (apart from executing unnecessary code), but our internal testers ran into issues with MariDB caused by this defect.
MariaDB's shutdown process does not block. It returns to the caller which then immediatelly proceeds with killall command before the shutdown finishes. This puts the database into a bad state resulting in an automated restart.

Solution:
Poll database status after executing shutdown and follow by 'killall' only if soft shutdown fails.

Petr Malik (pmalik)
Changed in trove:
assignee: nobody → Petr Malik (pmalik)
tags: added: backup mariadb mysql
Revision history for this message
Denis M. (dmakogon) wrote :

Here's a proble. Trove doesn't support MariaDB, it is not present at DatastoreCompatibility matrix. Another thing, there's no steps to reproduce. Also, there are couple things that were filed as MariaDB backlog (list of bugs, BPs that are in order to be fixed or implemented).

So, please concider on adding support of MariaDB to upstream code (Trove, Trove-integration) before actually trying to fix this bug, because this bug is not affecting any of supported datastore.

Revision history for this message
Petr Malik (pmalik) wrote :

The fact it was discovered on MariaDB (which is currently not supported, but likely will be in the future) is not the most relevant point. The current code potentially executes 'kill' unnecessarily after a successful shutdown. And in case of non-blocking shutdown may even cause database corruption.

Revision history for this message
Denis M. (dmakogon) wrote :

So, from this point of view this bug report seems to be reasonable. Please clear bug description and make it more MariaDB-agnostic to avoid confusion of other reviewers. Thanks.

OpenStack Infra (hudson-openstack)
Changed in trove:
status: New → In Progress
Nikhil Manchanda (slicknik)
Changed in trove:
importance: Undecided → Medium
milestone: none → kilo-2
Nikhil Manchanda (slicknik)
Changed in trove:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in trove:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in trove:
milestone: kilo-2 → 2015.1.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.