OpenStack Percona Cluster Charm

post-series-upgrade hook: catch mysql_upgrade error

Bug #1838041 reported by Alvaro Uria
38
This bug affects 7 people
Affects Status Importance Assigned to Milestone
OpenStack Charms Deployment Guide
Fix Released
High
Peter Matulis
OpenStack Percona Cluster Charm
Incomplete
High
Unassigned

Bug Description

I hit this bug when running OS Charms 19.04, on an upgrade from xenial to bionic.

post-series-upgrade hook can fail when "mysql_upgrade" was already run:
https://github.com/openstack/charm-percona-cluster/blob/stable/19.04/hooks/percona_hooks.py#L417

~# mysql_upgrade -p
Enter password:
Checking if update is needed.
This installation of MySQL is already upgraded to 5.7.20-18, use --force if you still need to run mysql_upgrade

subprocess.check_call should catch the output above and ignore the error instead of making the hook fail.

Alex Kavanagh (ajkavanagh)
Changed in charm-percona-cluster:
status: New → Triaged
importance: Undecided → High
Alex Kavanagh (ajkavanagh)
tags: added: series-upgrade
Revision history for this message
vinaya (agrahar) wrote :

does this going to fix ?

Revision history for this message
James Page (james-page) wrote :

We need to figure out why the mysql_upgrade is being executed twice - I believe it needs to be executed per unit to perform any required updates to the local db files which is why the mysqld is running in local mode during this step.

Billy Olsen (billy-olsen)
Changed in charm-percona-cluster:
assignee: nobody → Alex Kavanagh (ajkavanagh)
Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

I've tried reproducing this a couple of times (there's event a xenial->bionic upgrade bundle in percona-cluster now), but it doesn't seem to fail.

Is there any additional information you can provide as to what might have triggered the error?

Changed in charm-percona-cluster:
status: Triaged → In Progress
Alex Kavanagh (ajkavanagh)
Changed in charm-percona-cluster:
status: In Progress → Incomplete
Alex Kavanagh (ajkavanagh)
Changed in charm-percona-cluster:
assignee: Alex Kavanagh (ajkavanagh) → nobody
Revision history for this message
Xav Paice (xavpaice) wrote :

Please see https://bugs.launchpad.net/charm-percona-cluster/+bug/1949935. The process was simple, just ran the upgrade on the leader unit first, one step at a time. The same error occurred on the remaining two percona-cluster units.

Is it something as simple as the percona cluster packages running the upgrade steps as well as the charm? Could the charm be modified to catch the exception and continue, since it's just the tool reporting that the upgrade step doesn't need to be run?

Changed in charm-percona-cluster:
status: Incomplete → New
Revision history for this message
Billy Olsen (billy-olsen) wrote :

@xavpaice - would you be able to collect the logs from all of the percona units for further investigation?

Liam Young (gnuoy)
Changed in charm-percona-cluster:
assignee: nobody → Liam Young (gnuoy)
Revision history for this message
Liam Young (gnuoy) wrote :

I have failed to reproduce this bug too. I've documented the steps i went through (based on https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/upgrade-series-openstack.html#generalised-openstack-series-upgrade ). This is the steps I went through https://pastebin.ubuntu.com/p/H8Zqvsb6DB/ . Please can you confirm that you followed the steps and as billy-olsen mentioned please provide logs.

Changed in charm-percona-cluster:
assignee: Liam Young (gnuoy) → nobody
status: New → Incomplete
Revision history for this message
Peter Matulis (petermatulis) wrote :

There is an error in the logic of the steps in the documentation. I'm not sure if this is the cause of the problem reported here, but these steps will soon be corrected.

Changed in charm-deployment-guide:
assignee: nobody → Peter Matulis (petermatulis)
importance: Undecided → High
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-deployment-guide (master)

Reviewed: https://review.opendev.org/c/openstack/charm-deployment-guide/+/818848
Committed: https://opendev.org/openstack/charm-deployment-guide/commit/a655378cca96f8f2e518eca6fa5a32cf88a4626b
Submitter: "Zuul (22348)"
Branch: master

commit a655378cca96f8f2e518eca6fa5a32cf88a4626b
Author: Peter Matulis <email address hidden>
Date: Tue Nov 16 16:26:43 2021 -0500

    Review series upgrades

    Review and correct the upgrade-series page.

    Make miscellaneous improvements to various places
    for the sake of consistency.

    Also closes a doc bug and reverts changes made due to
    a second doc bug.

    Closes-Bug: #1838041
    Related-Bug: #1934764
    Change-Id: I88692573e8ae50cd77d3872b40361b464b8e0f19

Changed in charm-deployment-guide:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.