Ironic

Rolling upgrades: online schema migration

Bug #1502887 reported by Grzegorz Grasza
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ironic
Fix Released
Wishlist
Ruby Loo

Bug Description

Future incompatible changes in sqlalchemy models, like removing, renaming columns and tables can break rolling upgrades (upgrades when Ironic services are run at different versions simultaneously).

Besides multi-node grenade CI tests, I think we need some unit tests to check for incompatibilities.

See original description
Tags: db
Dmitry Tantsur (divius)
Changed in ironic:
status: New → Confirmed
importance: Undecided → Wishlist
tags: added: db
Revision history for this message
Grzegorz Grasza (xek) wrote :

To address this, we could also ban schema changes which cause incompatibilities like drops and alters:

https://github.com/openstack/nova/blob/stable/liberty/nova/tests/unit/db/test_migrations.py#L224-L225

Grzegorz Grasza (xek)
Changed in ironic:
assignee: nobody → Grzegorz Grasza (xek)
Revision history for this message
Jim Rollenhagen (jim-rollenhagen) wrote :

I don't think we need this any more, this is covered in the rolling upgrades spec, right?

Revision history for this message
Grzegorz Grasza (xek) wrote :

I'm thinking of writing an additional unit test, which will check for incompatible schema changes.

They have something like this in nova, where they are manually adding migrations with ALTER and DROP to a list of exceptions, so that core reviewers have a hint that they have to check if the patch doesn't introduce incompatible changes.

I'll have to port it to Alembic.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ironic (master)

Fix proposed to branch: master
Review: https://review.openstack.org/419993

Changed in ironic:
status: Confirmed → In Progress
Grzegorz Grasza (xek)
description: updated
OpenStack Infra (hudson-openstack)
Changed in ironic:
assignee: Grzegorz Grasza (xek) → Szymon Borkowski (szymon-borkowski)
OpenStack Infra (hudson-openstack)
Changed in ironic:
assignee: Szymon Borkowski (szymon-borkowski) → Ruby Loo (rloo)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/490270

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ironic (master)

Reviewed: https://review.openstack.org/490270
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=4686955d25aaa8330cb395869a2a8a6278324da1
Submitter: Jenkins
Branch: master

commit 4686955d25aaa8330cb395869a2a8a6278324da1
Author: Ruby Loo <email address hidden>
Date: Wed Aug 2 21:31:24 2017 -0400

    Remove WARNING from pin_release_version's help

    Rolling upgrades works so this removes the warning from the
    help for the [default]/pin_release_version config option.

    Change-Id: I0ba440a4136267e676370377394cee534505fdcd
    Partial-Bug: #1502887

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to ironic (master)

Reviewed: https://review.openstack.org/497949
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=e5b7aa203324b5d390e21ae725c910ea7e0e9502
Submitter: Jenkins
Branch: master

commit e5b7aa203324b5d390e21ae725c910ea7e0e9502
Author: Vasyl Saienko <email address hidden>
Date: Fri Aug 25 17:39:54 2017 +0300

    Test ironic-dbsync online_data_migrations

    Run ironic-dbsync online_data_migrations on single node grenade
    right after usual dbsync.
    Do not set pin_relase_version for single node grenade as we
    do cold upgrade and it is not required.

    Related-Bug: 1526283
    Related-Bug: 1502887
    Change-Id: I2e331252aeb66376c2d6cc3c16b6de0aefca5434

Revision history for this message
Ruby Loo (rloo) wrote :

This has been completed.

Changed in ironic:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on ironic (master)

Change abandoned by Grzegorz Grasza (<email address hidden>) on branch: master
Review: https://review.opendev.org/419993

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Related blueprints

Remote bug watches

Bug watches keep track of this bug in other bug trackers.