db_sync --expand may run downtime-incurring operations in upgrades to Newton

Bug #1676925 reported by Dolph Mathews
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
Fix Released
Low
Dolph Mathews

Bug Description

In discussing a fix for bug 1615014 with Richard, we realized that running:

  keystone-manage db_sync --expand

... automatically runs legacy (offline) migrations *specifically* when upgrading from Mitaka->Newton. Therefore, we don't technically support zero-downtime, rolling upgrades for Mitaka->Newton, despite having all the rolling upgrade commands available in Newton.

Therefore, all of the following migration paths are effectively identical in Mitaka->Newton upgrades, in that they all involve destructive migrations downtime in the first step:

Migration path A:

  keystone-manage db_sync # downtime-incurring operations normally occur here

Migration path B:

  keystone-manage db_sync # downtime-incurring operations normally occur here
  keystone-manage db_sync --expand # this becomes a no-op
  keystone-manage db_sync --migrate # this becomes a no-op
  keystone-manage db_sync --contract # this becomes a no-op

Migration path C:

  keystone-manage db_sync --expand # downtime-incurring operations only occur here in Mitaka->Newton
  keystone-manage db_sync --migrate
  keystone-manage db_sync --contract # downtime-incurring operations normally occur here

Migration path A is required for migrations up until Liberty->Mitaka, and is still supported in Ocata and beyond. Migration path B works, but there is no reason to recommend this path. Migration path C is recommended for Newton->Ocata upgrades and beyond.

Revision history for this message
Dolph Mathews (dolph) wrote :

At the moment, I think this is simply worth documenting -- I can't think of a "fix" that would enable truly zero-downtime, rolling upgrades for any part of the Mitaka->Newton upgrade process.

Changed in keystone:
status: New → Triaged
Revision history for this message
Lance Bragstad (lbragstad) wrote :

Based on the report - I'd be ok thoroughly documenting this for the Mitaka->Newton case.

tags: added: documentation
Dolph Mathews (dolph)
summary: - db_sync --expand may run destructive migrations
+ db_sync --expand may run downtime-incurring operations
summary: - db_sync --expand may run downtime-incurring operations
+ db_sync --expand may run downtime-incurring operations in upgrades to
+ Newton
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (master)

Fix proposed to branch: master
Review: https://review.openstack.org/450863

Changed in keystone:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (master)

Reviewed: https://review.openstack.org/450863
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=e6ef702f22929de831f07e6f56d7b696dfa50b89
Submitter: Jenkins
Branch: master

commit e6ef702f22929de831f07e6f56d7b696dfa50b89
Author: Dolph Mathews <email address hidden>
Date: Tue Mar 28 15:40:30 2017 +0000

    Doc db_sync --expand incurring downtime in upgrades to Newton

    The current implementation of db_sync --expand automatically runs any
    outstanding legacy operations to ensure it's safety. However, that
    behavior also means that legacy migrations may violate the safety
    assumptions around --expand.

    This is only true for Mitaka to Newton upgrades, as there should not be
    any newer migrations in the legacy repository to be triggered by this
    behavior.

    [1] https://github.com/openstack/keystone/blob/0ebd557325ff16c2535dac0eb190792e6d511a79/keystone/common/sql/upgrades.py#L269-L271

    Change-Id: Ib175f7fce96011d77eb4783123cb2d265d60aa42
    Closes-Bug: 1676925

Changed in keystone:
status: In Progress → Fix Released
Changed in keystone:
milestone: none → pike-1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/keystone 12.0.0.0b1

This issue was fixed in the openstack/keystone 12.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.