Juju Charms Collection
nova-cloud-controller package

nova-cloud-controller needs to run `nova-manage db migrate_flavor_data` on upgrade from kilo -> liberty

Bug #1580698 reported by David Ames
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
nova-cloud-controller (Juju Charms Collection)
Fix Released
Critical
Liam Young
Juju Charms Collection 16.07

Bug Description

The nova-cloud-controller charm is missing a pre-upgrade step when upgrading from kilo to liberty:

The following must be run when upgrading from kilo to liberty *before* the packages are upgraded to liberty.

 nova-manage db migrate_flavor_data

If not the db migrate step fails leaving the upgrade process stranded.

2016-05-11 12:19:37.136 150521 INFO migrate.versioning.api [-] 280 -> 281...
2016-05-11 12:19:37.142 150521 INFO migrate.versioning.api [-] done
2016-05-11 12:19:37.142 150521 INFO migrate.versioning.api [-] 281 -> 282...
2016-05-11 12:19:37.149 150521 INFO migrate.versioning.api [-] done
2016-05-11 12:19:37.149 150521 INFO migrate.versioning.api [-] 282 -> 283...
2016-05-11 12:19:37.154 150521 INFO migrate.versioning.api [-] done
2016-05-11 12:19:37.154 150521 INFO migrate.versioning.api [-] 283 -> 284...
2016-05-11 12:19:37.159 150521 INFO migrate.versioning.api [-] done
2016-05-11 12:19:37.160 150521 INFO migrate.versioning.api [-] 284 -> 285...
2016-05-11 12:19:37.164 150521 INFO migrate.versioning.api [-] done
2016-05-11 12:19:37.165 150521 INFO migrate.versioning.api [-] 285 -> 286...
2016-05-11 12:19:37.170 150521 INFO migrate.versioning.api [-] done
2016-05-11 12:19:37.170 150521 INFO migrate.versioning.api [-] 286 -> 287...
2016-05-11 12:19:37.175 150521 INFO migrate.versioning.api [-] done
2016-05-11 12:19:37.175 150521 INFO migrate.versioning.api [-] 287 -> 288...
2016-05-11 12:19:37.179 150521 INFO migrate.versioning.api [-] done
2016-05-11 12:19:37.180 150521 INFO migrate.versioning.api [-] 288 -> 289...
2016-05-11 12:19:37.184 150521 INFO migrate.versioning.api [-] done
2016-05-11 12:19:37.184 150521 INFO migrate.versioning.api [-] 289 -> 290...
2016-05-11 12:19:37.189 150521 INFO migrate.versioning.api [-] done
2016-05-11 12:19:37.189 150521 INFO migrate.versioning.api [-] 290 -> 291...
2016-05-11 12:19:37.204 150521 CRITICAL nova [-] ValidationError: There are still 97 unmigrated flavor records. Migration cannot continue until all instance flavor records have been migrated to the new format. Please run `nova-manage db migrate_flavor_data' first.
2016-05-11 12:19:37.204 150521 ERROR nova Traceback (most recent call last):
2016-05-11 12:19:37.204 150521 ERROR nova File "/usr/bin/nova-manage", line 10, in <module>
2016-05-11 12:19:37.204 150521 ERROR nova sys.exit(main())
2016-05-11 12:19:37.204 150521 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/cmd/manage.py", line 1443, in main
2016-05-11 12:19:37.204 150521 ERROR nova ret = fn(*fn_args, **fn_kwargs)
2016-05-11 12:19:37.204 150521 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/cmd/manage.py", line 910, in sync
2016-05-11 12:19:37.204 150521 ERROR nova return migration.db_sync(version)
2016-05-11 12:19:37.204 150521 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/db/migration.py", line 26, in db_sync
2016-05-11 12:19:37.204 150521 ERROR nova return IMPL.db_sync(version=version, database=database)
2016-05-11 12:19:37.204 150521 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/migration.py", line 106, in db_sync
2016-05-11 12:19:37.204 150521 ERROR nova version)
2016-05-11 12:19:37.204 150521 ERROR nova File "/usr/lib/python2.7/dist-packages/migrate/versioning/api.py", line 186, in upgrade
2016-05-11 12:19:37.204 150521 ERROR nova return _migrate(url, repository, version, upgrade=True, err=err, **opts)
2016-05-11 12:19:37.204 150521 ERROR nova File "<string>", line 2, in _migrate
2016-05-11 12:19:37.204 150521 ERROR nova File "/usr/lib/python2.7/dist-packages/migrate/versioning/util/__init__.py", line 160, in with_engine
2016-05-11 12:19:37.204 150521 ERROR nova return f(*a, **kw)
2016-05-11 12:19:37.204 150521 ERROR nova File "/usr/lib/python2.7/dist-packages/migrate/versioning/api.py", line 366, in _migrate
2016-05-11 12:19:37.204 150521 ERROR nova schema.runchange(ver, change, changeset.step)
2016-05-11 12:19:37.204 150521 ERROR nova File "/usr/lib/python2.7/dist-packages/migrate/versioning/schema.py", line 93, in runchange
2016-05-11 12:19:37.204 150521 ERROR nova change.run(self.engine, step)
2016-05-11 12:19:37.204 150521 ERROR nova File "/usr/lib/python2.7/dist-packages/migrate/versioning/script/py.py", line 148, in run
2016-05-11 12:19:37.204 150521 ERROR nova script_func(engine)
2016-05-11 12:19:37.204 150521 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/migrate_repo/versions/291_enforce_flavors_migrated.py", line 35, in upgrade
2016-05-11 12:19:37.204 150521 ERROR nova raise exception.ValidationError(detail=msg)
2016-05-11 12:19:37.204 150521 ERROR nova ValidationError: There are still 97 unmigrated flavor records. Migration cannot continue until all instance flavor records have been migrated to the new format. Please run `nova-manage db migrate_flavor_data' first.
2016-05-11 12:19:37.204 150521 ERROR nova

With the liberty packages installed there is no way to recover.

Potential workaround:
Create a trusty container; add kilo cloud-archive and install nova-common
steal nova.conf from nova-cc unit
Add extra permission to mysql db to allow access from your container
And then run the required flavor migrate thing from a kilo install
Then re-commence the db-migrate on the nova-cc units...
You'll have to do that by hand as the charm will only run that codepath if the binaries register as upgrade eligible which then won't any longer
On kilo nova-manage db migrate_flavor_data
And then on liberty nova-manage db sync

Possible related bug: https://bugs.launchpad.net/nova/+bug/1511466

David Ames (thedac)
Changed in nova-cloud-controller (Juju Charms Collection):
status: New → Triaged
importance: Undecided → Critical
milestone: none → 16.07
Billy Olsen (billy-olsen)
tags: added: backport-potential
tags: added: openstack
Peter Sabaini (peter-sabaini)
tags: added: canonical-bootstack
Revision history for this message
Peter Sabaini (peter-sabaini) wrote :

Thanks, FTR was able to migrate the schema the above workaround

Revision history for this message
Billy Olsen (billy-olsen) wrote :

Testing the icehouse -> liberty upgrade the same problem is seen:

2016-05-12 21:02:36 INFO config-changed 2016-05-12 21:02:36.448 21696 ERROR nova ValidationError: There are still 5 unmigrated flavor records. Migration cannot continue until all instance flavor records have been migrated to the new format. Please run `nova-manage db migrate_flavor_data' first.

Liam Young (gnuoy)
Changed in nova-cloud-controller (Juju Charms Collection):
assignee: nobody → Liam Young (gnuoy)
Revision history for this message
Liam Young (gnuoy) wrote :

<tl;dr> I don't think this bug would be triggered by upgrading from a fresh Kilo deploy and upgrading to Liberty. This bug seems specific to upgrades from Icehouse (or earlier) to Liberty, including upgrades that step through Kilo.

When an instance is created on Icehouse the flavour ID is stored in a table called instance_system_metadata. Kilo (or maybe Juno) creates a new table in the nova database called instance_extra. For new instances created after upgrading to Kilo the flavour information goes into the instance_extra table not instance_system_metadata. It is a prerequisite for upgrading to Liberty that this flavour data has been moved from instance_system_metadata to instance_extra. At upgrade at this point will fail because the instance created when the system was on Icehouse has its flavour information in the wrong place.

The flavour information, as thedac points out, can be moved with "nova-manage db migrate_flavor_data"

Revision history for this message
Liam Young (gnuoy) wrote :

Just to clarify, I haven't investigated Juno, so instances created using Juno may or may not have this issue. I believe if the following query returns any results then the flavour migration needs to be run:

select * from instance_system_metadata where instance_system_metadata.key='instance_type_flavorid';

I'll look into adding the flavour migration into the charm

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-nova-cloud-controller (master)

Fix proposed to branch: master
Review: https://review.openstack.org/329437

Changed in nova-cloud-controller (Juju Charms Collection):
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-nova-cloud-controller (master)

Reviewed: https://review.openstack.org/329437
Committed: https://git.openstack.org/cgit/openstack/charm-nova-cloud-controller/commit/?id=05bd039a4ebba88c3fce935624fd46198fa92420
Submitter: Jenkins
Branch: master

commit 05bd039a4ebba88c3fce935624fd46198fa92420
Author: Liam Young <email address hidden>
Date: Tue Jun 14 13:11:47 2016 +0000

    Run flavor migrate on upgrades.

    Force upgrades to Liberty to step through Kilo and force Kilo upgrades to
    migrate flavor information. The flavor migration needs to be run with a
    Kilo tool set as the flavor migration option appears to have been removed
    in Liberty . The charm now run the flavor migration if the current
    deployment is Kilo and the user is migrating to a higher release.

    When an instance is created on Icehouse the flavour ID is stored in a table
    called instance_system_metadata. Kilo (or maybe Juno) creates a new table
    in the nova database called instance_extra. For new instances created after
    upgrading to Kilo the flavour information goes into the instance_extra
    table not instance_system_metadata. It is a prerequisite for upgrading to
    Liberty that this flavour data has been moved from instance_system_metadata
    to instance_extra. At upgrade at this point will fail because the instance
    created when the system was on Icehouse has its flavour information in the
    wrong place.

    Change-Id: If486372b05d165bd17fa9466b435f22088532d89
    Closes-Bug: #1580698

Changed in nova-cloud-controller (Juju Charms Collection):
status: In Progress → Fix Committed
Liam Young (gnuoy)
Changed in nova-cloud-controller (Juju Charms Collection):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.