nova-manage db online_data_migrations can fail when upgrading to newton under certain conditions

Could you please provide more details about at least the stacktrace ?

There is no stacktrace to provide, the output of command is:

# nova-manage db online_data_migrations
Option "rpc_backend" from group "DEFAULT" is deprecated for removal. Its value may be silently ignored in the future.
Option "notification_topics" from group "DEFAULT" is deprecated. Use option "topics" from group "oslo_messaging_notifications".
Running batches of 50 until complete
Error attempting to run <function migrate_flavors at 0x63f3848>
Error attempting to run <function migrate_aggregates at 0x634f7d0>

Adding --verbose or --debug does not change the output either.

There's a workaround for this,
Run these in this order

nova-manage db sync;
nova-manage api_db sync;
nova-manage db online_data_migrations;

Recently ran into this out of nowhere, this seems to resolve the issue.

Automatically discovered version mitaka in description. If this is incorrect, please update the description to include 'nova version: ...'

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]

I also ran into this exact same issue. We have mysql as the backend. And the datetime type is not timezone aware. Neither "--verbose" and "--debug" are effective for the online_data_migrations command as they are not being taken into consideration. I ended up manually printing out the traceback by adding these two lines

import sys, traceback
traceback.print_exc(file=sys.stdout)

here https://github.com/openstack/nova/blob/stable/newton/nova/cmd/manage.py#L897

And able to see more information on the failure. Here's a sample traceback.

Running batches of 50 until complete
-------------- 2017-05-19 23:34:43+00:00 -----------------
Error attempting to run <function migrate_flavors at 0x7f5a0d758e60>
Traceback (most recent call last):
  File "/opt/stack/venv/nova-20170728T171245Z/lib/python2.7/site-packages/nova/cmd/manage.py", line 892, in _run_migration
    found, done = migration_meth(ctxt, count)
  File "/opt/stack/venv/nova-20170728T171245Z/lib/python2.7/site-packages/nova/objects/flavor.py", line 717, in migrate_flavors
    flavor._flavor_create(ctxt, flavor_values)
  File "/opt/stack/venv/nova-20170728T171245Z/lib/python2.7/site-packages/nova/objects/flavor.py", line 463, in _flavor_create
    return _flavor_create(context, updates)
  File "/opt/stack/venv/nova-20170728T171245Z/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 824, in wrapper
    return fn(*args, **kwargs)
  File "/opt/stack/venv/nova-20170728T171245Z/lib/python2.7/site-packages/nova/objects/flavor.py", line 166, in _flavor_create
    raise db_exc.DBError(e)
DBError: (_mysql_exceptions.OperationalError) (1292, "Incorrect datetime value: '2017-05-19 23:34:43+00:00' for column 'created_at' at row 1") [SQL: u'INSERT INTO flavors (created_at, updated_at, id, name, memory_mb, vcpus, root_gb, ephemeral_gb, flavorid, swap, rxtx_factor, vcpu_weight, disabled, is_public) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)'] [parameters: (datetime.datetime(2017, 5, 19, 23, 34, 43, tzinfo=<iso8601.Utc>), None, 16, 'foo', 1024, 1, 2, 0, '50aae60f-ba2a-40d8-a0c3-2117ba0dd2a6', 0, 1.0, 0, 0, 0)]

Looks like migrations code is attempting to insert a timezone aware datetime field into mysql.

But the fix is a bit complicated. I can think of a few option.

1. expose a global configurable option in oslo.versionedobjects to set the tzinfo_aware option.

https://github.com/openstack/oslo.versionedobjects/blob/master/oslo_versionedobjects/fields.py#L453

It shouldn't be per class as a given DB is either timezone-aware or not.

2. expose the option at the service level (i.e. in Nova config) and pass in the tzinfo_aware parameter when we instantiate the DateTimeField.

3. expose a global option in oslo.db and do the conversion there.

Pushed a patch for option #1 here

https://review.openstack.org/#/c/502189/