oslo-incubator

openstack.common.db.sqlalchemy.migration utf8 table check issue on initial migration

Bug #1301036 reported by Morgan Fainberg
30
This bug affects 5 people
Affects Status Importance Assigned to Milestone
Glance
Fix Released
Undecided
Shree Duth Awasthi
Ironic
Invalid
Undecided
Unassigned
OpenStack Heat
Fix Released
High
Thomas Herve
OpenStack Heat 2014.1 "icehouse"
OpenStack Identity (keystone)
Fix Released
High
Morgan Fainberg
OpenStack Identity (keystone) 2014.1 "icehouse"
oslo-incubator
Fix Released
Critical
Morgan Fainberg
oslo-incubator 2014.1 "icehouse"
keystone (Ubuntu)
Fix Released
High
Unassigned
Trusty
Fix Released
High
Unassigned

Bug Description

The code in openstack.common.db.sqlachemy.migration that does the sanity checking on the UTF8 table is a bit overzealous and checks the migration_version (and alembic equivalent) table for utf8 status. This will cause migrations to fail in any case where the db isn't forcing a default character set of utf8, the db engine isn't forced to utf8, or the migration_version table isn't fixed before migrations occur.

This was duplicated by doing a clean Ubuntu 12.04 install with mysql, using the default latin1 character set and simply creating the DB with ``create database keystone;``

The result is migrations fail at migration 0 unless the db sanity check is disabled (e.g. glance).

root@precise64:~/keystone# keystone-manage --config-file /etc/keystone.conf db_sync
2014-04-01 14:03:23.858 19840 CRITICAL keystone [-] ValueError: Tables "migrate_version" have non utf8 collation, please make sure all tables are CHARSET=utf8

This is unaffected by the character set in the connection string.

The solution is to explicitly ignore the migrate_version (and alembic equivalent) table in the sanity check.

Code in question is here:

http://git.openstack.org/cgit/openstack/keystone/tree/keystone/openstack/common/db/sqlalchemy/migration.py?id=51772e1addf8ab6f7483df44b7243fd7842eba4a#n200

This will affect any project using this code for migrations when using the mysql engine.

See original description
Morgan Fainberg (mdrnstm)
description: updated
Revision history for this message
aeva black (tenbrae) wrote :

Does not affect Ironic at the moment.

ironic.cmd.dbsync imports ironic.db.migration, which is lazy-loading ironic.db.sqlalchemy.migration, which is importing alembic.migration directly.

We'll need to resync oslo.openstack.common.db.migration to get the fix for this, but presumably we'll do that before switching to use it anyway.

Changed in ironic:
status: New → Invalid
Revision history for this message
Morgan Fainberg (mdrnstm) wrote :

I've looked through the following projects and added them to the bug as appropriate (only keystone, heat, and limitedly glance seem to be affected right now):

Appears to be affected (using the openstack.common.db.sqlalchemy.migration module):
heat
keystone
glance

Appears to be unaffected:
ceilometer
cinder
marconi
neutron
nova
sahara
swift
taskflow
trove
tuskar

Doug Hellmann (doug-hellmann)
Changed in oslo:
importance: Undecided → Critical
status: New → In Progress
assignee: nobody → Morgan Fainberg (mdrnstm)
milestone: none → icehouse-rc1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo-incubator (master)

Fix proposed to branch: master
Review: https://review.openstack.org/84596

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/84619

Changed in oslo:
assignee: Morgan Fainberg (mdrnstm) → Zhi Yan Liu (lzy-dev)
Dolph Mathews (dolph)
Changed in keystone (Ubuntu):
status: New → Confirmed
Thierry Carrez (ttx)
Changed in keystone:
milestone: none → icehouse-rc2
importance: Undecided → High
Revision history for this message
Lars Kellogg-Stedman (larsks) wrote :

Glance was tracking this in https://bugs.launchpad.net/glance/+bug/1279000

Changed in glance:
status: New → Fix Released
OpenStack Infra (hudson-openstack)
Changed in oslo:
assignee: Zhi Yan Liu (lzy-dev) → Doug Hellmann (doug-hellmann)
Doug Hellmann (doug-hellmann)
Changed in oslo:
assignee: Doug Hellmann (doug-hellmann) → Morgan Fainberg (mdrnstm)
James Page (james-page)
Changed in keystone (Ubuntu):
status: Confirmed → Triaged
importance: Undecided → High
Revision history for this message
Lars Kellogg-Stedman (larsks) wrote :

I looked at whether or not we could port the change from Glance into Heat, but
it turns out that's not going to be easy: because Heat from Havana -> Icehouse
creates several new tables, all explicitly set to utf8, these upgrade steps
will fail with an error along the lines of:

  ERROR: (OperationalError) (1005, "Can't create table 'heat.stack_lock'
  (errno: 150)") '\nCREATE TABLE stack_lock (\n\tstack_id VARCHAR(36)
  NOT NULL, \n\tcreated_at DATETIME, \n\tupdated_at DATETIME,
  \n\tengine_id VARCHAR(36), \n\tPRIMARY KEY (stack_id), \n\tFOREIGN
  KEY(stack_id) REFERENCES stack (id)\n)ENGINE=InnoDB CHARSET=utf8\n\n'
  ()

This happens when there is a foreign key reference from one of these
utf8 tables to a latin1 table.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (master)

Fix proposed to branch: master
Review: https://review.openstack.org/84869

Changed in keystone:
assignee: nobody → Morgan Fainberg (mdrnstm)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (milestone-proposed)

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/84870

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (master)

Reviewed: https://review.openstack.org/84869
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=908537c83cbf1628a9e9ea6fdeafa13de4e78996
Submitter: Jenkins
Branch: master

commit 908537c83cbf1628a9e9ea6fdeafa13de4e78996
Author: Morgan Fainberg <email address hidden>
Date: Wed Apr 2 14:10:12 2014 -0700

    Sync from oslo db.sqlalchemy.migration

    Sync the updated migration module to prevent errors on non-utf8
    migrate_version and alembic_version tables (not controlled by the
    schema migration directory).

    Change from oslo commits:
    [2fd457bf2cc] Ignore migrate versioning tables in utf8 sanity check

    Change-Id: I6aef126b8d09c7a44f3815c4143ef1d4e5463426
    Closes-Bug: #1301036

Changed in keystone:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (milestone-proposed)

Reviewed: https://review.openstack.org/84870
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=c15fac2b9a074de2d81b67b857bab38f1b1e3fb3
Submitter: Jenkins
Branch: milestone-proposed

commit c15fac2b9a074de2d81b67b857bab38f1b1e3fb3
Author: Morgan Fainberg <email address hidden>
Date: Wed Apr 2 14:10:12 2014 -0700

    Sync from oslo db.sqlalchemy.migration

    Sync the updated migration module to prevent errors on non-utf8
    migrate_version and alembic_version tables (not controlled by the
    schema migration directory).

    Change from oslo commits:
    [2fd457bf2cc] Ignore migrate versioning tables in utf8 sanity check

    Change-Id: I6aef126b8d09c7a44f3815c4143ef1d4e5463426
    Closes-Bug: #1301036

Changed in keystone:
status: Fix Committed → Fix Released
Steven Dake (sdake)
Changed in heat:
milestone: none → icehouse-rc2
importance: Undecided → High
Steve Baker (steve-stevebaker)
Changed in heat:
status: New → Triaged
Thierry Carrez (ttx)
Changed in oslo:
status: In Progress → Fix Released
Thomas Herve (therve)
Changed in heat:
assignee: nobody → Thomas Herve (therve)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/85941

Changed in heat:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master)

Reviewed: https://review.openstack.org/85941
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=4f9b62a2a5afab7a181ac30f9107fda51ed6403c
Submitter: Jenkins
Branch: master

commit 4f9b62a2a5afab7a181ac30f9107fda51ed6403c
Author: Thomas Herve <email address hidden>
Date: Tue Apr 8 09:02:58 2014 +0200

    Cherry pick oslo-incubator db fixes

    The patch cherrypicks 2 fixes related to utf8 migration:

     * Ib5c0f93c788afda92aad862e50dc086dc39605f1: Ignore migrate versioning tables in utf8 sanity check

     * I7f271d846141ac72dde3fb0d12159b125018eb2c: Make table utf-8 charset checking be optional for DB migration

    Change-Id: Icb811e846c05f863a42611054f2e3f30a47573db
    Closes-Bug: #1301036

Changed in heat:
status: In Progress → Fix Committed
Revision history for this message
James Page (james-page) wrote :

As trusty now has keystone rc2, marking fix released.

Changed in keystone (Ubuntu Trusty):
status: Triaged → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (milestone-proposed)

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/86457

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/86462

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (milestone-proposed)

Reviewed: https://review.openstack.org/86462
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=ec3a0c61393ee4b1ad33f79da256e0a510281ded
Submitter: Jenkins
Branch: milestone-proposed

commit ec3a0c61393ee4b1ad33f79da256e0a510281ded
Author: Thomas Herve <email address hidden>
Date: Tue Apr 8 09:02:58 2014 +0200

    Cherry pick oslo-incubator db fixes

    The patch cherrypicks 2 fixes related to utf8 migration:

     * Ib5c0f93c788afda92aad862e50dc086dc39605f1: Ignore migrate versioning tables in utf8 sanity check

     * I7f271d846141ac72dde3fb0d12159b125018eb2c: Make table utf-8 charset checking be optional for DB migration

    Change-Id: I5edf17c7491434ae4e1f9bb2819b3a415d245525
    Closes-Bug: #1301036

Changed in heat:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in oslo:
milestone: icehouse-rc1 → 2014.1
Thierry Carrez (ttx)
Changed in keystone:
milestone: icehouse-rc2 → 2014.1
Thierry Carrez (ttx)
Changed in heat:
milestone: icehouse-rc2 → 2014.1
Revision history for this message
Lee Thompson (stagr.lee) wrote :

I hit this issue on icehouse-rc2 for glance even though many of the above patches are incorporated into the RPMs. Think it is still lurking around somehow.

What got me around the problem was

# mysql -u root
mysql> use glance;
mysql> alter table migrate_version convert to character set utf8;
mysql> exit;
# glance-manage db_sync
# /etc/init.d/openstack-glance-api restart
# /etc/init.d/openstack-glance-registry restart
# glance image-list
+----+------+-------------+------------------+------+--------+
| ID | Name | Disk Format | Container Format | Size | Status |
 +----+------+-------------+------------------+------+--------+
+----+------+-------------+------------------+------+--------+

Revision history for this message
Shree Duth Awasthi (shreeduth-awasthi) wrote :

The bug affects me though the fix for glance is already in my source !

ubuntu@ubuntu-ThinkCentre-M92p:~$ glance --version
0.12.0.79

The above workaround by "Lee Thompson"helped me to solve it !

I will investigate further on this. Please correct me if my understanding is incorrect (Devstack: All in one installation, Ubuntu 14.04)

Thanks,
Shree Duth Awasthi.

Changed in glance:
assignee: nobody → Shree Duth Awasthi (shreeduth-awasthi)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.