Zun

zun-db-manage upgrade fails with AttributeError on 35cb52c5553f migration

Bug #2043209 reported by Dmitriy Rabotyagov
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Zun
Fix Released
Undecided
Unassigned

Bug Description

Zun on version 2023.2 fails to execute zun-db-manage upgrade with SQLAlchemy 1.4.41 and alembic 1.9.4, which are defined in upper-constraints:
https://opendev.org/openstack/requirements/raw/branch/stable/2023.2/upper-constraints.txt

The error when attempting to execute db upgrade is as follow:

# /openstack/venvs/zun-27.1.0.dev142/bin/zun-db-manage --config-dir /etc/zun upgrade
INFO [alembic.runtime.migration] Context impl MySQLImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.runtime.migration] Running upgrade -> a9a92eebd9a8, create_table_zun_service
INFO [alembic.runtime.migration] Running upgrade a9a92eebd9a8 -> 9fe371393a24, create_table_container
INFO [alembic.runtime.migration] Running upgrade 9fe371393a24 -> 5971a6844738, add container_id column to container
INFO [alembic.runtime.migration] Running upgrade 5971a6844738 -> 93fbb05b77b9, add memory field to container
INFO [alembic.runtime.migration] Running upgrade 93fbb05b77b9 -> 63a08e32cc43, add task state to container
INFO [alembic.runtime.migration] Running upgrade 63a08e32cc43 -> 1192ba19a6e9, Add cpu workdir ports hostname labels to container
INFO [alembic.runtime.migration] Running upgrade 1192ba19a6e9 -> 72c6947c6636, create_table_image
INFO [alembic.runtime.migration] Running upgrade 72c6947c6636 -> c5565cbaa3de, Insert status_reason to Container table
INFO [alembic.runtime.migration] Running upgrade c5565cbaa3de -> 43e1088c3389, add image_pull_policy column
INFO [alembic.runtime.migration] Running upgrade 43e1088c3389 -> 4a0c4f7a4a33, add meta addresses to container
INFO [alembic.runtime.migration] Running upgrade 4a0c4f7a4a33 -> 531e4a890480, add host to container
INFO [alembic.runtime.migration] Running upgrade 531e4a890480 -> bbcfa910a8a5, add_restart_policy_column
INFO [alembic.runtime.migration] Running upgrade bbcfa910a8a5 -> ad43a2179cf2, add_status_detail
INFO [alembic.runtime.migration] Running upgrade ad43a2179cf2 -> d1ef05fd92c8, add tty stdin_open
INFO [alembic.runtime.migration] Running upgrade d1ef05fd92c8 -> 5458f8394206, add image driver field
INFO [alembic.runtime.migration] Running upgrade 5458f8394206 -> 6fd4f7582eb0, Add resource provider table
INFO [alembic.runtime.migration] Running upgrade 6fd4f7582eb0 -> 7975b7f0f792, Add resource class table
INFO [alembic.runtime.migration] Running upgrade 7975b7f0f792 -> 09f196622a3f, create inventory table
INFO [alembic.runtime.migration] Running upgrade 09f196622a3f -> e4d145e195f4, Create allocation table
INFO [alembic.runtime.migration] Running upgrade e4d145e195f4 -> 8192905fd835, add uuid_to_resource_class
INFO [alembic.runtime.migration] Running upgrade 8192905fd835 -> eeac0d191f5a, add compute node table
INFO [alembic.runtime.migration] Running upgrade eeac0d191f5a -> 53a8b515057e, Add memory info to compute node
INFO [alembic.runtime.migration] Running upgrade 53a8b515057e -> 4bf34495d060, Add container number info to compute node
INFO [alembic.runtime.migration] Running upgrade 4bf34495d060 -> ce9944b346cb, combine tty and stdin_open
INFO [alembic.runtime.migration] Running upgrade ce9944b346cb -> 8c3d80e18eb5, Add container cpus,cpu_used to compute node
INFO [alembic.runtime.migration] Running upgrade 8c3d80e18eb5 -> 04ba87af76bb, Add container host operating system info
INFO [alembic.runtime.migration] Running upgrade 04ba87af76bb -> 17ab8b533cc8, Add container hosts label info
INFO [alembic.runtime.migration] Running upgrade 17ab8b533cc8 -> 5359d23b2322, add websocket_url and websocket_token
INFO [alembic.runtime.migration] Running upgrade 5359d23b2322 -> 174cafda0857, add security groups
INFO [alembic.runtime.migration] Running upgrade 174cafda0857 -> 648c25faa0be, add mem used to compute node
INFO [alembic.runtime.migration] Running upgrade 648c25faa0be -> 75315e219cfb, Add auto_remove to container
INFO [alembic.runtime.migration] Running upgrade 75315e219cfb -> a251f1f61217, create capsule table
INFO [alembic.runtime.migration] Running upgrade a251f1f61217 -> 945569b3669f, add runtime column
INFO [alembic.runtime.migration] Running upgrade 945569b3669f -> 10d65e285a59, create volume_mapping table
INFO [alembic.runtime.migration] Running upgrade 10d65e285a59 -> 37bce72463e3, add pci device
INFO [alembic.runtime.migration] Running upgrade 37bce72463e3 -> bcd6410d645e, add host to capsule
INFO [alembic.runtime.migration] Running upgrade bcd6410d645e -> fc27c7415d9c, change the properties of meta_labels
INFO [alembic.runtime.migration] Running upgrade fc27c7415d9c -> ff7b9665d504, add pci stats to compute node
INFO [alembic.runtime.migration] Running upgrade ff7b9665d504 -> f046346d1d87, add timestamp to pci device
INFO [alembic.runtime.migration] Running upgrade f046346d1d87 -> d2affd5b4172
INFO [alembic.runtime.migration] Running upgrade d2affd5b4172 -> cf46a28f46bc, add container_actions table
INFO [alembic.runtime.migration] Running upgrade cf46a28f46bc -> b6bfca998431, add container_actions_events table
INFO [alembic.runtime.migration] Running upgrade b6bfca998431 -> 8b0082d9e7c1, drop foreign key of container_actions container_uuid
INFO [alembic.runtime.migration] Running upgrade 8b0082d9e7c1 -> 10c9668a816d, add volumes info and addresses to capsule
INFO [alembic.runtime.migration] Running upgrade 10c9668a816d -> 71f8b4cf1dbf, upgrade
INFO [alembic.runtime.migration] Running upgrade 71f8b4cf1dbf -> d9714eadbdc2, add disk to container
INFO [alembic.runtime.migration] Running upgrade d9714eadbdc2 -> 6ff4d35f4334, change properties of restart policy in capsule
INFO [alembic.runtime.migration] Running upgrade 6ff4d35f4334 -> fb9ad4a050f8, drop_container_actions_foreign_key
INFO [alembic.runtime.migration] Running upgrade fb9ad4a050f8 -> 50829990c965, add ondelete to container_actions_events foreign key
INFO [alembic.runtime.migration] Running upgrade 50829990c965 -> 3f49fa520409, add availability_zone to service
INFO [alembic.runtime.migration] Running upgrade 3f49fa520409 -> d0c606fdec3c, add disk total and used to compute node
INFO [alembic.runtime.migration] Running upgrade d0c606fdec3c -> 372433c0afd2, add auto heal to container
INFO [alembic.runtime.migration] Running upgrade 372433c0afd2 -> 238f94009eab, add disk_quota_supported to compute_node
INFO [alembic.runtime.migration] Running upgrade 238f94009eab -> 2b045cb595db, Create quota & quota class tables
INFO [alembic.runtime.migration] Running upgrade 2b045cb595db -> cff60402dd86, add capsule_id to containers
INFO [alembic.runtime.migration] Running upgrade cff60402dd86 -> 271c7f45982d, add started_at to containers
INFO [alembic.runtime.migration] Running upgrade 271c7f45982d -> 3298c6a5c3d9, empty message
INFO [alembic.runtime.migration] Running upgrade 3298c6a5c3d9 -> 012a730926e8, Add quota usage
INFO [alembic.runtime.migration] Running upgrade 012a730926e8 -> 26896d5f9053, create exec_instance table
INFO [alembic.runtime.migration] Running upgrade 26896d5f9053 -> 3e80bbfd8da7, Convert type of 'command' from string to list
INFO [alembic.runtime.migration] Running upgrade 3e80bbfd8da7 -> 105626c4f972, add privileged to container
INFO [alembic.runtime.migration] Running upgrade 105626c4f972 -> 2fb377a5a519, add healthcheck to containers
INFO [alembic.runtime.migration] Running upgrade 2fb377a5a519 -> f746cd28bcac, add host to volume mapping
INFO [alembic.runtime.migration] Running upgrade f746cd28bcac -> bc56b9932dd9, add runtime to compute node
INFO [alembic.runtime.migration] Running upgrade bc56b9932dd9 -> a9c9fb54274a, add_contents_to_volume_mapping_table
INFO [alembic.runtime.migration] Running upgrade a9c9fb54274a -> a019998b09b5, add host to image
INFO [alembic.runtime.migration] Running upgrade a019998b09b5 -> 02134de8e7d3, add_exposed_ports_to_container
INFO [alembic.runtime.migration] Running upgrade 02134de8e7d3 -> 54bcb75afb32, Add init containers uuids to capsule
INFO [alembic.runtime.migration] Running upgrade 54bcb75afb32 -> 35cb52c5553f, rename volume_id to cinder_volume_id in volume_mapping
Traceback (most recent call last):
  File "/openstack/venvs/zun-27.1.0.dev142/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 854, in __getattr__
    return getattr(self.comparator, key)
AttributeError: 'Comparator' object has no attribute 'alter'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/openstack/venvs/zun-27.1.0.dev142/bin/zun-db-manage", line 8, in <module>
    sys.exit(main())
  File "/openstack/venvs/zun-27.1.0.dev142/lib/python3.10/site-packages/zun/cmd/db_manage.py", line 67, in main
    CONF.command.func()
  File "/openstack/venvs/zun-27.1.0.dev142/lib/python3.10/site-packages/zun/cmd/db_manage.py", line 29, in do_upgrade
    migration.upgrade(CONF.command.revision)
  File "/openstack/venvs/zun-27.1.0.dev142/lib/python3.10/site-packages/zun/db/migration.py", line 35, in upgrade
    return get_backend().upgrade(version)
  File "/openstack/venvs/zun-27.1.0.dev142/lib/python3.10/site-packages/zun/db/sqlalchemy/migration.py", line 75, in upgrade
    get_manager().upgrade(version)
  File "/openstack/venvs/zun-27.1.0.dev142/lib/python3.10/site-packages/oslo_db/sqlalchemy/migration_cli/manager.py", line 74, in upgrade
    results.append(plugin.upgrade(revision))
  File "/openstack/venvs/zun-27.1.0.dev142/lib/python3.10/site-packages/oslo_db/sqlalchemy/migration_cli/ext_alembic.py", line 50, in upgrade
    return alembic.command.upgrade(self.config, version or 'head')
  File "/openstack/venvs/zun-27.1.0.dev142/lib/python3.10/site-packages/alembic/command.py", line 378, in upgrade
    script.run_env()
  File "/openstack/venvs/zun-27.1.0.dev142/lib/python3.10/site-packages/alembic/script/base.py", line 569, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/openstack/venvs/zun-27.1.0.dev142/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 94, in load_python_file
    module = load_module_py(module_id, path)
  File "/openstack/venvs/zun-27.1.0.dev142/lib/python3.10/site-packages/alembic/util/pyfiles.py", line 110, in load_module_py
    spec.loader.exec_module(module) # type: ignore
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/openstack/venvs/zun-27.1.0.dev142/lib/python3.10/site-packages/zun/db/sqlalchemy/alembic/env.py", line 58, in <module>
    run_migrations_online()
  File "/openstack/venvs/zun-27.1.0.dev142/lib/python3.10/site-packages/zun/db/sqlalchemy/alembic/env.py", line 55, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/openstack/venvs/zun-27.1.0.dev142/lib/python3.10/site-packages/alembic/runtime/environment.py", line 867, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/openstack/venvs/zun-27.1.0.dev142/lib/python3.10/site-packages/alembic/runtime/migration.py", line 624, in run_migrations
    step.migration_fn(**kw)
  File "/openstack/venvs/zun-27.1.0.dev142/lib/python3.10/site-packages/zun/db/sqlalchemy/alembic/versions/35cb52c5553f_rename_volume_id_to_cinder_volume_id_in_.py", line 37, in upgrade
    volume_mapping_table.c.volume_id.alter(name='cinder_volume_id')
  File "/openstack/venvs/zun-27.1.0.dev142/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 856, in __getattr__
    util.raise_(
  File "/openstack/venvs/zun-27.1.0.dev142/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
    raise exception
AttributeError: Neither 'Column' object nor 'Comparator' object has an attribute 'alter'

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to zun (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/zun/+/900619

Changed in zun:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to zun (master)

Reviewed: https://review.opendev.org/c/openstack/zun/+/900619
Committed: https://opendev.org/openstack/zun/commit/397edf53c161e9dbc9b26ab152227142d8c38796
Submitter: "Zuul (22348)"
Branch: master

commit 397edf53c161e9dbc9b26ab152227142d8c38796
Author: Dmitriy Rabotyagov <email address hidden>
Date: Fri Nov 10 15:17:52 2023 +0100

    Use alembic alter_column for 35cb52c5553f migration

    With latest sqlalchemy there is no alter attribute for the Column object

    Thus, zun-db-manage upgrade fails with AttributeError. Instead of trying
    to use SQLAlchemy for altering Column we should use alembic instead
    for the migration

    This also fix py3.11 job by replacing libmysqlclient-dev with
    libmariadb-dev-compat which is available for both ubuntu and debian.

    Closes-Bug: #2043209
    Change-Id: Iaba736d391ad5820de5f481d2c5bac1714dc0809

Changed in zun:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to zun (stable/2023.2)

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/zun/+/900785

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to zun (stable/2023.2)

Reviewed: https://review.opendev.org/c/openstack/zun/+/900785
Committed: https://opendev.org/openstack/zun/commit/9a4d70f6d331c23de5210ee610b70b7e646c6db9
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 9a4d70f6d331c23de5210ee610b70b7e646c6db9
Author: Dmitriy Rabotyagov <email address hidden>
Date: Mon Nov 13 12:41:29 2023 +0100

    Use alembic alter_column for 35cb52c5553f migration

    With latest sqlalchemy there is no alter attribute for the Column object

    Thus, zun-db-manage upgrade fails with AttributeError. Instead of trying
    to use SQLAlchemy for altering Column we should use alembic instead
    for the migration

    This also fix py3.11 job by replacing libmysqlclient-dev with
    libmariadb-dev-compat which is available for both ubuntu and debian.

    Depends-On: https://review.opendev.org/c/openstack/kuryr-libnetwork/+/895662
    Closes-Bug: #2043209
    Change-Id: Iaba736d391ad5820de5f481d2c5bac1714dc0809
    (cherry picked from commit 397edf53c161e9dbc9b26ab152227142d8c38796)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/zun 13.0.0.0rc1

This issue was fixed in the openstack/zun 13.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/zun 12.0.1

This issue was fixed in the openstack/zun 12.0.1 Bobcat release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.