DB migration broken

Bug #1894056 reported by Slawek Kaplonski
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Rodolfo Alonso

Bug Description

Probably since we merged https://review.opendev.org/#/c/741676/ in Neutron db migration is broken for networking-bgpvpn project. Errors are like in https://zuul.opendev.org/t/openstack/build/141ddec140174c32a575ac70e38ee97a

Running upgrade for neutron ...
Traceback (most recent call last):
  File "/usr/local/bin/neutron-db-manage", line 10, in <module>
  File "/opt/stack/neutron/neutron/db/migration/cli.py", line 635, in main
    return_val |= bool(CONF.command.func(config, CONF.command.name))
  File "/opt/stack/neutron/neutron/db/migration/cli.py", line 179, in do_upgrade
    run_sanity_checks(config, revision)
  File "/opt/stack/neutron/neutron/db/migration/cli.py", line 619, in run_sanity_checks
  File "/usr/local/lib/python3.6/dist-packages/alembic/script/base.py", line 489, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/usr/local/lib/python3.6/dist-packages/alembic/util/pyfiles.py", line 98, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/local/lib/python3.6/dist-packages/alembic/util/compat.py", line 184, in load_module_py

  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/stack/networking-bagpipe/networking_bagpipe/db/migration/alembic_migrations/env.py", line 86, in <module>
  File "/opt/stack/networking-bagpipe/networking_bagpipe/db/migration/alembic_migrations/env.py", line 77, in run_migrations_online
  File "<string>", line 8, in run_migrations
  File "/usr/local/lib/python3.6/dist-packages/alembic/runtime/environment.py", line 846, in run_migrations
  File "/usr/local/lib/python3.6/dist-packages/alembic/runtime/migration.py", line 509, in run_migrations
    for step in self._migrations_fn(heads, self):
  File "/opt/stack/neutron/neutron/db/migration/cli.py", line 610, in check_sanity
    revision, rev, implicit_base=True):
  File "/usr/local/lib/python3.6/dist-packages/alembic/script/revision.py", line 777, in _iterate_revisions
    uppers = util.dedupe_tuple(self.get_revisions(upper))
  File "/usr/local/lib/python3.6/dist-packages/alembic/script/revision.py", line 321, in get_revisions
    resolved_id, branch_label = self._resolve_revision_number(id_)
  File "/usr/local/lib/python3.6/dist-packages/alembic/script/revision.py", line 501, in _resolve_revision_number
  File "/usr/local/lib/python3.6/dist-packages/alembic/util/langhelpers.py", line 230, in __get__
    obj.__dict__[self.__name__] = result = self.fget(obj)
  File "/usr/local/lib/python3.6/dist-packages/alembic/script/revision.py", line 156, in _revision_map
    down_revision = map_[downrev]
KeyError: 'd4d4d7f03b21'

Changed in bgpvpn:
status: New → Confirmed
Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

It broke kolla-ansible gates as well. :/

Changed in kolla-ansible:
status: New → Triaged
importance: Undecided → Critical
Revision history for this message
Radosław Piliszek (yoctozepto) wrote :
Download full text (3.7 KiB)

Thought we get this on VPNaaS:

"+++ neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --subproject neutron-vpnaas upgrade head",
    "INFO [alembic.runtime.migration] Context impl MySQLImpl.",
    "INFO [alembic.runtime.migration] Will assume non-transactional DDL.",
    "/var/lib/kolla/venv/lib/python3.8/site-packages/alembic/script/revision.py:152: UserWarning: Revision 24f28869838b referenced from 24f28869838b -> 41b509d10b5e (head), VPNaaS endpoint groups is not present",
    " util.warn(",
    "Traceback (most recent call last):",
    " File \"/var/lib/kolla/venv/bin/neutron-db-manage\", line 8, in <module>",
    " sys.exit(main())",
    " File \"/var/lib/kolla/venv/lib/python3.8/site-packages/neutron/db/migration/cli.py\", line 635, in main",
    " return_val |= bool(CONF.command.func(config, CONF.command.name))",
    " File \"/var/lib/kolla/venv/lib/python3.8/site-packages/neutron/db/migration/cli.py\", line 179, in do_upgrade",
    " run_sanity_checks(config, revision)",
    " File \"/var/lib/kolla/venv/lib/python3.8/site-packages/neutron/db/migration/cli.py\", line 619, in run_sanity_checks",
    " script_dir.run_env()",
    " File \"/var/lib/kolla/venv/lib/python3.8/site-packages/alembic/script/base.py\", line 489, in run_env",
    " util.load_python_file(self.dir, \"env.py\")",
    " File \"/var/lib/kolla/venv/lib/python3.8/site-packages/alembic/util/pyfiles.py\", line 98, in load_python_file",
    " module = load_module_py(module_id, path)",
    " File \"/var/lib/kolla/venv/lib/python3.8/site-packages/alembic/util/compat.py\", line 184, in load_module_py",
    " spec.loader.exec_module(module)",
    " File \"<frozen importlib._bootstrap_external>\", line 783, in exec_module",
    " File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed",
    " File \"/var/lib/kolla/venv/lib/python3.8/site-packages/neutron_vpnaas/db/migration/alembic_migrations/env.py\", line 88, in <module>",
    " run_migrations_online()",
    " File \"/var/lib/kolla/venv/lib/python3.8/site-packages/neutron_vpnaas/db/migration/alembic_migrations/env.py\", line 79, in run_migrations_online",
    " context.run_migrations()",
    " File \"<string>\", line 8, in run_migrations",
    " File \"/var/lib/kolla/venv/lib/python3.8/site-packages/alembic/runtime/environment.py\", line 846, in run_migrations",
    " self.get_context().run_migrations(**kw)",
    " File \"/var/lib/kolla/venv/lib/python3.8/site-packages/alembic/runtime/migration.py\", line 509, in run_migrations",
    " for step in self._migrations_fn(heads, self):",
    " File \"/var/lib/kolla/venv/lib/python3.8/site-packages/neutron/db/migration/cli.py\", line 609, in check_sanity",
    " for script in script_dir.revision_map.iterate_revisions(",
    " File \"/var/lib/kolla/venv/lib/python3.8/site-packages/alembic/script/revision.py\", line 777, in _iterate_revisions",
    " uppers = util.dedupe_tuple(self.get_revisions(upper))",
    " File \"/var/lib/kolla/venv/lib/python3.8/site-packages/alembic/script/revision.py\", line 321, in get_revisions",
    " resolved_id,...


Revision history for this message
Lajos Katona (lajos-katona) wrote :

By the exception in https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_141/749060/3/check/neutron-tempest-plugin-bgpvpn-bagpipe/141ddec/job-output.txt , it seems that bgpvpn and bagpipe liberty alembic branch must be squashed to Pike, but not sure about if that is really the only solution.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to networking-bgpvpn (master)

Fix proposed to branch: master
Review: https://review.opendev.org/749696

Changed in bgpvpn:
assignee: nobody → Rodolfo Alonso (rodolfo-alonso-hernandez)
status: Confirmed → In Progress
Revision history for this message
Lajos Katona (lajos-katona) wrote :

For bagpipe the patch from Rodolfo: https://review.opendev.org/749687

Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

How's it going?

Revision history for this message
Slawek Kaplonski (slaweq) wrote :

@Radosław: It should be fixed with revert https://review.opendev.org/#/c/749866/

Changed in bgpvpn:
status: In Progress → Fix Committed
status: Fix Committed → New
Revision history for this message
Slawek Kaplonski (slaweq) wrote :

Actually https://review.opendev.org/#/c/749866/ just avoids failures but I will keep this bug open as to fix it properly we will need to squash db migrations in that stadium project also and we can use this bug to track that.

Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

Ok, thanks @Sławek for confirming. It means we (kolla) can restore these optional migrations.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on networking-bgpvpn (master)

Change abandoned by Rodolfo Alonso Hernandez (<email address hidden>) on branch: master
Review: https://review.opendev.org/749696
Reason: We have too many external dependencies. [1] is not going to happen.


Changed in kolla-ansible:
status: Triaged → Invalid
importance: Critical → Undecided
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.