Upgrade from 1.95 to 2.35 failure because column maasserver_domain.ttl does not exist

Bug #1880495 reported by Xav Paice
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Triaged
Medium
Unassigned
maas (Ubuntu)
New
Undecided
Unassigned
Xenial
New
Undecided
Unassigned

Bug Description

Upgrading an older Trusty based Maas (1.9.5) to Xenial (2.3.5) resulted in a traceback during the do-release-upgrade procedure:

  Applying maasserver.0011_domain_data...Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
psycopg2.ProgrammingError: column maasserver_domain.ttl does not exist
LINE 1: ...ain"."name", "maasserver_domain"."authoritative", "maasserve...
                                                             ^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3/dist-packages/maasserver/__main__.py", line 9, in <module>
    raise SystemExit(execute_from_command_line())
  File "/usr/lib/python3/dist-packages/maasserver/__init__.py", line 73, in execute_from_command_line
    management.execute_from_command_line()
  File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 354, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 346, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 394, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 445, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python3/dist-packages/maasserver/management/commands/dbupgrade.py", line 426, in handle
    fake_initial=self._south_was_performed(database))
  File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 120, in call_command
    return command.execute(*args, **defaults)
  File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 445, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python3/dist-packages/django/core/management/commands/migrate.py", line 222, in handle
    executor.migrate(targets, plan, fake=fake, fake_initial=fake_initial)
  File "/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 110, in migrate
    self.apply_migration(states[migration], migration, fake=fake, fake_initial=fake_initial)
  File "/usr/lib/python3/dist-packages/django/db/migrations/executor.py", line 148, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/lib/python3/dist-packages/django/db/migrations/migration.py", line 115, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/lib/python3/dist-packages/django/db/migrations/operations/special.py", line 183, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/usr/lib/python3/dist-packages/maasserver/migrations/builtin/maasserver/0011_domain_data.py", line 109, in migrate_staticipaddress_hostname
    domain_id = maasserver.models.dnsresource.get_default_domain()
  File "/usr/lib/python3/dist-packages/maasserver/models/dnsresource.py", line 57, in get_default_domain
    return Domain.objects.get_default_domain().id
  File "/usr/lib/python3/dist-packages/maasserver/models/domain.py", line 108, in get_default_domain
    'updated': now,
  File "/usr/lib/python3/dist-packages/django/db/models/manager.py", line 127, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 405, in get_or_create
    return self.get(**lookup), False
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 328, in get
    num = len(clone)
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 144, in __len__
    self._fetch_all()
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 965, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 238, in iterator
    results = compiler.execute_sql()
  File "/usr/lib/python3/dist-packages/django/db/models/sql/compiler.py", line 840, in execute_sql
    cursor.execute(sql, params)
  File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/lib/python3/dist-packages/django/db/utils.py", line 98, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/lib/python3/dist-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: column maasserver_domain.ttl does not exist
LINE 1: ...ain"."name", "maasserver_domain"."authoritative", "maasserve...
                                                             ^

dpkg: error processing package maas-region-controller (--configure):
 subprocess installed post-installation script returned error exit status 1
No apport report written because the error message indicates its a followup error from a previous failure.
                                                                                                          dpkg: dependency problems prevent configuration of maas:
 maas depends on maas-region-controller (= 2.3.5-6511-gf466fdb-0ubuntu1); however:
  Package maas-region-controller is not configured yet.

To reproduce, install Maas on Trusty, with a working database, and run do-release-upgrade.

I suspect there's some edge case in this particular database, working now on trying to identify it. The particular database has been around for at least 3 years now.

Related branches

Revision history for this message
Xav Paice (xavpaice) wrote :
Download full text (11.7 KiB)

Given this is a test, I tried adding the missing column:

postgres=# \connect maasdb
You are now connected to database "maasdb" as user "postgres".
maasdb=# \d maasserver_domain
                                     Table "public.maasserver_domain"
    Column | Type | Modifiers
---------------+--------------------------+----------------------------------------------------------------
 id | integer | not null default nextval('maasserver_domain_id_seq'::regclass)
 created | timestamp with time zone | not null
 updated | timestamp with time zone | not null
 name | character varying(256) | not null
 authoritative | boolean |
Indexes:
    "maasserver_domain_pkey" PRIMARY KEY, btree (id)
    "maasserver_domain_name_key" UNIQUE CONSTRAINT, btree (name)
    "maasserver_domain_946f3fba" btree (authoritative)
    "maasserver_domain_name_4403c8e5d405ce97_like" btree (name varchar_pattern_ops)
Referenced by:
    TABLE "maasserver_dnsresource" CONSTRAINT "maasserver_d_domain_id_48f70cfc2a2704fa_fk_maasserver_domain_id" FOREIGN KEY (domain_id) REFERENCES maasserver_domain(id) DEFERRABLE INITIALLY DEFERRED
    TABLE "maasserver_node" CONSTRAINT "maasserver_n_domain_id_67d5fcf688c8fcd5_fk_maasserver_domain_id" FOREIGN KEY (domain_id) REFERENCES maasserver_domain(id) DEFERRABLE INITIALLY DEFERRED

maasdb=# alter table maasserver_domain add column ttl integer;
ALTER TABLE
maasdb=# \d maasserver_domain
                                     Table "public.maasserver_domain"
    Column | Type | Modifiers
---------------+--------------------------+----------------------------------------------------------------
 id | integer | not null default nextval('maasserver_domain_id_seq'::regclass)
 created | timestamp with time zone | not null
 updated | timestamp with time zone | not null
 name | character varying(256) | not null
 authoritative | boolean |
 ttl | integer |
Indexes:
    "maasserver_domain_pkey" PRIMARY KEY, btree (id)
    "maasserver_domain_name_key" UNIQUE CONSTRAINT...

Revision history for this message
Felipe Reyes (freyes) wrote :

I could reproduce this issue locally, and this is what I believe is happening with this environment (and not for other upgrades) is that the database has no domains defined for the interfaces related, so this piece of code [0] is falling into the "else" section which calls get_default_domain() method and internally ends up query the 'ttl' column. So the migration step 0011 is relying on it and expecting that it was created in step 0010 and it wasn't, it was added way later in migration step 0023.

[0] https://github.com/maas/maas/blob/2.3/src/maasserver/migrations/builtin/maasserver/0011_domain_data.py#L106-L109

Revision history for this message
Dougal Matthews (d0ugal) wrote :

Yup, as Felipe links I think the problem here is that we are using the models within the database migration. However, the models assume all migrations have been applied. The call to `maasserver.models.dnsresource.get_default_domain()` should be replaced with something that doesn't use the models.

Changed in maas:
status: New → Confirmed
Felipe Reyes (freyes)
Changed in maas:
assignee: nobody → Felipe Reyes (freyes)
Dougal Matthews (d0ugal)
Changed in maas:
status: Confirmed → Triaged
Felipe Reyes (freyes)
Changed in maas:
assignee: Felipe Reyes (freyes) → nobody
Revision history for this message
Jerzy Husakowski (jhusakowski) wrote :

MAAS upgrade logic should not rely on MAAS models sources. This issue appeared on an old version of MAAS, but may appear in other upgrade scenarios. We will fix this on the current MAAS versions and do not plan to backport the fix to older versions.

Changed in maas:
importance: Undecided → Medium
milestone: none → 3.3.0
importance: Medium → High
Revision history for this message
Jerzy Husakowski (jhusakowski) wrote :

This issue requires further analysis of MAAS upgrade mechanism, to determine if it's possible to prevent such problems, e.g. by eliminating references to MAAS code in migration logic

Changed in maas:
importance: High → Medium
milestone: 3.3.0 → 3.4.0
Alberto Donato (ack)
Changed in maas:
milestone: 3.4.0 → 3.4.x
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.