ImproperlyConfigured django error

Bug #1285248 reported by Tycho Andersen
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
Critical
Raphaël Badin

Bug Description

from celery.log:

[2014-02-26 18:43:01,409: CRITICAL/MainProcess] Can't decode message body: ImproperlyConfigured('Requested setting DEFAULT_INDEX_TABLESPACE, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.',) (type:u'application/x-python-serialize' encoding:u'binary' raw:"'\\x80\\x02}q\\x01(U\\x07expiresq\\x02NU\\x03utcq\\x03\\x88U\\x04argsq\\x04]q\\x05X\\x03\\x00\\x00\\x00amtq\\x06aU\\x05chordq\\x07NU\\tcallbacksq\\x08NU\\x08errbacksq\\tNU\\x07tasksetq\\nNU\\x02idq\\x0bU$3f297762-60bb-448d-8694-ba594757d937q\\x0cU\\x07retriesq\\rK\\x00U\\x04taskq\\x0eU!provisioningserver.tasks.power_onq\\x0fU\\ttimelimitq\\x10NN\\x86U\\x03etaq\\x11NU\\x06kwargsq\\x12}q\\x13(X\\x08\\x00\\x00\\x00usernameq\\x14X\\x00\\x00\\x00\\x00X\\t\\x00\\x00\\x00ipmipowerq\\x15X\\x13\\x00\\x00\\x00/usr/sbin/ipmipowerq\\x16X\\x08\\x00\\x00\\x00ipmitoolq\\x17X\\x11\\x00\\x00\\x00/usr/bin/ipmitoolq\\x18X\\t\\x00\\x00\\x00fence_cduq\\x19X\\x13\\x00\\x00\\x00/usr/sbin/fence_cduq\\x1aX\\r\\x00\\x00\\x00power_addressq\\x1bX\\x17\\x00\\x00\\x00qemu://localhost/systemq\\x1cX\\x08\\x00\\x00\\x00power_idq\\x1dX)\\x00\\x00\\x00node-0e2f2410-9f05-11e3-86b2-00ee4c62088dq\\x1eX\\x13\\x00\\x00\\x00ipmi_chassis_configq\\x1fX\\x1d\\x00\\x00\\x00/usr/sbin/ipmi-chassis-configq... (789b)"')
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 583, in _receive_callback
    decoded = None if on_m else message.decode()
  File "/usr/lib/python2.7/dist-packages/kombu/message.py", line 123, in decode
    self.content_encoding, accept=self.accept)
  File "/usr/lib/python2.7/dist-packages/kombu/serialization.py", line 165, in loads
    return decode(data)
  File "/usr/lib/python2.7/dist-packages/kombu/serialization.py", line 49, in pickle_loads
    return load(BytesIO(s))
  File "/usr/lib/python2.7/dist-packages/maasserver/fields.py", line 31, in <module>
    from django.db.models import (
  File "/usr/lib/python2.7/dist-packages/django/db/models/__init__.py", line 5, in <module>
    from django.db.models.query import Q
  File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 17, in <module>
    from django.db.models.deletion import Collector
  File "/usr/lib/python2.7/dist-packages/django/db/models/deletion.py", line 4, in <module>
    from django.db.models import signals, sql
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/__init__.py", line 4, in <module>
    from django.db.models.sql.subqueries import *
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/subqueries.py", line 12, in <module>
    from django.db.models.sql.query import Query
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 22, in <module>
    from django.db.models.sql import aggregates as base_aggregates_module
  File "/usr/lib/python2.7/dist-packages/django/db/models/sql/aggregates.py", line 9, in <module>
    ordinal_aggregate_field = IntegerField()
  File "/usr/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 116, in __init__
    self.db_tablespace = db_tablespace or settings.DEFAULT_INDEX_TABLESPACE
  File "/usr/lib/python2.7/dist-packages/django/conf/__init__.py", line 54, in __getattr__
    self._setup(name)
  File "/usr/lib/python2.7/dist-packages/django/conf/__init__.py", line 47, in _setup
    % (desc, ENVIRONMENT_VARIABLE))
ImproperlyConfigured: Requested setting DEFAULT_INDEX_TABLESPACE, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.

sudo service maas-cluster-celery restart seems to fix it.

Related branches

Revision history for this message
Tycho Andersen (tycho-s) wrote :

This seems to happen after I restart the celery server manually via 'sudo service maas-cluster-celery restart'.

Tycho Andersen (tycho-s)
tags: added: cloud-installer
Revision history for this message
Julian Edwards (julian-edwards) wrote :

So are you saying that restarting it manually *sometimes* makes it produce this error?

Revision history for this message
Tycho Andersen (tycho-s) wrote : Re: [Bug 1285248] Re: ImproperlyConfigured django error

On Thu, Feb 27, 2014 at 12:54:35AM -0000, Julian Edwards wrote:
> So are you saying that restarting it manually *sometimes* makes it
> produce this error?

Yeah, it seemed that way earlier, although now it seems like it
/always/ gets me this error.

Revision history for this message
Julian Edwards (julian-edwards) wrote :

I just tried and I don't get this, after many restarts.

Revision history for this message
Julian Edwards (julian-edwards) wrote :

Can you provide a bit more info about the package version, like it says in the bug reporting guidelines that nobody reads :)

Revision history for this message
Tycho Andersen (tycho-s) wrote :

On Thu, Feb 27, 2014 at 02:49:47AM -0000, Julian Edwards wrote:
> Can you provide a bit more info about the package version, like it says
> in the bug reporting guidelines that nobody reads :)

Yep, this is trusty maas:

CloudFive:/var/log/maas$ dpkg -l | grep maas
ii maas
1.5+bzr1977-0ubuntu2 all Ubuntu MAAS Server
ii maas-cli
1.5+bzr1977-0ubuntu2 all Ubuntu MAAS Client
Tool
ii maas-cluster-controller
1.5+bzr1977-0ubuntu2 all Ubuntu MAAS
Cluster Controller
ii maas-common
1.5+bzr1977-0ubuntu2 all Ubuntu MAAS Server
ii maas-dhcp
1.5+bzr1977-0ubuntu2 all Ubuntu MAAS Server
- DHCP Configuration (meta-package)
ii maas-dns
1.5+bzr1977-0ubuntu2 all Ubuntu MAAS Server
- DNS configuration (metapackage)
ii maas-region-controller
1.5+bzr1977-0ubuntu2 all Ubuntu MAAS Server
ii maas-region-controller-min
1.5+bzr1977-0ubuntu2 all Ubuntu MAAS Server
ii python-django-maas
1.5+bzr1977-0ubuntu2 all Ubuntu MAAS Server
- (django files)
ii python-maas-client
1.5+bzr1977-0ubuntu2 all Ubuntu MAAS API
Client - (python files)
ii python-maas-provisioningserver
1.5+bzr1977-0ubuntu2 all Ubuntu MAAS Server

I'd provide more info here, but to be honest I don't know what would
be useful. I know very little about django. I'd debug if I could, but
it just seems as though I restart either apache2 or
maas-cluster-celery and I get this.

Revision history for this message
Julian Edwards (julian-edwards) wrote :

Ok I'll wait for Raphaël to comment as he's the Django expert. I can't
see why this would only happen for you though :/

Out of interest, you only have the packages installed right? You're not
in a directory that has the dev environment? They don't mix well!

Revision history for this message
Tycho Andersen (tycho-s) wrote :

On Thu, Feb 27, 2014 at 03:41:15AM -0000, Julian Edwards wrote:
> Ok I'll wait for Raphaël to comment as he's the Django expert. I can't
> see why this would only happen for you though :/
>
> Out of interest, you only have the packages installed right? You're not
> in a directory that has the dev environment? They don't mix well!

Yep, just the packages.

Revision history for this message
Raphaël Badin (rvb) wrote :

 File "/usr/lib/python2.7/dist-packages/kombu/serialization.py", line 49, in pickle_loads
    return load(BytesIO(s))
  File "/usr/lib/python2.7/dist-packages/maasserver/fields.py", line 31, in <module>
    from django.db.models import (

^ as you can see, the de-serializer is trying to access maasserver's code and it shouldn't do that. The import partially succeeds because the cluster happens to be running on the same machine but the cluster celery doesn't have access to Django's settings, hence the error.

It means the "power up" task code was changed recently and it is either calling MAAS code directly or sending MAAS objects over the wire.

Changed in maas:
status: New → Triaged
importance: Undecided → Critical
Revision history for this message
Raphaël Badin (rvb) wrote :

btw, Diogo just got the same error in the lab.

Revision history for this message
Diogo Matsubara (matsubara) wrote :

I can reliably reproduce this in the qa lab: http://d-jenkins.ubuntu-ci:8080/view/MAAS/job/trusty-adt-maas-manual/65/console where the test fails to juju bootstrap a node and I can see the same exception Tycho reported in the celery.log. In the CI tests we don't restart celery but we do restart maas-pserv

Revision history for this message
Raphaël Badin (rvb) wrote :

This needs investigating but I suspect this change https://code.launchpad.net/~julian-edwards/maas/ipmi-with-mac/+merge/208286 is responsible for the problem. Well, the problem was there before this change but the change made the problem worse in a way.
'test_start_nodes_wakeonlan_falls_back_to_primary_mac' proves that the object passed to the task is an object of type maasserver.fields.MAC. It should be a string.

Raphaël Badin (rvb)
Changed in maas:
milestone: none → 14.04
assignee: nobody → Raphaël Badin (rvb)
Raphaël Badin (rvb)
Changed in maas:
status: Triaged → Fix Committed
Revision history for this message
Julian Edwards (julian-edwards) wrote :

!! Nicely sleuthed!

Revision history for this message
Julian Edwards (julian-edwards) wrote :

We need a test for this kind of cross-project pollution, it's far too easy to return objects that can't be unserialized in the clusters. Answers on a postcard ...

Changed in maas:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.