nova API fails to run with Postgres: "UPDATE/SHARE cannot be applied to the nullable side of an outer join"

Bug #948066 reported by Dan Prince
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Derek Higgins

Bug Description

Recent Nova API Essex builds (in the past week) no longer run with Postgres:

2012-03-03 00:40:08 DEBUG nova.api.openstack.common [req-8dfe6fa9-c1a3-4826-9f17-e7b351c7a255 admin admin] Generated BUILD from vm_state=building task_state=scheduling. from (pid=10309) status_from_state /usr/lib/python2.7/dist-packages/nova/api/openstack/common.py:96
2012-03-03 00:40:08 ERROR nova.api.openstack [req-8dfe6fa9-c1a3-4826-9f17-e7b351c7a255 admin admin] Caught error: (NotSupportedError) SELECT FOR UPDATE/SHARE cannot be applied to the nullable side of an outer join
 'SELECT compute_nodes.created_at AS compute_nodes_created_at, compute_nodes.updated_at AS compute_nodes_updated_at, compute_nodes.deleted_at AS compute_nodes_deleted_at, compute_nodes.deleted AS compute_nodes_deleted, compute_nodes.id AS compute_nodes_id, compute_nodes.service_id AS compute_nodes_service_id, compute_nodes.vcpus AS compute_nodes_vcpus, compute_nodes.memory_mb AS compute_nodes_memory_mb, compute_nodes.local_gb AS compute_nodes_local_gb, compute_nodes.vcpus_used AS compute_nodes_vcpus_used, compute_nodes.memory_mb_used AS compute_nodes_memory_mb_used, compute_nodes.local_gb_used AS compute_nodes_local_gb_used, compute_nodes.hypervisor_type AS compute_nodes_hypervisor_type, compute_nodes.hypervisor_version AS compute_nodes_hypervisor_version, compute_nodes.hypervisor_hostname AS compute_nodes_hypervisor_hostname, compute_nodes.free_ram_mb AS compute_nodes_free_ram_mb, compute_nodes.free_disk_gb AS compute_nodes_free_disk_gb, compute_nodes.current_workload AS compute_nodes_current_workload, compute_nodes.running_vms AS compute_nodes_running_vms, compute_nodes.cpu_info AS compute_nodes_cpu_info, compute_nodes.disk_available_least AS compute_nodes_disk_available_least, services_1.created_at AS services_1_created_at, services_1.updated_at AS services_1_updated_at, services_1.deleted_at AS services_1_deleted_at, services_1.deleted AS services_1_deleted, services_1.id AS services_1_id, services_1.host AS services_1_host, services_1."binary" AS services_1_binary, services_1.topic AS services_1_topic, services_1.report_count AS services_1_report_count, services_1.disabled AS services_1_disabled, services_1.availability_zone AS services_1_availability_zone \nFROM services, compute_nodes LEFT OUTER JOIN services AS services_1 ON compute_nodes.service_id = services_1.id AND compute_nodes.deleted = %(deleted_1)s \nWHERE services.host IS NULL AND compute_nodes.deleted = %(deleted_2)s \n LIMIT 1 OFFSET 0 FOR UPDATE' {'deleted_2': False, 'deleted_1': False}
(nova.api.openstack): TRACE: Traceback (most recent call last):
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 41, in __call__
(nova.api.openstack): TRACE: return req.get_response(self.application)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1053, in get_response
(nova.api.openstack): TRACE: application, catch_exc_info=False)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1022, in call_application
(nova.api.openstack): TRACE: app_iter = application(self.environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE: return resp(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE: return resp(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE: return resp(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.7/routes/middleware.py", line 131, in __call__
(nova.api.openstack): TRACE: response = self.app(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE: return resp(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
(nova.api.openstack): TRACE: resp = self.call_func(req, *args, **self.kwargs)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 208, in call_func
(nova.api.openstack): TRACE: return self.func(req, *args, **kwargs)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 803, in __call__
(nova.api.openstack): TRACE: content_type, body, accept)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 878, in _process_stack
(nova.api.openstack): TRACE: request, action_args)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 773, in post_process_extensions
(nova.api.openstack): TRACE: **action_args)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/extended_server_attributes.py", line 80, in show
(nova.api.openstack): TRACE: self._extend_server(context, resp_obj.obj['server'], instance)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/extended_server_attributes.py", line 57, in _extend_server
(nova.api.openstack): TRACE: server[key] = self._get_hypervisor_hostname(context, instance)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/extended_server_attributes.py", line 48, in _get_hypervisor_hostname
(nova.api.openstack): TRACE: compute_node = db.compute_node_get_by_host(context, instance["host"])
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 206, in compute_node_get_by_host
(nova.api.openstack): TRACE: return IMPL.compute_node_get_by_host(context, host)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 534, in compute_node_get_by_host
(nova.api.openstack): TRACE: return node.first()
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 1652, in first
(nova.api.openstack): TRACE: ret = list(self[0:1])
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 1560, in __getitem__
(nova.api.openstack): TRACE: return list(res)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 1721, in __iter__
(nova.api.openstack): TRACE: return self._execute_and_instances(context)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 1726, in _execute_and_instances
(nova.api.openstack): TRACE: mapper=self._mapper_zero_or_none())
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 724, in execute
(nova.api.openstack): TRACE: clause, params or {})
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1191, in execute
(nova.api.openstack): TRACE: params)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1271, in _execute_clauseelement
(nova.api.openstack): TRACE: return self.__execute_context(context)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1302, in __execute_context
(nova.api.openstack): TRACE: context.parameters[0], context=context)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1401, in _cursor_execute
(nova.api.openstack): TRACE: context)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1394, in _cursor_execute
(nova.api.openstack): TRACE: context)
(nova.api.openstack): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 299, in do_execute
(nova.api.openstack): TRACE: cursor.execute(statement, parameters)
(nova.api.openstack): TRACE: NotSupportedError: (NotSupportedError) SELECT FOR UPDATE/SHARE cannot be applied to the nullable side of an outer join
(nova.api.openstack): TRACE: 'SELECT compute_nodes.created_at AS compute_nodes_created_at, compute_nodes.updated_at AS compute_nodes_updated_at, compute_nodes.deleted_at AS compute_nodes_deleted_at, compute_nodes.deleted AS compute_nodes_deleted, compute_nodes.id AS compute_nodes_id, compute_nodes.service_id AS compute_nodes_service_id, compute_nodes.vcpus AS compute_nodes_vcpus, compute_nodes.memory_mb AS compute_nodes_memory_mb, compute_nodes.local_gb AS compute_nodes_local_gb, compute_nodes.vcpus_used AS compute_nodes_vcpus_used, compute_nodes.memory_mb_used AS compute_nodes_memory_mb_used, compute_nodes.local_gb_used AS compute_nodes_local_gb_used, compute_nodes.hypervisor_type AS compute_nodes_hypervisor_type, compute_nodes.hypervisor_version AS compute_nodes_hypervisor_version, compute_nodes.hypervisor_hostname AS compute_nodes_hypervisor_hostname, compute_nodes.free_ram_mb AS compute_nodes_free_ram_mb, compute_nodes.free_disk_gb AS compute_nodes_free_disk_gb, compute_nodes.current_workload AS compute_nodes_current_workload, compute_nodes.running_vms AS compute_nodes_running_vms, compute_nodes.cpu_info AS compute_nodes_cpu_info, compute_nodes.disk_available_least AS compute_nodes_disk_available_least, services_1.created_at AS services_1_created_at, services_1.updated_at AS services_1_updated_at, services_1.deleted_at AS services_1_deleted_at, services_1.deleted AS services_1_deleted, services_1.id AS services_1_id, services_1.host AS services_1_host, services_1."binary" AS services_1_binary, services_1.topic AS services_1_topic, services_1.report_count AS services_1_report_count, services_1.disabled AS services_1_disabled, services_1.availability_zone AS services_1_availability_zone \nFROM services, compute_nodes LEFT OUTER JOIN services AS services_1 ON compute_nodes.service_id = services_1.id AND compute_nodes.deleted = %(deleted_1)s \nWHERE services.host IS NULL AND compute_nodes.deleted = %(deleted_2)s \n LIMIT 1 OFFSET 0 FOR UPDATE' {'deleted_2': False, 'deleted_1': False}
(nova.api.openstack): TRACE:
2012-03-03 00:40:08 INFO nova.api.openstack [req-8dfe6fa9-c1a3-4826-9f17-e7b351c7a255 admin admin] http://172.19.0.3:8774/v1.1/admin/servers/069425e1-6764-4aae-ad42-8de24185f352 returned with HTTP 500

Dan Prince (dan-prince)
Changed in nova:
importance: Undecided → High
Derek Higgins (derekh)
Changed in nova:
assignee: nobody → Derek Higgins (derekh)
Thierry Carrez (ttx)
Changed in nova:
milestone: none → essex-rc1
status: New → In Progress
Revision history for this message
Derek Higgins (derekh) wrote :

The patch I'm submitting removes the lock mode, I don't think an update cursor is required where this function is used in _get_hypervisor_hostname

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/5205

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/5205
Committed: http://github.com/openstack/nova/commit/5e7fd2584ad8b20635d284bbe448e45a59c37792
Submitter: Jenkins
Branch: master

commit 5e7fd2584ad8b20635d284bbe448e45a59c37792
Author: Derek Higgins <email address hidden>
Date: Sat Mar 10 22:49:29 2012 +0000

    Remove update lockmode from compute_node_get_by_host

    Fixes bug #948066

    This commit removes the usage of an update cursor so postgres
    doesn't raise the error below
    SELECT FOR UPDATE/SHARE cannot be applied to the nullable side of
    an outer join

    Change-Id: Ia1ede22df38d5f6a24372e0ad8ec25151ad195c7

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: essex-rc1 → 2012.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.