OpenStack Compute (nova)

nova list --deleted as admin fails with 404

Bug #1548980 reported by Matt Riedemann on 2016-02-23

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Fix Released	High	Matt Riedemann

Bug Description

Mitaka devstack created about a week ago:

stack@neutron:~/python-novaclient$ cd /opt/stack/nova
stack@neutron:~/nova$ git log -1
commit 11019fab7a2415cbca8b93e9346b21327e79688d
Author: bhagyashris <email address hidden>
Date: Tue Feb 16 01:13:23 2016 -0800

Remove duplicate key from dictionary

There is a duplicate dictionary key entry in test_vmops.py.
Removed duplicate key 'display_name' from dictionary.

TrivialFix

Change-Id: I4e779bceb26077b95bd3ae4ab19e60152c126e34
stack@neutron:~/nova$

I have a deleted instance:

I try to list deleted instances using 'nova list --deleted' and it fails with a 404.

Checking the n-api logs there is an InstanceNotFound, it looks like when lazy-loading the instance.services field:

2016-02-23 20:17:25.103 DEBUG nova.objects.instance [req-4f701f32-d988-4ae0-93f5-11a4591b297e admin alt_demo] Lazy-loading 'services' on Instance uuid 55b9808b-7e01-44ba-ab84-c0bac34d57f1 from (pid=17965) obj_load_attr /opt/stack/nova/nova/objects/instance.py:879
2016-02-23 20:17:25.168 ERROR nova.api.openstack [req-4f701f32-d988-4ae0-93f5-11a4591b297e admin alt_demo] Caught error: Instance 55b9808b-7e01-44ba-ab84-c0bac34d57f1 could not be found.
2016-02-23 20:17:25.168 TRACE nova.api.openstack Traceback (most recent call last):
2016-02-23 20:17:25.168 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/__init__.py", line 140, in __call__
2016-02-23 20:17:25.168 TRACE nova.api.openstack return req.get_response(self.application)
2016-02-23 20:17:25.168 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1317, in send
2016-02-23 20:17:25.168 TRACE nova.api.openstack application, catch_exc_info=False)
2016-02-23 20:17:25.168 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1281, in call_application
2016-02-23 20:17:25.168 TRACE nova.api.openstack app_iter = application(self.environ, start_response)
2016-02-23 20:17:25.168 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2016-02-23 20:17:25.168 TRACE nova.api.openstack return resp(environ, start_response)
2016-02-23 20:17:25.168 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2016-02-23 20:17:25.168 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2016-02-23 20:17:25.168 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2016-02-23 20:17:25.168 TRACE nova.api.openstack return self.func(req, *args, **kwargs)
2016-02-23 20:17:25.168 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 457, in __call__
2016-02-23 20:17:25.168 TRACE nova.api.openstack response = req.get_response(self._app)
2016-02-23 20:17:25.168 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1317, in send
2016-02-23 20:17:25.168 TRACE nova.api.openstack application, catch_exc_info=False)
2016-02-23 20:17:25.168 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1281, in call_application
2016-02-23 20:17:25.168 TRACE nova.api.openstack app_iter = application(self.environ, start_response)
2016-02-23 20:17:25.168 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2016-02-23 20:17:25.168 TRACE nova.api.openstack return resp(environ, start_response)
2016-02-23 20:17:25.168 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2016-02-23 20:17:25.168 TRACE nova.api.openstack return resp(environ, start_response)
2016-02-23 20:17:25.168 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/routes/middleware.py", line 136, in __call__
2016-02-23 20:17:25.168 TRACE nova.api.openstack response = self.app(environ, start_response)
2016-02-23 20:17:25.168 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2016-02-23 20:17:25.168 TRACE nova.api.openstack return resp(environ, start_response)
2016-02-23 20:17:25.168 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2016-02-23 20:17:25.168 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2016-02-23 20:17:25.168 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2016-02-23 20:17:25.168 TRACE nova.api.openstack return self.func(req, *args, **kwargs)
2016-02-23 20:17:25.168 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 672, in __call__
2016-02-23 20:17:25.168 TRACE nova.api.openstack content_type, body, accept)
2016-02-23 20:17:25.168 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 756, in _process_stack
2016-02-23 20:17:25.168 TRACE nova.api.openstack request, action_args)
2016-02-23 20:17:25.168 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 619, in post_process_extensions
2016-02-23 20:17:25.168 TRACE nova.api.openstack **action_args)
2016-02-23 20:17:25.168 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/compute/extended_server_attributes.py", line 90, in detail
2016-02-23 20:17:25.168 TRACE nova.api.openstack instances.values())
2016-02-23 20:17:25.168 TRACE nova.api.openstack File "/opt/stack/nova/nova/compute/api.py", line 3461, in get_instances_host_statuses
2016-02-23 20:17:25.168 TRACE nova.api.openstack host_status = self.get_instance_host_status(instance)
2016-02-23 20:17:25.168 TRACE nova.api.openstack File "/opt/stack/nova/nova/compute/api.py", line 3439, in get_instance_host_status
2016-02-23 20:17:25.168 TRACE nova.api.openstack service = [service for service in instance.services if
2016-02-23 20:17:25.168 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 67, in getter
2016-02-23 20:17:25.168 TRACE nova.api.openstack self.obj_load_attr(name)
2016-02-23 20:17:25.168 TRACE nova.api.openstack File "/opt/stack/nova/nova/objects/instance.py", line 902, in obj_load_attr
2016-02-23 20:17:25.168 TRACE nova.api.openstack self._load_generic(attrname)
2016-02-23 20:17:25.168 TRACE nova.api.openstack File "/opt/stack/nova/nova/objects/instance.py", line 746, in _load_generic
2016-02-23 20:17:25.168 TRACE nova.api.openstack expected_attrs=[attrname])
2016-02-23 20:17:25.168 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 181, in wrapper
2016-02-23 20:17:25.168 TRACE nova.api.openstack result = fn(cls, context, *args, **kwargs)
2016-02-23 20:17:25.168 TRACE nova.api.openstack File "/opt/stack/nova/nova/objects/instance.py", line 399, in get_by_uuid
2016-02-23 20:17:25.168 TRACE nova.api.openstack use_slave=use_slave)
2016-02-23 20:17:25.168 TRACE nova.api.openstack File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 283, in wrapper
2016-02-23 20:17:25.168 TRACE nova.api.openstack return f(*args, **kwargs)
2016-02-23 20:17:25.168 TRACE nova.api.openstack File "/opt/stack/nova/nova/objects/instance.py", line 391, in _db_instance_get_by_uuid
2016-02-23 20:17:25.168 TRACE nova.api.openstack columns_to_join=columns_to_join)
2016-02-23 20:17:25.168 TRACE nova.api.openstack File "/opt/stack/nova/nova/db/api.py", line 670, in instance_get_by_uuid
2016-02-23 20:17:25.168 TRACE nova.api.openstack return IMPL.instance_get_by_uuid(context, uuid, columns_to_join)
2016-02-23 20:17:25.168 TRACE nova.api.openstack File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 227, in wrapper
2016-02-23 20:17:25.168 TRACE nova.api.openstack return f(*args, **kwargs)
2016-02-23 20:17:25.168 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 709, in wrapper
2016-02-23 20:17:25.168 TRACE nova.api.openstack return fn(*args, **kwargs)
2016-02-23 20:17:25.168 TRACE nova.api.openstack File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1746, in instance_get_by_uuid
2016-02-23 20:17:25.168 TRACE nova.api.openstack columns_to_join=columns_to_join)
2016-02-23 20:17:25.168 TRACE nova.api.openstack File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1755, in _instance_get_by_uuid
2016-02-23 20:17:25.168 TRACE nova.api.openstack raise exception.InstanceNotFound(instance_id=uuid)
2016-02-23 20:17:25.168 TRACE nova.api.openstack InstanceNotFound: Instance 55b9808b-7e01-44ba-ab84-c0bac34d57f1 could not be found.
2016-02-23 20:17:25.168 TRACE nova.api.openstack

Tags:

Revision history for this message

Matt Riedemann (mriedem) wrote on 2016-02-23:

This is the compute API call to get the instance.services:

https://github.com/openstack/nova/blob/11019fab7a2415cbca8b93e9346b21327e79688d/nova/compute/api.py#L3439

Which tries to generically get the updated instance object from the database here:

https://github.com/openstack/nova/blob/11019fab7a2415cbca8b93e9346b21327e79688d/nova/objects/instance.py#L744

But because it's deleted that results in InstanceNotFound.

So we should either update the nova.objects.instance.Instance.obj_load_attr method to specifically handle the services field, or when we get the instance in the first place we should join on the services table.

Changed in nova:
status:	New → Confirmed
status:	Confirmed → Triaged
importance:	Undecided → High

Revision history for this message

Matt Riedemann (mriedem) wrote on 2016-02-23:

The regression was introduced with this change:

https://github.com/openstack/nova/commit/9345d5835fb8ff6a3534122c3c13620547862e95

Matt Riedemann (mriedem) on 2016-02-23

Changed in nova:
assignee:	nobody → Matt Riedemann (mriedem)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-02-23: Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/283820

Changed in nova:
status:	Triaged → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-02-23: Related fix proposed to nova (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/283837

Matt Riedemann (mriedem) on 2016-03-03

tags:

added: mitaka-rc-potential

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-03-04: Related fix merged to nova (master)

Reviewed: https://review.openstack.org/283837
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d6c334765559df07572441647a48debfe51829cb
Submitter: Jenkins
Branch: master

commit d6c334765559df07572441647a48debfe51829cb
Author: Matt Riedemann <email address hidden>
Date: Tue Feb 23 17:27:50 2016 -0500

Add functional regression test for list deleted instances on v2.16

    There is a bug such that listing deleted instances with microversion
    2.16 and greater is broken. It results in a 404. This adds a functional
    regression test to exhibit the failure. The fix will build on the test
    to show it's working again.

Change-Id: Ifca42c23e2ead2b533776365de29f874819cb2a8
Related-Bug: #1548980

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-03-04: Fix merged to nova (master)

Reviewed: https://review.openstack.org/283820
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3d6bb233828ce63ae649e98e02dc59e04f3db2f5
Submitter: Jenkins
Branch: master

commit 3d6bb233828ce63ae649e98e02dc59e04f3db2f5
Author: Matt Riedemann <email address hidden>
Date: Tue Feb 23 16:34:39 2016 -0500

Don't lazy-load instance.services if the instance is deleted

The 2.16 microversion added the host_status extended
server attribute which relies on the instance.services field.

The primary join in the database for that field is dependent on
the instance not being deleted.

    When listing deleted instances at microversion>=2.16, the
    compute API attempts to lazy-load the instance.services field
    which fails with an InstanceNotFound because the instance
    is deleted.

    In this case, it's best to just set instance.services to an
    empty ServiceList when lazy loading the services field on a
    deleted instance since the DB object won't have any value for
    the services attribute anyway.

Change-Id: Ic2f239f634f917a5771b0401a5073546c710c036
Closes-Bug: #1548980

Changed in nova:
status:	In Progress → Fix Released

Revision history for this message

Thierry Carrez (ttx) wrote on 2016-03-16: Fix included in openstack/nova 13.0.0.0rc1

This issue was fixed in the openstack/nova 13.0.0.0rc1 release candidate.

Matt Riedemann (mriedem) on 2016-03-21

tags:

removed: mitaka-rc-potential

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.