leak of vswitch port if delete an instance while resizing

I've been testing with a modified version of stable/kilo, but I believe the bug is present in upstream stable/kilo.

When using nova with neutron, if I boot an instance, then trigger a resize, and then delete the instance at just the right point during the resize it ends up causing a vswitch port to be "leaked".

So far I've been able to show that if I issue the "nova delete" command while the resize operation is anywhere in nova.compute.manager.ComputeManager._finish_resize() up to the point where we set "instance.vm_state = vm_states.RESIZED" then I end up "leaking" a vswitch port. (By "leaking" I mean that it stays allocated even after the instance that it was allocated for is deleted.) I've been testing this by calling pdb.set_trace() to pause the resize while the nova delete runs, then letting the resize continue. Yes, this exaggerates the timing issues, but it shouldn't introduce any new races if the code is correct.

I think the problem occurs because the deletion path can't confirm the migration/resize because it hasn't gotten to the proper state yet. The resize code takes various exceptions depending on the exact timing of when the deletion happens, but it doesn't trigger a revert of the resize and it doesn't clean up the vswitch port on the source host. See sample log below.

I'm not sure what the proper fix is for this case. It seems that until we set "instance.vm_state = vm_states.RESIZED" it should be up to the resize code to clean up all resources if the instance gets deleted while a resize is in progress.

Sample log on source compute node. This is with a pause right at the beginning of _finish_resize().

(Pdb) c
2015-11-03 23:28:37.968 17000 INFO nova.compute.resource_tracker [req-2d9812a5-eadf-4eb2-96a3-d46e496b292d - - - - -] Auditing locally available compute resources for node compute-1
2015-11-03 23:28:38.511 17000 INFO nova.network.neutronv2.api [req-46519d0e-6e80-40e1-bbb3-dc39184eb046 41f42dfc41f9428fb143623f0a83d2fa 726f4a1ce23f4f12acb9139dcfcdb313 - - -] [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] Port dad525cf-75af-47a7-a57c-cc6fa26a6cf2 from network info_cache is no longer associated with instance in Neutron. Removing from network info_cache.
2015-11-03 23:28:38.677 17000 ERROR nova.compute.manager [req-46519d0e-6e80-40e1-bbb3-dc39184eb046 41f42dfc41f9428fb143623f0a83d2fa 726f4a1ce23f4f12acb9139dcfcdb313 - - -] [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] Setting instance vm_state to ERROR
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] Traceback (most recent call last):
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] File "/usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 4350, in finish_resize
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] disk_info, image)
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] File "/usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 4247, in _finish_resize
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] resize_instance = False
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] File "./usr/lib64/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7866, in macs_for_instance
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] File "./usr/lib64/python2.7/site-packages/nova/pci/manager.py", line 380, in get_instance_pci_devs
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] File "./usr/lib64/python2.7/site-packages/nova/objects/base.py", line 72, in getter
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] File "./usr/lib64/python2.7/site-packages/nova/objects/instance.py", line 1022, in obj_load_attr
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] File "./usr/lib64/python2.7/site-packages/nova/objects/instance.py", line 904, in _load_generic
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] File "./usr/lib64/python2.7/site-packages/nova/objects/base.py", line 161, in wrapper
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] File "./usr/lib64/python2.7/site-packages/nova/conductor/rpcapi.py", line 335, in object_class_action
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] File "/usr/lib64/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] retry=self.retry)
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] File "/usr/lib64/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] timeout=timeout, retry=retry)
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] File "/usr/lib64/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] retry=retry)
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] File "/usr/lib64/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 341, in _send
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] raise result
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] InstanceNotFound_Remote: Instance 21a4ff59-9057-4bc2-8b5e-185374c2d557 could not be found.
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] Traceback (most recent call last):
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] File "./usr/lib64/python2.7/site-packages/nova/conductor/manager.py", line 426, in _object_dispatch
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] File "./usr/lib64/python2.7/site-packages/nova/objects/base.py", line 163, in wrapper
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] File "./usr/lib64/python2.7/site-packages/nova/objects/instance.py", line 581, in get_by_uuid
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] File "./usr/lib64/python2.7/site-packages/nova/db/api.py", line 664, in instance_get_by_uuid
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] File "./usr/lib64/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 233, in wrapper
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] File "./usr/lib64/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1757, in instance_get_by_uuid
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] File "./usr/lib64/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1769, in _instance_get_by_uuid
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557] InstanceNotFound: Instance 21a4ff59-9057-4bc2-8b5e-185374c2d557 could not be found.
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]
2015-11-03 23:28:38.677 17000 TRACE nova.compute.manager [instance: 21a4ff59-9057-4bc2-8b5e-185374c2d557]
2015-11-03 23:28:39.053 17000 ERROR oslo_messaging.rpc.dispatcher [req-46519d0e-6e80-40e1-bbb3-dc39184eb046 41f42dfc41f9428fb143623f0a83d2fa 726f4a1ce23f4f12acb9139dcfcdb313 - - -] Exception during message handling: Instance 21a4ff59-9057-4bc2-8b5e-185374c2d557 could not be found.
Traceback (most recent call last):

  File "./usr/lib64/python2.7/site-packages/nova/conductor/manager.py", line 426, in _object_dispatch

  File "./usr/lib64/python2.7/site-packages/nova/objects/base.py", line 163, in wrapper

  File "./usr/lib64/python2.7/site-packages/nova/objects/instance.py", line 581, in get_by_uuid

  File "./usr/lib64/python2.7/site-packages/nova/db/api.py", line 664, in instance_get_by_uuid

  File "./usr/lib64/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 233, in wrapper

  File "./usr/lib64/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1757, in instance_get_by_uuid

  File "./usr/lib64/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1769, in _instance_get_by_uuid

InstanceNotFound: Instance 21a4ff59-9057-4bc2-8b5e-185374c2d557 could not be found.
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 7263, in finish_resize
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher reservations, migration)
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher File "./usr/lib64/python2.7/site-packages/nova/exception.py", line 88, in wrapped
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher File "./usr/lib64/python2.7/site-packages/nova/exception.py", line 71, in wrapped
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 345, in decorated_function
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance_uuid=instance_uuid)
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 316, in decorated_function
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 395, in decorated_function
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 304, in decorated_function
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher migration.instance_uuid, exc_info=True)
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 280, in decorated_function
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 361, in decorated_function
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 4362, in finish_resize
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher self._set_instance_error_state(context, instance)
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 4350, in finish_resize
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher disk_info, image)
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 4247, in _finish_resize
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher resize_instance = False
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher File "./usr/lib64/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7866, in macs_for_instance
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher File "./usr/lib64/python2.7/site-packages/nova/pci/manager.py", line 380, in get_instance_pci_devs
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher File "./usr/lib64/python2.7/site-packages/nova/objects/base.py", line 72, in getter
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher File "./usr/lib64/python2.7/site-packages/nova/objects/instance.py", line 1022, in obj_load_attr
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher File "./usr/lib64/python2.7/site-packages/nova/objects/instance.py", line 904, in _load_generic
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher File "./usr/lib64/python2.7/site-packages/nova/objects/base.py", line 161, in wrapper
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher File "./usr/lib64/python2.7/site-packages/nova/conductor/rpcapi.py", line 335, in object_class_action
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher retry=self.retry)
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher timeout=timeout, retry=retry)
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher retry=retry)
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 341, in _send
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher raise result
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher InstanceNotFound_Remote: Instance 21a4ff59-9057-4bc2-8b5e-185374c2d557 could not be found.
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher File "./usr/lib64/python2.7/site-packages/nova/conductor/manager.py", line 426, in _object_dispatch
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher File "./usr/lib64/python2.7/site-packages/nova/objects/base.py", line 163, in wrapper
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher File "./usr/lib64/python2.7/site-packages/nova/objects/instance.py", line 581, in get_by_uuid
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher File "./usr/lib64/python2.7/site-packages/nova/db/api.py", line 664, in instance_get_by_uuid
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher File "./usr/lib64/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 233, in wrapper
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher File "./usr/lib64/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1757, in instance_get_by_uuid
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher File "./usr/lib64/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1769, in _instance_get_by_uuid
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher InstanceNotFound: Instance 21a4ff59-9057-4bc2-8b5e-185374c2d557 could not be found.
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher
2015-11-03 23:28:39.053 17000 TRACE oslo_messaging.rpc.dispatcher
2015-11-03 23:28:39.165 17000 INFO nova.scheduler.client.report [req-2d9812a5-eadf-4eb2-96a3-d46e496b292d - - - - -] Compute_service record updated for ('compute-1', 'compute-1')