During compute/terminate_instance networking is de-allocated by a call to network/api/deallocate_for_instance(), which is implemented as an rpc.cast to the network manager. This in turn will eventually call network/manager/deallocate_fixed_ip(), which in turn call the compute API to trigger a security group refresh, which will get the instance from the database
However because original call to the network manager is a cast there is a chance that the compute manager will delete the instance record in the DB before the compute API (in the network manager) tries to retrieve the instance. At this point the security group refresh fails (leaving rules in place which are a security risk when the IP is reused), and potentially stopping other aspects of the network deallocation from completing.
This is one of the few places left in the network api where a cast is used, and it seems to me that an rpc.call would be better – the compute manager really needs to know that the network has been properly deallocated and cleaned up before it marks the VM as deleted.
In addition to this race condition, in general if there is an error during network clean up it is better that the VM remains in an error condition so that delete can be retried than for the VM to be deleted with the network left in some indeterminate state with no trigger available to retry the clean up. Hence some exception handling for the call to deallocate_for_instance() is required.
Note that aside from this specific use of a cast there are 4 other casts in the network API:
add_fixed_ip_to_instance
remove_fixed_ip_from_instance
add_network_to_project
release_floating_ip
Maybe these should all be considered for conversion to rpc.call to avoid race conditions and inconsistencies between the compute and network manager.
Agreed, lets convert everything to call. We may be sacrificing a little performance, but It will prevent a lot of annoying race conditions like this one.