VM isn't connected to tenant network

Tried to reboot the VM. It seems the VM reached metadata (see screenshot), but steel unreachable from the tenant network

Have found the following in neutron-server logs: http://paste.openstack.org/show/593483/
Probably it's the root cause of the issue, but not sure.
We reduced number of neutron servers to 1 from default=3 in this case.

http://paste.openstack.org/show/593483/ isn't root cause. We found this error when a VM was accessible and didn't see the error when a VM was unaccessible

Looks like the root cause was VMs failing to connect to a metadata agent on boot and thus not getting an SSH key:

2016-12-27 16:10:45,281 - util.py[WARNING]: Failed fetching metadata from url http://169.254.169.254/2009-04-04/meta-data/

As only one router was used in the testing process, all metadata requests from instances were being handled by only one metadata agent that as a result couldn't process some of them due to high load.

So the recommendation here would be to create several routers to balance the load among all metadata-agents.

Thanks for recommendation.
After increasing the number of routers, this bug have not been reproduced.

It's reproduced again.

Now, we spawned 288 VMs on top of 12 compute nodes. Also, we have 48 networks and 12 routers.
After that, we spawned new VMs using heat, and it's unreachable from tenant network.

root@node1:~/ayasakov$ ssh -i key.pem ubuntu@10.144.1.115
Connection closed by 10.144.1.115 port 22

This issue can be explained by the way images used for VMs were configured:

1. Cloud-init was configured to use only Ec2 as its datasource, which means that on instance's boot all metadata was fetched from metadata-agent, even when config-drive was configured.
   ubuntu@wordpress-vm-0:~$ cat /etc/cloud/cloud.cfg.d/91-dib-cloud-init-datasources.cfg
   datasource_list: [ Ec2, None ]
2. At the same time, all instances were running an os-collect-config daemon that is constantly polling Neutron metadata agent and heat-api-cfn.

With several hundreds of simultaneous requests to handle metadata agent shows poor performance and can fail to serve requests within reasonable time (often taking over 10 seconds). These performance issues lead to some instances failing to get public keys on boot stage.

Note: Benchmarking of neutron metadata agent from a VM on a multinode DevStack and on CCP (both in default configuration) has shown significant performance drop on CCP:
DevStack http://paste.openstack.org/show/594838/
CCP http://paste.openstack.org/show/594839/

With images that are built to include ConfigDrive in a list of datasources this issue wasn't reproduced. Additionally it is possible to completely refrain from polling neutron metadata-agent using os-collect-config>=6.0.0b2.

I suggest to close this bug and open a new one concerning metadata-agent performance.

According last comment let's move this bug to Won't Fix.

Reproduced on my env. Not all instances have access to metadata.