VMware: Fetching images from Glance is slower than it should be

also tracked here - https://bugs.launchpad.net/fuel/+bug/1370684

When spawning a new instance an image for it must be transferred from Glance to vCenter datastore first. VMWareAPI Nova compute driver is implemented in a way, so that images *are not* stored on nodes running nova-compute+vmwareapi, but rather directly uploaded to vCenter datastore (i.e. nova-compute opens two connections: one to glance-api and another to vCenter). Nova implementation of this file transfer is indeed inefficient and can be optimized (the first step is done here https://review.openstack.org/#/c/72545/ ; we could probably go even further and try to use sendfile() system call, if needed).

However, our observations show that nova-compute *is not* the root cause of the issue, but vCenter is. We used tcpdump and strace to monitor nova-compute behavior when the image is uploaded to vCenter datastore. It turns out, that vCenter datastore is not able to keep the pace and tends to reduce the TCP window periodically up to setting it to 0 (i.e. it asks nova-compute to stop sending more data) periodically: when transferring a 17 Mb image, we've seen TCP window size set to 0 29 times. This leads to inefficient bandwidth usage and slow upload speed values. In other words, vCenter datastore is a bottleneck here.

Please see the figure attached (moments when TCP win is set to 0 are shown as red lines, blue markers show the upload speed value in a moment of time, the average upload speed value is shown as a green line).

The bottom line is we can tweak nova-compute a bit, but this won't help much. From what we see, the problem is on vCenter side. Marking this as invalid until we get more data from Partner Integration team.

It was fixed in vCenter 5.5u2 at Sep, 9 2014.
See the "vCenter Server takes a long time to upload file" item in resolved issues:
https://www.vmware.com/support/vsphere5/doc/vsphere-vcenter-server-55u2-release-notes.html