deployment takes ~20mins to create containers

Opened by odyssey4me on 2014-11-27 11:47:17+00:00 at https://github.com/rcbops/ansible-lxc-rpc/issues/658

------------------------------------------------------------

Currently TASK: [container_create | Create Container LVM] takes approximately 20 minutes to complete. We need to find a way to optimise this task to reduce jenkins build times, thereby reducing time to feedback when doing pull requests.

Tags: enhancement

====================== COMMENTS ============================

Comment created by odyssey4me on 2014-11-27 18:44:12+00:00

@hughsaunders has suggested that the time taken may be due to a limitation in the way the containers are being built - essentially one being built per host at any one time. If the container build mechanism was changed to delegate the container builds to the hosts, then ansible could fork on a per host basis, essentially doing up to 15 containers per host at any one time. This would be far more efficient.

Some work to use this method has been done in order to work around #166 but the delegation has a bug which has prevented further progress. For reference, the work done is here: https://github.com/hughsaunders/ansible-lxc-rpc/compare/166

------------------------------------------------------------

Comment created by cloudnull on 2014-11-28 16:24:43+00:00

@odyssey4me @hughsaunders - I agree with being able to build the containers faster by delegating to the hosts is the best way to go as we get the benefits of Ansible's forking. In ansbile 1.6.6 of 1.6.10 the delegate_to function works great; it should work in 1.8.0 but i've not tested it as of yet. We presently have a requirement of ansible 1.6.10 so in normal deployments we should have no issues with the delegate_to function.

A while back I too attempted to make things faster as part of the old CDC branch, many of those improvements I have ported over as a smaller pieces though the container build process was not one that I've done as of yet. I put together a gist of the container build process from the CDC branch and have updated it a bit to fit in some of the more recent changes, "https://gist.github.com/cloudnull/d8e088da95f52dc7e908" if you guys wouldn't mind giving it some review and the changes seem sensible enough we can put together a pull request whenever.

I will mention is that the build process is still not crazy fast when building more than 20 some containers on a single host. This is mainly because of the subsystem locks in LXC that are used when creating and attaching to a container. But doing the build process this way does saves a bit of time. The other slowness is in our current LXC module. I have done a lot of refactoring in the LXC module since it was originally written much of which improves the speed of the module when managing containers. Sadly its not a drop in replacement and would require additional changes to some of our roles to get it to work in our current stack though maybe worth the investment to improve deployment performance. The work that was done in the refactored module has also been PR'd agains upstream ansible @ "ansible/ansible-modules-extras#123" which is a continuation of our original PR regarding native LXC support in ansible.