IBP images build is way too slow
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Fix Released
|
High
|
Alexei Sheplyakov | ||
6.1.x |
Won't Fix
|
High
|
MOS Maintenance | ||
7.0.x |
Fix Released
|
High
|
Alexei Sheplyakov |
Bug Description
Due to a very suboptimal IO layering building the target OS (Ubuntu)
IBP images takes an unreasonably long time (more than an hour!) if
the master node runs in the KVM VM.
fuel_agent (which builds Ubuntu images) runs in the mcollective
container. A container filesystem is backed by dm-snapshot which
is known to have a poor write performance, especially for large
files (like OS images). To add an insult to injury dm-snapshot
storage is a loopback file residing in /var/lib/docker.
Thus the data written to the OS image being built goes through
a number of layers:
ext4 -> loop -> dm-snapshot (/var/lib/docker) -> loop ->
-> ext4 (/var/lib/docker) -> dm-linear (os-varlibdocker) ->
-> hard drive
If the master node runs on a VM rather than a physcial hardware
the above chain gets even more complicated (and write performance
gets really awful)
ext4 -> loop -> dm-snapshot (/var/lib/docker) -> loop ->
-> ext4 (/var/lib/docker) -> dm-linear (os-varlibdocker) ->
-> virtual hard drive (qemu) -> qcow2 -> ext4 (host)
We need to remove the extra layers to make IO performance reasonable.
tags: |
added: docs removed: fuel-docs |
description: | updated |
description: | updated |
Changed in fuel: | |
assignee: | Fuel Documentation Team (fuel-docs) → Evgeny Konstantinov (evkonstantinov) |
Changed in fuel: | |
status: | Triaged → In Progress |
summary: |
- Use cache='unsafe' for KVM based master node + IBP images build is way too slow |
description: | updated |
Actually it's better to add a dedicated raw or lvm2 backed virtual hard drive for building images.
This way we can avoid the overhead due to qcow2. Using a host LV also avoids overhead due to fsync'img the image (by the host kernel)