[library] Large qcow2 images booted to and create volume fail.

More info:

When I try to boot > 6 of my (Centos_base) image and create volume, some may or may not fail as well. The raw images don't seem to have this restriction.

could you add diagnostic snapshot of the environment, please?

IRC discussion of this issue can be found here:
http://irclog.perlgeek.de/fuel/2014-02-14#i_8286110

IIRC the conclusion of the discussion was that the most likely cause of the failure would be a timeout while cinder volume manager waits for qemu-img convert to finish converting a qcow2 image to raw.

There is enough information here to test that this isn't a problem elsewhere. Two people in IRC discussed having variations of this issue.

Ok, guys, we need some openstack logs to identify where the failure is happening. If we do not have them - it will require more time to debug the issue. Thus, we can try to fix it in 4.1 if we have diagnostic snapshot or we postpone it for 5.0 as we need time to reproduce it and collect all the debug information.

Managed to reproduce it. On a large images that require large volumes, spawning of new instance may timeout while waiting for block device mapping to finish. But cinder continues to create volume until it's complete:

root@node-6:~# cinder list
+--------------------------------------+-------------+--------------+------+-------------+----------+-------------+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-------------+--------------+------+-------------+----------+-------------+
| d9fb1238-91e7-4ae5-99d2-23faa7aab550 | downloading | | 40 | None | false | |
+--------------------------------------+-------------+--------------+------+-------------+----------+-------------+

That's why you can use this volume later to spawn new instance from it.

So we just need to increase timeout value for block device mapping in nova config, using configuration parameters introduced in Juno (https://review.openstack.org/#/c/102891/). Default values are:

block_device_allocate_retries=60
block_device_allocate_retries_interval=3

Which is 3 minutes. After increasing block_device_allocate_retries from 60 to 300 (15 minutes) on my local env, spawning new instances from 1.8G qcow2 image and 40G volume started to work just fine.

Fix proposed to branch: master
Review: https://review.openstack.org/171233

Reviewed: https://review.openstack.org/171233
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=a3322f28b5548dd83e89b225b88cf067cf9da43d
Submitter: Jenkins
Branch: master

commit a3322f28b5548dd83e89b225b88cf067cf9da43d
Author: Aleksandr Didenko <email address hidden>
Date: Tue Apr 7 17:15:07 2015 +0300

    Make block device mapping timeout configurable

    When booting instances passing in block-device and increasing the
    volume size, instances can go in to error state if the volume takes
    longer to create than 3 minutes (current default value).

    This change adds support for 2 new cluster configutation options:
    - block_device_allocate_retries (default: 300)
    - block_device_allocate_retries_interval (default: 3)

    Thess options are configurable via fuel cli or Hiera. They are added
    to nova.conf on compute nodes and allow to adjust block device
    allocation timeout value.

    DocImpact

    Change-Id: If9148da046d31aab60a13bd70e94700331770000
    Closes-bug: #1280399