Create volume and boot instance from it failed on step server deletion

Seems the root of issue not only in time-outs. We are waiting for dictionary as response, but got the error message:
ERROR: Gateway Time-out (HTTP 504)

Steps to reproduce:
1). Deploy cluster: 3Controllers, 2Computes, 1Cinder
2). Destroy (Force Shutoff) one controller
3). Create cinder volume

Expected:
Volume will be created in ~2min
Response will be like(in cli):
http://paste.openstack.org/show/483768/
in rest api - dictionary

Actual:
Response: error message:
http://paste.openstack.org/show/483769/
but the volume was created.
Approximate time of volume creation ~5 min

https://drive.google.com/file/d/0BzWDM1PONYEub0FpdXlMdzQ3S1U/view?usp=sharing

This is incorrect behaviour of cinder. Moving to mos-cinder team.

Looks like the used env is slow according to the appropriate time of volume creation (the above mentioned ~2-5min). So I don't think that it is incorrect behaviour of cinder.

I think that the used env is slow according to the appropriate time of volume creation (the above mentioned ~2-5min). Please check it out on a faster environment. The appropriate time of volume creation should be < 1 min.

@Ivan - please use snapshots and described steps to reproduce, also ping mos -qa for the help here

Note according to test rail, the same issue happens on baremetal env https://mirantis.testrail.com/index.php?/tests/view/2465860&group_by=tests:status_id&group_order=asc&group_id=8 - so do not thinks that problem is in slow bm environment that appears only after destructive actions

Reproduced again on the baremetal environment, sometimes it is possible reproduce the problem not from the first time.
Destroy controller start test if was not reproduced:
start destroyed controller wait it is online and destroy another one after start the test.

https://drive.google.com/file/d/0BzWDM1PONYEuM0pYOUNwTGt0aG8/view?usp=sharing

Another failure here: http://jenkins-product.srt.mirantis.net:8080/job/8.0.custom_system_test/1669/consoleFull

Upstream bug: https://bugs.launchpad.net/cinder/+bug/1539057

Fix proposed to branch: openstack-ci/fuel-8.0/liberty
Change author: Ivan Kolodyazhny <email address hidden>
Review: https://review.fuel-infra.org/16539

Do we have a stable repro? Also, why RabbitMQ is down?

The last thing I want to do here is introduce a last minute fix, which can break other things. Maybe it makes to move this to -updates?

+ occurrence https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.ha_neutron_destructive/123/testReport/%28root%29/ha_neutron_destroy_controllers/ha_neutron_destroy_controllers/

+ occurrence
https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.ha_neutron_destructive/123/testReport/%28root%29/ha_neutron_test_3_1_rabbit_failover/ha_neutron_test_3_1_rabbit_failover/

Change abandoned by Ivan Kolodyazhny <email address hidden> on branch: openstack-ci/fuel-8.0/liberty
Review: https://review.fuel-infra.org/16539

We haven't seen in this in the field yet, only on CI (rarely). The proposed fix is too risky to be merged at this stage of the release cycle. We believe it's safe to move this to a MU and continue investigation of this issue.

We've got a hacky workaround. Proper fix [1] requires fix in oslo.messaging to support 'retry' and 'timeout' params per client. For now, these params are global.

[1] https://review.openstack.org/274148 - PoC

Given the fact we haven't actually seen this in the field, I suggest we skip the release notes part.

Also applicable for 7.0
https://mirantis.testrail.com/index.php?/tests/view/4044632

The original issue is definitely the same as #1560097. Since we already tracking all the work there, I marking current bug as duplicate.

Re issue that Egor have found in comment #2 - it is al least a very different issue, but it might be that that one is a wrong test as well. For instance, it is not clear if Egor waited for RabbitMQ to recover after destructive action. Please file a separate issue if you experience it again and also please post snippets of the logs with timestamps indicating when the issue occurred, because it is completely unclear from the comment.