[10.0] [Swarm] Heat typical actions OSTF is failed

neutronclient.v2_0.client: DEBUG: DELETE call to neutron for https://public.fuel.local:9696/v2.0/subnets/7e7b1620-d428-49c2-9d4f-0d35e3724dfe.json used request id req-c567c16a-aac4-48e2-b805-db30d5939ac5
neutronclient.client: DEBUG: REQ: curl -i https://public.fuel.local:9696/v2.0/networks/4f3b7377-d852-449c-a676-3f678c7142f0.json -X DELETE -H "User-Agent: python-neutronclient" -H "X-Auth-Token: {SHA1}46a8637edce16ff28db006fa906c96fce62f344f"
neutronclient.client: DEBUG: RESP: 409 {'Date': 'Thu, 05 Jan 2017 02:52:45 GMT', 'Content-Length': '205', 'Connection': 'close', 'Content-Type': 'application/json; charset=UTF-8', 'X-Openstack-Request-Id': 'req-5018dcd5-ca1e-4d39-afa8-07dac470f94e'} {"NeutronError": {"message": "Unable to complete operation on network 4f3b7377-d852-449c-a676-3f678c7142f0. There are one or more ports still in use on the network.", "type": "NetworkInUse", "detail": ""}}
neutronclient.v2_0.client: DEBUG: Error message: {"NeutronError": {"message": "Unable to complete operation on network 4f3b7377-d852-449c-a676-3f678c7142f0. There are one or more ports still in use on the network.", "type": "NetworkInUse", "detail": ""}}
neutronclient.v2_0.client: DEBUG: DELETE call to neutron for https://public.fuel.local:9696/v2.0/networks/4f3b7377-d852-449c-a676-3f678c7142f0.json used request id req-5018dcd5-ca1e-4d39-afa8-07dac470f94e
fuel_plugin.ostf_adapter.nose_plugin.nose_storage_plugin: ERROR: fuel_health.tests.tests_platform.test_heat.HeatSmokeTests.test_actions
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 67, in testPartExecutor
    yield
  File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 601, in run
    testMethod()
  File "/usr/lib/python2.7/site-packages/fuel_health/tests/tests_platform/test_heat.py", line 243, in test_actions
    template, parameters=parameters
  File "/usr/lib/python2.7/site-packages/fuel_health/common/test_mixins.py", line 180, in verify
    " Please refer to OpenStack logs for more details.")
  File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 666, in fail
    raise self.failureException(msg)
AssertionError: Step 2 failed: Stack was not created properly. Please refer to OpenStack logs for more details.

Original error from ostf log: http://paste.openstack.org/show/594418/

then ostf test starts to cleanup resources while heat stack is still being created -> hence we have conflicts when deleting neutron net. Need to investigate initial failure with heat

Looks like that neutron network is not deleted during cleanup since it is sill used by instance that is created by heat stack. And the instance is missed in the cleanup for some reason.
But it seems to be not the source of the problem in this case.

Initial creation of the stack is failed:

2017-02-09 11:09:49 FAILURE Typical stack actions: create, delete, show details, etc. (fuel_health.tests.tests_platform.test_heat.HeatSmokeTests.test_actions) Stack was not created properly. Please refer to OpenStack logs for more details. File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 67, in testPartExecutor
    yield
  File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 601, in run
    testMethod()
  File "/usr/lib/python2.7/site-packages/fuel_health/tests/tests_platform/test_heat.py", line 243, in test_actions
    template, parameters=parameters
  File "/usr/lib/python2.7/site-packages/fuel_health/common/test_mixins.py", line 180, in verify
    " Please refer to OpenStack logs for more details.")
  File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 666, in fail
    raise self.failureException(msg)
Step 2 failed: Stack was not created properly. Please refer to OpenStack logs for more details.

After that it is impossible to get the stack after creation with error: Unable to retrieve stack.
And it is impossible to delete that stack even if related instance and network are deleted.

Diagnostic snapshot with fresh reproduction is here: https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.services_ha/175/artifact/logs/fail_error_deploy_heat_ha-fuel-snapshot-2017-02-09_02-54-42.tar

Also feel free to ask me for env for reproduction.

Current investigation result:

1. Before [201] build heat tests failed.

2. [201] build image contains https://review.openstack.org/#/c/439653/ patch, which just disable all ceilometer and, suddenly, ALL heat tests, so skipped tests don't fail, of course.

3. After discovery of cases above, Anastasia Kuznetsova pushed patch on review, which enable heat tests and remove "mongo" role from it [1].

4. Patch has been merged, but have a lack with checking ceilometer service availability, so [216] build failed. But @akuznetsova pushed fix for this on review [2].

5. Also, ostf tests, manually started, complete (on1 reverted [216] snapshot).

Conclusion: if after [2] patch heat_ha tests will pass successfully, then I will close this bug as complete. Else continue investigation.

[1] https://review.openstack.org/#/c/448070/

[2] https://review.openstack.org/#/c/448116/

[201] https://product-ci.infra.mirantis.net/view/10.0_swarm/job/10.0.system_test.ubuntu.services_ha/201/

[216] https://product-ci.infra.mirantis.net/view/10.0_swarm/job/10.0.system_test.ubuntu.services_ha/216/

As I said yesterday, I will close bug, if [217] build completes successful. [217] is green => I close this bug.

[217] https://product-ci.infra.mirantis.net/view/10.0_swarm/job/10.0.system_test.ubuntu.services_ha/217/