Stack creation is failed due to Resource CREATE failed: WaitConditionTimeout: 0 of 5 received

Bug https://bugs.launchpad.net/mos/+bug/1497273 was verified on MOS 7.0 + MU6 updates on cluster without TLS (despite of fact that issue from description was observed several times). After this bug will be fixed, bug 1497273 also should be checked on cluster with TLS.

Fix proposed to branch: openstack-ci/fuel-8.0/liberty
Change author: danny <email address hidden>
Review: https://review.fuel-infra.org/28088

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: danny <email address hidden>
Review: https://review.fuel-infra.org/28089

Here is an upstream bug: https://bugs.launchpad.net/heat/+bug/1594441
As we can see it was fixed in Mitaka, so 9.x branch already contains the patch. That why status for 9.2 has been set as Invalid.

Note for QA: it's necessary to add next configuration to /etc/heat/heat.conf in order to changes make effect and restart heat-engine:

[clients_heat]
url=https://<Public Endpoing IP>:8004/v1/%(tenant_id)s
insecure = True

Sergii, I want to note, that using this option in config in production is not secure obviously, because it disables SSL everywhere in Heat, not only for Wait conditions.

So you may use it for testing, but not for production state.
Also I several times repeated, that it's expected behavior.

Wait condition should not work with SSL by default without any changes, because it requires SSl certificates on the booted VM. So there are several options how to deal with it in production:
 - enable TLS:
    a. Put ssl certificates to VM - it should be done by user and it's not so obvious, because these SSL certificates should be the same as on Contollers/computes of Openstack deployment
    b. manually set "--insecure" flag inside of Heat template (if we take a look on the https://paste.mirantis.net/show/2712/ this flag should be added in each line from 36 to 40).
    c. Use insecure option in config, as suggested in previous comment from Sergii. However he you need to take in mind, that it's real insecure - it disables SSl for whole Heat, not only waitconditions.
 - disable TLS:
     use mentioned template and Heat without changes :)

P.s. I will approve backports for proposed patch, it will allow to support case "c" in production.

Reviewed: https://review.fuel-infra.org/28088
Submitter: Pkgs Jenkins <email address hidden>
Branch: openstack-ci/fuel-8.0/liberty

Commit: e0a538e51f36847523d5aff6bd8ffb224643b8ce
Author: danny <email address hidden>
Date: Tue Nov 1 18:24:20 2016

Add --insecure in CURL if set True in client option

The CURL attribute is offen used as part of
the user data to signal back to heat in wait
handle. If the insecure option is set True in
the clients_heat option, we need to add
--insecure to the CURL url to make it work.

Change-Id: I153a9c71837ee61632e0cf39254bbbc66427b1de
Closes-Bug: #1636428

Reviewed: https://review.fuel-infra.org/28089
Submitter: Denis V. Meltsaykin <email address hidden>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: 81838f83ccf38c2097b6399084a30bb3fdf87415
Author: danny <email address hidden>
Date: Tue Nov 1 18:34:00 2016

Add --insecure in CURL if set True in client option

The CURL attribute is offen used as part of
the user data to signal back to heat in wait
handle. If the insecure option is set True in
the clients_heat option, we need to add
--insecure to the CURL url to make it work.

Change-Id: I153a9c71837ee61632e0cf39254bbbc66427b1de
Closes-Bug: #1636428

Verified on MOS 7.0 + MU7 updates.

Actual results:
After the fix it's possible to create stack for cluster with Enabled TLS if to add next configuration to /etc/heat/heat.conf and restart heat-engine:
   [clients_heat]
   url=https://<Public Endpoing IP>:8004/v1/%(tenant_id)s
   insecure = True

Verified on MOS 8.0 + MU4 updates.

Actual results:
After the fix it's possible to create stack for cluster with Enabled TLS if to add next configuration to /etc/heat/heat.conf and restart heat-engine:
   [clients_heat]
   url=https://<Public Endpoing IP>:8004/v1/%(tenant_id)s
   insecure = True

“Heat scripts are failing in WaitCondition“

“Create_Failed: Resource CREATE failed: WaitConditionTimeout: resources.serverpair.resources[7].resources.serverpair.resources[1].resources.waitcondition: 0 of 1 received”

1. Fuel 7.0 MU 7 is used for deployment in openstack kilo with http endpoints
2. 3 controller cluster
3. All nodes have heat_metadata_server_url and heat_waitcondition_server_url pointed to public routable IP
4. Instances getting created successfully,stack is failing in wait condition.
5. 401 message seen in heat engine logs

Instance console log: -

+ echo 'Hello, World!'
+ cfn-create-aws-symlinks
+ cfn-signal -e 0 -r 'FINISH!!' 'http://xxxxxxxx:8000/v1/waitcondition/arn%3Aopenstack%3Aheat%3A%3A876e627764f742238cff5ab5e7b814f8%3Astacks%2Fsfsdfsf%2Fe0e28440-2670-41d7-8209-ad9117f22149%2Fresources%2Fwait_handle?Timestamp=2017-04-07T04%3A30%3A51Z&SignatureMethod=HmacSHA256&AWSAccessKeyId=5d1f9faab31d4947a7f55ffe973ebdb7&SignatureVersion=2&Signature=peXuxltNbUb0Yv5sH86TifKRJq4NMkiBBQBoQ0ZGfQE%3D'
DEBUG [2017-04-07 04:31:53,734] cfn-signal called Namespace(data='Application has completed configuration.', exit=None, exit_code='0', reason='FINISH!!', success='true', unique_id='00000', url='http://10.142.194.148:8000/v1/waitcondition/arn%3Aopenstack%3Aheat%3A%3A876e627764f742238cff5ab5e7b814f8%3Astacks%2Fsfsdfsf%2Fe0e28440-2670-41d7-8209-ad9117f22149%2Fresources%2Fwait_handle?Timestamp=2017-04-07T04%3A30%3A51Z&SignatureMethod=HmacSHA256&AWSAccessKeyId=5d1f9faab31d4947a7f55ffe973ebdb7&SignatureVersion=2&Signature=peXuxltNbUb0Yv5sH86TifKRJq4NMkiBBQBoQ0ZGfQE%3D')
DEBUG [2017-04-07 04:31:53,734] Running command: curl -X PUT -H 'Content-Type:' --data-binary '{"Status": "SUCCESS", "Reason": "FINISH!!", "Data": "Application has completed configuration.", "UniqueId": "00000"}' "http://xxxxxxxx:8000/v1/waitcondition/arn%3Aopenstack%3Aheat%3A%3A876e627764f742238cff5ab5e7b814f8%3Astacks%2Fsfsdfsf%2Fe0e28440-2670-41d7-8209-ad9117f22149%2Fresources%2Fwait_handle?Timestamp=2017-04-07T04%3A30%3A51Z&SignatureMethod=HmacSHA256&AWSAccessKeyId=5d1f9faab31d4947a7f55ffe973ebdb7&SignatureVersion=2&Signature=peXuxltNbUb0Yv5sH86TifKRJq4NMkiBBQBoQ0ZGfQE%3D"

Heat engine log : -

heat-api-cfn.log:2017-04-07 04:31:54.720 7167 INFO eventlet.wsgi.server [-] 192.168.12.19 - - [07/Apr/2017 04:31:54] "PUT /v1/waitcondition/arn%3Aopenstack%3Aheat%3A%3A876e627764f742238cff5ab5e7b814f8%3Astacks%2Fsfsdfsf%2Fe0e28440-2670-41d7-8209-ad9117f22149%2Fresources%2Fwait_handle?Timestamp=2017-04-07T04%3A30%3A51Z&SignatureMethod=HmacSHA256&AWSAccessKeyId=5d1f9faab31d4947a7f55ffe973ebdb7&SignatureVersion=2&Signature=peXuxltNbUb0Yv5sH86TifKRJq4NMkiBBQBoQ0ZGfQE%3D HTTP/1.1" 401 219 0.232454

Following bug investigation get next results:

I added to raw user_data several test lines: create new "check" file with test string, after that touch new log-file and write curl verbose output to it:

...
      user_data_format: RAW
      user_data:
        str_replace:
          template: |
            #!/bin/sh
            echo "Test!" > /check
            echo "Starting user_data script" > /wc_logs.log
            wc_notify --data-binary '{"status": "SUCCESS"}' --verbose >> /wc_logs.log 2>&1
            wc_notify --data-binary '{"status": "SUCCESS", "reason": "signal2"}' --verbose >> /wc_logs.log 2>&1
            wc_notify --data-binary '{"status": "SUCCESS", "reason": "signal3", "data": "data3"}' --verbose >> /wc_logs.log 2>&1
            wc_notify --data-binary '{"status": "SUCCESS", "reason": "signal4", "data": "data4"}' --verbose >> /wc_logs.log 2>&1
            wc_notify --data-binary '{"status": "SUCCESS", "id": "5"}' --verbose >> /wc_logs.log 2>&1
            wc_notify --data-binary '{"status": "SUCCESS", "id": "5"}' --verbose >> /wc_logs.log 2>&1
            echo "user_data script finished" >> /wc_logs.log
          params:
            wc_notify: { get_attr: ['wait_handle', 'curl_cli'] }
...

Get 7/10 CREATE_COMPLETE stacks. 3 failed stacks interrupted with waitcondition timeout => 0 of 5 signals received. When I ssh to nova instance of failed stack, there are neither "check" file nor "wc_logs.log" file with logs. After some debugging I found that heat correctly send user_data value to nova, but nova do nothing with it (or instance hanging and refuses run user_data script).

Based on this, I can concluse that basic incorrect behaviour is into nova, not heat.

Nova team, please take a look.

Also I'm going to move this bug to 8.0-MU-5 since it will take much more time to investigate and fix.

Moving this bug to backlog because it needs a thorough investigation which doesn't seem like it is worth it right now.

We no longer support MOS5.1, MOS6.0, MOS6.1
We deliver only Critical/Security fixes to MOS7.0, MOS8.0.
We deliver only High/Critical/Security fixes to MOS9.2.