provision_config_node script fails as it tried connecting before contrail-api came up

R4.0 Build 10 Ubuntu 16.04.2 multi controller container setup

On two of the three controller containers, the provision_config_node.py script was run before contrail-api was up on 8082 and thus, it failed.

The provision script was run at May 23 06:25:46
But contrail-api seems to have started listening around May 23 06:26:16

contrail-api logs are attached

May 23 06:25:23 nodec2 contrailctl[26]: TASK [contrail/config : include] ***********************************************
May 23 06:25:23 nodec2 contrailctl[26]: included: /contrail-ansible-internal/playbooks/roles/contrail/config/tasks/provision/main.yml for localhost
May 23 06:25:23 nodec2 contrailctl[26]: TASK [contrail/config : Include cloud_orchestrator specific code to provision config objects] ***
May 23 06:25:23 nodec2 contrailctl[26]: included: /contrail-ansible-internal/playbooks/roles/contrail/config/tasks/provision/noauth.yml for localhost
May 23 06:25:23 nodec2 contrailctl[26]: TASK [contrail/config : Wait till config api server answers before creating config objects] ***
May 23 06:25:24 nodec2 contrailctl[26]: FAILED - RETRYING: TASK: contrail/config : Wait till config api server answers before creating config objects (300 retries left).
May 23 06:25:25 nodec2 contrailctl[26]: FAILED - RETRYING: TASK: contrail/config : Wait till config api server answers before creating config objects (299 retries left).
May 23 06:25:27 nodec2 contrailctl[26]: FAILED - RETRYING: TASK: contrail/config : Wait till config api server answers before creating config objects (298 retries left).
May 23 06:25:28 nodec2 contrailctl[26]: FAILED - RETRYING: TASK: contrail/config : Wait till config api server answers before creating config objects (297 retries left).
May 23 06:25:30 nodec2 contrailctl[26]: FAILED - RETRYING: TASK: contrail/config : Wait till config api server answers before creating config objects (296 retries left).
May 23 06:25:31 nodec2 contrailctl[26]: FAILED - RETRYING: TASK: contrail/config : Wait till config api server answers before creating config objects (295 retries left).
May 23 06:25:32 nodec2 contrailctl[26]: FAILED - RETRYING: TASK: contrail/config : Wait till config api server answers before creating config objects (294 retries left).
May 23 06:25:34 nodec2 contrailctl[26]: FAILED - RETRYING: TASK: contrail/config : Wait till config api server answers before creating config objects (293 retries left).
May 23 06:25:35 nodec2 contrailctl[26]: FAILED - RETRYING: TASK: contrail/config : Wait till config api server answers before creating config objects (292 retries left).
May 23 06:25:37 nodec2 contrailctl[26]: FAILED - RETRYING: TASK: contrail/config : Wait till config api server answers before creating config objects (291 retries left).
May 23 06:25:38 nodec2 contrailctl[26]: FAILED - RETRYING: TASK: contrail/config : Wait till config api server answers before creating config objects (290 retries left).
May 23 06:25:39 nodec2 contrailctl[26]: FAILED - RETRYING: TASK: contrail/config : Wait till config api server answers before creating config objects (289 retries left).
May 23 06:25:41 nodec2 contrailctl[26]: FAILED - RETRYING: TASK: contrail/config : Wait till config api server answers before creating config objects (288 retries left).
May 23 06:25:42 nodec2 contrailctl[26]: FAILED - RETRYING: TASK: contrail/config : Wait till config api server answers before creating config objects (287 retries left).
May 23 06:25:44 nodec2 contrailctl[26]: FAILED - RETRYING: TASK: contrail/config : Wait till config api server answers before creating config objects (286 retries left).
May 23 06:25:45 nodec2 contrailctl[26]: ok: [localhost]
May 23 06:25:45 nodec2 contrailctl[26]: TASK [contrail/config : register config node to config api server] *************
May 23 06:25:46 nodec2 contrailctl[26]: fatal: [localhost]: FAILED! => {"changed": true, "cmd": "/usr/share/contrail-utils/provision_config_node.py --api_server_ip 127.0.0.1 --host_name nodec2 --host_ip 10.204.216.59 --admin_user admin --admin_password admin --admin_tenant_name admin --oper add", "delta": "0:00:00.415309", "end": "2017-05-23 06:25:45.993293", "failed": true, "rc": 1, "start": "2017-05-23 06:25:45.577984", "stderr": "Traceback (most recent call last):\n File \"/usr/share/contrail-utils/provision_config_node.py\", line 172, in <module>\n main()\n File \"/usr/share/contrail-utils/provision_config_node.py\", line 168, in main\n ConfigNodeProvisioner(args_str)\n File \"/usr/share/contrail-utils/provision_config_node.py\", line 35, in __init__\n api_server_use_ssl=self._args.api_server_use_ssl)\n File \"/usr/share/contrail-utils/vnc_admin_api.py\", line 35, in __init__\n auth_host=self.auth_host)\n File \"/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py\", line 449, in __init__\n retry_on_error=False)\n File \"/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py\", line 914, in _request\n raise ConnectionError\nrequests.exceptions.ConnectionError", "stdout": "", "stdout_lines": [], "warnings": []}
May 23 06:25:46 nodec2 contrailctl[26]: to retry, use: --limit @/contrail-ansible-internal/playbooks/contrail_controller.retry
May 23 06:25:46 nodec2 contrailctl[26]: PLAY RECAP *********************************************************************
May 23 06:25:46 nodec2 contrailctl[26]: localhost : ok=96 changed=43 unreachable=0 failed=1
May 23 06:25:46 nodec2 systemd[1]: contrail-ansible.service: Main process exited, code=exited, status=1/FAILURE
May 23 06:25:46 nodec2 systemd[1]: Failed to start Contrail controller configuration.
May 23 06:25:48 nodec2 systemd[1]: contrail-ansible.service: Unit entered failed state.
May 23 06:25:48 nodec2 systemd[1]: contrail-ansible.service: Failed with result 'exit-code'.