Vcenter-as-compute: SM provisioning fails @ ansible PLAY [Setup ceph compute node]

Version: 4.0.1.0-43
Topology: 3 node contrail + openstack HA

shown the setup in problem state to Development Kamlesh

we don't have any CEPH configs and failure seen as below snippet,

::
"2017-08-23 14:49:01,755-INFO-sm_ansible_callback.py:43-append(): TASK [Reboot node]"
"2017-08-23 14:49:02,976-INFO-sm_ansible_callback.py:43-append(): changed: [10.87.36.19]"
"2017-08-23 14:49:02,994-INFO-sm_ansible_callback.py:43-append(): TASK [Waiting for server to come back]"
"2017-08-23 14:49:03,858-DEBUG-server_mgr_main.py:1272-validate_smgr_request(): validate_smgr_request"
"2017-08-23 14:49:06,220-DEBUG-server_mgr_main.py:1272-validate_smgr_request(): validate_smgr_request"
"2017-08-23 14:49:08,582-DEBUG-server_mgr_main.py:1272-validate_smgr_request(): validate_smgr_request"
"2017-08-23 14:49:09,398-INFO-server_mgr_ssh_client.py:57-connect(): CONNECT FAILED: Host => 10.87.36.19, option => key, ERROR => Authentication failed."
"2017-08-23 14:49:10,979-DEBUG-server_mgr_main.py:1272-validate_smgr_request(): validate_smgr_request"
"2017-08-23 14:49:13,373-DEBUG-server_mgr_main.py:1272-validate_smgr_request(): validate_smgr_request"
"2017-08-23 14:49:15,772-DEBUG-server_mgr_main.py:1272-validate_smgr_request(): validate_smgr_request"
"2017-08-23 14:49:18,135-DEBUG-server_mgr_main.py:1272-validate_smgr_request(): validate_smgr_request"
"2017-08-23 14:49:20,493-DEBUG-server_mgr_main.py:1272-validate_smgr_request(): validate_smgr_request"
"2017-08-23 14:49:22,856-DEBUG-server_mgr_main.py:1272-validate_smgr_request(): validate_smgr_request"
"2017-08-23 14:49:25,254-DEBUG-server_mgr_main.py:1272-validate_smgr_request(): validate_smgr_request"
"2017-08-23 14:49:27,652-DEBUG-server_mgr_main.py:1272-validate_smgr_request(): validate_smgr_request"
"2017-08-23 14:49:30,014-DEBUG-server_mgr_main.py:1272-validate_smgr_request(): validate_smgr_request"
"2017-08-23 14:49:32,382-DEBUG-server_mgr_main.py:1272-validate_smgr_request(): validate_smgr_request"
"2017-08-23 14:49:34,739-DEBUG-server_mgr_main.py:1272-validate_smgr_request(): validate_smgr_request"
"2017-08-23 14:49:37,097-DEBUG-server_mgr_main.py:1272-validate_smgr_request(): validate_smgr_request"
"2017-08-23 14:49:39,496-DEBUG-server_mgr_main.py:1272-validate_smgr_request(): validate_smgr_request"
"2017-08-23 14:49:41,859-DEBUG-server_mgr_main.py:1272-validate_smgr_request(): validate_smgr_request"
"2017-08-23 14:49:42,274-INFO-sm_ansible_callback.py:43-append(): ok: [10.87.36.19 -> localhost]"
"2017-08-23 14:49:42,283-INFO-sm_ansible_callback.py:43-append(): PLAY [Setup ceph compute node]"
"2017-08-23 14:49:42,287-INFO-sm_ansible_utils.py:462-send_REST_request(): Sending post request to http://10.87.36.15:9002/ansible_status?server_id=10.87.36.10&state=provision_failed"
"2017-08-23 14:49:42,290-DEBUG-server_mgr_status.py:132-put_ansible_status(): Server status Data 5a10s31 provision_failed 2017_08_23__14_49_42"
"2017-08-23 14:49:42,412-DEBUG-server_mgr_status.py:135-put_ansible_status(): ######### cluster is cluster-vcenter-compute ##########"
"2017-08-23 14:49:42,416-INFO-sm_ansible_callback.py:43-append(): PLAY RECAP [10.87.36.10] : ok: 79 changed: 6 unreachable: 0 skipped: 78 failed: 1"
"2017-08-23 14:49:42,418-INFO-sm_ansible_utils.py:462-send_REST_request(): Sending post request to http://10.87.36.15:9002/ansible_status?server_id=10.87.36.11&state=provision_failed"
"2017-08-23 14:49:42,421-DEBUG-server_mgr_status.py:132-put_ansible_status(): Server status Data 5a10s30 provision_failed 2017_08_23__14_49_42"
"2017-08-23 14:49:42,513-DEBUG-server_mgr_status.py:135-put_ansible_status(): ######### cluster is cluster-vcenter-compute ##########"
"2017-08-23 14:49:42,516-INFO-sm_ansible_callback.py:43-append(): PLAY RECAP [10.87.36.11] : ok: 79 changed: 6 unreachable: 0 skipped: 78 failed: 1"
"2017-08-23 14:49:42,518-INFO-sm_ansible_utils.py:462-send_REST_request(): Sending post request to http://10.87.36.15:9002/ansible_status?server_id=10.87.36.12&state=provision_completed"
"2017-08-23 14:49:42,521-DEBUG-server_mgr_status.py:132-put_ansible_status(): Server status Data 5a10s29 provision_completed 2017_08_23__14_49_42"
"2017-08-23 14:49:42,622-DEBUG-server_mgr_status.py:135-put_ansible_status(): ######### cluster is cluster-vcenter-compute ##########"
"2017-08-23 14:49:42,625-INFO-sm_ansible_callback.py:43-append(): PLAY RECAP [10.87.36.12] : ok: 142 changed: 17 unreachable: 0 skipped: 211 failed: 0"
"2017-08-23 14:49:42,627-INFO-sm_ansible_utils.py:462-send_REST_request(): Sending post request to http://10.87.36.15:9002/ansible_status?server_id=10.87.36.19&state=provision_completed"
"2017-08-23 14:49:42,630-DEBUG-server_mgr_status.py:132-put_ansible_status(): Server status Data contrailvm-5a10s27 provision_completed 2017_08_23__14_49_42"
"2017-08-23 14:49:42,722-DEBUG-server_mgr_status.py:135-put_ansible_status(): ######### cluster is cluster-vcenter-compute ##########"
"2017-08-23 14:49:42,726-INFO-sm_ansible_callback.py:43-append(): PLAY RECAP [10.87.36.19] : ok: 99 changed: 9 unreachable: 0 skipped: 260 failed: 0"
"2017-08-23 14:49:42,728-INFO-sm_ansible_utils.py:462-send_REST_request(): Sending post request to http://10.87.36.15:9002/ansible_status?server_id=localhost&state=provision_completed"
"2017-08-23 14:49:42,731-DEBUG-server_mgr_status.py:111-put_ansible_status(): Server status Data provision_completed localhost 2017_08_23__14_49_42"
"2017-08-23 14:49:42,733-INFO-sm_ansible_callback.py:43-append(): PLAY RECAP [localhost] : ok: 52 changed: 17 unreachable: 0 skipped: 7 failed: 0"
"2017-08-23 14:49:42,736-INFO-sm_ansible_utils.py:462-send_REST_request(): Sending post request to http://10.87.36.15:9002/ansible_status?server_id=10.87.36.10&state=provision_failed"
"2017-08-23 14:49:42,738-DEBUG-server_mgr_status.py:132-put_ansible_status(): Server status Data 5a10s31 provision_failed 2017_08_23__14_49_42"
"2017-08-23 14:49:42,840-DEBUG-server_mgr_status.py:135-put_ansible_status(): ######### cluster is cluster-vcenter-compute ##########"
"2017-08-23 14:49:42,843-INFO-sm_ansible_utils.py:462-send_REST_request(): Sending post request to http://10.87.36.15:9002/ansible_status?server_id=10.87.36.11&state=provision_failed"
"2017-08-23 14:49:42,845-DEBUG-server_mgr_status.py:132-put_ansible_status(): Server status Data 5a10s30 provision_failed 2017_08_23__14_49_42"
"2017-08-23 14:49:42,940-DEBUG-server_mgr_status.py:135-put_ansible_status(): ######### cluster is cluster-vcenter-compute ##########"
"2017-08-23 14:49:42,943-INFO-sm_ansible_utils.py:462-send_REST_request(): Sending post request to http://10.87.36.15:9002/ansible_status?server_id=10.87.36.12&state=provision_failed"
"2017-08-23 14:49:42,946-DEBUG-server_mgr_status.py:132-put_ansible_status(): Server status Data 5a10s29 provision_failed 2017_08_23__14_49_42"
"2017-08-23 14:49:43,057-DEBUG-server_mgr_status.py:135-put_ansible_status(): ######### cluster is cluster-vcenter-compute ##########"
"2017-08-23 14:49:43,061-INFO-sm_ansible_utils.py:462-send_REST_request(): Sending post request to http://10.87.36.15:9002/ansible_status?server_id=10.87.36.19&state=provision_failed"
"2017-08-23 14:49:43,063-DEBUG-server_mgr_status.py:132-put_ansible_status(): Server status Data contrailvm-5a10s27 provision_failed 2017_08_23__14_49_43"
"2017-08-23 14:49:43,183-DEBUG-server_mgr_status.py:135-put_ansible_status(): ######### cluster is cluster-vcenter-compute ##########"
"2017-08-23 14:49:43,186-ERROR-sm_ansible_playbook.py:751-run_playbook(): Playbook Failed: /opt/contrail/server_manager/ansible/playbooks/contrail_4_0_1_43_vc_new/playbooks/site.yml"
::
::

root@5a10s26:~# server-manager display server --select id,status,ip_address,cluster_id,roles
+--------------------+------------------+-------------+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------+
| id | status | ip_address | cluster_id | roles |
+--------------------+------------------+-------------+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------+
| 5a10s31 | provision_failed | 10.87.36.10 | cluster-vcenter-compute | [u'contrail-controller', u'contrail-analytics', u'contrail-analyticsdb', u'openstack'] |
| 5a10s30 | provision_failed | 10.87.36.11 | cluster-vcenter-compute | [u'contrail-controller', u'contrail-analytics', u'contrail-analyticsdb', u'openstack'] |
| 5a10s29 | provision_failed | 10.87.36.12 | cluster-vcenter-compute | [u'contrail-controller', u'contrail-analytics', u'contrail-analyticsdb', u'contrail-vcenter-plugin', u'contrail-vcenter-compute', u'openstack'] |
| contrailvm-5a10s27 | provision_failed | 10.87.36.19 | cluster-vcenter-compute | [u'contrail-compute'] |
+--------------------+------------------+-------------+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------+