Vcenter-as-compute: 14-04/Mitaka : SM provisioning broken @ create contrail VM

Bug #1716285 reported by Sarath
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R4.0
Fix Committed
Critical
kamlesh parmar
Trunk
Fix Committed
Critical
kamlesh parmar

Bug Description

Version: 4.0.1.0-69-mitaka
Topology: 3node HA with multiple computes (multi-cluster Esxi) & Kvm

This is regression bug which used to work in #50 build last tested.
this looks broken in Mitaka possibly due to below bug commit for Newton,
---------
add unverified ssl context for urllib request. This is required for
xenial. Daemonize the keepalive thread.

Change-Id: Id2a8d8447209a29e6f35c3d2dcc44bee84bb795f
Closes-Bug: #1714854
------------

Please find below the debugging / Tracebacks,

"2017-09-10 01:49:26,812-INFO-sm_ansible_callback.py:43-append(): TASK [vcenter : Create contrail vm in the esxi host {'username': u'root', 'datacenter': u'A6s32-VC-Compute-Sanity', 'std_switch_list': [{'pg_name': u'mgmt-pg', 'switch_name': u'vSwitch0'}], 'name': u'10.87.36.29', 'contrail_vm': {'vmdk': u'/root/vmdk/vmdk.tar', 'mode': u'vcenter', 'mgmt_pg': u'mgmt-pg', 'networks': [{'mac': u'00:77:56:aa:bb:03', 'sw_type': u'standard', 'role': u'mgmt', 'switch_name': u'vSwitch0', 'pg': u'mgmt-pg'}]}, 'cluster': u'cluster-2', 'id_in_sm': u'contrailvm-5a10s31', 'datastore': u'datastore4', 'password': u'c0ntrail123', 'validate_certs': False, 'skip_reimage': u'true', 'vcenter_server': {'username': <email address hidden>', 'datacentername': u'A6s32-VC-Compute-Sanity', 'password': u'Contrail123!', 'vcenter_compute_ip': u'10.87.36.27', 'hostname': u'10.87.131.15', 'dv_port_group_mgmt': None, 'dv_switch_control_data': {'dv_switch_name': None}, 'dv_switch_mgmt': {'dv_switch_name': None}, 'clusternames': [u'cluster-2'], 'dv_switch': {'dv_switch_name': u'Sanity-dvswitch-2'}, 'dv_port_group_control_data': None, 'validate_certs': False, 'dv_port_group': {'dv_portgroup_name': u'Sanity-dvportgroup-2', 'number_of_ports': u'1800'}}}]"
"2017-09-10 01:49:27,938-INFO-sm_ansible_callback.py:43-append(): changed: [localhost]"
"2017-09-10 01:49:27,950-INFO-sm_ansible_callback.py:43-append(): TASK [vcenter : Save task results {u'started': 1, u'finished': 0, u'results_file': u'/root/.ansible_async/247251544613.23958', u'ansible_job_id': u'247251544613.23958', 'changed': True}]"
"2017-09-10 01:49:27,989-INFO-sm_ansible_callback.py:43-append(): ok: [localhost]"
"2017-09-10 01:49:28,004-INFO-sm_ansible_callback.py:43-append(): TASK [vcenter : Wait for contrailVm create completion [{u'started': 1, u'finished': 0, u'results_file': u'/root/.ansible_async/515287490246.23884', u'ansible_job_id': u'515287490246.23884', 'changed': True}, {u'started': 1, u'finished': 0, u'results_file': u'/root/.ansible_async/779700746384.23927', u'ansible_job_id': u'779700746384.23927', 'changed': True}, {u'started': 1, u'finished': 0, u'results_file': u'/root/.ansible_async/247251544613.23958', u'ansible_job_id': u'247251544613.23958', 'changed': True}]]"
"2017-09-10 01:49:32,739-INFO-server_mgr_ssh_client.py:57-connect(): CONNECT FAILED: Host => 10.87.36.20, option => password, ERROR => [Errno None] Unable to connect to port 22 on or 10.87.36.20"
"2017-09-10 01:49:32,739-ERROR-server_mgr_mon_base_plugin.py:464-copy_ssh_keys_to_server(): COPY-KEYS: Host : contrailvm-5a10s25 Try: 100: ERROR Copying Keys: [Errno None] Unable to connect to port 22 on or 10.87.36.20"
"2017-09-10 01:49:33,369-INFO-sm_ansible_callback.py:43-append(): fatal: [localhost]/usr/lib/python2.7/dist-packages/urllib3/util/ssl_.py:315: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
/usr/lib/python2.7/dist-packages/urllib3/util/ssl_.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:793: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
Traceback (most recent call last):
  File "/opt/contrail/server_manager/deploy_contrail_vm.py", line 260, in <module>
    exit(main())
  File "/opt/contrail/server_manager/deploy_contrail_vm.py", line 247, in main
    response = urllib2.urlopen(req, context = context)
UnboundLocalError: local variable 'context' referenced before assignment"
"2017-09-10 01:49:33,523-INFO-sm_ansible_callback.py:43-append(): fatal: [localhost]/usr/lib/python2.7/dist-packages/urllib3/util/ssl_.py:315: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
/usr/lib/python2.7/dist-packages/urllib3/util/ssl_.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:793: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
Traceback (most recent call last):
  File "/opt/contrail/server_manager/deploy_contrail_vm.py", line 260, in <module>
    exit(main())
  File "/opt/contrail/server_manager/deploy_contrail_vm.py", line 247, in main
    response = urllib2.urlopen(req, context = context)
UnboundLocalError: local variable 'context' referenced before assignment"
"2017-09-10 01:49:33,686-INFO-sm_ansible_callback.py:43-append(): fatal: [localhost]/usr/lib/python2.7/dist-packages/urllib3/util/ssl_.py:315: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
/usr/lib/python2.7/dist-packages/urllib3/util/ssl_.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:793: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
Traceback (most recent call last):
  File "/opt/contrail/server_manager/deploy_contrail_vm.py", line 260, in <module>
    exit(main())
  File "/opt/contrail/server_manager/deploy_contrail_vm.py", line 247, in main
    response = urllib2.urlopen(req, context = context)
UnboundLocalError: local variable 'context' referenced before assignment"
"2017-09-10 01:49:33,689-INFO-sm_ansible_callback.py:43-append(): fatal: [localhost]: FAILED! => (item - None) {"changed": true, "failed": true, "msg": "One or more items failed"}"
"2017-09-10 01:49:33,694-INFO-sm_ansible_utils.py:473-send_REST_request(): Sending post request to http://10.87.36.15:9002/ansible_status?server_id=localhost&state=provision_failed"
"2017-09-10 01:49:33,697-DEBUG-server_mgr_status.py:111-put_ansible_status(): Server status Data provision_failed localhost 2017_09_10__01_49_33"
"2017-09-10 01:49:33,699-INFO-sm_ansible_callback.py:43-append(): PLAY RECAP [localhost] : ok: 70 changed: 18 unreachable: 0 skipped: 17 failed: 1"
"2017-09-10 01:49:33,702-INFO-sm_ansible_utils.py:473-send_REST_request(): Sending post request to http://10.87.36.15:9002/ansible_status?server_id=10.87.36.10&state=provision_failed"
"2017-09-10 01:49:33,704-DEBUG-server_mgr_status.py:132-put_ansible_status(): Server status Data 5a10s31 provision_failed 2017_09_10__01_49_33"
"2017-09-10 01:49:33,828-DEBUG-server_mgr_status.py:135-put_ansible_status(): ######### cluster is cluster-vcenter-compute ##########"
"2017-09-10 01:49:33,831-INFO-sm_ansible_utils.py:473-send_REST_request(): Sending post request to http://10.87.36.15:9002/ansible_status?server_id=10.87.36.11&state=provision_failed"
"2017-09-10 01:49:33,834-DEBUG-server_mgr_status.py:132-put_ansible_status(): Server status Data 5a10s30 provision_failed 2017_09_10__01_49_33"
"2017-09-10 01:49:33,962-DEBUG-server_mgr_status.py:135-put_ansible_status(): ######### cluster is cluster-vcenter-compute ##########"
"2017-09-10 01:49:33,964-INFO-server_mgr_ssh_client.py:57-connect(): CONNECT FAILED: Host => 10.87.36.21, option => password, ERROR => [Errno None] Unable to connect to port 22 on or 10.87.36.21"
"2017-09-10 01:49:33,965-ERROR-server_mgr_mon_base_plugin.py:464-copy_ssh_keys_to_server(): COPY-KEYS: Host : contrailvm-5a10s31 Try: 100: ERROR Copying Keys: [Errno None] Unable to connect to port 22 on or 10.87.36.21"
"2017-09-10 01:49:33,965-INFO-sm_ansible_utils.py:473-send_REST_request(): Sending post request to http://10.87.36.15:9002/ansible_status?server_id=10.87.36.12&state=provision_failed"
"2017-09-10 01:49:33,967-DEBUG-server_mgr_status.py:132-put_ansible_status(): Server status Data 5a10s29 provision_failed 2017_09_10__01_49_33"
"2017-09-10 01:49:34,096-DEBUG-server_mgr_status.py:135-put_ansible_status(): ######### cluster is cluster-vcenter-compute ##########"
"2017-09-10 01:49:34,099-INFO-sm_ansible_utils.py:473-send_REST_request(): Sending post request to http://10.87.36.15:9002/ansible_status?server_id=10.87.36.18&state=provision_failed"
"2017-09-10 01:49:34,101-DEBUG-server_mgr_status.py:132-put_ansible_status(): Server status Data 5a10s23 provision_failed 2017_09_10__01_49_34"
"2017-09-10 01:49:34,221-DEBUG-server_mgr_status.py:135-put_ansible_status(): ######### cluster is cluster-vcenter-compute ##########"
"2017-09-10 01:49:34,224-INFO-sm_ansible_utils.py:473-send_REST_request(): Sending post request to http://10.87.36.15:9002/ansible_status?server_id=10.87.36.19&state=provision_failed"
"2017-09-10 01:49:34,227-DEBUG-server_mgr_status.py:132-put_ansible_status(): Server status Data contrailvm-5a10s27 provision_failed 2017_09_10__01_49_34"
"2017-09-10 01:49:34,347-DEBUG-server_mgr_status.py:135-put_ansible_status(): ######### cluster is cluster-vcenter-compute ##########"
"2017-09-10 01:49:34,350-INFO-sm_ansible_utils.py:473-send_REST_request(): Sending post request to http://10.87.36.15:9002/ansible_status?server_id=10.87.36.20&state=provision_failed"
"2017-09-10 01:49:34,353-DEBUG-server_mgr_status.py:132-put_ansible_status(): Server status Data contrailvm-5a10s25 provision_failed 2017_09_10__01_49_34"
"2017-09-10 01:49:34,687-DEBUG-server_mgr_status.py:135-put_ansible_status(): ######### cluster is cluster-vcenter-compute ##########"
"2017-09-10 01:49:34,690-INFO-sm_ansible_utils.py:473-send_REST_request(): Sending post request to http://10.87.36.15:9002/ansible_status?server_id=10.87.36.21&state=provision_failed"
"2017-09-10 01:49:34,692-DEBUG-server_mgr_status.py:132-put_ansible_status(): Server status Data contrailvm-5a10s31 provision_failed 2017_09_10__01_49_34"
"2017-09-10 01:49:34,807-DEBUG-server_mgr_status.py:135-put_ansible_status(): ######### cluster is cluster-vcenter-compute ##########"
"2017-09-10 01:49:34,810-INFO-sm_ansible_utils.py:473-send_REST_request(): Sending post request to http://10.87.36.15:9002/ansible_status?server_id=10.87.36.27&state=provision_failed"
"2017-09-10 01:49:34,812-DEBUG-server_mgr_status.py:132-put_ansible_status(): Server status Data 5a10s01b provision_failed 2017_09_10__01_49_34"
"2017-09-10 01:49:34,924-DEBUG-server_mgr_status.py:135-put_ansible_status(): ######### cluster is cluster-vcenter-compute ##########"
"2017-09-10 01:49:34,927-ERROR-sm_ansible_playbook.py:778-run_playbook(): Playbook Failed: /opt/contrail/server_manager/ansible/playbooks/contrail_4_0_1_50_vc_new/playbooks/site.yml"
"2017-09-10 01:49:34,937-INFO-sm_ansible_server.py:42-run(): Process Done"
"2017-09-10 01:49:38,691-INFO-server_mgr_ssh_client.py:57-connect(): CONNECT FAILED: Host => 10.87.36.19, option => password, ERROR => [Errno None] Unable to connect to port 22 on or 10.87.36.19"
"2017-09-10 01:49:38,691-ERROR-server_mgr_mon_base_plugin.py:464-copy_ssh_keys_to_server(): COPY-KEYS: Host : contrailvm-5a10s27 Try: 100: ERROR Copying Keys: [Errno None] Unable to connect to port 22 on or 10.87.36.19"
"2017-09-10 01:49:39,669-INFO-server_mgr_ssh_client.py:62-connect(): CONNECT SUCCESS: Host => 10.87.36.11, option => key"
"2017-09-10 01:50:17,688-INFO-server_mgr_ssh_client.py:62-connect(): CONNECT SUCCESS: Host => 10.87.36.10, option => key"
"2017-09-10 01:50:35,695-INFO-server_mgr_ssh_client.py:57-connect(): CONNECT FAILED: Host => 10.87.36.19, option => password, ERROR => [Errno None] Unable to connect to port 22 on or 10.87.36.19"
"2017-09-10 01:50:35,695-ERROR-server_mgr_mon_base_plugin.py:464-copy_ssh_keys_to_server(): COPY-KEYS: Host : contrailvm-5a10s27 Try: 28: ERROR Copying Keys: [Errno None] Unable to connect to port 22 on or 10.87.36.19"
"2017-09-10 01:50:36,315-INFO-server_mgr_ssh_client.py:57-connect(): CONNECT FAILED: Host => 10.87.36.20, option => password, ERROR => [Errno None] Unable to connect to port 22 on or 10.87.36.20"
"2017-09-10 01:50:36,315-ERROR-server_mgr_mon_base_plugin.py:464-copy_ssh_keys_to_server(): COPY-KEYS: Host : contrailvm-5a10s25 Try: 28: ERROR Copying Keys: [Errno None] Unable to connect to port 22 on or 10.87.36.20"
"2017-09-10 01:50:36,879-INFO-server_mgr_ssh_client.py:57-connect(): CONNECT FAILED: Host => 10.87.36.21, option => password, ERROR => [Errno None] Unable to connect to port 22 on or 10.87.36.21"
"2017-09-10 01:50:36,879-ERROR-server_mgr_mon_base_plugin.py:464-copy_ssh_keys_to_server(): COPY-KEYS: Host : contrailvm-5a10s31 Try: 28: ERROR Copying Keys: [Errno None] Unable to connect to port 22 on or 10.87.36.21"
"2017-09-10 01:50:58,511-INFO-server_mgr_ssh_client.py:57-connect(): CONNECT FAILED: Host => 10.87.36.20, option => key, ERROR => [Errno None] Unable to connect to port 22 on or 10.87.36.20"
"2017-09-10 01:51:35,739-INFO-server_mgr_ssh_client.py:57-connect(): CONNECT FAILED: Host => 10.87.36.20, option => password, ERROR => [Errno None] Unable to connect to port 22 on or 10.87.36.20"
"2017-09-10 01:51:35,740-ERROR-server_mgr_mon_base_plugin.py:464-copy_ssh_keys_to_server(): COPY-KEYS: Host : contrailvm-5a10s25 Try: 101: ERROR Copying Keys: [Errno None] Unable to connect to port 22 on or 10.87.36.20"
"2017-09-10 01:51:36,511-INFO-server_mgr_ssh_client.py:57-connect(): CONNECT FAILED: Host => 10.87.36.19, option => key, ERROR => [Errno None] Unable to connect to port 22 on or 10.87.36.19"
"2017-09-10 01:51:36,971-INFO-server_mgr_ssh_client.py:57-connect(): CONNECT FAILED: Host => 10.87.36.21, option => password, ERROR => [Errno None] Unable to connect to port 22 on or 10.87.36.21"
"2017-09-10 01:51:36,972-ERROR-server_mgr_mon_base_plugin.py:464-copy_ssh_keys_to_server(): COPY-KEYS: Host : contrailvm-5a10s31 Try: 101: ERROR Copying Keys: [Errno None] Unable to connect to port 22 on or 10.87.36.21"
"2017-09-10 01:51:39,511-INFO-server_mgr_ssh_client.py:57-connect(): CONNECT FAILED: Host => 10.87.36.19, option => password, ERROR => [Errno None] Unable to connect to port 22 on or 10.87.36.19"
"2017-09-10 01:51:39,511-ERROR-server_mgr_mon_base_plugin.py:464-copy_ssh_keys_to_server(): COPY-KEYS: Host : contrailvm-5a10s27 Try: 101: ERROR Copying Keys: [Errno None] Unable to connect to port 22 on or 10.87.36.19"
"2017-09-10 01:52:11,511-DEBUG-server_mgr_mon_base_plugin.py:706-create_server_dict(): Created server dictionary."
"2017-09-10 01:52:11,665-INFO-server_mgr_ssh_client.py:62-connect(): CONNECT SUCCESS: Host => 10.87.36.18, option => key"
"2017-09-10 01:52:38,695-INFO-server_mgr_ssh_client.py:57-connect(): CONNECT FAILED: Host => 10.87.36.19, option => password, ERROR => [Errno None] Unable to connect to port 22 on or 10.87.36.19"
"2017-09-10 01:52:38,695-ERROR-server_mgr_mon_base_plugin.py:464-copy_ssh_keys_to_server(): COPY-KEYS: Host : contrailvm-5a10s27 Try: 29: ERROR Copying Keys: [Errno None] Unable to connect to port 22 on or 10.87.36.19"

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.0

Review in progress for https://review.opencontrail.org/35455
Submitter: kamlesh parmar (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/35456
Submitter: kamlesh parmar (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/35455
Committed: http://github.com/Juniper/contrail-server-manager/commit/bd41339e8d6d8b76e7de3495d540b512f58ba670
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit bd41339e8d6d8b76e7de3495d540b512f58ba670
Author: Kamlesh Parmar <email address hidden>
Date: Sun Sep 10 15:33:38 2017 -0700

no unverified context reqd for 1404

Change-Id: I2e51f621430736228e6b2a3ee86963867fac25ba
Closes-Bug: #1716285

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/35456
Committed: http://github.com/Juniper/contrail-server-manager/commit/0b31a162b5e5d873331d83e9b800ee4828a948dd
Submitter: Zuul (<email address hidden>)
Branch: master

commit 0b31a162b5e5d873331d83e9b800ee4828a948dd
Author: Kamlesh Parmar <email address hidden>
Date: Sun Sep 10 15:33:38 2017 -0700

no unverified context reqd for 1404

Change-Id: I2e51f621430736228e6b2a3ee86963867fac25ba
Closes-Bug: #1716285

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.