Comment 0 for bug 1613724

Revision history for this message
Ilya Bumarskov (ibumarskov) wrote : Can't create instance in nova az after shutdown primary controller after scale environment.

Fuel 9.1 (snapshot-id#138) with DVS plugin 3.0.0

Detailed bug description:
OSTF failed on step "Create instance".

Steps to reproduce:
   - Create env with vCenter and DVS plugin
   - Add following nodes:
         * Controller
         * Compute + Cinder
         * Cinder-vmware
         * Compute-vmware
   - Configure VMware vCenter Settings. Add 2 vSphere clusters and configure Nova Compute instances on controller and compute-vmware.
   - Deploy env
   - Add two nodes with controller role.
   - Deploy changes
   - Shutdown primary controller
   - Run OSTF

Observed behavior:
Instance in nova az can't be created.
Nova-compute logs:

nova.compute.manager [req-a6012749-c2bd-4caa-b7e5-dda2dcb40d34 629f4e545c0e499c8450c890439e6e1f 58c4468a004e44888bf6723925513638 - - -] Instance failed network setup after 1 attempt(s) 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager Traceback (most recent call last): 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1570, in _allocate_network_async 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager bind_host_id=bind_host_id) 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 668, in allocate_for_instance 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager self._delete_ports(neutron, instance, created_port_ids) 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager self.force_reraise() 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager six.reraise(self.type_, self.value, self.tb) 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 657, in allocate_for_instance 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager security_group_ids, available_macs, dhcp_opts) 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 259, in _create_port 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager raise exception.PortBindingFailed(port_id=port_id) 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager PortBindingFailed: Binding failed for port bce50f48-71f5-4044-ad1a-89f2301aeb16,

Neutron-server logs:

neutron.plugins.ml2.managers [req-0442810d-f7a2-4260-9183-0a7788cee79d 39c82b217fc5421ca668297e97b0aed8 d5f913b567b84343bcd98abe18ea3603 - - -] Failed to bind port bce50f48-71f5-4044-ad1a-89f2301aeb16 on host node-1.test.domain.local for vnic_type normal using segments [{'segmentation_id': 1000L, 'physical_network': u'physnet2', 'id': u'5bbb93c3-bc9e-4094-ac1c-114ea342046f', 'network_type': u'vlan'}]

root@node-5:~# neutron agent-list
+--------------------------------------+--------------------+--------------------------+-------------------+-------+----------------+---------------------------+
| id | agent_type | host | availability_zone | alive | admin_state_up | binary |
+--------------------------------------+--------------------+--------------------------+-------------------+-------+----------------+---------------------------+
| 07348e67-48cb-4739-afdc-cc2b813196ab | L3 agent | node-6.test.domain.local | nova | :-) | True | neutron-l3-agent |
| 1811da1a-58ed-4229-a0ed-432d5b9217e6 | DVS agent | vcenter-srv_cluster1 | | :-) | True | neutron-dvs-agent |
| 1951a29e-dc47-42f9-a882-396090c71942 | DHCP agent | node-2.test.domain.local | nova | xxx | True | neutron-dhcp-agent |
| 615d66ff-572d-4519-9446-fad3322e1304 | Metadata agent | node-6.test.domain.local | | :-) | True | neutron-metadata-agent |
| 6ed8a8c6-b6c3-48ce-9540-8225c296223c | L3 agent | node-5.test.domain.local | nova | :-) | True | neutron-l3-agent |
| 8183f0ac-ad4c-41c7-97b9-93515f4bc868 | Open vSwitch agent | node-6.test.domain.local | | :-) | True | neutron-openvswitch-agent |
| 8ec51ad3-40b9-40bf-9f41-d8d82830b8c2 | DVS agent | vcenter-srv_cluster2 | | :-) | True | neutron-dvs-agent |
| 973bc2e9-29fc-4e3c-91d6-bdfedeef4624 | Open vSwitch agent | node-2.test.domain.local | | xxx | True | neutron-openvswitch-agent |
| 975ca510-c505-4a41-8390-0f1c6f9a566a | Open vSwitch agent | node-5.test.domain.local | | :-) | True | neutron-openvswitch-agent |
| a80bb7ba-deb8-4c81-a5d0-aba0a1554d7d | DHCP agent | node-6.test.domain.local | nova | :-) | True | neutron-dhcp-agent |
| b0ef01a6-e5ca-453b-9380-b0c20d3aea48 | Open vSwitch agent | node-1.test.domain.local | | xxx | True | neutron-openvswitch-agent |
| b5d9e95c-b145-48ad-a3e0-3488b4ec47ac | L3 agent | node-2.test.domain.local | nova | xxx | True | neutron-l3-agent |
| bab1ee1e-4a5b-4232-8801-1ee0ea384899 | Open vSwitch agent | node-4.test.domain.local | | xxx | True | neutron-openvswitch-agent |
| c0b37a63-11f6-4e5d-bc0d-54926bef7107 | Metadata agent | node-5.test.domain.local | | :-) | True | neutron-metadata-agent |
| e3fd635d-26b6-4d07-9231-e59d5fb54d8c | Metadata agent | node-2.test.domain.local | | xxx | True | neutron-metadata-agent |
| e7a377d3-cad3-49b8-a98f-66b3001a866c | DHCP agent | node-5.test.domain.local | nova | :-) | True | neutron-dhcp-agent |
+--------------------------------------+--------------------+--------------------------+-------------------+-------+----------------+---------------------------+