vmware-dvs

[DVS-2.1] Can't launch instances after launch->remove instances of Cluster(compute-vmware).

Bug #1548730 reported by Olesia Tsvigun
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
vmware-dvs
Fix Released
High
Vyacheslav Tabolin
vmware-dvs liberty-agent

Bug Description

Fuel ISO#570

id | name | version | package_version
---|------------------------|---------|----------------
1 | fuel-plugin-vmware-dvs | 2.1.14 | 4.0.0

Description
Can't launch instances after launch->remove instances from CLuster(compute-vmware).

Steps to reproduce
1. Upload plugins to the master node
2. Install plugin.
3. Create cluster with vcenter.
4. Add 1 node with controller role.
5. Add 2 node with compute role.
6. Add 1 node with compute-vmware role.
8. Fill all fields in vmware tab.
9. Fill field 'Enter the Cluster to dvSwitch mapping.'
10. Use the VMware DVS firewall driver is enabled.
11. Deploy the cluster.
12. Create net_1: net01__subnet, 192.168.1.0/24
13. Create SG1 with remote sg rules.
14. Launch 10-20 instances with default+SG1.
15. Remove all instances.
16. Launch new 10-20 instances with default+SG1.

Actual Result
During removing instances from cluster(compute-vmware) error was occured 'Error: Failed to perform requested operation on instance "df-5", the instance has an error status: Please try again later [Error: <html><body><h1>504 Gateway Time-out</h1> The server didn't respond in time. </body></html> ]. ' Try to remove instance again. Instances were removed. Launch 10-20 instances with deafult SG/SG1. Error message was occured
Message
    <html><body><h1>504 Gateway Time-out</h1> The server didn't respond in time. </body></html>
Code
    500
Details
    File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 366, in decorated_function return function(self, context, *args, **kwargs) File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2440, in terminate_instance do_terminate_instance(instance, bdms) File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 254, in inner return f(*args, **kwargs) File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2438, in do_terminate_instance self._set_instance_obj_error_state(context, instance) File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__ six.reraise(self.type_, self.value, self.tb) File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2428, in do_terminate_instance self._delete_instance(context, instance, bdms, quotas) File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 149, in inner rv = f(*args, **kwargs) File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2391, in _delete_instance quotas.rollback() File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__ six.reraise(self.type_, self.value, self.tb) File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2354, in _delete_instance self._shutdown_instance(context, instance, bdms) File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2281, in _shutdown_instance self._try_deallocate_network(context, instance, requested_networks) File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2210, in _try_deallocate_network self._set_instance_obj_error_state(context, instance) File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__ six.reraise(self.type_, self.value, self.tb) File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2205, in _try_deallocate_network self._deallocate_network(context, instance, requested_networks) File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1812, in _deallocate_network context, instance, requested_networks=requested_networks) File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 837, in deallocate_for_instance self._delete_ports(neutron, instance, ports, raise_if_fail=True) File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 811, in _delete_ports raise exceptions[0]

From Neutron logs:
http://paste.openstack.org/show/487868/

Also ports were not cleaned on cluster(compute-vmware). See attachment.

See original description
Tags: dvs
Revision history for this message
Olesia Tsvigun (otsvigun) wrote :
Changed in vmware-dvs:
milestone: none → liberty-agent
assignee: nobody → Vyachestav Tabolin (slavchick)
importance: Undecided → High
tags: added: dvs
description: updated
Revision history for this message
Olesia Tsvigun (otsvigun) wrote :

Scenario2
1. Launch few instances in net1.
2. Detached instance port from net1 and attach to internal default net.
3. Remove instances.
4. Try launch few instances.

Actual Result
During removing instances from cluster(compute-vmware) error was occured
    Build of instance dfe5bc15-85b6-4745-b1aa-4d2ada4787f6 aborted: Could not clean up failed build, not rescheduling
Code
    500
Details
    File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1907, in _do_build_and_run_instance filter_properties) File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2027, in _build_and_run_instance 'create.error', fault=e) File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__ six.reraise(self.type_, self.value, self.tb) File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2011, in _build_and_run_instance block_device_info=block_device_info) File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__ self.gen.throw(type, value, traceback) File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2181, in _build_resources instance_uuid=instance.uuid, reason=msg)
Created
    Feb. 23, 2016, 2:14 p.m.

Vyacheslav Tabolin (slavchick)
Changed in vmware-dvs:
status: New → Incomplete
Olesia Tsvigun (otsvigun)
Changed in vmware-dvs:
status: Incomplete → Confirmed
Revision history for this message
Olesia Tsvigun (otsvigun) wrote :

Issue was reproduced at:
[root@nailgun ~]# fuel plugins list
id | name | version | package_version
---|------------------------|---------|----------------
1 | fuel-plugin-vmware-dvs | 2.1.17 | 4.0.0

Steps to reproduce.
1. Create 10 instances in one batch with 3 SG(default+ sg with remoute rules+sg with remoute rules)

Actual result
All instance of CLuster2(compute-vmware) has failed.

From neutron logs:
http://paste.openstack.org/show/489278/

SlOPS (vtabolin)
Changed in vmware-dvs:
assignee: Vyacheslav Tabolin (slavchick) → Vyacheslav Tabolin (vtabolin)
Revision history for this message
SlOPS (vtabolin) wrote :

Can not reproduce

Changed in vmware-dvs:
status: Confirmed → Incomplete
Vyacheslav Tabolin (slavchick)
Changed in vmware-dvs:
assignee: Vyacheslav Tabolin (vtabolin) → Vyacheslav Tabolin (slavchick)
Olesia Tsvigun (otsvigun)
summary: [DVS-2.1] Can't launch instances after launch->remove instances of
- CLuster(compute-vmware).
+ Cluster(compute-vmware).
Olesia Tsvigun (otsvigun)
Changed in vmware-dvs:
status: Incomplete → Confirmed
Revision history for this message
Olesia Tsvigun (otsvigun) wrote :

Reproduced at fuel-plugin-vmware-dvs-2.1.0
Fuel ISO#570

Steps to reproduce
1.Deploy cluster with:
 fuel-plugin-vmware-dvs-2.1.0
 1 controller role.
 2 node with compute role.
 1 node with compute-vmware role
 storage default
 two clusters mapping to dvSwitches

2. Create net_1: net01__subnet, 192.168.1.0/24, and attach it to the router01.
3. Create security SG1 group with rules:
Ingress rule with ip protocol 'icmp ', port range any, SG group 'SG1'
Egress rule with ip protocol 'icmp ', port range any, SG group 'SG1'
Ingress rule with ssh protocol 'tcp', port range 22, SG group 'SG1'
Egress rule with ssh protocol 'tcp ', port range 22, SG group 'SG1'
4. Create security Sg2 group with rules:
Ingress rule with ssh protocol 'tcp', port range 22, SG group 'SG2'
Egress rule with ssh protocol 'tcp ', port range 22, SG group 'SG2'
5. Launch 20 instances of vcenter availability zone with Default SG +SG1+SG2 in net1 in one batch.
6. Launch few instances of nova availability zone with Default SG +SG1+SG2 in net1 in one batch.
7. Verify that icmp/ssh is enabled between instances.
8. Remove all instances.
9. Launch 20 instances of nova availability zone with Default SG +SG1+SG2 in net1 in one batch.

 Actual Result
One DVS agent of controller node has gone Down state. Instances have got error state.

10

Launch few instances of vcenter availability zone with Default SG +SG1+SG2 in net1 in one batch.

Vyacheslav Tabolin (slavchick)
Changed in vmware-dvs:
status: Confirmed → In Progress
Vyacheslav Tabolin (slavchick)
Changed in vmware-dvs:
status: In Progress → Fix Committed
Revision history for this message
Olesia Tsvigun (otsvigun) wrote :

Verified at [root@nailgun var]# fuel plugins list
id | name | version | package_version
---|------------------------|---------|----------------
1 | fuel-plugin-vmware-dvs | 2.1.0 | 4.0.0

Steps to reproduce
1.Deploy cluster with:
 fuel-plugin-vmware-dvs-2.1.0
 1 controller role.
 2 node with compute role.
 1 node with compute-vmware role
 storage default
 two clusters mapping to dvSwitches

2. Create net_1: net01__subnet, 192.168.1.0/24, and attach it to the router01.
3. Create security SG1 group with rules:
Ingress rule with ip protocol 'icmp ', port range any, SG group 'SG1'
Egress rule with ip protocol 'icmp ', port range any, SG group 'SG1'
Ingress rule with ssh protocol 'tcp', port range 22, SG group 'SG1'
Egress rule with ssh protocol 'tcp ', port range 22, SG group 'SG1'
4. Create security Sg2 group with rules:
Ingress rule with ssh protocol 'tcp', port range 22, SG group 'SG2'
Egress rule with ssh protocol 'tcp ', port range 22, SG group 'SG2'
5. Launch 20 instances of vcenter availability zone with Default SG +SG1+SG2 in net1 in one batch.
6. Launch few instances of nova availability zone with Default SG +SG1+SG2 in net1 in one batch.
7. Verify that icmp/ssh is enabled between instances.
8. Remove all instances.
9. Launch 20 instances of vcenter availability zone with Default SG +SG1+SG2 in net1 in one batch.

 Actual Result
One DVS agent of controller node has gone Down state. Instances have got error state.

Changed in vmware-dvs:
status: Fix Committed → Confirmed
Revision history for this message
Olesia Tsvigun (otsvigun) wrote :

Verified at Fuel ISO#570
Vcenter 5.5, 6.0
 md5sum fuel-plugin-vmware-dvs-2.1-2.1.0-1.noarch.rpm
b79312dcaa67e666e537d653f5acc74d fuel-plugin-vmware-dvs-2.1-2.1.0-1.noarch.rpm

Changed in vmware-dvs:
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.