Live migration's assigned ports conflicts

This requires more investigation so changing to Incomplete. Please set back to New when you have clear problem statement and steps to reproduce.

Vitaly, this bug was found during scale tests on 200 nodes - NovaServers.boot_and_live_migrate_server rally test. Steps to reproduce were added to the description.

>> Actually this happens during rally testing of pretty big env (~200 nodes) one per 200 iterations so chances for getting that on scale are pretty big. So it should be easily reproduced under following circumastances:
1. Very high rate of migrations.
2. A lot of running VMs/other services with large amount of used TCP ports.

Both of these statements will lead to the higher chances of getting collision for qemu migration port allocation procedure.

we have no other way to reproduce it right now. The problem is that instance was said to be migrated, although that was not true in fact :) That's the issue

@dbelova, could you explain how that was said that instance had been migrated successfully? nova show {instance_uuid} shows another host, or smth. else?

@tdurakov, the only way we could see it was the following traceback from the rally:

Traceback (most recent call last):
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/runners/base.py", line 79, in _run_scenario_once
    method_name)(**kwargs) or scenario_output
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/nova/servers.py", line 440, in boot_and_live_migrate_server
    block_migration, disk_over_commit)
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/base.py", line 262, in func_atomic_actions
    f = func(self, *args, **kwargs)
  File "/opt/stack/.venv/lib/python2.7/site-packages/rally/benchmark/scenarios/nova/utils.py", line 647, in _live_migrate
    host_pre_migrate)
LiveMigrateException: Live Migration failed: Migration complete but instance did not change host: node-169.domain.tld

@dbelova, ok, now i see, need to check snapshot first, thanks for fast response.

tried to reproduce, no results. moving bug to incomplete. If it reproduces again, please reopen

No repro in a month, closing.

Reproduced on customer environment. Error from Rally:

Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/rally/task/runner.py", line 66, in _run_scenario_once deprecated_output = getattr(scenario_inst, method_name)(**kwargs) File "/usr/local/lib/python2.7/dist-packages/rally/plugins/openstack/scenarios/nova/servers.py", line 619, in boot_server_from_volume_and_live_migrate block_migration, disk_over_commit) File "/usr/local/lib/python2.7/dist-packages/rally/task/atomic.py", line 84, in func_atomic_actions f = func(self, *args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/rally/plugins/openstack/scenarios/nova/utils.py", line 733, in _live_migrate host_pre_migrate) LiveMigrateException: Live Migration failed: Migration complete but instance did not change host: <host_name>

Environment: MOS 7.0 mu2, nfs share for cinder, glance, nova.

Scenario:
"concurrency": 10,
"times": 100
 "block_migration": false
"flavor": {"name": "m1.tiny"}
"force_delete": false
"image": {"name": "TestVM"}
"volume_size": 10
"users_per_tenant": 2
"tenants": 2

We got 7-10 fails per this tests run.

Oleksii, are you sure you are seeing the very same problem? (qemu files to bind a socket for performing a live migration) Could you please provide a diagnostic snapshot?

More than a month in the incomplete state. Moving to Invalid. Please move back to confirmed if that is still reproducible + please provide the information about Roman's request.

No feedback here, thus, I think this should also be closed as Invalid for 7.0. Feel free to re-open the bug, if it's still reproduced.