OpenStack Compute (nova)

VMware: Ports are not cleaned up after reschedule

Bug #1321381 reported by Ryan Hsu
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Critical
Gary Kotton
OpenStack Compute (nova) 2014.2 "juno"
Havana
Fix Released
Undecided
Unassigned
OpenStack Compute (nova) 2013.2.4
Icehouse
Fix Released
Undecided
Unassigned
OpenStack Compute (nova) 2014.1.2

Bug Description

On Havana, when an instance errors out and gets rescheduled, attached ports are currently not being cleaned up and is causing NVS/OVS to have multiple ports with the same UUID and MAC.

This is currently occurring with the VC Driver + Neutron and is blocking VMware Minesweeper CI.

See original description
Revision history for this message
Aaron Rosen (arosen) wrote :

Fwiw, this bug is only in havana.

Ryan Hsu (rhsu)
description: updated
Tracy Jones (tjones-i)
Changed in nova:
status: New → Incomplete
status: Incomplete → Confirmed
importance: Undecided → Critical
importance: Critical → High
Revision history for this message
Aaron Rosen (arosen) wrote :

I'm guessing you're missing this patch in your setup: https://review.openstack.org/#/c/84583/

Revision history for this message
Ryan Hsu (rhsu) wrote :

Aaron, thanks for identifying the missing patch. The patch has been applied and we're monitoring for the next few days to ensure the problem has gone away.

Revision history for this message
Gary Kotton (garyk) wrote :

https://review.openstack.org/#/c/99400/

Changed in nova:
importance: High → Critical
assignee: nobody → Gary Kotton (garyk)
milestone: none → juno-2
tags: added: havana-backport-potential icehouse-backport-potential
Revision history for this message
Gary Kotton (garyk) wrote :

With the patch in #4 the issue is resolved and minesweeper looks to be up and running again

OpenStack Infra (hudson-openstack)
Changed in nova:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/99400
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=803d59e9f409a59403b2ce55040acc02e14eee28
Submitter: Jenkins
Branch: master

commit 803d59e9f409a59403b2ce55040acc02e14eee28
Author: Gary Kotton <email address hidden>
Date: Wed Jun 11 05:59:27 2014 -0700

    Network: ensure that ports are 'unset' when instance is deleted

    Commit a141206e9dfd31955b9b31d9e5a7f73bbd8510ca ensured that user defined
    ports were not deleted when an instance is deleted or rescheduled. The
    problem with this commit is that it did not reset the 'device_id' and
    'device_owner' of the port. This leads to port leakage and eventually a tenant
    is unable to launch any instances as their port quota has exceeded the
    allocated amount.

    The solution is for the deallocation to reset the 'device_id' and the
    'device_owner' of the port. This is allocated when the port is actually assigned
    to an instance.

    In the case of minesweeper (which runs on OpenStack), the tenant was leaking ports.
    The root cause was that the user defined ports were assigned to a specific instance,
    which was deleted, as a result of a failed reschedule (which is legitimate).

    When an instance is deleted all artificats of that instance should be reset on the
    port.

    Change-Id: Ie1516c299ce6d79faf2d5103a14d6d8931d3ede2
    Closes-bug: #1321381

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/icehouse)

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/100782

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/havana)

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/100783

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/icehouse)

Reviewed: https://review.openstack.org/100782
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=4aeefa71614be9c8c5072c34f26db2fb8b79167d
Submitter: Jenkins
Branch: stable/icehouse

commit 4aeefa71614be9c8c5072c34f26db2fb8b79167d
Author: Gary Kotton <email address hidden>
Date: Wed Jun 11 05:59:27 2014 -0700

    Network: ensure that ports are 'unset' when instance is deleted

    Commit a141206e9dfd31955b9b31d9e5a7f73bbd8510ca ensured that user defined
    ports were not deleted when an instance is deleted or rescheduled. The
    problem with this commit is that it did not reset the 'device_id' and
    'device_owner' of the port. This leads to port leakage and eventually a tenant
    is unable to launch any instances as their port quota has exceeded the
    allocated amount.

    The solution is for the deallocation to reset the 'device_id' and the
    'device_owner' of the port. This is allocated when the port is actually assigned
    to an instance.

    In the case of minesweeper (which runs on OpenStack), the tenant was leaking ports.
    The root cause was that the user defined ports were assigned to a specific instance,
    which was deleted, as a result of a failed reschedule (which is legitimate).

    When an instance is deleted all artificats of that instance should be reset on the
    port.

    Change-Id: Ie1516c299ce6d79faf2d5103a14d6d8931d3ede2
    Closes-bug: #1321381
    (cherry picked from commit 803d59e9f409a59403b2ce55040acc02e14eee28)

tags: added: in-stable-icehouse
tags: added: in-stable-havana
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/havana)

Reviewed: https://review.openstack.org/100783
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b818c8d807e3c67fc1dbdd007d04158cbb80f367
Submitter: Jenkins
Branch: stable/havana

commit b818c8d807e3c67fc1dbdd007d04158cbb80f367
Author: Gary Kotton <email address hidden>
Date: Wed Jun 11 05:59:27 2014 -0700

    Network: ensure that ports are 'unset' when instance is deleted

    Commit a141206e9dfd31955b9b31d9e5a7f73bbd8510ca ensured that user defined
    ports were not deleted when an instance is deleted or rescheduled. The
    problem with this commit is that it did not reset the 'device_id' and
    'device_owner' of the port. This leads to port leakage and eventually a tenant
    is unable to launch any instances as their port quota has exceeded the
    allocated amount.

    The solution is for the deallocation to reset the 'device_id' and the
    'device_owner' of the port. This is allocated when the port is actually assigned
    to an instance.

    In the case of minesweeper (which runs on OpenStack), the tenant was leaking ports.
    The root cause was that the user defined ports were assigned to a specific instance,
    which was deleted, as a result of a failed reschedule (which is legitimate).

    When an instance is deleted all artificats of that instance should be reset on the
    port.

    Change-Id: Ie1516c299ce6d79faf2d5103a14d6d8931d3ede2
    Closes-bug: #1321381
    (cherry picked from commit 803d59e9f409a59403b2ce55040acc02e14eee28)

Russell Bryant (russellb)
Changed in nova:
status: Fix Committed → Fix Released
Chuck Short (zulcss)
tags: removed: icehouse-backport-potential
Thierry Carrez (ttx)
Changed in nova:
milestone: juno-2 → 2014.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.