Failing deployment "Unable to create the flat network. Physical network tenant is in use."

Bug #1817631 reported by Alex Schultz
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Alex Schultz

Bug Description

If the user fails to include their network-isolation or network environment files when performing a stack update, it will fail when Heat attempts to create the default flat network in the overcloud configuration. This is a very destructive action as recovery from this condition requires that a user restore their undercloud database from a backup or manually remove the resources created.

How to reproduce the problem:

1) Deploy the cloud
openstack overcloud deploy --templates -e /usr/share/openstack-tripleo-heat-templates/environments/enable-swap.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-environment.yaml -e /home/centos/container-defaults.yaml -e /home/centos/params.yaml

2) Run an update without network-isolation.yaml
openstack overcloud deploy --templates -e /usr/share/openstack-tripleo-heat-templates/environments/enable-swap.yaml -e /home/centos/container-defaults.yaml -e /home/centos/params.yaml

Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to python-tripleoclient (master)

Reviewed: https://review.openstack.org/638858
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=1a71c441fa8958896d7f380636075612c67bf7da
Submitter: Zuul
Branch: master

commit 1a71c441fa8958896d7f380636075612c67bf7da
Author: Alex Schultz <email address hidden>
Date: Sat Feb 23 12:54:23 2019 -0700

    Check that the networks are defined on update

    Check defined networks in the resource registry of the stack against the
    networks defined in the environment files. If the environment provided
    doesn't have the networks defined, it's likely they were improperly
    dropped which can lead to deployment issues. This is a light check that
    only checks for the existance of the networks but not the contents of
    those networks. This handles the case where a user forgets to include
    the network-isolation configuration on a subsequent update to the cloud.
    This does not prevent a user from changing the contents of the networks
    to something that breaks their deployment.

    Partial-Bug: #1817631
    Change-Id: Ia97a2367770e37bf8c55f2fd04c9a9efde914a67

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to python-tripleoclient (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/640517

tags: added: queens-backport-potential rocky-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to python-tripleoclient (stable/rocky)

Reviewed: https://review.openstack.org/640517
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=ab72340021351a82e51920d56ec2247ac265df08
Submitter: Zuul
Branch: stable/rocky

commit ab72340021351a82e51920d56ec2247ac265df08
Author: Alex Schultz <email address hidden>
Date: Sat Feb 23 12:54:23 2019 -0700

    Check that the networks are defined on update

    Check defined networks in the resource registry of the stack against the
    networks defined in the environment files. If the environment provided
    doesn't have the networks defined, it's likely they were improperly
    dropped which can lead to deployment issues. This is a light check that
    only checks for the existance of the networks but not the contents of
    those networks. This handles the case where a user forgets to include
    the network-isolation configuration on a subsequent update to the cloud.
    This does not prevent a user from changing the contents of the networks
    to something that breaks their deployment.

    Conflicts:
     tripleoclient/tests/v1/overcloud_deploy/test_overcloud_deploy.py

    Partial-Bug: #1817631
    Change-Id: Ia97a2367770e37bf8c55f2fd04c9a9efde914a67
    (cherry picked from commit 1a71c441fa8958896d7f380636075612c67bf7da)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to python-tripleoclient (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/640787

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to python-tripleoclient (stable/queens)

Reviewed: https://review.openstack.org/640787
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=6851fc1b4179c5e4119cff5e2c0d6fc43a622569
Submitter: Zuul
Branch: stable/queens

commit 6851fc1b4179c5e4119cff5e2c0d6fc43a622569
Author: Alex Schultz <email address hidden>
Date: Sat Feb 23 12:54:23 2019 -0700

    Check that the networks are defined on update

    Check defined networks in the resource registry of the stack against the
    networks defined in the environment files. If the environment provided
    doesn't have the networks defined, it's likely they were improperly
    dropped which can lead to deployment issues. This is a light check that
    only checks for the existance of the networks but not the contents of
    those networks. This handles the case where a user forgets to include
    the network-isolation configuration on a subsequent update to the cloud.
    This does not prevent a user from changing the contents of the networks
    to something that breaks their deployment.

    Conflicts:
     tripleoclient/tests/v1/overcloud_deploy/test_overcloud_deploy.py

    Partial-Bug: #1817631
    Change-Id: Ia97a2367770e37bf8c55f2fd04c9a9efde914a67
    (cherry picked from commit 1a71c441fa8958896d7f380636075612c67bf7da)
    (cherry picked from commit ab72340021351a82e51920d56ec2247ac265df08)

tags: added: in-stable-queens
Changed in tripleo:
milestone: stein-3 → stein-rc1
Revision history for this message
Alex Schultz (alex-schultz) wrote :

Going to mark this as done for now since this will prevent folks from dropping the files. We'll need to follow up with a different bug if folks manage to mess up their environments with the networks in a different fashion.

Changed in tripleo:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.