Bug #1632327 “Defining networks inconvenient when deploying mult...” : Bugs : tripleo

Revision history for this message

Steven Hardy (shardy) wrote on 2016-10-12:

#1

Thanks for the report Rhys - I'm not certain tho if this is a bug or a usability issue - I think we by design don't want multiple overclouds to use the same networks (or they will no longer be isolated), but we do need a better way to easily set the parameters so that things (names in this case) don't collide or overlap.

To get past the issue you're encountering, I think it's probably enough to simply pass a non-default name (e.g something that's unique per stack):

$ cat net-env.yaml
parameter_defaults:
  ExternalNetName: external2
  InternalApiNetName: internal2
  ManagementNetName: management2
  StorageMgmtNetName: storagemgmt2
  StorageNetName: storage2
  TenantNetName: tenant2

etc

Would that resolve your problem?

Revision history for this message

Rhys Oxenham (rdoxenham) wrote on 2016-10-12:

#2

Download full text (9.9 KiB)

Thanks, Steve. I tried this but unfortunately it still wouldn't let me override the name of the network it's trying to create. Perhaps it's hard coded somewhere?

[stack@undercloud templates]$ heat stack-show 8d7d3fa7-ae27-4361-a271-8468f07354b3
WARNING (shell) "heat stack-show" is deprecated, please use "openstack stack show" instead
+-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| capabilities | [] |
| creation_time | 2016-10-12T12:52:08Z |
| deletion_time | None |
| description | External network. Public traffic, Neutron l3router for |
| | floating IPs/SNAT, etc. |
| disable_rollback | True |
| id | 8d7d3fa7-ae27-4361-a271-8468f07354b3 |
| links | http://172.16.0.1:8004/v1/8d35712bab764e4cb8666dcd0c9a6a50/stacks/region-two-Networks-gp55udj5jo3g-ExternalNetwork-o6mko4qco7i5/8d7d3fa7-ae27-4361-a271-8468f07354b3 (self) |
| notification_topics | [] |
| outputs | [ |
| | { |
| | "output_value": null, ...

Thanks, Steve. I tried this but unfortunately it still wouldn't let me override the name of the network it's trying to create. Perhaps it's hard coded somewhere?

[stack@undercloud templates]$ heat stack-show 8d7d3fa7-ae27-4361-a271-8468f07354b3
WARNING (shell) "heat stack-show" is deprecated, please use "openstack stack show" instead
+-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property              | Value                                                                                                                                                                       |
+-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| capabilities          | []                                                                                                                                                                          |
| creation_time         | 2016-10-12T12:52:08Z                                                                                                                                                        |
| deletion_time         | None                                                                                                                                                                        |
| description           | External network. Public traffic, Neutron l3router for                                                                                                                      |
|                       | floating IPs/SNAT, etc.                                                                                                                                                     |
| disable_rollback      | True                                                                                                                                                                        |
| id                    | 8d7d3fa7-ae27-4361-a271-8468f07354b3                                                                                                                                        |
| links                 | http://172.16.0.1:8004/v1/8d35712bab764e4cb8666dcd0c9a6a50/stacks/region-two-Networks-gp55udj5jo3g-ExternalNetwork-o6mko4qco7i5/8d7d3fa7-ae27-4361-a271-8468f07354b3 (self) |
| notification_topics   | []                                                                                                                                                                          |
| outputs               | [                                                                                                                                                                           |
|                       |   {                                                                                                                                                                         |
|                       |     "output_value": null,                                                                                                                                                   |
|                       |     "output_key": "OS::stack_id",                                                                                                                                           |
|                       |     "description": "Neutron external network"                                                                                                                               |
|                       |   }                                                                                                                                                                         |
|                       | ]                                                                                                                                                                           |
| parameters            | {                                                                                                                                                                           |
|                       |   "OS::project_id": "8d35712bab764e4cb8666dcd0c9a6a50",                                                                                                                     |
|                       |   "ExternalNetName": "external2",                                                                                                                                           |
|                       |   "OS::stack_id": "8d7d3fa7-ae27-4361-a271-8468f07354b3",                                                                                                                   |
|                       |   "OS::stack_name": "region-two-Networks-gp55udj5jo3g-ExternalNetwork-o6mko4qco7i5",                                                                                        |
|                       |   "ExternalNetAdminStateUp": "False",                                                                                                                                       |
|                       |   "ExternalAllocationPools": "[{u'start': u'192.168.122.130', u'end': u'192.168.122.159'}]",                                                                                |
|                       |   "ExternalNetValueSpecs": "{u'provider:physical_network': u'external', u'provider:network_type': u'flat'}",                                                                |
|                       |   "ExternalNetShared": "False",                                                                                                                                             |
|                       |   "ExternalInterfaceDefaultRoute": "192.168.122.1",                                                                                                                         |
|                       |   "ExternalNetCidr": "192.168.122.0/24",                                                                                                                                    |
|                       |   "ExternalSubnetName": "external_subnet",                                                                                                                                  |
|                       |   "ExternalNetEnableDHCP": "False"                                                                                                                                          |
|                       | }                                                                                                                                                                           |
| parent                | 04f8259f-8340-48f5-90de-411063ce4b23                                                                                                                                        |
| stack_name            | region-two-Networks-gp55udj5jo3g-ExternalNetwork-o6mko4qco7i5                                                                                                               |
| stack_owner           | admin                                                                                                                                                                       |
| stack_status          | CREATE_FAILED                                                                                                                                                               |
| stack_status_reason   | Resource CREATE failed: Conflict:                                                                                                                                           |
|                       | resources.ExternalNetwork: Unable to create the flat                                                                                                                        |
|                       | network. Physical network external is in use. Neutron                                                                                                                       |
|                       | server returns request_ids: ['req-6924165a-                                                                                                                                 |
|                       | 0f05-4667-8b96-b5604b14d0f1']                                                                                                                                               |
| stack_user_project_id | 1a92b336ce2c4713aba6149556e61438                                                                                                                                            |
| tags                  | null                                                                                                                                                                        |
| template_description  | External network. Public traffic, Neutron l3router for                                                                                                                      |
|                       | floating IPs/SNAT, etc.                                                                                                                                                     |
| timeout_mins          | 240                                                                                                                                                                         |
| updated_time          | None                                                                                                                                                                        |
+-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Emilien Macchi (emilienm) on 2016-10-12

Changed in tripleo:
milestone:	none → ocata-1
status:	New → Triaged

Revision history for this message

Steven Hardy (shardy) wrote on 2016-10-13:

#3

Download full text (3.5 KiB)

So I think this is actually a configuration problem, not a bug as such.

There's two ways to configure the overlay networks, you can either use Neutron flat networks or define a vlan per network, but in both cases the underlying network (flat name or vlan) must be unique, and just changing the *NetName parameter isn't enough.

And to override it you need to change *NetName and also *NetValueSpecs (because *NetValueSpecs has hard-coded references to the flat network name).

I tested this, which uses flat networking for all except the external network (where I created a new vlan.

parameter_defaults:
  ExternalNetName: external2
  InternalApiNetName: internal2
  ManagementNetName: management2
  StorageMgmtNetName: storagemgt2
  StorageNetName: storage2
  TenantNetName: tenant2

  ExternalSubnetName: external_subnet2
  InternalSubnetName: internal_subnet2
  ManagementSubnetName: management_subnet2
  StorageMgmtSubnetName: storagemgmt_subnet2
  StorageSubnetName: storage_subnet2
  TenantSubnetName: tenant_subnet2

  ExternalAllocationPools: [{'start': '10.0.1.4', 'end': '10.0.1.250'}]
  ExternalInterfaceDefaultRoute: '10.0.1.1'
  # Note this has been created via /etc/sysconfig/network-scripts/ifcfg-vlan20
  # DEVICE=vlan20
  # ONBOOT=yes
  # HOTPLUG=no
  # TYPE=OVSIntPort
  # OVS_BRIDGE=br-ctlplane
  # OVS_OPTIONS="tag=10"
  # BOOTPROTO=static
  # IPADDR=10.0.1.1
  # PREFIX=24
  # NM_CONTROLLED=no
  ExternalNetworkVlanID: 20

  InternalApiNetCidr: '172.16.4.0/24'
  StorageMgmtNetCidr: '172.16.5.0/24'
  StorageNetCidr: '172.16.6.0/24'
  TenantNetCidr: '172.16.7.0/24'
  ManagementNetCidr: '172.16.8.0/24'

  InternalAllocationPools: [{'start': '172.16.4.4', 'end': '172.16.4.250'}]
  StorageMgmtAllocationPools: [{'start': '172.16.5.4', 'end': '172.16.5.250'}]
  StorageAllocationPools: [{'start': '172.16.6.4', 'end': '172.16.6.250'}]
  TenantAllocationPools: [{'start': '172.16.7.4', 'end': '172.16.7.250'}]
  ManagementAllocationPools: [{'start': '172.16.8.4', 'end': '172.16.8.250'}]

  InternalApiNetValueSpecs: {'provider:physical_network': 'internal_api2', 'provider:network_type': 'flat'}
  ManagementNetValueSpecs: {'provider:physical_network': 'management2', 'provider:network_type': 'flat'}
  StorageMgmtNetValueSpecs: {'provider:physical_network': 'storagemgmt2', 'provider:network_type': 'flat'}
  StorageNetValueSpecs: {'provider:physical_network': 'storage2', 'provider:network_type': 'flat'}
  TenantNetValueSpecs: {'provider:physical_network': 'tenant2', 'provider:network...

So I think this is actually a configuration problem, not a bug as such.

There's two ways to configure the overlay networks, you can either use Neutron flat networks or define a vlan per network, but in both cases the underlying network (flat name or vlan) must be unique, and just changing the *NetName parameter isn't enough.

And to override it you need to change *NetName and also *NetValueSpecs (because *NetValueSpecs has hard-coded references to the flat network name).

I tested this, which uses flat networking for all except the external network (where I created a new vlan.

parameter_defaults:
  ExternalNetName: external2
  InternalApiNetName: internal2
  ManagementNetName: management2
  StorageMgmtNetName: storagemgt2
  StorageNetName: storage2
  TenantNetName: tenant2

ExternalSubnetName: external_subnet2
  InternalSubnetName: internal_subnet2
  ManagementSubnetName: management_subnet2
  StorageMgmtSubnetName: storagemgmt_subnet2
  StorageSubnetName: storage_subnet2
  TenantSubnetName: tenant_subnet2

ExternalAllocationPools: [{'start': '10.0.1.4', 'end': '10.0.1.250'}]
  ExternalInterfaceDefaultRoute: '10.0.1.1'
  # Note this has been created via /etc/sysconfig/network-scripts/ifcfg-vlan20
  # DEVICE=vlan20
  # ONBOOT=yes
  # HOTPLUG=no
  # TYPE=OVSIntPort
  # OVS_BRIDGE=br-ctlplane
  # OVS_OPTIONS="tag=10"
  # BOOTPROTO=static
  # IPADDR=10.0.1.1
  # PREFIX=24
  # NM_CONTROLLED=no
  ExternalNetworkVlanID: 20

InternalApiNetCidr: '172.16.4.0/24'
  StorageMgmtNetCidr: '172.16.5.0/24'
  StorageNetCidr: '172.16.6.0/24'
  TenantNetCidr: '172.16.7.0/24'
  ManagementNetCidr: '172.16.8.0/24'

InternalAllocationPools: [{'start': '172.16.4.4', 'end': '172.16.4.250'}]
  StorageMgmtAllocationPools: [{'start': '172.16.5.4', 'end': '172.16.5.250'}]
  StorageAllocationPools: [{'start': '172.16.6.4', 'end': '172.16.6.250'}]
  TenantAllocationPools: [{'start': '172.16.7.4', 'end': '172.16.7.250'}]
  ManagementAllocationPools: [{'start': '172.16.8.4', 'end': '172.16.8.250'}]

InternalApiNetValueSpecs: {'provider:physical_network': 'internal_api2', 'provider:network_type': 'flat'}
  ManagementNetValueSpecs: {'provider:physical_network': 'management2', 'provider:network_type': 'flat'}
  StorageMgmtNetValueSpecs: {'provider:physical_network': 'storagemgmt2', 'provider:network_type': 'flat'}
  StorageNetValueSpecs: {'provider:physical_network': 'storage2', 'provider:network_type': 'flat'}
  TenantNetValueSpecs: {'provider:physical_network': 'tenant2', 'provider:network_type': 'flat'}

The other approach is to use vlans for everything, which is documented here (WIP docs patch):

https://review.openstack.org/#/c/368840/

So, I think this is primarily a docs issue, and perhaps also we could do better at defaulting *ValueSpecs based on the *NetName, but I'm not sure if that really buys us much given that folks need to make a decision on how to ensure all the other stuff doesn't conflict.

Revision history for this message

Steven Hardy (shardy) wrote on 2016-10-13:

#4

paste version of the environment file above https://review.openstack.org/#/c/368840/

Changed in tripleo:
status:	Triaged → Incomplete

Revision history for this message

Steven Hardy (shardy) wrote on 2016-10-13:

#5

So, the paste above actually doesn't work, here's an updated version:

http://paste.fedoraproject.org/449607/63726021

You also need to increase the network quota:

openstack quota set --network 100 --subnets 100 admin

You can then deploy an additional overcloud via

[stack@instack ~]$ cat deploy_netiso2.sh
TEMPLATES=/tmp/tripleo-heat-templates
openstack overcloud deploy \
--templates=$TEMPLATES \
-e $TEMPLATES/environments/network-isolation.yaml \
-e $TEMPLATES/environments/net-single-nic-with-vlans.yaml \
-e /home/stack/custom.yaml \
-e netiso2.yaml \
--stack netiso2
[stack@instack ~]$ cat custom.yaml
parameter_defaults:
EC2MetadataIp: 192.0.2.1
ControlPlaneDefaultRoute: 192.0.2.1

Changed in tripleo:
status:	Incomplete → Triaged

Revision history for this message

Steven Hardy (shardy) wrote on 2016-10-13:

#6

Rhys if you're happy with the above, I'd like to reword the bug description to say switching the network names is inconvenient and error prone, so it'd be better if we optionally allowed a network prefix (which could just be the stack name), the a lot of the stuff above could be removed.

Changed in tripleo:
assignee:	nobody → Steven Hardy (shardy)
summary:	- Cannot reuse existing networks when deploying multiple overclouds + Defining networks inconvenient when deploying multiple overclouds

Revision history for this message

Rhys Oxenham (rdoxenham) wrote on 2016-10-13:

#7

Thanks a lot for your support, Steve.

Steven Hardy (shardy) on 2016-10-28

Changed in tripleo:
milestone:	ocata-1 → ocata-2
importance:	Low → Medium

Emilien Macchi (emilienm) on 2016-12-14

Changed in tripleo:
milestone:	ocata-2 → ocata-3

Steven Hardy (shardy) on 2016-12-15

Changed in tripleo:
milestone:	ocata-3 → pike-1

Revision history for this message

Joe Talerico (jtaleric) wrote on 2017-03-15:

#8

Download full text (3.1 KiB)

I have used this method to deploy two different stacks.

Stack 1) All Baremetal nodes for OpenShift
*This stack used the default networks

Stack 2) OpenStack deployment
* This stack uses
parameter_defaults:

  ExternalNetName: external2
  InternalApiNetName: internal_api2
  StorageMgmtNetName: storagemgt2
  StorageNetName: storage2
  TenantNetName: tenant2

  ExternalSubnetName: external_subnet2
  InternalSubnetName: internal_api_subnet2
  StorageMgmtSubnetName: storagemgmt_subnet2
  StorageSubnetName: storage_subnet2
  TenantSubnetName: tenant_subnet2

  InternalApiNetValueSpecs:
      'provider:physical_network': 'internal_api2'
      'provider:network_type': 'flat'
  ExternalNetValueSpecs:
      'provider:physical_network': 'external2'
      'provider:network_type': 'flat'
  StorageNetValueSpecs:
      'provider:physical_network': 'storage2'
      'provider:network_type': 'flat'
  StorageMgmtNetValueSpecs:
      'provider:physical_network': 'storagemgt2'
      'provider:network_type': 'flat'
  TenantNetValueSpecs:
      'provider:physical_network': 'tenant2'
      'provider:network_type': 'flat'

However, on my controllers, it seems to be using resources from Stack1, such as the internal_api.

| 065b7c04-9354-4663-8b43-1a1546f47468 | redis_virtual_ip | fa:16:3e:06:2b:0f | {"subnet_id": "62713943-bcaa-4fc
b-88a7-329fe8f144a8", "ip_address": "172.16.0.106"} |

^ Is a IP on my controller. However, Stack2 should be using internal_api2,215f46a3-ec1b-42c1-8406-66263819a...

tripleo

Defining networks inconvenient when deploying multiple overclouds

Bug Description

Other bug subscribers

Related blueprints

Remote bug watches

Changed in tripleo:
assignee:	Steven Hardy (shardy) → nobody

Changed in tripleo:
importance:	Medium → Undecided
status:	Triaged → Expired