Fuel Contrail plugin

[contrail] Cannot deploy cluster with 2 node groups with Contrail

Bug #1521119 reported by Iryna Vovk on 2015-11-30

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Fuel Contrail plugin	Fix Released	High	Illia Polliul	Fuel Contrail plugin 4.0.0

Bug Description

Steps to reproduce:
1. Create 2 devops envs
2. Shutdown fuel master env#2
3. Clean up iptables, set sysctls on lab host
4. Kill DNSMASQ, start DHCP-helper, start DNSMASQ-admin on lab host
5. Update cobler on fuel master
6. Create fuel env
7. Add a node group
8. Boot slaves and verify ips
9. Add slaves, assign roles, verify group id
10. Deploy cluster with plugin

Actual result:
Cannot deploy due error "Network 1 does not exist."

Additional info:

api: '1.0'
astute_sha: 1ea8017fe8889413706d543a5b9f557f5414beae
auth_required: true
build_id: 2015-06-19_13-02-31
build_number: '525'
feature_groups:
- mirantis
fuel-library_sha: 2e7a08ad9792c700ebf08ce87f4867df36aa9fab
fuel-ostf_sha: 8fefcf7c4649370f00847cc309c24f0b62de718d
fuelmain_sha: a3998372183468f56019c8ce21aa8bb81fee0c2f
nailgun_sha: dbd54158812033dd8cfd7e60c3f6650f18013a37
openstack_version: 2014.2.2-6.1
production: docker
python-fuelclient_sha: 4fc55db0265bbf39c369df398b9dc7d6469ba13b
release: '6.1'
release_versions:
  2014.2.2-6.1:
    VERSION:
      api: '1.0'
      astute_sha: 1ea8017fe8889413706d543a5b9f557f5414beae
      build_id: 2015-06-19_13-02-31
      build_number: '525'
      feature_groups:
      - mirantis
      fuel-library_sha: 2e7a08ad9792c700ebf08ce87f4867df36aa9fab
      fuel-ostf_sha: 8fefcf7c4649370f00847cc309c24f0b62de718d
      fuelmain_sha: a3998372183468f56019c8ce21aa8bb81fee0c2f
      nailgun_sha: dbd54158812033dd8cfd7e60c3f6650f18013a37
      openstack_version: 2014.2.2-6.1
      production: docker
      python-fuelclient_sha: 4fc55db0265bbf39c369df398b9dc7d6469ba13b
      release: '6.1'

Tags:

Iryna Vovk (ivovk) on 2015-11-30

Changed in fuel-plugins:
assignee:	nobody → Oleksandr Martsyniuk (omartsyniuk)
summary:	- [contrail] Cannot deploy cluster with 2 node groups + [contrail] Cannot deploy cluster with 2 node groups with Contrail v2.1

Oleksandr Martsyniuk (omartsyniuk) on 2015-11-30

Changed in fuel-plugins:
assignee:	Oleksandr Martsyniuk (omartsyniuk) → Illia Polliul (ipolliul)

Revision history for this message

Illia Polliul (ipolliul) wrote on 2015-11-30: Re: [contrail] Cannot deploy cluster with 2 node groups with Contrail v2.1

Please attach log snapshot.

Changed in fuel-plugins:
status:	New → Incomplete

Revision history for this message

Iryna Vovk (ivovk) wrote on 2015-11-30:

We don't have any logs because deployment could not be started

Changed in fuel-plugins:
importance:	High → Medium

okosse (okosse) on 2015-11-30

Changed in fuel-plugins:
status:	Incomplete → Confirmed

Oleksandr Martsyniuk (omartsyniuk) on 2015-12-08

Changed in fuel-plugins:
assignee:	Illia Polliul (ipolliul) → Oleksandr Martsyniuk (omartsyniuk)

Przemysław Szypowicz (pszypowicz) on 2015-12-22

Changed in fuel-plugins:
assignee:	Oleksandr Martsyniuk (omartsyniuk) → Przemysław Szypowicz (pszypowicz)

Yevgeniy (yshapovalov) on 2016-04-11

affects:	fuel-plugins → fuel-plugin-contrail
Changed in fuel-plugin-contrail:
milestone:	6.1 → none

okosse (okosse) on 2016-04-21

Changed in fuel-plugin-contrail:
milestone:	none → 3.0.1

Revision history for this message

Illia Polliul (ipolliul) wrote on 2016-06-02:

Olesya, please try to reproduce this problem with current 4.0.0 plug-in build.

Changed in fuel-plugin-contrail:
milestone:	3.0.1 → 4.0.0
assignee:	Przemysław Szypowicz (pszypowicz) → Olesia Tsvigun (otsvigun)
status:	Confirmed → Incomplete

Revision history for this message

Olesia Tsvigun (otsvigun) wrote on 2016-06-15:

Reproduced at Fuel 8.0 ISO#570(mu1)
contrail-4.0-4.0.0-1-404.noarch.rpm

Steps to reproduce
            1. Revert snapshot with ready master node
            2. Install contrail plugin
            3. Bootstrap slaves from default nodegroup
            4. Create cluster with Neutron GRE and custom nodegroups
            5. Activate plugin and configure plugins setings
            6. Remove 2nd custom nodegroup which is added automatically
            7. Bootstrap slave nodes from custom nodegroup
            8. Download network configuration
            9. Update network.json with customized ip ranges
            10. Put new json on master node and update network data
            11. Verify that new IP ranges are applied for network config
            12. Add following nodes to default nodegroup:
                * 3 controller+ceph
            13. Add following nodes to custom nodegroup:
                * 1 compute
                * 1 contrail-config+contrail-control+contrail-db
            14. Deploy cluster

Actual result
Deploymet failed on compute node with error 'Error running RPC method granular_deploy: Failed to execute hook 'ceph-compute' Puppet run failed. Check puppet logs for details'

From puppet logs:
http://paste.openstack.org/show/516132/

Expected result
Cluster should be deployed.

Changed in fuel-plugin-contrail:
status:	Incomplete → Confirmed
importance:	Medium → High
summary:	- [contrail] Cannot deploy cluster with 2 node groups with Contrail v2.1 + [contrail] Cannot deploy cluster with 2 node groups with Contrail

Revision history for this message

Olesia Tsvigun (otsvigun) wrote on 2016-06-15:

fuel-snapshot-2016-06-15_05-51-54.tar.xz Edit (97.7 MiB, application/octet-stream)

Olesia Tsvigun (otsvigun) on 2016-06-15

Changed in fuel-plugin-contrail:
assignee:	Olesia Tsvigun (otsvigun) → Illia Polliul (ipolliul)

Revision history for this message

Illia Polliul (ipolliul) wrote on 2016-06-15:

After checking the environment I see that gateway for admin network in 2nd node-group is set to 10.109.7.2, while on the lab host 10.109.7.1 is set.

Please, correct the settings and check once again.

Changed in fuel-plugin-contrail:
status:	Confirmed → Invalid

Revision history for this message

Olesia Tsvigun (otsvigun) wrote on 2016-06-16:

Gateway is not wrong. Contrail node node-group-2 has access to internet, but compute node of node-group-2 hasn't access. Seems issue is related for compute node configuration.
master node
default via 10.109.0.1 dev enp0s3 proto static metric 100
10.109.0.0/24 dev enp0s3 proto kernel scope link src 10.109.0.2 metric 100
10.109.7.0/24 dev enp0s4 proto kernel scope link src 10.109.7.2
10.109.10.0/24 dev enp0s5 proto kernel scope link src 10.109.10.2
169.254.0.0/16 dev enp0s4 scope link metric 1003
169.254.0.0/16 dev enp0s5 scope link metric 1004
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.42.1

contrail node
root@node-4:~# ip route
default via 10.109.7.2 dev br-fw-admin
10.109.0.0/24 via 10.109.7.2 dev br-fw-admin
10.109.1.0/24 via 10.109.6.1 dev br-mgmt
10.109.2.0/24 dev br-storage proto kernel scope link src 10.109.2.8
10.109.4.0/24 via 10.109.5.1 dev br-mesh
10.109.4.250 via 10.109.5.1 dev br-mesh
10.109.5.0/24 dev br-mesh proto kernel scope link src 10.109.5.3
10.109.6.0/24 dev br-mgmt proto kernel scope link src 10.109.6.5
10.109.7.0/24 dev br-fw-admin proto kernel scope link src 10.109.7.3
unreachable 169.254.169.254 scope host

compute node
root@node-5:~# ip route
10.109.0.0/24 via 10.109.7.2 dev br-fw-admin
10.109.1.0/24 via 10.109.6.1 dev br-mgmt
10.109.2.0/24 dev br-storage proto kernel scope link src 10.109.2.9
10.109.4.0/24 via 10.109.5.1 dev vhost0
10.109.5.0/24 dev vhost0 proto kernel scope link src 10.109.5.2
10.109.6.0/24 dev br-mgmt proto kernel scope link src 10.109.6.6
10.109.7.0/24 dev br-fw-admin proto kernel scope link src 10.109.7.4
169.254.0.3 dev vhost0 proto 109 scope link
169.254.0.4 dev vhost0 proto 109 scope link
unreachable 169.254.169.254 scope host