From: Ritam Gangopadhyay 
Sent: Thursday, May 31, 2018 12:09 AM
To: Ramprakash Ram Mohan <email address hidden>
Cc: Michael Henkel <email address hidden>; Andrey Pavlov <email address hidden>; Sudheendra Rao <email address hidden>; Vimal Appachan <email address hidden>; Abhay Joshi <email address hidden>
Subject: Re: Multi Interface Setup with Openstack HA.



Hi Ram, 

       The configuration mentioned in your email is for which the bug was filled. 

Open stack nodes on mgmt/api n/w
Internal VIP on ctl-data n/w
External VIP on mgmt/api n/w

With this we see haproxy failing to come up while waiting for vip:3306

In the bug you asked me to configure open stack nodes on 10.10.10.X i.e. the ctl-data n/w. And with this I see the problem illustrated in my first mail. 

Apart from these configurations we have also tried setting network interface,  omitting open stack nodes in the instances.yaml and few others, without any success. 

Regards, 
Ritam. 
       

On May 30, 2018 23:11, Ramprakash Ram Mohan <email address hidden> wrote:
+Abhay
 
Hi Ritam,
 
In your setup, if you want openstack to run its services on the mgmt. network, then you need to change the following in your config like I’ve mentioned in the bug:
 
contrail_configuration:
    OPENSTACK_NODES: 10.204.216.103,10.204.216.95,10.204.216.96
kolla_config:
   kolla_internal_vip_address: 10.204.216.140
   kolla_external_vip_address: 10.10.10.20
 
Basic theory is this: 
Openstack runs services on the “kolla_internal_vip_address” which should correspond to an IP address on the “network_interface” parameter. 
kolla_external_vip_address is the address you want access from. In “kolla” terminology, this should be your management network, but in contrail, we want to run the services themselves on the management network hence the swapped configs. (If you do not want access to openstack services on the 10.10.10.20 via haproxy, then you can avoid that altogether - I have not tested this though)
“network_interface” can be derived from the OPENSTACK_NODES parameter to avoid having to specify network names, so make sure kolla_internal_vip_address and OPENSTACK_NODES are on the same network.
 
 
Let me know if this helps.
 
Thanks,
Ram
 
 
From: Ritam Gangopadhyay <email address hidden>
Date: Wednesday, May 30, 2018 at 9:55 AM
To: Michael Henkel <email address hidden>, Ramprakash Ram Mohan <email address hidden>, Andrey Pavlov <email address hidden>
Cc: Sudheendra Rao <email address hidden>, Vimal Appachan <email address hidden>
Subject: Multi Interface Setup with Openstack HA.
 
instances.yaml_with_openstack_on_mgmt-api_subnet.yaml instances.yaml_with_opnstack_nodes_set_to_ctl_data_subnet.yaml 
 
Hi,
 
       We are trying to get clarity on how services(contrail and openstack) are supposed to come up on a multi interface openstack HA setup and what would be the right deployment scenario.
 
       Ideally we should have mgmt, API and CTL/DATA as three separate networks on contrail, but as of now we have mgmt/API in one subnet and CTL/DATA in the other subnet. With that in mind, as of today what we understand from a multi interface setup is:-
 
1.       Openstack services should open up north bound connections on the Mgmt/API network subnet-A and east-west communication on a separate subnet-B
2.       Contrail controller services (config, analytics, collector) should come up on the Mgmt/API network subnet-A
3.       Contrail control services and compute services should communicate on the CTL/DATA subnet-C
 
       With above assumption we have tried many combination and each one has lead us to haproxy service failing to provision because it doesn’t see the vip come up properly as mentioned in bug - https://bugs.launchpad.net/juniperopenstack/+bug/1773425
 
       The most recent configuration that we have tried is as below:-
 
1.       Openstack on mgmt./API network 10.204.216.0/24 – eno1
2.       Openstack east-west communication on 9.9.9.0/24 subnet – eno2
3.       Contrail controller on mgmt./api network 10.204.216.0/24 – eno1
4.       Contrail control data n/w on 10.10.10.0/24 – ens2f1
 
         Even with the above config we see haproxy fails with the mentioned error. Attached the instances.yaml file for this to the mail
 
         On Ram’s suggestion in the bug to use 10.10.10.0/24 n/w (i.e. the CTL/DATA subnet for that particular setup) for openstack nodes, we were able to bring up a setup. Though I am not fully convinced about this configuration as it beats the purpose of having openstack on the mgmt./API network.
         The instances.yaml file for this setup is also attached to this mail. The main issue we see here is all the openstack services run on 192.168.100.0/24 n/w (i.e. the CTL/DATA n/w) whereas contrail controller runs on 10.204.217.0/24 n/w which is the mgmt./API subnet. 
 
[root@nodec33 ~]# netstat -anp | grep 8775
tcp        0      0 192.168.100.17:8775     0.0.0.0:*               LISTEN      32280/python2
tcp        0      0 192.168.100.17:8775     192.168.100.17:43540    SYN_RECV    -
tcp        0      0 192.168.100.17:8775     192.168.100.12:57480    SYN_RECV    -
tcp        0      0 192.168.100.17:8775     192.168.100.15:37638    SYN_RECV    -
tcp        0      0 192.168.100.17:8775     192.168.100.12:57506    SYN_RECV    -
tcp        0      0 10.204.217.184:8775     0.0.0.0:*               LISTEN      20874/haproxy
tcp        0      0 192.168.100.20:8775     0.0.0.0:*               LISTEN      20874/haproxy
[root@nodec33 ~]# netstat -anp | grep 35357
tcp        0      0 192.168.100.17:35357    0.0.0.0:*               LISTEN      17391/httpd
tcp        0      0 192.168.100.17:35357    192.168.100.12:60476    SYN_RECV    -
tcp        0      0 192.168.100.17:35357    192.168.100.15:34448    SYN_RECV    -
tcp        0      0 192.168.100.20:35357    0.0.0.0:*               LISTEN      20874/haproxy
[root@nodec33 ~]# netstat -anp | grep 8082
tcp        0      0 10.204.217.168:8082     0.0.0.0:*               LISTEN      28966/python
tcp        0      0 10.204.217.168:50238    10.204.217.168:8082     ESTABLISHED 28966/python
tcp        0      0 10.204.217.168:8082     10.204.217.13:38660     ESTABLISHED 28966/python
tcp        0      0 10.204.217.168:50240    10.204.217.168:8082     ESTABLISHED 28966/python
tcp        0      0 10.204.217.168:54176    10.204.217.13:8082      ESTABLISHED 5591/python
tcp        0      0 10.204.217.168:8082     10.204.217.176:59740    ESTABLISHED 28966/python
tcp        0      0 10.204.217.168:8082     10.204.217.176:59872    ESTABLISHED 28966/python
tcp        0      0 10.204.217.168:56088    10.204.217.168:8082     ESTABLISHED 5591/python
tcp        0      0 10.204.217.168:45208    10.204.217.176:8082     ESTABLISHED 5591/python
tcp        0      0 10.204.217.168:8082     10.204.217.176:60284    ESTABLISHED 28966/python
tcp        0      0 10.204.217.168:8082     10.204.217.168:56088    ESTABLISHED 28966/python
tcp        0      0 10.204.217.168:8082     10.204.217.168:50240    ESTABLISHED 28966/python
tcp        0      0 10.204.217.168:8082     10.204.217.168:50238    ESTABLISHED 28966/python
[root@nodec33 ~]# netstat -anp | grep 8086
tcp        0      0 0.0.0.0:8086            0.0.0.0:*               LISTEN      6077/contrail-colle
tcp        0      0 10.204.217.168:51194    10.204.217.13:8086      ESTABLISHED 28756/python
tcp        0      0 10.204.217.168:56640    10.204.217.13:8086      ESTABLISHED 31452/python
tcp        0      0 10.204.217.168:50742    10.204.217.13:8086      ESTABLISHED 5447/contrail-query
tcp        0      0 10.204.217.168:54064    10.204.217.13:8086      ESTABLISHED 28966/python
tcp        0      0 10.204.217.168:50604    10.204.217.13:8086      ESTABLISHED 3731/python
 
 
 
          We need to get clarity on the deployment of the services and how we can bring openstack and contrail controllers to run on same subnet.
 
Regards,
Ritam