Fuel for OpenStack

Neutron: Deployment on Centos takes in twice more than on Ubuntu

  1. Series future
  2. Bug #1459644
Bug #1459644 reported by Nastya Urlapova
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Invalid
High
Fuel Library (Deprecated)
Fuel for OpenStack 7.0
Future
Confirmed
High
Fuel Library (Deprecated)
Fuel for OpenStack next

Bug Description

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  openstack_version: "2014.2.2-6.1"
  api: "1.0"
  build_number: "477"
  build_id: "2015-05-28_06-58-38"
  nailgun_sha: "2f5ffd9f238cd2b0428937bc31580182a5e1d2c1"
  python-fuelclient_sha: "e19f1b65792f84c4a18b5a9473f85ef3ba172fce"
  astute_sha: "5d570ae5e03909182db8e284fbe6e4468c0a4e3e"
  fuel-library_sha: "7eb54c7b30c3668e3d282e9302a04265112e2582"
  fuel-ostf_sha: "4cd2fef040ae9e7645a6b17a7cb44d3cd8fbe0be"
  fuelmain_sha: "6b5712a7197672d588801a1816f56f321cbceebd"

One set of tests was run on Ubuntu/Centos, we got results:
centos:
deploy_neutron_gre_ha:
  deploy_cluster_wait_00: '6055.27'
deploy_neutron_gre_ha_with_public_network:
  deploy_cluster_wait_00: '6270.63'
deploy_neutron_vlan_ha:
  deploy_cluster_wait_00: '6295.81'
deploy_neutron_vlan_ha_with_public_network:
  deploy_cluster_wait_00: '6421.49'

ubuntu:
deploy_neutron_gre_ha:
  deploy_cluster_wait_00: '3707.81'
deploy_neutron_gre_ha_with_public_network:
  deploy_cluster_wait_00: '706.86'
deploy_neutron_vlan_ha:
  deploy_cluster_wait_00: '3449.60'
deploy_neutron_vlan_ha_with_public_network:
  deploy_cluster_wait_00: '3506.76'

Revision history for this message
Nastya Urlapova (aurlapova) wrote :
Revision history for this message
Nastya Urlapova (aurlapova) wrote :
Revision history for this message
Nastya Urlapova (aurlapova) wrote :
Revision history for this message
Nastya Urlapova (aurlapova) wrote :
Andrey Maximov (maximov)
Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Sergey Vasilenko (xenolog)
Revision history for this message
Nastya Urlapova (aurlapova) wrote :
Revision history for this message
Nastya Urlapova (aurlapova) wrote :
Revision history for this message
Nastya Urlapova (aurlapova) wrote :
Revision history for this message
Andrey Maximov (maximov) wrote :

looks like systematic issue exists. might be regression..

Changed in fuel:
importance: Undecided → High
status: New → Confirmed
Revision history for this message
Sergey Vasilenko (xenolog) wrote :

started custom tests (HA neutron GRE) for both OSes:
#156 -- centos
#157 -- ubuntu

After end of tests we should re-run ones tests with increased RAM up to 3G for each node and compare results.

Revision history for this message
Vladimir Kuklin (vkuklin) wrote :

Qa team please evaluate test results for increased RAM size for CentOS

Changed in fuel:
assignee: Sergey Vasilenko (xenolog) → Fuel QA Team (fuel-qa)
Revision history for this message
Nastya Urlapova (aurlapova) wrote :

After Sergey's analysis we will find solution.

Changed in fuel:
assignee: Fuel QA Team (fuel-qa) → Sergey Vasilenko (xenolog)
Revision history for this message
Vladimir Kuklin (vkuklin) wrote :

Comparing atop files of neutron vlan with public network:

Average swappiness of Centos installation is 500-700 megabytes
Average swappiness of Ubuntu installation is 200 megabytes

We can see in the logs that CentOS starts swapping when it installs keystone and heat on the controllers. This means that for some reason heat consumes a lot of memory. Also, there is significant amount of memory consumed by httpd.

So far we have two solutions here:

Increase amount of memory we use for CentOS in our tests
Tune heat, httpd and other services - by the way https://bugs.launchpad.net/fuel/+bug/1459977 should decrease httpd memory consumption.

Revision history for this message
Sergey Vasilenko (xenolog) wrote :

increasing RAM up to 3G lead to decrease controllers deployment time to 17min.

Changed in fuel:
assignee: Sergey Vasilenko (xenolog) → nobody
assignee: nobody → Fuel Library Team (fuel-library)
Revision history for this message
Nastya Urlapova (aurlapova) wrote :

@Sergey,
can you give a link to job results?

Revision history for this message
Nastya Urlapova (aurlapova) wrote :

After switching keystone to eventlet tests are become faster on Centos:
deploy_ha_dns_ntp:
  deploy_cluster_wait_00: '3335.99'
deploy_neutron_gre_ha:
  deploy_cluster_wait_00: '4092.93'
deploy_neutron_gre_ha_with_public_network:
  deploy_cluster_wait_00: '4162.63'
deploy_neutron_vlan_ha:
  deploy_cluster_wait_00: '4122.17'
deploy_neutron_vlan_ha_with_public_network:
  deploy_cluster_wait_00: '4182.30'
prepare_slaves_5:
  wait_for_nodes_to_start_and_register_in_nailgun_00: '303.45'
setup_master:
  setup_environment: '995.96'

Revision history for this message
Nastya Urlapova (aurlapova) wrote :

Moved to 7.0 due we will refactor keystone behaviour in 7.0.

Changed in fuel:
milestone: 6.1 → 7.0
Revision history for this message
Aleksandr Didenko (adidenko) wrote :

Since we're not going to support CentOS in 7.0, I'm marking it as invalid and moving to Future.

Changed in fuel:
status: Confirmed → Invalid
To post a comment you must log in.