[OSTF] "Can not find any online database" when hostname is not the default

Bug #1606195 reported by Kevin Decherf
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Confirmed
High
Fuel QA Team
Mitaka
Confirmed
High
Fuel QA Team

Bug Description

Detailed bug description:
When running OSTF on an operational cluster (3 controllers, 3 computes, 6
OSDs), it fails on all HA-related MySQL tests with the error "Can not find any
online database. Check that at least one database is operable".

All names and hostnames of nodes were updated before deployment and don't have
the default "node-%d" form.

After further investigation the issue seems to be lying in get_database_nodes()
from /usr/lib/python2.7/site-packages/fuel_health/tests/ha/test_mysql_status.py
where the following code block (L62-64):

----
hiera_cmd = ('ruby -e \'require "hiera";'
             'db = Hiera.new().lookup("database_nodes", {}, {}).keys;'
             'if db != [] then puts db else puts "None" end\'')
----

produces the following output (in my case):
----
node-54
node-58
node-60
----

Unfortunately the next code block does not match these hosts as long as it
explicitly checks hostnames (L69-72):
----
for node in cls.config.compute.nodes:
    hostname = node['hostname']
    if hostname in database_nodes and node['online']:
        databases.append(hostname)
----

Steps to reproduce:
- Create a new env
- Add nodes
- Change name and hostname of nodes (fuel node --name, fuel node --hostname)
- Deploy the env
- Launch OSTF

Expected result:
All checks pass

Actual result:
All HA-related MySQL checks fail:
- Check data replication over mysql
- Check if amount of tables in databases is the same on each node
- Check galera environment state

Workaround:

The following heira_cmd fixes the issue:
----
hiera_cmd = ('ruby -e \'require "hiera";'
             'db = Hiera.new().lookup("database_nodes", {}, {});'
             'if db.keys != [] then db.each_value {|value| puts value["name"]} else puts "None" end\'')
----

In my case hostnames and node names are the same, I'm not sure if it works for
the case when name differs.

Description of the environment:
[root@fuel ~]# shotgun2 short-report
cat /etc/fuel_build_id:
 4243
cat /etc/fuel_build_number:
 4243
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-release-9.0.0-1.mos6345.noarch
 fuel-misc-9.0.0-1.mos8420.noarch
 fuel-bootstrap-cli-9.0.0-1.mos284.noarch
 fuel-migrate-9.0.0-1.mos8420.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8717.noarch
 fuel-mirror-9.0.0-1.mos137.noarch
 fuel-notify-9.0.0-1.mos8420.noarch
 nailgun-mcagents-9.0.0-1.mos748.noarch
 python-fuelclient-9.0.0-1.mos319.noarch
 fuelmenu-9.0.0-1.mos270.noarch
 fuel-9.0.0-1.mos6345.noarch
 fuel-utils-9.0.0-1.mos8420.noarch
 fuel-setup-9.0.0-1.mos6345.noarch
 fuel-library9.0-9.0.0-1.mos8420.noarch
 shotgun-9.0.0-1.mos90.noarch
 fuel-agent-9.0.0-1.mos284.noarch
 fuel-ui-9.0.0-1.mos2708.noarch
 fuel-ostf-9.0.0-1.mos934.noarch
 python-packetary-9.0.0-1.mos137.noarch
 fuel-nailgun-9.0.0-1.mos8717.noarch
 rubygem-astute-9.0.0-1.mos748.noarch
 network-checker-9.0.0-1.mos72.x86_64
 fuel-openstack-metadata-9.0.0-1.mos8717.noarch

Revision history for this message
ivano (l-ivan) wrote :

this is reproducible in my setup also.

ivano (l-ivan)
Changed in fuel:
status: New → Confirmed
Changed in fuel:
milestone: none → 9.1
Changed in fuel:
importance: Undecided → High
assignee: nobody → Fuel QA Team (fuel-qa)
milestone: 9.1 → 10.0
Revision history for this message
Vladimir Khlyunev (vkhlyunev) wrote :

10.0 already fixed in https://review.openstack.org/#/c/317660/
backport is required

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.