[regression][Keystone] Keystone detached plugin cannot connect to Mysql

snapshot https://drive.google.com/file/d/0B8UU8DiYLmsWeVdKNUd2ZWpqdUE/view?usp=sharing

This bug is blocker for QA acceptance

Evgeny,

Can you please share the criteria you used to decide this is a keystone
issue?
You are not the first, so I think it's a system problem that leads to such
decisions.

On Wed, Jun 22, 2016 at 8:13 PM, Sergii Turivnyi <email address hidden>
wrote:

> ** Tags added: fuel-dev-tools
>
> --
> You received this bug notification because you are a member of MOS
> Keystone, which is a bug assignee.
> https://bugs.launchpad.net/bugs/1595219
>
> Title:
> [Keystone] Keystone plugin cannot connect to Mysql
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/fuel/+bug/1595219/+subscriptions
>

--
Kind Regards,
Alexander Makarov,
Senior Software Developer,

Mirantis, Inc.
35b/3, Vorontsovskaya St., 109147, Moscow, Russia

Tel.: +7 (495) 640-49-04
Tel.: +7 (926) 204-50-60

Skype: MAKAPOB.AJIEKCAHDP

Aleksandr, we just wanted to get help in debugging from team who understand the context of KEystone requirements/deployment workflow.

It looks like fuel-library regression issue.

Can you please provide the exact environment setup that this fails with? The snapshot does not contain all the logs so we'll need to be able to reproduce it.

Also it should be noted that we test this in swarm, which is currently passing so we need more details as to how this was configured.

I got access to an environment and it looks like the keystone/db task ran before all the database tasks were fully run so the database was not up yet when the task was run.

The detached-db database task started at
2016-06-22T16:03:28.400247+00:00 notice: (Scope(Class[Osnailyfacter::Database::Database])) MODULAR: database/database.pp
but does not complete until
2016-06-22T16:04:37.059760+00:00 notice: Finished catalog run in 64.94 seconds

Meanwhile the keystone/db task was run at...
2016-06-22T16:03:37.869731+00:00 notice: (Scope(Class[Openstack_tasks::Keystone::Db])) MODULAR: keystone/db.pp
2016-06-22T16:03:47.721773+00:00 notice: (Scope(Class[Openstack_tasks::Keystone::Db])) MODULAR: keystone/db.pp
2016-06-22T16:03:52.346289+00:00 notice: (Scope(Class[Openstack_tasks::Keystone::Db])) MODULAR: keystone/db.pp

so all retries occurred before the database was run.

RCA: detached keystone keystone-db task does not properly wait for all database tasks before attempting to to use the database. It only waited for the database tasks on the controllers so when this paired with the detached-database plugin the keystone-db task may run before the database is actually setup. As part of Bug 1591256, we updated the core keystone-db task to wait for all the database tasks before starting and so this needs to get ported to the plugin as well. See https://github.com/openstack/fuel-library/commit/bedd29639a500038a4d9f864f1d3d5a6e50345cc

ETA: 6/24/16

master: https://review.openstack.org/#/c/332969/
stable/mitaka: https://review.openstack.org/#/c/332970/

Marking non-release since it doesn't affect core fuel functionality

can reproduce
Could not prefetch mysql_grant provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 2

Steps to reproduce
1. git clone https://github.com/stackforge/fuel-plugin-detach-keystone
2. git fetch https://git.openstack.org/openstack/fuel-plugin-detach-keystone refs/changes/70/332970/1 && git checkout FETCH_HEAD
3. fpb --build fuel-plugin-detach-keystone
4. copy plugin to the master-node
5. Create cluster
6. Deploy cluster with plugin

Actual result:
Could not prefetch mysql_grant provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 2

Related fix proposed to branch: master
Review: https://review.openstack.org/333364

@Sergii, Please try recreating the described issue from the bug with the plugin generated from the change. You can download the rpm from https://plugin-ci.fuel-infra.org/job/9.0.fuel-plugin.detach-keystone.build/4/artifact/plugin/detach-keystone-1.0-1.0.2-1.noarch.rpm

I wrote a fuel-qa test that describes an environment from the original report as well, https://review.openstack.org/#/c/333364/

I was unable to reproduce this error with these, I'm continuing to try and reproduce it.

So I took the bug conditions and created a fuel-qa test case (https://review.openstack.org/#/c/333364/) for this scenario. I then ran it using the existing detached plugins with fuel-qa and was able to reproduce the failure. The fuel-qa log can be viewed here: http://paste.openstack.org/show/521760/

I then took the rpm from the plugin-ci from https://review.openstack.org/#/c/332970 and ran it through the same test case about 4 times and was unable to reproduce the issue. The log can be viewed here, http://paste.openstack.org/show/521761/

I believe the proposed detached-keystone fixes address the issue and we now have an automated way to test this scenario.

Moved from Crit to High after discussions with release, dev and product managers.

Since we don't support detached keystone plugin in 9.1 release the issue moved to "Fix Released"

Change abandoned by Alex Schultz (<email address hidden>) on branch: master
Review: https://review.openstack.org/333364