percona cluster host not allowed even when unit in allowed_hosts

Bug #1513239 reported by David Ames
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Percona Cluster Charm
Fix Released
High
James Page
percona-cluster (Juju Charms Collection)
Invalid
High
David Ames

Bug Description

Errors relating keystone with percona-cluster

ERROR 1130 (HY000): Host 'juju-thedac-machine-2.openstacklocal' is not allowed to connect to this MySQL server

OperationalError: (OperationalError) (1130, "Host 'juju-thedac-machine-2.openstacklocal' is not allowed to connect to this MySQL server") None None

Keystone getting these errors even when relation-get shows keystone/n in allowed_hosts.

Requires "FLUSH PRIVILEGES;" after GRANT statements.
http://www.thegeekstuff.com/2010/08/allow-mysql-client-connection/

Tags: oil
David Ames (thedac)
affects: charms → percona-cluster (Juju Charms Collection)
Changed in percona-cluster (Juju Charms Collection):
importance: Undecided → High
assignee: nobody → David Ames (thedac)
status: New → In Progress
Revision history for this message
David Ames (thedac) wrote :

The issue is percona cluster is not allowing connection via the VIP without a restart of the node with the VIP.

Connections work to all mysql ips. But not to the VIP.

# mysql/0
root@juju-thedac-machine-2:~# nc -v 10.5.48.250 3306
Connection to 10.5.48.250 3306 port [tcp/mysql] succeeded!
O
5.5.37-35.0dS|;OYTy%��Gu8X1-.=)SHOmysql_native_password
^C

# mysql/1
root@juju-thedac-machine-2:~# nc -v 10.5.48.251 3306
Connection to 10.5.48.251 3306 port [tcp/mysql] succeeded!
O
5.5.37-35.0Mb#%0b{dc��Die>Dz\?e^'Dmysql_native_password
^C

# mysql/2
root@juju-thedac-machine-2:~# nc -v 10.5.48.252 3306
Connection to 10.5.48.252 3306 port [tcp/mysql] succeeded!
O
5.5.37-35.0VRFGq%`83��9dkne1bz`RW@mysql_native_password
^C

# VIP
root@juju-thedac-machine-2:~# nc -v 10.5.100.1 3306
Connection to 10.5.100.1 3306 port [tcp/mysql] succeeded!
]�jHost 'juju-thedac-machine-2.openstacklocal' is not allowed to connect to this MySQL serverroot@juju-thedac-machine-2:~#

Larry Michel (lmic)
tags: added: oil
Revision history for this message
Larry Michel (lmic) wrote :
Download full text (8.5 KiB)

We have recreated this for OpenStack deployment with mitaka and trusty.

  mysql:
    charm: cs:trusty/percona-cluster
    num_units: 1
    options:
      max-connections: 20000
      source: cloud:trusty-mitaka
    to:
    - lxc:1

The error is from unit-cinder.log:

2016-09-14 12:30:01 INFO shared-db-relation-changed exception.NotSupportedWarning
2016-09-14 12:30:01 INFO shared-db-relation-changed
2016-09-14 12:30:01 INFO shared-db-relation-changed 2016-09-14 12:30:01.436 58735 ERROR oslo_db.sqlalchemy.exc_filters [-] DBAPIError exception wrapped from (pymysql.err.InternalError) (1130, u"Host 'emerson.oil' is not allowed to connect to this MySQL server")
2016-09-14 12:30:01 INFO shared-db-relation-changed 2016-09-14 12:30:01.436 58735 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
2016-09-14 12:30:01 INFO shared-db-relation-changed 2016-09-14 12:30:01.436 58735 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect
2016-09-14 12:30:01 INFO shared-db-relation-changed 2016-09-14 12:30:01.436 58735 ERROR oslo_db.sqlalchemy.exc_filters return fn()
2016-09-14 12:30:01 INFO shared-db-relation-changed 2016-09-14 12:30:01.436 58735 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 318, in unique_connection
2016-09-14 12:30:01 INFO shared-db-relation-changed 2016-09-14 12:30:01.436 58735 ERROR oslo_db.sqlalchemy.exc_filters return _ConnectionFairy._checkout(self)
2016-09-14 12:30:01 INFO shared-db-relation-changed 2016-09-14 12:30:01.436 58735 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in _checkout
2016-09-14 12:30:01 INFO shared-db-relation-changed 2016-09-14 12:30:01.436 58735 ERROR oslo_db.sqlalchemy.exc_filters fairy = _ConnectionRecord.checkout(pool)
2016-09-14 12:30:01 INFO shared-db-relation-changed 2016-09-14 12:30:01.436 58735 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 480, in checkout
2016-09-14 12:30:01 INFO shared-db-relation-changed 2016-09-14 12:30:01.436 58735 ERROR oslo_db.sqlalchemy.exc_filters rec = pool._do_get()
2016-09-14 12:30:01 INFO shared-db-relation-changed 2016-09-14 12:30:01.436 58735 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1060, in _do_get
2016-09-14 12:30:01 INFO shared-db-relation-changed 2016-09-14 12:30:01.436 58735 ERROR oslo_db.sqlalchemy.exc_filters self._dec_overflow()
2016-09-14 12:30:01 INFO shared-db-relation-changed 2016-09-14 12:30:01.436 58735 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2016-09-14 12:30:01 INFO shared-db-relation-changed 2016-09-14 12:30:01.436 58735 ERROR oslo_db.sqlalchemy.exc_filters compat.reraise(exc_type, exc_value, exc_tb)
2016-09-14 12:30:01 INFO shared-db-relation-changed 2016-09-14 12:30:01.436 58735 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1057, in _do_get
2...

Read more...

Revision history for this message
Larry Michel (lmic) wrote :

Attaching logs.

Revision history for this message
Larry Michel (lmic) wrote :

In this case Keystone is deployed on a container in that host.

  '7':
    constraints: tags=emerson
    series: trusty

  keystone:
    charm: cs:trusty/keystone
    num_units: 1
    options:
      admin-password: ******
      admin-token: *******
      openstack-origin: cloud:trusty-mitaka
    to:
    - lxc:7

Revision history for this message
Ashley Lai (alai) wrote :
Download full text (3.3 KiB)

I'm seeing the same issue.

cinder/0* error idle 1 10.244.128.6 8776/tcp hook failed: "shared-db-relation-changed" for mysql:shared-db

Below is the traceback from the log. Please see full log attached.

2017-01-20 05:22:41 INFO shared-db-relation-changed 2017-01-20 05:22:41.864 14904 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/db/migration.py", line 57, in db_sync
2017-01-20 05:22:41 INFO shared-db-relation-changed 2017-01-20 05:22:41.864 14904 ERROR cinder engine = db_api.get_engine()
2017-01-20 05:22:41 INFO shared-db-relation-changed 2017-01-20 05:22:41.864 14904 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/db/sqlalchemy/api.py", line 88, in get_engine
2017-01-20 05:22:41 INFO shared-db-relation-changed 2017-01-20 05:22:41.864 14904 ERROR cinder facade = _create_facade_lazily()
2017-01-20 05:22:41 INFO shared-db-relation-changed 2017-01-20 05:22:41.864 14904 ERROR cinder File "/usr/lib/python2.7/dist-packages/cinder/db/sqlalchemy/api.py", line 75, in _create_facade_lazily
2017-01-20 05:22:41 INFO shared-db-relation-changed 2017-01-20 05:22:41.864 14904 ERROR cinder **dict(CONF.database)
2017-01-20 05:22:41 INFO shared-db-relation-changed 2017-01-20 05:22:41.864 14904 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 943, in __init__
2017-01-20 05:22:41 INFO shared-db-relation-changed 2017-01-20 05:22:41.864 14904 ERROR cinder slave_connection=slave_connection)
2017-01-20 05:22:41 INFO shared-db-relation-changed 2017-01-20 05:22:41.864 14904 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 338, in _start
2017-01-20 05:22:41 INFO shared-db-relation-changed 2017-01-20 05:22:41.864 14904 ERROR cinder engine_args, maker_args)
2017-01-20 05:22:41 INFO shared-db-relation-changed 2017-01-20 05:22:41.864 14904 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 362, in _setup_for_connection
2017-01-20 05:22:41 INFO shared-db-relation-changed 2017-01-20 05:22:41.864 14904 ERROR cinder sql_connection=sql_connection, **engine_kwargs)
2017-01-20 05:22:41 INFO shared-db-relation-changed 2017-01-20 05:22:41.864 14904 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", line 152, in create_engine
2017-01-20 05:22:41 INFO shared-db-relation-changed 2017-01-20 05:22:41.864 14904 ERROR cinder test_conn = _test_connection(engine, max_retries, retry_interval)
2017-01-20 05:22:41 INFO shared-db-relation-changed 2017-01-20 05:22:41.864 14904 ERROR cinder File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", line 334, in _test_connection
2017-01-20 05:22:41 INFO shared-db-relation-changed 2017-01-20 05:22:41.864 14904 ERROR cinder six.reraise(type(de_ref), de_ref)
2017-01-20 05:22:41 INFO shared-db-relation-changed 2017-01-20 05:22:41.864 14904 ERROR cinder File "<string>", line 2, in reraise
2017-01-20 05:22:41 INFO shared-db-relation-changed 2017-01-20 05:22:41.864 14904 ERROR cinder DBConnectionError: (pymysql.err.OperationalError) (2003, 'Can\'t connect to My...

Read more...

James Page (james-page)
Changed in charm-percona-cluster:
assignee: nobody → David Ames (thedac)
importance: Undecided → High
status: New → In Progress
Changed in percona-cluster (Juju Charms Collection):
status: In Progress → Invalid
James Page (james-page)
Changed in charm-percona-cluster:
assignee: David Ames (thedac) → James Page (james-page)
milestone: none → 17.05
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-percona-cluster (master)

Reviewed: https://review.openstack.org/440333
Committed: https://git.openstack.org/cgit/openstack/charm-percona-cluster/commit/?id=bda27479a45b69eff58a46a73813b1cc331bba8e
Submitter: Jenkins
Branch: master

commit bda27479a45b69eff58a46a73813b1cc331bba8e
Author: James Page <email address hidden>
Date: Thu Mar 2 10:34:31 2017 +0000

    Add tuning-level configuration option

    Inline with the mysql charm, add a tuning-level configuration option
    that allows end users to change the configuration profile for PXC.

    This option supports three values

       safest (default): use configuration options with best data
                         integrity guarantees.
       fast: compromise some data integrity guarantees
                         to improve performance.
       unsafe: pretty much throw away all data integrity
                         guarantees to maximise performance.

    In clustered deployments, 'fast' and 'unsafe' may be appropriate to
    use but should be considered carefully before reconfiguration away
    from the default 'safest' option.

    Right now, this option tweaks the innodb_flush_log_at_trx_commit
    value for PXC:

       safest (default): 1
       fast: 2
       unsafe: 0

    but should be used for other tuning optimizations that come along
    in the future.

    Also add direct configuration options for:

       innodb-change-buffering
       innodb-io-capacity

    to allow end users to tweak other performance optimizations that
    we can't yet do automatically using charm options.

    This commit also includes a resync of charm-helpers, which includes
    the fix to flush priviledges after adding grants to resolve bug
    1513239.

    Change-Id: I7c31e3bfbb825ae7091913e678dd1b7893892d1d
    Closes-Bug: 1599222
    Closes-Bug: 1513239

Changed in charm-percona-cluster:
status: In Progress → Fix Committed
James Page (james-page)
Changed in charm-percona-cluster:
milestone: 17.05 → 17.08
James Page (james-page)
Changed in charm-percona-cluster:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.