Juniper Openstack

ServerManager :: Keystone Authentication failures seen on multi interface HA setup.

Bug #1642509 reported by Ritam Gangopadhyay
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Invalid
High
Dheeraj Gautam
Juniper Openstack r4.0.0.0-fcs "r4.0.0.0"
R3.1
Won't Fix
High
Dheeraj Gautam
Juniper Openstack r3.1.2.0
R3.1.1.x
Won't Fix
High
Dheeraj Gautam
Juniper Openstack r3.1.1.1
R3.2
Won't Fix
High
Dheeraj Gautam
Trunk
Won't Fix
High
Dheeraj Gautam
Juniper Openstack r4.0.0.0-fcs "r4.0.0.0"

Bug Description

Build 42 R3.1 mitaka

host1 = 'root@10.204.217.13'
host2 = 'root@10.204.217.77'
host3 = 'root@10.204.217.176'
host4 = 'root@10.204.217.129'
host5 = 'root@10.204.217.131'
host6 = 'root@10.204.217.132'

env.roledefs = {
    'all': [host1, host2, host3, host4, host5, host6],
    'cfgm': [host1, host2, host3],
    'openstack': [host1, host2, host3],
    'control': [host1, host2, host3],
    'compute': [host4, host5, host6],
    'collector': [host1, host2, host3],
    'webui': [host1, host2, host3],
    'database': [host1, host2, host3],
    'build': [host_build],
}

env.hostnames ={
    'all': ['nodec28', 'nodeg37', 'nodec10', 'nodei17', 'nodei19', 'nodei20']
}

bond= {
    host6 : { 'name': 'bond0', 'member': ['eth1','eth2','eth3'],'mode':'802.3ad' },
}

control_data = {
    host1 : { 'ip': '192.168.100.11/24', 'gw' : '', 'device':'eth1' },
    host2 : { 'ip': '192.168.100.12/24', 'gw' : '', 'device':'eth1' },
    host3 : { 'ip': '192.168.100.13/24', 'gw' : '', 'device':'eth1' },
    host4 : { 'ip': '192.168.100.14/24', 'gw' : '', 'device':'eth1' },
    host5 : { 'ip': '192.168.100.15/24', 'gw' : '', 'device':'eth1' },
    host6 : { 'ip': '192.168.100.16/24', 'gw' : '', 'device':'bond0' },
}

env.ha = {
    'internal_vip' : '192.168.100.10',
}

root@nodec28:~# nova service-list --debug
DEBUG (extension:157) found extension EntryPoint.parse('v2token = keystoneauth1.loading._plugins.identity.v2:Token')
DEBUG (extension:157) found extension EntryPoint.parse('admin_token = keystoneauth1.loading._plugins.admin_token:AdminToken')
DEBUG (extension:157) found extension EntryPoint.parse('v3oidcauthcode = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectAuthorizationCode')
DEBUG (extension:157) found extension EntryPoint.parse('v2password = keystoneauth1.loading._plugins.identity.v2:Password')
DEBUG (extension:157) found extension EntryPoint.parse('v3password = keystoneauth1.loading._plugins.identity.v3:Password')
DEBUG (extension:157) found extension EntryPoint.parse('v3oidcpassword = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectPassword')
DEBUG (extension:157) found extension EntryPoint.parse('token = keystoneauth1.loading._plugins.identity.generic:Token')
DEBUG (extension:157) found extension EntryPoint.parse('v3token = keystoneauth1.loading._plugins.identity.v3:Token')
DEBUG (extension:157) found extension EntryPoint.parse('password = keystoneauth1.loading._plugins.identity.generic:Password')
DEBUG (extension:157) found extension EntryPoint.parse('password-ceilometer-legacy = ceilometer.keystone_client:LegacyCeilometerKeystoneLoader')
DEBUG (session:248) REQ: curl -g -i -X GET http://192.168.100.10:5000/v2.0/ -H "Accept: application/json" -H "User-Agent: keystoneauth1/2.4.0 python-requests/2.9.1 CPython/2.7.6"
INFO (connectionpool:207) Starting new HTTP connection (1): 192.168.100.10
DEBUG (connectionpool:387) "GET /v2.0/ HTTP/1.1" 200 340
DEBUG (session:277) RESP: [200] Content-Length: 340 Vary: X-Auth-Token X-Distribution: Ubuntu Date: Thu, 17 Nov 2016 08:31:10 GMT Content-Type: application/json X-Openstack-Request-Id: req-17074a06-e98e-4ad6-ab76-5dcbc30fb4fe
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://192.168.100.10:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}

DEBUG (v2:63) Making authentication request to http://192.168.100.10:5000/v2.0/tokens
DEBUG (shell:1082) Request to http://192.168.100.10:5000/v2.0/tokens timed out
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 1080, in main
    OpenStackComputeShell().main(argv)
  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 914, in main
    api_version = api_versions.discover_version(self.cs, api_version)
  File "/usr/lib/python2.7/dist-packages/novaclient/api_versions.py", line 267, in discover_version
    client)
  File "/usr/lib/python2.7/dist-packages/novaclient/api_versions.py", line 248, in _get_server_version_range
    version = client.versions.get_current()
  File "/usr/lib/python2.7/dist-packages/novaclient/v2/versions.py", line 84, in get_current
    return self._get_current()
  File "/usr/lib/python2.7/dist-packages/novaclient/v2/versions.py", line 56, in _get_current
    url = "%s" % self.api.client.get_endpoint()
  File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 132, in get_endpoint
    return self.session.get_endpoint(auth or self.auth, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 661, in get_endpoint
    return auth.get_endpoint(self, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 210, in get_endpoint
    service_catalog = self.get_access(session).service_catalog
  File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 136, in get_access
    self.auth_ref = self.get_auth_ref(session)
  File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py", line 181, in get_auth_ref
    return self._plugin.get_auth_ref(session, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/v2.py", line 65, in get_auth_ref
    authenticated=False, log=False)
  File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 572, in post
    return self.request(url, 'POST', **kwargs)
  File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 452, in request
    resp = send(**kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 493, in _send_request
    raise exceptions.ConnectTimeout(msg)
ConnectTimeout: Request to http://192.168.100.10:5000/v2.0/tokens timed out
ERROR (ConnectTimeout): Request to http://192.168.100.10:5000/v2.0/tokens timed out
root@nodec28:~#

/var/log/keystone/keystone-all.log
***********************************
***********************************

2016-11-17 00:31:10.734 22507 DEBUG keystone.middleware.auth [req-17074a06-e98e-4ad6-ab76-5dcbc30fb4fe - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set. _build_auth_context /usr/lib/python2.7/dist-packages/keystone/middleware/auth.py:71
2016-11-17 00:31:10.736 22507 INFO keystone.common.wsgi [req-17074a06-e98e-4ad6-ab76-5dcbc30fb4fe - - - - -] GET http://192.168.100.10:5000/v2.0/
2016-11-17 00:31:10.737 22507 INFO eventlet.wsgi.server [req-17074a06-e98e-4ad6-ab76-5dcbc30fb4fe - - - - -] 192.168.100.11 - - [17/Nov/2016 00:31:10] "GET /v2.0/ HTTP/1.1" 200 559 0.003378
2016-11-17 00:31:49.427 22498 DEBUG keystone.middleware.auth [req-c880b397-a1b0-4ab8-89ae-deb2fc4c050b - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set. _build_auth_context /usr/lib/python2.7/dist-packages/keystone/middleware/auth.py:71
2016-11-17 00:31:49.429 22498 INFO keystone.common.wsgi [req-c880b397-a1b0-4ab8-89ae-deb2fc4c050b - - - - -] POST http://192.168.100.10:35357/v2.0/tokens
2016-11-17 00:31:49.522 22498 INFO eventlet.wsgi.server [req-c880b397-a1b0-4ab8-89ae-deb2fc4c050b - - - - -] 192.168.100.11 - - [17/Nov/2016 00:31:49] "POST /v2.0/tokens HTTP/1.1" 200 4304 0.095897
2016-11-17 00:31:58.641 22506 DEBUG keystone.middleware.auth [req-34cba6b0-ad71-460b-bb52-1e47b44c685e 48ad5b806b1c4ddcb8c6325364fdbdf4 d0d3ef7d6cc64192ac108ab67dc209c7 - default default] RBAC: auth_context: {'is_delegated_auth': False, 'access_token_id': None, 'user_id': u'48ad5b806b1c4ddcb8c6325364fdbdf4', 'roles': [u'admin'], 'user_domain_id': 'default', 'trustee_id': None, 'trustor_id': None, 'consumer_id': None, 'token': <KeystoneToken (audit_id=o8XX_6LESA-Pp4wza4jp6w, audit_chain_id=o8XX_6LESA-Pp4wza4jp6w) at 0x7fe6c9a96e30>, 'project_id': u'd0d3ef7d6cc64192ac108ab67dc209c7', 'trust_id': None, 'project_domain_id': 'default'} process_request /usr/lib/python2.7/dist-packages/keystone/middleware/auth.py:221
2016-11-17 00:31:58.642 22506 INFO keystone.common.wsgi [req-34cba6b0-ad71-460b-bb52-1e47b44c685e 48ad5b806b1c4ddcb8c6325364fdbdf4 d0d3ef7d6cc64192ac108ab67dc209c7 - default default] GET http://192.168.100.10:35357/v3/auth/tokens
2016-11-17 00:31:58.643 22506 DEBUG keystone.common.controller [req-34cba6b0-ad71-460b-bb52-1e47b44c685e 48ad5b806b1c4ddcb8c6325364fdbdf4 d0d3ef7d6cc64192ac108ab67dc209c7 - default default] RBAC: Authorizing identity:validate_token() _build_policy_check_credentials /usr/lib/python2.7/dist-packages/keystone/common/controller.py:80
2016-11-17 00:31:58.643 22506 DEBUG keystone.common.controller [req-34cba6b0-ad71-460b-bb52-1e47b44c685e 48ad5b806b1c4ddcb8c6325364fdbdf4 d0d3ef7d6cc64192ac108ab67dc209c7 - default default] RBAC: using auth context from the request environment _build_policy_check_credentials /usr/lib/python2.7/dist-packages/keystone/common/controller.py:85
2016-11-17 00:31:58.653 22506 DEBUG keystone.policy.backends.rules [req-34cba6b0-ad71-460b-bb52-1e47b44c685e 48ad5b806b1c4ddcb8c6325364fdbdf4 d0d3ef7d6cc64192ac108ab67dc209c7 - default default] enforce identity:validate_token: {'is_delegated_auth': False, 'access_token_id': None, 'user_id': u'48ad5b806b1c4ddcb8c6325364fdbdf4', 'roles': [u'admin'], 'user_domain_id': 'default', 'trustee_id': None, 'trustor_id': None, 'consumer_id': None, 'token': <KeystoneToken (audit_id=o8XX_6LESA-Pp4wza4jp6w, audit_chain_id=o8XX_6LESA-Pp4wza4jp6w) at 0x7fe6c9a96e30>, 'project_id': u'd0d3ef7d6cc64192ac108ab67dc209c7', 'trust_id': None, 'project_domain_id': 'default'} enforce /usr/lib/python2.7/dist-packages/keystone/policy/backends/rules.py:76
2016-11-17 00:31:58.654 22506 DEBUG keystone.common.controller [req-34cba6b0-ad71-460b-bb52-1e47b44c685e 48ad5b806b1c4ddcb8c6325364fdbdf4 d0d3ef7d6cc64192ac108ab67dc209c7 - default default] RBAC: Authorization granted inner /usr/lib/python2.7/dist-packages/keystone/common/controller.py:180
2016-11-17 00:31:58.737 22506 INFO eventlet.wsgi.server [req-34cba6b0-ad71-460b-bb52-1e47b44c685e 48ad5b806b1c4ddcb8c6325364fdbdf4 d0d3ef7d6cc64192ac108ab67dc209c7 - default default] 192.168.100.11 - - [17/Nov/2016 00:31:58] "GET /v3/auth/tokens HTTP/1.1" 200 7415 0.109045

sundarkh (sundar-kh)
Changed in juniperopenstack:
milestone: r3.1.1.0 → none
Abhay Joshi (abhayj)
Changed in juniperopenstack:
assignee: Abhay Joshi (abhayj) → Dheeraj Gautam (dgautam)
Revision history for this message
Dheeraj Gautam (dgautam) wrote :
Download full text (4.3 KiB)

keystone endpoint-list was being stuck because of mysql. mysql was stuck for some reason. on killing mysql and starting it back, system recovered. many of other mysql commands were stuck as well.

root@nodec10:/var/log/mysql# ps ax| grep mysql
  609 ? S 0:00 mysql -ucmon -px xx -hlocalhost -P3306 -e use cmon; truncate table cmon_log;
 5059 ? S 0:00 mysql -ucmon -px xx -hlocalhost -P3306 -e use cmon; truncate table cmon_log;
 8319 ? S 0:00 /bin/sh /usr/bin/mysqld_safe
 9683 ? Sl 67:41 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --log-error=/var/log/mysql/error.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306 --wsrep_start_position=00000000-0000-0000-0000-000000000000:-1
14121 ? S 0:00 mysql -ucmon -px xx -hlocalhost -P3306 -e use cmon; truncate table cmon_log;
15248 ? S 0:00 mysql -ucmon -px xx -hlocalhost -P3306 -e use cmon; truncate table cmon_log;
26339 pts/16 S+ 0:00 grep --color=auto mysql
26677 ? S 0:00 mysql -ucmon -px xx -hlocalhost -P3306 -e use cmon; truncate table cmon_log;
27280 ? S 0:00 mysql -ucmon -px xx -hlocalhost -P3306 -e use cmon; truncate table cmon_log;
27396 ? S 0:00 mysql -ucmon -px xx -hlocalhost -P3306 -e use cmon; truncate table cmon_log;
28155 ? S 0:00 mysql -ucmon -px xx -hlocalhost -P3306 -e use cmon; truncate table cmon_log;
28204 ? S 0:00 mysql -ucmon -px xx -hlocalhost -P3306 -e use cmon; truncate table cmon_log;
28501 ? S 0:00 mysql -ucmon -px xx -hlocalhost -P3306 -e use cmon; truncate table cmon_log;
28681 ? S 0:00 mysql -ucmon -px xx -hlocalhost -P3306 -e use cmon; truncate table cmon_log;
28942 ? S 0:00 mysql -ucmon -px xx -hlocalhost -P3306 -e use cmon; truncate table cmon_log;
29165 ? S 0:00 mysql -ucmon -px xx -hlocalhost -P3306 -e use cmon; truncate table cmon_log;
29380 ? S 0:00 mysql -ucmon -px xx -hlocalhost -P3306 -e use cmon; truncate table cmon_log;
30772 ? S 0:00 mysql -ucmon -px xx -hlocalhost -P3306 -e use cmon; truncate table cmon_log;
31788 ? S 0:00 mysql -ucmon -px xx -hlocalhost -P3306 -e use cmon; truncate table cmon_log;
32297 ? S 0:00 mysql -ucmon -px xx -hlocalhost -P3306 -e use cmon; truncate table cmon_log;

keystone-all.log reported following errors:
2016-11-17 11:46:05.224 30182 WARNING oslo_db.sqlalchemy.engines [req-898fff2f-fc93-42ce-a707-5dfcd8935aa7 - - - - -] SQL connection failed. -14 attempts left.
2016-11-17 11:46:06.231 30179 WARNING oslo_db.sqlalchemy.engines [req-f386c607-f517-4a83-83eb-281f984527b2 - - - - -] SQL connection failed. -14 attempts left.
2016-11-17 11:46:08.245 30181 WARNING oslo_db.sqlalchemy.engines [req-f66b6d86-872c-4bcd-9506-afea2aaf7a49 - - - - -] SQL connection failed. -15 attempts left.
2016-11-17 11:46:08.246 30180 WARNING oslo_db.sqlalchemy.engines [req-cb3a0244-836e-4490-ada0-4dd7edaba110 - - - - -] SQL connection failed. -15 attempts left.
2016-11-17 11:46:09.688 30183 WARNING oslo_db.sqlalchemy.engines [req-67023e8...

Read more...

Revision history for this message
Jeba Paulaiyan (jebap) wrote :

Workaround: Restart mysql

Changed in juniperopenstack:
importance: Critical → High
Revision history for this message
Ritam Gangopadhyay (ritam) wrote :

Even after restarting MySql seeing intermittent issues in connection

root@nodec28:~# source /etc/contrail/openstackrc
root@nodec28:~# nova service-list
ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_db.exception.DBConnectionError'> (HTTP 500) (Request-ID: req-732ffdca-f8a8-4c8a-859e-fb11da4f5bdf)
root@nodec28:~#

root@nodec28:~# contrail-status | grep api
contrail-analytics-api active
contrail-api:0 initializing (Generic Connection:Keystone[Error: An unexpected error prevented the server from fulfilling your request: (pymysql.err.OperationalError) (2003, 'Can\'t connect to MySQL server on \'192.168.100.11\' ((2013, "Lost connection to MySQL server during query (error(104, \'Connection reset by peer\'))"))') (Disable debug mode to suppress these details.) (HTTP 500) (Request-ID: req-11deeccd-5305-4f98-8b95-e885ffa1fb58) at UTC 2016-11-18 13:45:27.912092] connection down)
root@nodec28:~#

Revision history for this message
Ritam Gangopadhyay (ritam) wrote :
Download full text (3.4 KiB)

Seeing this trace in Nova-api.log

2016-11-18 06:07:48.407 2228 DEBUG oslo.messaging._drivers.impl_rabbit [-] Received recoverable error from kombu: on_error /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py:635
2016-11-18 06:07:48.407 2228 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2016-11-18 06:07:48.407 2228 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 436, in _ensured
2016-11-18 06:07:48.407 2228 ERROR oslo.messaging._drivers.impl_rabbit return fun(*args, **kwargs)
2016-11-18 06:07:48.407 2228 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 508, in __call__
2016-11-18 06:07:48.407 2228 ERROR oslo.messaging._drivers.impl_rabbit return fun(*args, channel=channels[0], **kwargs), channels[0]
2016-11-18 06:07:48.407 2228 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 684, in execute_method
2016-11-18 06:07:48.407 2228 ERROR oslo.messaging._drivers.impl_rabbit method()
2016-11-18 06:07:48.407 2228 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 910, in _consume
2016-11-18 06:07:48.407 2228 ERROR oslo.messaging._drivers.impl_rabbit self.connection.drain_events(timeout=poll_timeout)
2016-11-18 06:07:48.407 2228 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 275, in drain_events
2016-11-18 06:07:48.407 2228 ERROR oslo.messaging._drivers.impl_rabbit return self.transport.drain_events(self.connection, **kwargs)
2016-11-18 06:07:48.407 2228 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 95, in drain_events
2016-11-18 06:07:48.407 2228 ERROR oslo.messaging._drivers.impl_rabbit return connection.drain_events(**kwargs)
2016-11-18 06:07:48.407 2228 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 302, in drain_events
2016-11-18 06:07:48.407 2228 ERROR oslo.messaging._drivers.impl_rabbit chanmap, None, timeout=timeout,
2016-11-18 06:07:48.407 2228 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 365, in _wait_multiple
2016-11-18 06:07:48.407 2228 ERROR oslo.messaging._drivers.impl_rabbit channel, method_sig, args, content = read_timeout(timeout)
2016-11-18 06:07:48.407 2228 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 336, in read_timeout
2016-11-18 06:07:48.407 2228 ERROR oslo.messaging._drivers.impl_rabbit return self.method_reader.read_method()
2016-11-18 06:07:48.407 2228 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 189, in read_method
2016-11-18 06:07:48.407 2228 ERROR oslo.messaging._drivers.impl_rabbit raise m
2016-11-18 06:07:48.407 2228 ERROR oslo.messaging._drivers.impl_rabbit IOError: Soc...

Read more...

Revision history for this message
Ritam Gangopadhyay (ritam) wrote :

Rebooting the cfgm-0 on which we were seeing mysql issues gets us out of the problem.

sundarkh (sundar-kh)
information type: Proprietary → Public
Revision history for this message
sundarkh (sundar-kh) wrote :
Download full text (5.2 KiB)

Seen with R3.2 SM build 2, and Contrail package R3.2 build 5 as well

nodec28 (cfgm0) in the above mentioned setup becomes, very slow to the execution of commands

contrail-api.log shows same trace as mentioned in the comment #3

Before start of test case

root@nodec28:~# ps -aux | grep mysql
mysql 1363 2.1 0.7 12656876 258324 ? Sl Nov30 4:13 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --log-error=/var/log/mysql/error.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306 --wsrep_start_position=e133cfb1-b736-11e6-872d-3b4849414c8d:98762
root 26284 0.0 0.0 10464 936 pts/17 S+ 02:26 0:00 grep --color=auto mysql
root 31396 0.0 0.0 4440 752 ? S Nov30 0:00 /bin/sh /usr/bin/mysqld_safe
root@nodec28:~#
root@nodec28:~#
root@nodec28:~# After the Start of the test cases

root@nodec28:~# ps -aux | grep mysql
mysql 1363 2.1 0.7 12656876 258324 ? Sl Nov30 4:14 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --log-error=/var/log/mysql/error.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306 --wsrep_start_position=e133cfb1-b736-11e6-872d-3b4849414c8d:98762
root 29605 0.0 0.0 99716 2052 ? S 02:27 0:00 /usr/bin/mysql --connect_timeout 2 -h localhost --port 3306 -u cmon -px xx -e show global status where variable_name='wsrep_local_state'
root 29612 0.0 0.0 99716 2048 ? S 02:27 0:00 /usr/bin/mysql --connect_timeout 2 -h localhost --port 3306 -u cmon -px xx -e show global status where variable_name='wsrep_local_state'
root 29619 0.0 0.0 99716 2048 ? S 02:27 0:00 /usr/bin/mysql --connect_timeout 2 -h localhost --port 3306 -u cmon -px xx -e show global status where variable_name='wsrep_local_state'
root 29664 0.0 0.0 99716 2052 ? S 02:27 0:00 /usr/bin/mysql --connect_timeout 2 -h localhost --port 3306 -u cmon -px xx -e show global status where variable_name='wsrep_local_state'
root 29671 0.0 0.0 99716 2044 ? S 02:27 0:00 /usr/bin/mysql --connect_timeout 2 -h localhost --port 3306 -u cmon -px xx -e show global status where variable_name='wsrep_local_state'
root 29678 0.0 0.0 99716 2048 ? S 02:27 0:00 /usr/bin/mysql --connect_timeout 2 -h localhost --port 3306 -u cmon -px xx -e show global status where variable_name='wsrep_local_state'
root 29685 0.0 0.0 99716 2048 ? S 02:27 0:00 /usr/bin/mysql --connect_timeout 2 -h localhost --port 3306 -u cmon -px xx -e show global status where variable_name='wsrep_local_state'
root 29693 0.0 0.0 99716 2044 ? S 02:27 0:00 /usr/bin/mysql --connect_timeout 2 -h localhost --port 3306 -u cmon -px xx -e show global status where variable_name='wsrep_local_state'
root 29701 0.0 0.0 99716 2052 ? S 02:27 0:00 /usr/bin/mysql --connect_timeout 2 -h localhost --port 3306 -u cmon -px xx -e show global status where variable_name='wsrep_local_state'
root 29707 0.0 0.0 99716 2048 ? S 02:27 0:...

Read more...

Revision history for this message
sundarkh (sundar-kh) wrote :

With R3.2 mitaka Build 8 in this setup:
----------------------------------------
== Contrail Config ==
supervisor-config: active
contrail-api:0 initializing (Generic Connection:Keystone[Error: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-dfa812b0-a405-4193-8375-2f16167850b3) at UTC 2016-12-04 18:47:21.369316] connection down)
contrail-config-nodemgr active
contrail-device-manager backup
contrail-discovery active
contrail-schema backup
contrail-svc-monitor backup
ifmap active

Mysql processes is dead

sundarkh (sundar-kh)
tags: added: blocker
Revision history for this message
sundarkh (sundar-kh) wrote :

contrail services after provision is fine.
Setup goes to this issue state, during sanity is run.
Once this state is reached, nodes become slow in response for any commands issue, and reboot of cfgm0 helps to recover

Revision history for this message
Dheeraj Gautam (dgautam) wrote :

Removing blocker tag from this as this happens only in the case of sanity, manual test cases doesn't fail and other testing so far didn't found keystone issue.

tags: removed: blocker
Revision history for this message
Abhay Joshi (abhayj) wrote :

We will continue debugging, but this is old issue from 3.1 onwards. It seems to happen in sanity setup only. Provisioning seems to complete successfully, but MySQL goes down when sanity tests (possibly parallel run) are executed. Needs more debugging, but not high priority. Move out of 3.2.

Revision history for this message
Dheeraj Gautam (dgautam) wrote :

This was a setup issue of SM cluster used sanity. on using high end servers, this problem goes away. So closing the bug now.

Changed in juniperopenstack:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.