OpenStack Percona Cluster Charm

  1. Bug #1413620
  2. Comment #4

Comment 4 for bug 1413620

Revision history for this message
David Britton (dpb) wrote : Re: [Bug 1413620] Re: Error after reboot: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

I see something like the following on all nodes of the cluster.

150513 18:23:29 mysqld_safe Starting mysqld daemon with databases from
/var/lib/mysql
150513 18:23:29 mysqld_safe WSREP: Running position recovery with
--log_error='/var/lib/mysql/wsrep_recovery.x65OEf'
--pid-file='/var/lib/mysql/juju-machine-13-lxc-1-recover.pid'
150513 18:23:31 mysqld_safe WSREP: Recovered position
b68fd0cd-f8ff-11e4-bf32-8b63f4f6930e:265378
150513 18:23:31 [Note] WSREP: wsrep_start_position var submitted:
'b68fd0cd-f8ff-11e4-bf32-8b63f4f6930e:265378'
150513 18:23:31 [Note] WSREP: Read nil XID from storage engines, skipping
position init
150513 18:23:31 [Note] WSREP: wsrep_load(): loading provider library
'/usr/lib/libgalera_smm.so'
150513 18:23:31 [Note] WSREP: wsrep_load(): Galera 2.8(r165) by Codership
Oy <email address hidden> loaded successfully.
150513 18:23:31 [Note] WSREP: Found saved state:
b68fd0cd-f8ff-11e4-bf32-8b63f4f6930e:-1
150513 18:23:31 [Note] WSREP: Reusing existing
'/var/lib/mysql//galera.cache'.
150513 18:23:31 [Note] WSREP: Passing config to GCS: base_host =
172.27.101.175; base_port = 4567; cert.log_conflicts = no; gcache.dir =
/var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0;
gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M;
gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16;
gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle =
0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit =
0.25; gcs.sync_donor = NO; replicator.causal_read_timeout = PT30S;
replicator.commit_order = 3
150513 18:23:31 [Note] WSREP: Assign initial position for certification:
265378, protocol version: -1
150513 18:23:31 [Note] WSREP: wsrep_sst_grab()
150513 18:23:31 [Note] WSREP: Start replication
150513 18:23:31 [Note] WSREP: Setting initial position to
b68fd0cd-f8ff-11e4-bf32-8b63f4f6930e:265378
150513 18:23:31 [Note] WSREP: protonet asio version 0
150513 18:23:31 [Note] WSREP: backend: asio
150513 18:23:31 [Note] WSREP: GMCast version 0
150513 18:23:31 [Note] WSREP: (28905a6f-f99d-11e4-8794-afa4dadb86b5, 'tcp://
0.0.0.0:4567') listening at tcp://0.0.0.0:4567
150513 18:23:31 [Note] WSREP: (28905a6f-f99d-11e4-8794-afa4dadb86b5, 'tcp://
0.0.0.0:4567') multicast: , ttl: 1
150513 18:23:31 [Note] WSREP: EVS version 0
150513 18:23:31 [Note] WSREP: PC version 0
150513 18:23:31 [Note] WSREP: gcomm: connecting to group 'juju_cluster',
peer '172.27.101.175:,172.27.101.134:,172.27.101.146:'
150513 18:23:31 [Warning] WSREP: (28905a6f-f99d-11e4-8794-afa4dadb86b5,
'tcp://0.0.0.0:4567') address 'tcp://172.27.101.175:4567' points to own
listening address, blacklisting
150513 18:23:31 [Note] WSREP: (28905a6f-f99d-11e4-8794-afa4dadb86b5, 'tcp://
0.0.0.0:4567') address 'tcp://172.27.101.175:4567' pointing to uuid
28905a6f-f99d-11e4-8794-afa4dadb86b5 is blacklisted, skipping
150513 18:23:31 [Note] WSREP: declaring
264c0c94-f99d-11e4-8da9-0345cc984ee5 stable
150513 18:23:31 [Warning] WSREP: no nodes coming from prim view, prim not
possible
150513 18:23:31 [Note] WSREP:
view(view_id(NON_PRIM,264c0c94-f99d-11e4-8da9-0345cc984ee5,2) memb {
        264c0c94-f99d-11e4-8da9-0345cc984ee5,
        28905a6f-f99d-11e4-8794-afa4dadb86b5,
} joined {
} left {
} partitioned {
})
150513 18:23:32 [Note] WSREP: (28905a6f-f99d-11e4-8794-afa4dadb86b5, 'tcp://
0.0.0.0:4567') address 'tcp://172.27.101.175:4567' pointing to uuid
28905a6f-f99d-11e4-8794-afa4dadb86b5 is blacklisted, skipping
150513 18:23:34 [Note] WSREP: (28905a6f-f99d-11e4-8794-afa4dadb86b5, 'tcp://
0.0.0.0:4567') address 'tcp://172.27.101.175:4567' pointing to uuid
28905a6f-f99d-11e4-8794-afa4dadb86b5 is blacklisted, skipping
150513 18:23:35 [Note] WSREP: (28905a6f-f99d-11e4-8794-afa4dadb86b5, 'tcp://
0.0.0.0:4567') address 'tcp://172.27.101.175:4567' pointing to uuid
28905a6f-f99d-11e4-8794-afa4dadb86b5 is blacklisted, skipping
150513 18:23:37 [Note] WSREP: (28905a6f-f99d-11e4-8794-afa4dadb86b5, 'tcp://
0.0.0.0:4567') address 'tcp://172.27.101.175:4567' pointing to uuid
28905a6f-f99d-11e4-8794-afa4dadb86b5 is blacklisted, skipping
150513 18:23:38 [Note] WSREP: (28905a6f-f99d-11e4-8794-afa4dadb86b5, 'tcp://
0.0.0.0:4567') address 'tcp://172.27.101.175:4567' pointing to uuid
28905a6f-f99d-11e4-8794-afa4dadb86b5 is blacklisted, skipping
150513 18:23:40 [Note] WSREP: (28905a6f-f99d-11e4-8794-afa4dadb86b5, 'tcp://
0.0.0.0:4567') address 'tcp://172.27.101.175:4567' pointing to uuid
28905a6f-f99d-11e4-8794-afa4dadb86b5 is blacklisted, skipping
150513 18:23:41 [Note] WSREP: (28905a6f-f99d-11e4-8794-afa4dadb86b5, 'tcp://
0.0.0.0:4567') address 'tcp://172.27.101.175:4567' pointing to uuid
28905a6f-f99d-11e4-8794-afa4dadb86b5 is blacklisted, skipping
150513 18:23:43 [Note] WSREP: (28905a6f-f99d-11e4-8794-afa4dadb86b5, 'tcp://
0.0.0.0:4567') address 'tcp://172.27.101.175:4567' pointing to uuid
28905a6f-f99d-11e4-8794-afa4dadb86b5 is blacklisted, skipping
150513 18:23:44 [Note] WSREP: (28905a6f-f99d-11e4-8794-afa4dadb86b5, 'tcp://
0.0.0.0:4567') address 'tcp://172.27.101.175:4567' pointing to uuid
28905a6f-f99d-11e4-8794-afa4dadb86b5 is blacklisted, skipping
150513 18:23:46 [Note] WSREP: (28905a6f-f99d-11e4-8794-afa4dadb86b5, 'tcp://
0.0.0.0:4567') address 'tcp://172.27.101.175:4567' pointing to uuid
28905a6f-f99d-11e4-8794-afa4dadb86b5 is blacklisted, skipping
150513 18:23:47 [Note] WSREP: (28905a6f-f99d-11e4-8794-afa4dadb86b5, 'tcp://
0.0.0.0:4567') address 'tcp://172.27.101.175:4567' pointing to uuid
28905a6f-f99d-11e4-8794-afa4dadb86b5 is blacklisted, skipping
150513 18:23:49 [Note] WSREP: (28905a6f-f99d-11e4-8794-afa4dadb86b5, 'tcp://
0.0.0.0:4567') address 'tcp://172.27.101.175:4567' pointing to uuid
28905a6f-f99d-11e4-8794-afa4dadb86b5 is blacklisted, skipping
150513 18:23:50 [Note] WSREP: (28905a6f-f99d-11e4-8794-afa4dadb86b5, 'tcp://
0.0.0.0:4567') address 'tcp://172.27.101.175:4567' pointing to uuid
28905a6f-f99d-11e4-8794-afa4dadb86b5 is blacklisted, skipping
150513 18:23:52 [Note] WSREP: (28905a6f-f99d-11e4-8794-afa4dadb86b5, 'tcp://
0.0.0.0:4567') address 'tcp://172.27.101.175:4567' pointing to uuid
28905a6f-f99d-11e4-8794-afa4dadb86b5 is blacklisted, skipping
150513 18:23:53 [Note] WSREP: (28905a6f-f99d-11e4-8794-afa4dadb86b5, 'tcp://
0.0.0.0:4567') address 'tcp://172.27.101.175:4567' pointing to uuid
28905a6f-f99d-11e4-8794-afa4dadb86b5 is blacklisted, skipping
150513 18:23:55 [Note] WSREP: (28905a6f-f99d-11e4-8794-afa4dadb86b5, 'tcp://
0.0.0.0:4567') address 'tcp://172.27.101.175:4567' pointing to uuid
28905a6f-f99d-11e4-8794-afa4dadb86b5 is blacklisted, skipping
150513 18:23:56 [Note] WSREP: (28905a6f-f99d-11e4-8794-afa4dadb86b5, 'tcp://
0.0.0.0:4567') address 'tcp://172.27.101.175:4567' pointing to uuid
28905a6f-f99d-11e4-8794-afa4dadb86b5 is blacklisted, skipping
150513 18:23:58 [Note] WSREP: (28905a6f-f99d-11e4-8794-afa4dadb86b5, 'tcp://
0.0.0.0:4567') address 'tcp://172.27.101.175:4567' pointing to uuid
28905a6f-f99d-11e4-8794-afa4dadb86b5 is blacklisted, skipping
150513 18:23:59 [Note] WSREP: (28905a6f-f99d-11e4-8794-afa4dadb86b5, 'tcp://
0.0.0.0:4567') address 'tcp://172.27.101.175:4567' pointing to uuid
28905a6f-f99d-11e4-8794-afa4dadb86b5 is blacklisted, skipping
150513 18:24:00 [Note] WSREP: (28905a6f-f99d-11e4-8794-afa4dadb86b5, 'tcp://
0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://
172.27.101.146:4567
150513 18:24:01 [Warning] WSREP: no nodes coming from prim view, prim not
possible
150513 18:24:01 [Note] WSREP:
view(view_id(NON_PRIM,28905a6f-f99d-11e4-8794-afa4dadb86b5,3) memb {
        28905a6f-f99d-11e4-8794-afa4dadb86b5,
} joined {
} left {
} partitioned {
        264c0c94-f99d-11e4-8da9-0345cc984ee5,
})
150513 18:24:01 [Note] WSREP: (28905a6f-f99d-11e4-8794-afa4dadb86b5, 'tcp://
0.0.0.0:4567') reconnecting to 264c0c94-f99d-11e4-8da9-0345cc984ee5 (tcp://
172.27.101.146:4567), attempt 0
150513 18:24:01 [Note] WSREP: view((empty))
150513 18:24:01 [ERROR] WSREP: failed to open gcomm backend connection:
110: failed to reach primary view: 110 (Connection timed out)
         at gcomm/src/pc.cpp:connect():141
150513 18:24:01 [ERROR] WSREP: gcs/src/gcs_core.c:gcs_core_open():196:
Failed to open backend connection: -110 (Connection timed out)
150513 18:24:01 [ERROR] WSREP: gcs/src/gcs.c:gcs_open():1292: Failed to
open channel 'juju_cluster' at
'gcomm://172.27.101.175,172.27.101.134,172.27.101.146':
-110 (Connection timed out)
150513 18:24:01 [ERROR] WSREP: gcs connect failed: Connection timed out
150513 18:24:01 [ERROR] WSREP: wsrep::connect() failed: 7
150513 18:24:01 [ERROR] Aborting

150513 18:24:01 [Note] WSREP: Service disconnected.
150513 18:24:02 [Note] WSREP: Some threads may fail to exit.
150513 18:24:02 [Note] /usr/sbin/mysqld: Shutdown complete

150513 18:24:02 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid
ended

On Fri, Apr 10, 2015 at 4:21 AM, Adam Collard <email address hidden>
wrote:

> ** Changed in: percona-cluster (Juju Charms Collection)
> Status: New => Incomplete
>
> --
> You received this bug notification because you are a member of OpenStack
> Charmers, which is subscribed to percona-cluster in Juju Charms
> Collection.
> https://bugs.launchpad.net/bugs/1413620
>
> Title:
> Error after reboot: Can't connect to local MySQL server through socket
> '/var/run/mysqld/mysqld.sock' (2)
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/charms/+source/percona-cluster/+bug/1413620/+subscriptions
>

--
David Britton <email address hidden>