SET key_cache_segments leads to crashes in get_partitioned_key_cache_statistics, or safe_mutex_lock, or partitioned_key_cache_statistics

Bug #1008293 reported by Jean Weisbuch
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
MariaDB
Fix Committed
Critical
Igor Babaev

Bug Description

On 3 of the 4 servers i tried to do a "set global key_cache_segments = 8", 3 have crashed and only one didnt.

The crashing servers were :
 - One on MariaDB 5.3.7 (immediate crash with "server has gone away" when sending the set command)
 - One on MariaDB 5.5.23 (set command returns "OK" but server crashes just after that)
 - One on MariaDB 5.5.24 (same as on the previous one)
The server that has not crashed was on 5.5.23.

All the servers are on Debian Squeeze amd64 using MariaDB repository packages.

All servers have different datas and workloads (and are not using replication), these are used on a shared hosting environement so there is many databases/tables on each with many random accesses on them.

Did put the value on my.cnf and restarted completely the servers and it didnt crash (some have been up for weeks without a crash), so it seems to be only crashing when changing the parameter on a live server.

(attached two crash reports: 5.5.24 and 5.3.7)

Revision history for this message
Jean Weisbuch (m-jea0-p) wrote :
Revision history for this message
Elena Stepanova (elenst) wrote :

Hi Jean,

Could you please provide your my.cnf file from a server where you observed the crash?

Thanks.

Revision history for this message
Jean Weisbuch (m-jea0-p) wrote :

Here is one of the servers conf (the others have similar ones with mainly different buffer_pool_size, key_buffer_size, table_cache and max_connections) :

[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmpfs
skip-external-locking
key_buffer = 512M
max_allowed_packet = 64M
thread_cache_size = 12
myisam-recover = BACKUP
max_connections = 150
table_cache = 16384
table_definition_cache = 16384
query_cache_limit = 1M
query_cache_size = 128M
query_cache_strip_comments = 1
local-infile = 0
open_files_limit = 131072
skip-name-resolve
max_delayed_threads = 1
wait_timeout = 60
interactive_timeout = 60
log_error = /var/log/mysql/error.log
join_buffer_size = 1M
read_buffer_size = 256K
tmp_table_size = 32M
max_heap_table_size = 32M
max_connect_errors = 1000
query_prealloc_size = 131072
query_alloc_block_size = 16384
myisam_sort_buffer_size = 4M
concurrent_insert = 2
read_rnd_buffer_size = 1M
default_storage_engine = InnoDB
innodb_buffer_pool_size = 1280M
innodb_log_file_size = 32M
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 4M
innodb_file_per_table = 1
innodb_open_files = 400
innodb_file_format = barracuda
key_cache_segments = 8 (wasnt there before i tried to set the value)
log_slow_queries = /var/log/mysql/mysql-slow.log
long_query_time = 2
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
[mysql]
[isamchk]
key_buffer = 16M
!includedir /etc/mysql/conf.d/

And in the conf.d dir :
/etc/mysql/conf.d/mariadb.cnf:[client]
/etc/mysql/conf.d/mariadb.cnf:[mysqld]
/etc/mysql/conf.d/mysqld_safe_syslog.cnf:[mysqld_safe]
/etc/mysql/conf.d/mysqld_safe_syslog.cnf:syslog

Revision history for this message
sbester (sbester) wrote :

Very easy to reproduce. Any myisam workload with 2 or more threads will hit this. Try setting the key_cache_segments to a random value in 1 thread and select/insert/update/show global status/cache index in another thread.

Tttached two examples of a crash.

Revision history for this message
Elena Stepanova (elenst) wrote :

Jean, Shane, thank you.

Reproducible with the RQG grammar and command line below on maria/5.2 revno 3139, maria/5.3 revno 3539, maria/5.5 revno 3426, and on release versions of 5.2, 5.3, 5.5.

# cat 1.yy
thread1:

  SET GLOBAL key_cache_segments = _digit;

query:
  SHOW GLOBAL STATUS;

# End of RQG grammar

# Run as
perl ./runall.pl \
--grammar=1.yy \
--duration=60 \
--queries=10M \
--threads=2 \
--skip-gendata \
--basedir=<your basedir> \
--vardir=<your vardir>

Revision history for this message
Elena Stepanova (elenst) wrote :

The other variation of crash can be hit with the following grammar and command line:

# cat 2.yy
query_init:
  SET GLOBAL keycache1.key_buffer_size = 1024*1024;
thread1:
  SET GLOBAL key_cache_segments = _digit;

query:
  CACHE INDEX _table IN keycache1;

# End of RQG grammar

# Run as:

perl ./runall.pl \
--grammar=2.yy \
--duration=60 \
--queries=10M \
--engine=MyISAM \
--threads=2 \
--basedir=<your basedir> \
--vardir=<your vardir>

Elena Stepanova (elenst)
summary: - Changing key_cache_segments value using set leads to crashes
+ SET key_cache_segments leads to crashes in
+ get_partitioned_key_cache_statistics, or safe_mutex_lock, or
+ partitioned_key_cache_statistics
Changed in maria:
importance: Undecided → Critical
milestone: none → 5.2
Elena Stepanova (elenst)
Changed in maria:
assignee: nobody → Igor Babaev (igorb-seattle)
Changed in maria:
status: New → Confirmed
Changed in maria:
status: Confirmed → In Progress
Changed in maria:
status: In Progress → Fix Committed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.