Startup fails due to hard-coded timeout in init script

Bug #1434022 reported by Patrick McConnell
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Percona Server moved to https://jira.percona.com/projects/PS
Fix Released
Medium
Tomislav Plavcic
5.1
Won't Fix
Undecided
Unassigned
5.5
Fix Released
Medium
Tomislav Plavcic
5.6
Fix Released
Medium
Tomislav Plavcic

Bug Description

/etc/init.d/mysql now has a hard-coded startup timeout of 10 seconds. Previously it was 14 seconds, which was also too short for our configuration.

Our my.cnf contains:

innodb_buffer_pool_size = 193392M

MySQL takes around 15 seconds to start on this server with this option in place. After waiting 10 seconds, the init script reports that startup failed:

 * Starting MySQL (Percona Server) database server mysqld [fail]

But in fact mysqld is running and responsive a few seconds later. The guilty code is in /etc/init.d/mysql:

# wait 10sec before start checking if pid file created or server is dead
if [ $dead_check_counter -lt 10 ]; then

If I remove "innodb_buffer_pool_size = 193392M" from my.cnf, the init script completes successfully:

 * Starting MySQL (Percona Server) database server mysqld [ OK ]

However we need this option, so we are currently working around this by changing the guilty code in the init script to:

# wait 60sec before start checking if pid file created or server is dead
if [ $dead_check_counter -lt 60 ]; then

This problem has been discussed before in bugs #1072538 and #1328262. Both of these bugs were reported fixed in 5.6.22-71, but it seems to have worked its way back to the current version in a slightly altered form.

Bug #1328262 mentions a configurable timeout using a value in /etc/default/mysql. I think this is probably the best solution, since as Vladimir mentions, "whatever value you hardcode won't cut it for edge cases".

Tags: pkg i60266
tags: added: pkg
Revision history for this message
Tomislav Plavcic (tplavcic) wrote :

10sec here should be only for server pid to show up not for the full server to start, when pid show's up then it waits unlimited time for server to become pingable.
Need to check that part maybe there's something wrong with it.

Revision history for this message
Patrick McConnell (j-patrick) wrote :

@Tomislav based on your comment, I think the 10sec loop should look like this:

# wait 10sec before start checking if pid file created or server is dead
if [ $dead_check_counter -lt 10 ]; then
    dead_check_counter=$(( dead_check_counter + 1 ))
else
    if mysqld_status check_alive nowarn; then break; fi
fi

i.e. the else clause should run "mysqld_status check_alive" instead of "mysqld_status check_dead". This will cause it to loop forever until the server responds to ping.

Revision history for this message
Patrick McConnell (j-patrick) wrote :

Also when you mentioned the server pid, are you referring to mysqld_status watching for the pid file? If so, I see a couple of problems with this:

- I don't see the pid file showing up until after the server has started (specifically, until after the log shows "ready for connections"), which in our case means the pid file doesn't exist until after the 10sec loop is finished and the init script has exited.

- mysqld_get_param doesn't know what our pid file is unless I specify it in my.cnf, and even then, I need to specify it as pid-file and not pid_file (i.e. hyphen and not underscore). If I don't specify it in my.cnf at all, it doesn't show up in mysqld --print-defaults. If I specify it as pid_file then mysqld_get_param still doesn't find an option called pid-file. In both cases mysqld_status doesn't end up knowing the name of the pid file.

Revision history for this message
Sveta Smirnova (svetasmirnova) wrote :

I agree with the reporter this is a bug. I saw many cases when MySQL Server does not create PID in 10 seconds. This parameter should be configurable.

Changed in percona-server:
status: New → Confirmed
Changed in percona-server:
assignee: nobody → Tomislav Plavcic (tplavcic)
Revision history for this message
Sveta Smirnova (svetasmirnova) wrote :

See also old MySQL bug, fixed long time ago: https://bugs.mysql.com/bug.php?id=25341 Maybe time to port this fix into Percona Server too?

tags: added: i60266
Revision history for this message
Michael Wang (xw73) wrote :

This bug affects more on the install and upgrade process than starting up the database routinely, because if you start up the database manually, you can always wait and check the log file after it reports startup "failure", however during the upgrade, after the startup failure, the post configuration process also fails and package state is in limbo. Next time when you try to run the package command, it will try to re-run the command to finish the unfinished business (at least this is the case on Ubuntu).

The workaround is to have a version of modified /etc/init.d/mysql in place, and run the installation command with the option NOT to replace the configuration file. It is like below on Ubuntu:

# export DEBIAN_FRONTEND=noninteractive
# apt-get -y -o Dpkg::Options::="--force-confold" install percona-server-server-5.6

I spent a lot of time figured out all these, hope this helps.

Revision history for this message
Tomislav Plavcic (tplavcic) wrote :
Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-1617

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.