| 2016-04-16 16:08:59 |
guessi |
bug |
|
|
added bug |
| 2016-04-16 16:08:59 |
guessi |
attachment added |
|
allow-change-sockfile_check_retries.diff https://bugs.launchpad.net/bugs/1571209/+attachment/4638300/+files/allow-change-sockfile_check_retries.diff |
|
| 2016-04-16 16:26:52 |
Ubuntu Foundations Team Bug Bot |
tags |
|
patch |
|
| 2016-04-16 16:27:00 |
Ubuntu Foundations Team Bug Bot |
bug |
|
|
added subscriber Ubuntu Review Team |
| 2016-04-16 16:42:39 |
guessi |
tags |
patch |
patch trusty xenial |
|
| 2016-04-16 16:49:54 |
guessi |
description |
[ problem description ]
sockfile_check_retries is first introduced by #1455608, for preventing the failure case of sockfile not ready,
but it was default to a hard-coded value "5", it might be too short for a busy system boot,
#1455608 - https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1455608
[ step to reproduce ]
setup a clean install system (Ubuntu Server 14.04.4 LTS),
boot up multiple guest instances, and set as start-at-boot,
force shutdown host by pressing power-button for 3s ~ 5s, then power-on afterward,
it may sometimes failed to get sockfile ready after in "post-start" script,
with an line of error in /var/log/syslog,
==> Apr 16 21:02:09 host2 kernel: [ 313.059830] init: libvirt-bin post-start process (2430) terminated with status 1
[ possible solution ]
extend the retry times for sockfile waiting
<please see the patch file as attachment>
[ sysinfo ]
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.4 LTS
Release: 14.04
Codename: trusty
$ uname -a
Linux host2 4.2.0-35-generic #40~14.04.1-Ubuntu SMP Fri Mar 18 16:37:35 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
[ related issue ]
#1386465 - https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1386465 |
[ problem description ]
sockfile_check_retries is first introduced by #1455608, for preventing the failure case of sockfile not ready, but it was default to a hard-coded value "5", it might be too short for a busy system boot.
#1455608 - https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1455608
[ step to reproduce ]
setup a clean install system (Ubuntu Server 14.04.4 LTS), and assemble os disk as RAID-1, boot up some guest instances (count > 10, start-at-boot), force shutdown host by pressing power-button for 3s ~ 5s, or via IPMI command, then power-on afterward. it may sometimes failed to get sockfile ready after in "post-start" script, with an line of error in /var/log/syslog,
==> kernel: [ 313.059830] init: libvirt-bin post-start process (2430) terminated with status 1 <==
since there's multiple VMs Read/Write before a non-graceful shutdown, RAID devices need to re-sync after boot, and lead to a slow response, but start-up script for libvirt-bin can only wait 5 cycles, 2 seconds wait for each cycle, so it will timed-out after 10s, and exit with "1".
[ possible solution ]
extend the retry times for sockfile waiting, and make it possible to change via editing `/etc/default/libvirt-bin` file.
<please see the patch file as attachment>
[ sysinfo ]
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.4 LTS
Release: 14.04
Codename: trusty
$ uname -a
Linux host2 4.2.0-35-generic #40~14.04.1-Ubuntu SMP Fri Mar 18 16:37:35 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
[ related issue ]
#1386465 - https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1386465 |
|
| 2016-04-18 15:12:45 |
Serge Hallyn |
libvirt (Ubuntu): assignee |
|
Edward Hope-Morley (hopem) |
|
| 2016-04-18 15:17:18 |
Serge Hallyn |
libvirt (Ubuntu): importance |
Undecided |
High |
|
| 2016-04-18 15:17:22 |
Serge Hallyn |
libvirt (Ubuntu): status |
New |
Confirmed |
|
| 2016-04-19 15:51:20 |
guessi |
attachment added |
|
trusty patch https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1571209/+attachment/4639935/+files/libvirt-bin-trusty.diff |
|
| 2016-04-21 01:57:21 |
guessi |
attachment added |
|
trusty patch https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1571209/+attachment/4641118/+files/libvirt-bin-trusty.diff |
|
| 2016-04-21 01:57:37 |
guessi |
attachment added |
|
libvirt-bin-precise.diff https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1571209/+attachment/4641119/+files/libvirt-bin-precise.diff |
|
| 2016-04-21 01:57:48 |
guessi |
attachment added |
|
libvirt-bin-vivid.diff https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1571209/+attachment/4641120/+files/libvirt-bin-vivid.diff |
|
| 2016-04-21 01:58:05 |
guessi |
attachment added |
|
libvirt-bin-wily.diff https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1571209/+attachment/4641121/+files/libvirt-bin-wily.diff |
|
| 2016-04-21 01:58:15 |
guessi |
attachment added |
|
libvirt-bin-xenial.diff https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1571209/+attachment/4641122/+files/libvirt-bin-xenial.diff |
|
| 2016-04-21 02:03:06 |
guessi |
tags |
patch trusty xenial |
patch precise trusty vivid wily xenial |
|
| 2016-04-25 14:27:11 |
Serge Hallyn |
nominated for series |
|
Ubuntu Xenial |
|
| 2016-04-25 14:27:11 |
Serge Hallyn |
bug task added |
|
libvirt (Ubuntu Xenial) |
|
| 2016-04-25 14:27:11 |
Serge Hallyn |
nominated for series |
|
Ubuntu Wily |
|
| 2016-04-25 14:27:11 |
Serge Hallyn |
bug task added |
|
libvirt (Ubuntu Wily) |
|
| 2016-04-25 14:27:11 |
Serge Hallyn |
nominated for series |
|
Ubuntu Trusty |
|
| 2016-04-25 14:27:11 |
Serge Hallyn |
bug task added |
|
libvirt (Ubuntu Trusty) |
|
| 2016-04-25 14:27:39 |
Serge Hallyn |
nominated for series |
|
Ubuntu Precise |
|
| 2016-04-25 14:27:39 |
Serge Hallyn |
bug task added |
|
libvirt (Ubuntu Precise) |
|
| 2016-05-23 18:51:50 |
Serge Hallyn |
libvirt (Ubuntu): assignee |
Edward Hope-Morley (hopem) |
Serge Hallyn (serge-hallyn) |
|
| 2016-05-23 18:51:54 |
Serge Hallyn |
libvirt (Ubuntu): status |
Confirmed |
In Progress |
|
| 2016-05-23 18:51:59 |
Serge Hallyn |
libvirt (Ubuntu Precise): importance |
Undecided |
High |
|
| 2016-05-23 18:52:01 |
Serge Hallyn |
libvirt (Ubuntu Trusty): importance |
Undecided |
High |
|
| 2016-05-23 18:52:04 |
Serge Hallyn |
libvirt (Ubuntu Wily): importance |
Undecided |
High |
|
| 2016-05-23 18:52:06 |
Serge Hallyn |
libvirt (Ubuntu Xenial): importance |
Undecided |
High |
|
| 2016-05-23 22:14:45 |
Launchpad Janitor |
libvirt (Ubuntu): status |
In Progress |
Fix Released |
|
| 2016-09-30 15:49:51 |
Jon Grimm |
libvirt (Ubuntu Wily): status |
New |
Won't Fix |
|
| 2017-09-06 08:00:38 |
Christian Ehrhardt |
nominated for series |
|
Ubuntu Zesty |
|
| 2017-09-06 08:00:38 |
Christian Ehrhardt |
bug task added |
|
libvirt (Ubuntu Zesty) |
|
| 2017-09-06 08:00:38 |
Christian Ehrhardt |
nominated for series |
|
Ubuntu Artful |
|
| 2017-09-06 08:00:38 |
Christian Ehrhardt |
bug task added |
|
libvirt (Ubuntu Artful) |
|
| 2017-09-06 08:00:49 |
Christian Ehrhardt |
libvirt (Ubuntu Artful): status |
Fix Released |
Incomplete |
|
| 2017-09-06 08:00:52 |
Christian Ehrhardt |
libvirt (Ubuntu Zesty): status |
New |
Incomplete |
|
| 2017-09-06 08:00:53 |
Christian Ehrhardt |
libvirt (Ubuntu Xenial): status |
New |
Incomplete |
|
| 2017-09-06 08:00:56 |
Christian Ehrhardt |
libvirt (Ubuntu Trusty): status |
New |
Triaged |
|
| 2017-09-06 08:01:00 |
Christian Ehrhardt |
libvirt (Ubuntu Precise): status |
New |
Won't Fix |
|
| 2017-09-06 08:01:06 |
Christian Ehrhardt |
libvirt (Ubuntu Artful): assignee |
Serge Hallyn (serge-hallyn) |
|
|
| 2017-09-06 08:44:38 |
Christian Ehrhardt |
libvirt (Ubuntu Artful): status |
Incomplete |
Fix Released |
|
| 2017-09-06 09:05:41 |
Christian Ehrhardt |
libvirt (Ubuntu Xenial): status |
Incomplete |
Fix Released |
|
| 2017-09-06 09:05:43 |
Christian Ehrhardt |
libvirt (Ubuntu Zesty): status |
Incomplete |
Fix Released |
|
| 2017-09-06 09:08:09 |
Christian Ehrhardt |
libvirt (Ubuntu Trusty): status |
Triaged |
Incomplete |
|
| 2017-09-06 09:46:51 |
Christian Ehrhardt |
bug |
|
|
added subscriber Edward Hope-Morley |
| 2017-09-06 12:40:37 |
Christian Ehrhardt |
libvirt (Ubuntu Trusty): status |
Incomplete |
Triaged |
|
| 2017-09-07 14:01:10 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~paelzer/ubuntu/+source/libvirt/+git/libvirt/+merge/330359 |
|
| 2017-09-13 13:06:37 |
Christian Ehrhardt |
description |
[ problem description ]
sockfile_check_retries is first introduced by #1455608, for preventing the failure case of sockfile not ready, but it was default to a hard-coded value "5", it might be too short for a busy system boot.
#1455608 - https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1455608
[ step to reproduce ]
setup a clean install system (Ubuntu Server 14.04.4 LTS), and assemble os disk as RAID-1, boot up some guest instances (count > 10, start-at-boot), force shutdown host by pressing power-button for 3s ~ 5s, or via IPMI command, then power-on afterward. it may sometimes failed to get sockfile ready after in "post-start" script, with an line of error in /var/log/syslog,
==> kernel: [ 313.059830] init: libvirt-bin post-start process (2430) terminated with status 1 <==
since there's multiple VMs Read/Write before a non-graceful shutdown, RAID devices need to re-sync after boot, and lead to a slow response, but start-up script for libvirt-bin can only wait 5 cycles, 2 seconds wait for each cycle, so it will timed-out after 10s, and exit with "1".
[ possible solution ]
extend the retry times for sockfile waiting, and make it possible to change via editing `/etc/default/libvirt-bin` file.
<please see the patch file as attachment>
[ sysinfo ]
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.4 LTS
Release: 14.04
Codename: trusty
$ uname -a
Linux host2 4.2.0-35-generic #40~14.04.1-Ubuntu SMP Fri Mar 18 16:37:35 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
[ related issue ]
#1386465 - https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1386465 |
[Impact]
* Libvirt service reports to be ready, but it has not spawned the libvirt
socket yet. Depending services fail. There was an SRU (#1455608) meant
to fix that but it has many deficiencies (not considering config,
giving up after 10 seconds, being an unconditional sleep 2, taking up
to 2 seconds to a service stop while in pist-start).
* This is the backport and improvement of a change that was brought to
Yakkety already, but there due to systemd it doesn't matter too much.
[Test Case]
* There are two very different ways to "test" this due to the overload
based scenario where this really becomes important.
* Version #1 - being lame
One can just modify the upstart script and exchange the check for the
socket with /bin/true.
That way it waits forever which allows you to check the log entries,
the abort responsiveness and similar.
* Version #2 - recreating the case
- This mostly means the system has to be very slow and overloaded.
You can either just slow down the system (e.g. run a qemu with nice
MAX). Stress your host with other things burning CPU/memory/disk.
- we worked with adding autostart guests (see comment #35) but that
actually takes place after the socket is created. The reported acse
had a raid rebuilding.
- TL;DR get your system slow enough so that libvirt exceeds 10 seconds
to start properly (the old limit is 5*2 seconds)
[Regression Potential]
* I'd think that there might exist (super rare) cases were the post-start
now does spin forever. But by the definition
http://upstart.ubuntu.com/cookbook/#post-start this is correct. It is
started (yes) but not yet ready. Yet this might appear as a regression
to some.
* Other than that clearly this should fix more issues than it (hopefully
not) causes.
[Other Info]
* n/a
--- END SRU Template ---
[ problem description ]
sockfile_check_retries is first introduced by #1455608, for preventing the failure case of sockfile not ready, but it was default to a hard-coded value "5", it might be too short for a busy system boot.
#1455608 - https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1455608
[ step to reproduce ]
setup a clean install system (Ubuntu Server 14.04.4 LTS), and assemble os disk as RAID-1, boot up some guest instances (count > 10, start-at-boot), force shutdown host by pressing power-button for 3s ~ 5s, or via IPMI command, then power-on afterward. it may sometimes failed to get sockfile ready after in "post-start" script, with an line of error in /var/log/syslog,
==> kernel: [ 313.059830] init: libvirt-bin post-start process (2430) terminated with status 1 <==
since there's multiple VMs Read/Write before a non-graceful shutdown, RAID devices need to re-sync after boot, and lead to a slow response, but start-up script for libvirt-bin can only wait 5 cycles, 2 seconds wait for each cycle, so it will timed-out after 10s, and exit with "1".
[ possible solution ]
extend the retry times for sockfile waiting, and make it possible to change via editing `/etc/default/libvirt-bin` file.
<please see the patch file as attachment>
[ sysinfo ]
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.4 LTS
Release: 14.04
Codename: trusty
$ uname -a
Linux host2 4.2.0-35-generic #40~14.04.1-Ubuntu SMP Fri Mar 18 16:37:35 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
[ related issue ]
#1386465 - https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1386465 |
|
| 2017-09-21 19:21:18 |
Brian Murray |
libvirt (Ubuntu Trusty): status |
Triaged |
Fix Committed |
|
| 2017-09-21 19:21:21 |
Brian Murray |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2017-09-21 19:21:23 |
Brian Murray |
bug |
|
|
added subscriber SRU Verification |
| 2017-09-21 19:21:26 |
Brian Murray |
tags |
patch precise trusty vivid wily xenial |
patch precise trusty verification-needed verification-needed-trusty vivid wily xenial |
|
| 2017-09-28 15:45:52 |
Christian Ehrhardt |
tags |
patch precise trusty verification-needed verification-needed-trusty vivid wily xenial |
patch precise trusty verification-done verification-done-trusty vivid wily xenial |
|
| 2017-10-05 16:45:33 |
Launchpad Janitor |
libvirt (Ubuntu Trusty): status |
Fix Committed |
Fix Released |
|
| 2017-10-05 16:45:39 |
Brian Murray |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
