libvirtd crashed with SIGABRT in __assert_fail_base()

Hi Peter,
I beg your pardon but I don't know why this was missed since you reported - I'm afraid now most chances to debug are gone but lets continue anyway.

I checked through the trace and it didn't ring a bell for anything that might be known already.
Is (was) this issue reproducible multiple times then?

If so hve you identified some sort of guest/net config that was triggering it so that we might have a chance to reproduce for debugging?

Christian, First, thank you for taking time to look into this issue. Since
the upgrade to 17.10 from 17.04 libvirtd will not start on boot. The work
around is to select an older kernel from grub, e.g., 4.10. It seems they
separated out virtualization perhaps to make the kernel more efficient. If
you would like any tests or screenshots let me know. Thanks again, Peter

On Dec 8, 2017 8:00 AM, "ChristianEhrhardt" <email address hidden>
wrote:

> Hi Peter,
> I beg your pardon but I don't know why this was missed since you reported
> - I'm afraid now most chances to debug are gone but lets continue anyway.
>
> I checked through the trace and it didn't ring a bell for anything that
> might be known already.
> Is (was) this issue reproducible multiple times then?
>
> If so hve you identified some sort of guest/net config that was
> triggering it so that we might have a chance to reproduce for debugging?
>
> ** Changed in: libvirt (Ubuntu)
> Status: New => Incomplete
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1727138
>
> Title:
> libvirtd crashed with SIGABRT in __assert_fail_base()
>
> Status in libvirt package in Ubuntu:
> Incomplete
>
> Bug description:
> During upgrade to 17.10, a couple packages didn't install correctly
> due to dependencies. This was the first reboot when libvirt failed.
>
> ProblemType: Crash
> DistroRelease: Ubuntu 17.10
> Package: libvirt-daemon 3.6.0-1ubuntu5
> ProcVersionSignature: Ubuntu 4.13.0-16.19-lowlatency 4.13.4
> Uname: Linux 4.13.0-16-lowlatency x86_64
> ApportVersion: 2.20.7-0ubuntu3
> Architecture: amd64
> Date: Tue Oct 24 22:34:59 2017
> ExecutablePath: /usr/sbin/libvirtd
> InstallationDate: Installed on 2016-02-09 (623 days ago)
> InstallationMedia: Ubuntu-MATE 15.04 "Vivid Vervet" - Release amd64
> (20150422.1)
> ProcAttrCurrent: /usr/sbin/libvirtd (enforce)
> ProcCmdline: placeholder root=UUID=8edb633f-0b5e-4e9b-a0e3-ef6f9b14dcb8
> ro quiet splash
> ProcEnviron:
> LANG=en_US.UTF-8
> PATH=(custom, no user)
> Signal: 6
> SourcePackage: libvirt
> Title: libvirtd crashed with SIGABRT in __assert_fail_base()
> UpgradeStatus: Upgraded to artful on 2017-10-25 (0 days ago)
> UserGroups:
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/
> 1727138/+subscriptions
>

So to confirm:
1. install a 17.04
2. install libvirt-daemon-system
3. reboot
4. check that the servive libvirtd runs correctly
5. upgrade to 17.10
6. reboot
X. libvirtd is not running

Is that a correct summary for me to retry?

Christian,
If you mean by repeatable: reinstall 17.04, install qemu then upgrade to
17.10, then no I did not repeat it, though my desktop is still in the
same state. Booting version 4.10 brings up the usable qemu:///system. It
shows warnings, but is working.

output after booting kernel 4.10.xx
------------------------------
peter@vm-host:~$ sudo libvirtd
[sudo] password for peter:
2017-12-12 03:36:59.805+0000: 4504: info : libvirt version: 3.6.0,
package: 1ubuntu6 (Christian Ehrhardt <email address hidden>
Tue, 24 Oct 2017 14:30:34 +0200)
2017-12-12 03:36:59.805+0000: 4504: info : hostname: vm-host
2017-12-12 03:36:59.805+0000: 4504: error : virPidFileAcquirePath:422 :
Failed to acquire pid file '/var/run/libvirtd.pid': Resource temporarily
unavailable
peter@vm-host:~$
-----------------------------
output of kernel 4.13.xx
----------------------
$ sudo libvirtd
libvirtd: libxl_fork.c:353: sigchld_installhandler_core: Assertion
`((void)"application must negotiate with libxl about SIGCHLD",
!(sigchld_saved_action.sa_flags & SA_SIGINFO) &&
(sigchld_saved_action.sa_handler == SIG_DFL ||
sigchld_saved_action.sa_handler == SIG_IGN))' failed.
Aborted
peter@vm-host:~$
---------------------
output of 4.10

Peter
--

On 12/11/2017 10:23 AM, ChristianEhrhardt wrote:
> So to confirm:
> 1. install a 17.04
> 2. install libvirt-daemon-system
> 3. reboot
> 4. check that the servive libvirtd runs correctly
> 5. upgrade to 17.10
> 6. reboot
> X. libvirtd is not running
>
> Is that a correct summary for me to retry?
>
> ** Changed in: libvirt (Ubuntu)
> Assignee: (unassigned) => ChristianEhrhardt (paelzer)
>

Hi,
I wonder about the "Resource temporarily unavailable" error in this.
This could be something consuming all FDs, or multiple libvirts daemons starting.

What does "ps axlf | grep libvirtd" show you?

Does /var/run/libvirtd.pid exist and if so what is in there.
If it is a pid, which process is that?

[1] had a similar case.

For the issue you see with the 4.10 kernel hat is actually a xen library (not libvirt).
libxl_fork.c:353 is from source:xen ./tools/libxl/libxl_fork.c

341 static void sigchld_installhandler_core(void) /* idempotent */
342 {
343 if (sigchld_installed)
344 return;
345
346 sigchld_installed = 1;
347
348 sigchld_sethandler_raw(sigchld_handler, &sigchld_saved_action);
349
350 assert(((void)"application must negotiate with libxl about SIGCHLD",
351 !(sigchld_saved_action.sa_flags & SA_SIGINFO) &&
352 (sigchld_saved_action.sa_handler == SIG_DFL ||
353 sigchld_saved_action.sa_handler == SIG_IGN)));
354 }

Are you driving xen with your libvirt, or is this just the basic initialization code?

IMHO that is a known upstream issue:
https://www.redhat.com/archives/libvir-list/2015-January/msg00868.html
https://www.redhat.com/archives/libvir-list/2015-November/msg00336.html
https://bugzilla.redhat.com/show_bug.cgi?id=1278847

Due to Xen + vbox + libvirt not playing together nicely.

I don't know what guests you really drive, but as a last resort workaround until one day there is an upstream fix you might disable what you don't use, which means either:
- do not use a xen kernel
- remove one of /usr/lib/libvirt/connection-driver/libvirt_driver_xen.so /usr/lib/libvirt/connection-driver/libvirt_driver_vbox.so (or both if you need none)
  They initialize when they see something they could handle, maybe there is a config to avoid loading them but I don't know (the same search distance away as you)

[1]: https://www.redhat.com/archives/libvirt-users/2012-October/msg00146.html

Hello Christian,
Unfortunately I have re-installed since the error came up. You are right,
due to my ignorance I originally installed multiple related virtualization
tools xen qemu, kdm, not being sure which one would work.
The good news is I am prepared to try to reproduce the problem now. In my
case this would be:

Install Ubuntu 14.04 install libvirt and friends including xen, and upgrade
each OS version until 17.10.

I will go through this procedure tonight if it would be helpful in your
diagnosis.

Peter
--

On Fri, Mar 16, 2018 at 5:25 AM, ChristianEhrhardt <
<email address hidden>> wrote:

> Hi,
> I wonder about the "Resource temporarily unavailable" error in this.
> This could be something consuming all FDs, or multiple libvirts daemons
> starting.
>
> What does "ps axlf | grep libvirtd" show you?
>
> Does /var/run/libvirtd.pid exist and if so what is in there.
> If it is a pid, which process is that?
>
> [1] had a similar case.
>
> For the issue you see with the 4.10 kernel hat is actually a xen library
> (not libvirt).
> libxl_fork.c:353 is from source:xen ./tools/libxl/libxl_fork.c
>
> 341 static void sigchld_installhandler_core(void) /* idempotent */
> 342 {
> 343 if (sigchld_installed)
> 344 return;
> 345
> 346 sigchld_installed = 1;
> 347
> 348 sigchld_sethandler_raw(sigchld_handler, &sigchld_saved_action);
> 349
> 350 assert(((void)"application must negotiate with libxl about
> SIGCHLD",
> 351 !(sigchld_saved_action.sa_flags & SA_SIGINFO) &&
> 352 (sigchld_saved_action.sa_handler == SIG_DFL ||
> 353 sigchld_saved_action.sa_handler == SIG_IGN)));
> 354 }
>
> Are you driving xen with your libvirt, or is this just the basic
> initialization code?
>
> IMHO that is a known upstream issue:
> https://www.redhat.com/archives/libvir-list/2015-January/msg00868.html
> https://www.redhat.com/archives/libvir-list/2015-November/msg00336.html
> https://bugzilla.redhat.com/show_bug.cgi?id=1278847
>
> Due to Xen + vbox + libvirt not playing together nicely.
>
> I don't know what guests you really drive, but as a last resort workaround
> until one day there is an upstream fix you might disable what you don't
> use, which means either:
> - do not use a xen kernel
> - remove one of /usr/lib/libvirt/connection-driver/libvirt_driver_xen.so
> /usr/lib/libvirt/connection-driver/libvirt_driver_vbox.so (or both if you
> need none)
> They initialize when they see something they could handle, maybe there
> is a config to avoid loading them but I don't know (the same search
> distance away as you)
>
>
> [1]: https://www.redhat.com/archives/libvirt-users/2012-
> October/msg00146.html
>
> ** Bug watch added: Red Hat Bugzilla #1278847
> https://bugzilla.redhat.com/show_bug.cgi?id=1278847
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1727138
>
> Title:
> libvirtd crashed with SIGABRT in __assert_fail_base()
>
> Status in libvirt package in Ubuntu:
> New
>
> Bug description:
> During upgrade to 17.10, a couple packages didn't install correctly
> due to dependencies. This was the first reboot when libvir...

There is no private info in this crash, lets dup it onto the main bug (that probably stays open due to low prio).