dnsmasq on Ubuntu Jammy crashes on neutron-dhcp-agent updates
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Ironic |
Triaged
|
Critical
|
Unassigned | ||
neutron |
New
|
Low
|
Unassigned | ||
dnsmasq (Ubuntu) |
Invalid
|
Undecided
|
Unassigned | ||
Jammy |
Incomplete
|
Undecided
|
Unassigned | ||
Kinetic |
Won't Fix
|
Undecided
|
Unassigned | ||
Lunar |
Invalid
|
Undecided
|
Unassigned | ||
Mantic |
Invalid
|
Undecided
|
Unassigned |
Bug Description
The Ironic project's CI has been having major blocking issues moving to utilizing Ubuntu Jammy and with some investigation we were able to isolate the issues down to the dhcp updates causing dnsmasq to crash on Ubuntu Jammy, which ships with dnsmasq 2.86. This issue sounds similar to an issue known about to the dnsmasq maintainers, where dnsmasq would crash with updates occurring due to configuration refresh[0].
This resulted in us upgrading dnsmasq to the version which ships with Ubuntu Lunar.
Which was no better. Dnsmasq still crashed upon record updates for addresses and ports getting configuration added/changed/
We later downgraded to the version of dnsmasq shipped in Ubuntu Focal, and dnsmasq stopped crashing and appeared stable enough to utilize for CI purposes.
** Kernel log from Ubuntu Jammy Package **
[229798.876726] dnsmasq[81586]: segfault at 7c28 ip 00007f6e8313147e sp 00007fffb3d6f830 error 4 in libc.so.
[229798.876745] Code: 98 13 00 e8 04 b9 ff ff 0f 1f 40 00 f3 0f 1e fa 48 85 ff 0f 84 bb 00 00 00 55 48 8d 77 f0 53 48 83 ec 18 48 8b 1d 92 39 17 00 <48> 8b 47 f8 64 8b 2b a8 02 75 57 48 8b 15 18 39 17 00 64 48 83 3a
[229805.444912] dnsmasq[401428]: segfault at dce8 ip 00007fe63bf6a47e sp 00007ffdb105b440 error 4 in libc.so.
[229805.444933] Code: 98 13 00 e8 04 b9 ff ff 0f 1f 40 00 f3 0f 1e fa 48 85 ff 0f 84 bb 00 00 00 55 48 8d 77 f0 53 48 83 ec 18 48 8b 1d 92 39 17 00 <48> 8b 47 f8 64 8b 2b a8 02 75 57 48 8b 15 18 39 17 00 64 48 83 3a
[230414.213448] dnsmasq[401538]: segfault at 78b8 ip 00007f12160e447e sp 00007ffed6ef2190 error 4 in libc.so.
[230414.213467] Code: 98 13 00 e8 04 b9 ff ff 0f 1f 40 00 f3 0f 1e fa 48 85 ff 0f 84 bb 00 00 00 55 48 8d 77 f0 53 48 83 ec 18 48 8b 1d 92 39 17 00 <48> 8b 47 f8 64 8b 2b a8 02 75 57 48 8b 15 18 39 17 00 64 48 83 3a
[230465.098989] dnsmasq[402665]: segfault at c378 ip 00007f81458f047e sp 00007fff0db334a0 error 4 in libc.so.
[230465.099005] Code: 98 13 00 e8 04 b9 ff ff 0f 1f 40 00 f3 0f 1e fa 48 85 ff 0f 84 bb 00 00 00 55 48 8d 77 f0 53 48 83 ec 18 48 8b 1d 92 39 17 00 <48> 8b 47 f8 64 8b 2b a8 02 75 57 48 8b 15 18 39 17 00 64 48 83 3a
[231787.247374] dnsmasq[402863]: segfault at 7318 ip 00007f3940b9147e sp 00007ffc8df4f010 error 4 in libc.so.
[231787.247392] Code: 98 13 00 e8 04 b9 ff ff 0f 1f 40 00 f3 0f 1e fa 48 85 ff 0f 84 bb 00 00 00 55 48 8d 77 f0 53 48 83 ec 18 48 8b 1d 92 39 17 00 <48> 8b 47 f8 64 8b 2b a8 02 75 57 48 8b 15 18 39 17 00 64 48 83 3a
[231844.886399] dnsmasq[405182]: segfault at dc58 ip 00007f32a29e147e sp 00007ffddedd7480 error 4 in libc.so.
[231844.886420] Code: 98 13 00 e8 04 b9 ff ff 0f 1f 40 00 f3 0f 1e fa 48 85 ff 0f 84 bb 00 00 00 55 48 8d 77 f0 53 48 83 ec 18 48 8b 1d 92 39 17 00 <48> 8b 47 f8 64 8b 2b a8 02 75 57 48 8b 15 18 39 17 00 64 48 83 3a
[234692.482154] dnsmasq[405289]: segfault at 67d8 ip 00007fab0c5c447e sp 00007fffd6fd8fa0 error 4 in libc.so.
[234692.482173] Code: 98 13 00 e8 04 b9 ff ff 0f 1f 40 00 f3 0f 1e fa 48 85 ff 0f 84 bb 00 00 00 55 48 8d 77 f0 53 48 83 ec 18 48 8b 1d 92 39 17 00 <48> 8b 47 f8 64 8b 2b a8 02 75 57 48 8b 15 18 39 17 00 64 48 83 3a
** Kernel log entries from Ubuntu Lunar package **
[234724.842339] dnsmasq[409843]: segfault at fffffffffffffffd ip 00007f35a147647e sp 00007ffd536038c0 error 5 in libc.so.
[234724.842368] Code: 98 13 00 e8 04 b9 ff ff 0f 1f 40 00 f3 0f 1e fa 48 85 ff 0f 84 bb 00 00 00 55 48 8d 77 f0 53 48 83 ec 18 48 8b 1d 92 39 17 00 <48> 8b 47 f8 64 8b 2b a8 02 75 57 48 8b 15 18 39 17 00 64 48 83 3a
[234784.918116] dnsmasq[410019]: segfault at fffffffffffffffd ip 00007f634233947e sp 00007fff33877f20 error 5 in libc.so.
[234784.918133] Code: 98 13 00 e8 04 b9 ff ff 0f 1f 40 00 f3 0f 1e fa 48 85 ff 0f 84 bb 00 00 00 55 48 8d 77 f0 53 48 83 ec 18 48 8b 1d 92 39 17 00 <48> 8b 47 f8 64 8b 2b a8 02 75 57 48 8b 15 18 39 17 00 64 48 83 3a
[235022.163339] dnsmasq[410151]: segfault at fffffffffffffffd ip 00007f21dd37f47e sp 00007fff9bf416d0 error 5 in libc.so.
[235022.163362] Code: 98 13 00 e8 04 b9 ff ff 0f 1f 40 00 f3 0f 1e fa 48 85 ff 0f 84 bb 00 00 00 55 48 8d 77 f0 53 48 83 ec 18 48 8b 1d 92 39 17 00 <48> 8b 47 f8 64 8b 2b a8 02 75 57 48 8b 15 18 39 17 00 64 48 83 3a
[235024.831325] dnsmasq[410445]: segfault at fffffffffffffffd ip 00007f7edf02147e sp 00007ffc4fb19cd0 error 5 in libc.so.
[235024.831354] Code: 98 13 00 e8 04 b9 ff ff 0f 1f 40 00 f3 0f 1e fa 48 85 ff 0f 84 bb 00 00 00 55 48 8d 77 f0 53 48 83 ec 18 48 8b 1d 92 39 17 00 <48> 8b 47 f8 64 8b 2b a8 02 75 57 48 8b 15 18 39 17 00 64 48 83 3a
[236052.793683] dnsmasq[410630]: segfault at fffffffffffffffd ip 00007f3046ca147e sp 00007ffe5583df50 error 5 in libc.so.
[236052.793704] Code: 98 13 00 e8 04 b9 ff ff 0f 1f 40 00 f3 0f 1e fa 48 85 ff 0f 84 bb 00 00 00 55 48 8d 77 f0 53 48 83 ec 18 48 8b 1d 92 39 17 00 <48> 8b 47 f8 64 8b 2b a8 02 75 57 48 8b 15 18 39 17 00 64 48 83 3a
[236105.451351] dnsmasq[412107]: segfault at fffffffffffffffd ip 00007f4425bcd47e sp 00007fffd5337560 error 5 in libc.so.
[236105.451368] Code: 98 13 00 e8 04 b9 ff ff 0f 1f 40 00 f3 0f 1e fa 48 85 ff 0f 84 bb 00 00 00 55 48 8d 77 f0 53 48 83 ec 18 48 8b 1d 92 39 17 00 <48> 8b 47 f8 64 8b 2b a8 02 75 57 48 8b 15 18 39 17 00 64 48 83 3a
** The command line the process is launched with **
dnsmasq --no-hosts --pid-file=
** Neutron Logging **
Jul 10 15:26:01 np0034614991 neutron-
Jul 10 15:26:01 np0034614991 neutron-
Jul 10 15:26:01 np0034614991 neutron-
Jul 10 15:26:01 np0034614991 neutron-
Jul 10 15:26:01 np0034614991 neutron-
Jul 10 15:26:01 np0034614991 neutron-
Jul 10 15:26:01 np0034614991 neutron-
Jul 10 15:26:01 np0034614991 neutron-
Jul 10 15:26:02 np0034614991 neutron-
Jul 10 15:26:02 np0034614991 neutron-
Jul 10 15:26:02 np0034614991 neutron-
Jul 10 15:26:02 np0034614991 neutron-
Jul 10 15:27:00 np0034614991 neutron-
Jul 10 15:27:00 np0034614991 neutron-
Jul 10 15:27:00 np0034614991 neutron-
Jul 10 15:27:00 np0034614991 neutron-
Jul 10 15:27:00 np0034614991 neutron-
We don't believe this is a neutron bug, at least outright, but suspect neutron is also likely encountering this issue as well, at least with any sort of exhaustive test jobs. Most of Ironic's one job tests would pass with this dnsmasq, it was only where we continually ran new test scenarios that we would see this issue crop up and cause failures.
In the mean time, the ironic project will likely downgrade dnsmasq to unblock it's CI.
[0]: https:/
no longer affects: | dnsmasq |
Changed in dnsmasq (Ubuntu Jammy): | |
status: | New → Confirmed |
Status changed to 'Confirmed' because the bug affects multiple users.