epmd does not support binding to an IPv4 address anymore

Bug #1374109 reported by alexander
86
This bug affects 16 people
Affects Status Importance Assigned to Milestone
erlang (Ubuntu)
Confirmed
High
Unassigned

Bug Description

The bug is actually located in erlang itself. epmd fails to bind to any IPv4 address if instructed to do so

[Impact]
Users of RabbitMQ in IPv4 environments are unable to start rabbitmq unless the environment variable ERL_EPMD_ADDRESS is unset. In case of disabled IPv6 support there is no valid setting for ERL_EPMD_ADDRESS as
* IPv4 addresses will be rejected as syntacitcally invalid
* IPv6 addresses will fail to bind

[Test Case]
service rabbitmq-server stop && killall epmd

ERL_EPMD_ADDRESS="127.0.0.1" epmd
epmd: Thu Sep 25 21:00:16 2014: cannot parse IP address "127.0.0.1"

same could be acieved with the epmd's -address option

In case IPv6 support is disabled:
ERL_EPMD_ADDRESS="::1" epmd
epmd: Thu Sep 25 21:01:40 2014: failed to bind socket: Cannot assign requested address

-- rabbitmq case:
cat rabbitmq-env.conf
export RABBITMQ_NODENAME=rabbit@localhost
export RABBITMQ_NODE_IP_ADDRESS=127.0.0.1
export ERL_EPMD_ADDRESS=127.0.0.1

service rabbitmq-server start
 * Starting message broker rabbitmq-server
 * FAILED - check /var/log/rabbitmq/startup_\{log, _err\} [fail]

Commenting/removing the "export ERL_EPMD_ADDRESS=127.0.0.1" line resolves the issue

[Possible cause]
In this bug: https://bugs.launchpad.net/ubuntu/+source/rabbitmq-server/+bug/1312507?comments=all

The issue was fixed using -DEPMD6" in CPPFLAGS (https://bugs.launchpad.net/ubuntu/+source/rabbitmq-server/+bug/1312507/comments/5) which perhaps (I speculate only) activates some of the erlang's IPV6_V6ONLY definitions.

There is another complaint regarding the same issue here: https://bugs.launchpad.net/ubuntu/+source/rabbitmq-server/+bug/1312507/comments/14

[System info]
lsb_release -rd
Description: Ubuntu 14.04.1 LTS
Release: 14.04

apt-cache policy rabbitmq-server
rabbitmq-server:
  Installed: 3.2.4-1
  Candidate: 3.2.4-1

apt-cache policy erlang-base
erlang-base:
  Installed: 1:16.b.3-dfsg-1ubuntu2.1
  Candidate: 1:16.b.3-dfsg-1ubuntu2.1

description: updated
description: updated
affects: rabbitmq-server (Ubuntu) → erlang (Ubuntu)
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in erlang (Ubuntu):
status: New → Confirmed
Revision history for this message
Mihaly Arva-Toth (mihaly-arva-toth+ulaunchpad) wrote :

Workaround for me: recomplied whole erlang source _without_ CPPFLAGS=-DEPMD6 in debian/rules file. It would be great if epmd could listen ipv4 and ipv6 at same time.

Louis Bouchard (louis)
tags: added: cts
Revision history for this message
James Page (james-page) wrote :

Poking at this a bit it appears that epmd will bind everywhere by default, specific IPv6 addresses but not specific IPv4 addresses; so by enabling IPv6 support, we've broken anyone who's a) disabled IPv6 or b) trying to limit access to a single IPv4 address.

Changed in erlang (Ubuntu):
importance: Undecided → High
Revision history for this message
Arthur Lutz (arthur-lutz) wrote :

wow

Revision history for this message
alexander (alex-g-marinov) wrote :

@James Page,

on my specific setup epmd was starting just fine if there is no excplicit IPv6 bind address set (via the ERL_EPMD_ADDRESS env). On `netstat -ntlp` epmd was visible and listening to ::1. IPv6 on box was disabled per https://wiki.ubuntu.com/IPv6#Disabling_IPv6

I'll assume it may be working fine for a) if you don't specify a bind address tough I didn't test this. Unfortunately I've changed my setup and I'm unable to confirm this.

FTR: upstream packages (via Erlang's Ubuntu repository) are compiled with IPv4 and this is probably the easiest workaround for the case.

Changed in erlang (Ubuntu):
assignee: nobody → Dariusz Gadomski (dgadomski)
Revision history for this message
Dariusz Gadomski (dgadomski) wrote :

Since by default IPV6_V6ONLY option is *turned off* by default for ipv6 sockets in Linux it is possible to use the IPV4-mapped IPV6 addresses to do the binding [1].

What I have tried was:
$ export ERL_EPMD_ADDRESS=::ffff:127.0.1.1
$ epmd -d
$ netstat -npta | grep 127.0.1.1:4369
tcp6 0 0 127.0.1.1:4369 :::* LISTEN 3544/epmd

[1] http://en.wikipedia.org/wiki/IPv6#IPv4-mapped_IPv6_addresses

Revision history for this message
Dariusz Gadomski (dgadomski) wrote :

I have found an interesting merge request for erlang [1].

Looks like during the implementation it's author has fixed the assumption that if epmd is built with IPv6 support it losses the possibility to bind with IPv4 address.

I have cherry-picked that change and applied it to Ubuntu's erlang-base. Seems to solve the issue.

Can anyone give it a try? It is available in my ppa [2].

[1] https://github.com/msantos/otp/compare/erlang:maint...epmd-IPv6-node-reg
[2] ppa:dgadomski/erlang

Revision history for this message
Marcin Jaworski (yaworski) wrote :

@Dariusz Gadomski,

I've upgraded erlang from your PPA and I can confirm that it epmd (and rabbitmq) is back again starting and responding correctly.

Revision history for this message
Dariusz Gadomski (dgadomski) wrote :

Thanks Marcin.

I got some feedback from the feature author on the erlang-bugs mailing list. He will make a new pull request against the latest erlang. Hopefully it will get merged and we will be able to cherry-pick it and deliver as SRU to Ubuntu.

Revision history for this message
David Medberry (med) wrote :

Has there been any update on this?
(Iknow that's a sad pathetic update.)

Revision history for this message
Dariusz Gadomski (dgadomski) wrote :

David, unfortunately I don't see any updates in this area in upstream erlang.

At this point I can suggest you using either:
1) the version from my ppa [1] or
2) the workaround from comment #6

I will update this bug as soon as the fix will be merged to the upstream erlang.

[1] ppa:dgadomski/erlang

Revision history for this message
Daniel T. (dthul) wrote :

I am totally stuck with this issue. I disabled IPv6 via a Grub boot option.

When I try to set an IPv4 address for ERL_EPMD_ADDRESS it tells me that it can't be parsed. If I try an IPv6 address (like comment #6 recommends) it tells me "Address family not supported by protocol".

I tried Dariusz' Erlang repository but now I get "Address family not supported by protocol" for IPv4 and IPv6 addresses.

Revision history for this message
Dariusz Gadomski (dgadomski) wrote :

Daniel: can you please try the latest version available in my ppa [1]. It contains the latest approach taken upstream to solve this issue. Please provide your feedback regarding this. Thanks!

[1] ppa:dgadomski/erlang

Revision history for this message
Dr. Jens Harbott (j-harbott) wrote :

Dariusz: I'm seeing the same issue, but running on Vivid, could you add packages for that to your repo so I could test your solution?

Revision history for this message
Dariusz Gadomski (dgadomski) wrote :

A build for Vivid is available in my ppa [1] for testing. I will appreciate any feedback. Thanks!

[1] ppa:dgadomski/erlang

Revision history for this message
L-reimann (l-reimann) wrote :

hi guys,

please take a look at the date the bug was reported:
2014-09-25, status is auto-confirmed and prio is high.
Is there chance that this gets fixed before one year is over?
I would also argue the estimated number of unreported cases is much higher!
Greetings,
LR

Revision history for this message
Dr. Jens Harbott (j-harbott) wrote :

@Dariusz: The package for Vivid works fine for me, thanks.

tags: added: sts
removed: cts
Revision history for this message
Dr. Jens Harbott (j-harbott) wrote :

Running with 1:18.3-dfsg-1ubuntu3 on Xenial, the issue seems to have been fixed for me, can anyone confirm this?

Revision history for this message
John Lewis (jlewis-johnlewis-deactivatedaccount) wrote :

Just to let you know about a possibly interesting edge case which *may* be related to this bug - if I use "ipv6.disable=1" to disable ipv6, "export ERL_EPMD_ADDRESS=127.0.0.1" works as expected, if I use "net.ipv6.conf.all.disable_ipv6 = 1" via sysctl to disable ipv6, I get "epmd: failed to bind socket".

Using RabbitMQ 3.5.7-1ubuntu0.16.04.1 on 16.04.

Tried the workaround from comment 6 but obviously that's unlikely to work with ipv6 disabled.

Revision history for this message
John Lewis (jlewis-johnlewis-deactivatedaccount) wrote :

Additionally, using "net.ipv6.conf.lo.disable_ipv6 = 0" to enable ipv6 just for loopback works around the problem, and lsof -i does show epmd using ipv6 despite specifying only the ipv4 loopback in ERL_EPMD_ADDRESS.

lsof -i |grep epmd
epmd 30418 rabbitmq 3u IPv4 458312 0t0 TCP localhost:epmd (LISTEN)
epmd 30418 rabbitmq 4u IPv6 458313 0t0 TCP localhost:epmd (LISTEN)
epmd 30418 rabbitmq 6u IPv4 457617 0t0 TCP localhost:epmd->localhost:59565 (ESTABLISHED)
beam.smp 30476 rabbitmq 13u IPv4 459345 0t0 TCP localhost:59565->localhost:epmd (ESTABLISHED)

Dan Streetman (ddstreet)
tags: removed: sts
Changed in erlang (Ubuntu):
assignee: Dariusz Gadomski (dgadomski) → nobody
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.