juju ssh/scp proxy ip address selection oddities

In your above list of IP addresses, I have the feeling that we can't really tell what is "public" and what is "private" given that 10.100.* doesn't look any more public/private than 10.96.*

I believe we use a few of the "what is a private IP address" detection, but when everything falls under RFC1918, we are unable to tell which is actually routable.

(I presume that all of these machines actually have multiple network interfaces, and multiple potential addresses.)

It is true that in 2.0 we don't model preferred spaces for accessing machines very well. We do model what spaces applications should interact with, but that doesn't give information about the machine itself.

Hi John --

> (I presume that all of these machines actually have multiple
> network interfaces, and multiple potential addresses.)

In the case of what I presented above, 10.96.* are floating IP addresses, the openstack instance itself doesn't even have an interface bound to that IP. Openstack takes care of the routing above it.

10.100.* is the private IP range inside the cloud and does map to a real interface on the system. It also is another range I can see on my corporate VPN. The juju command line client seems to be dynamically calculating that it thinks it can connect to the 10.100.* range for these machines, maybe through a direct socket test or something. I don't know. I believe this to be where the bug lies. Since, it can connect the socket, but it's actually the *wrong* host.

So, going further, SSH attempts to connect to 10.100.* and it's the wrong machine (the one on the corporate VPN). This action attempts to verify the key, and that fails.

This bug is crazy :)

juju ssh <n> to a remote machine tries to connect to my local virbr0 ip :)

$ juju ssh --debug 5
17:33:44 INFO juju.cmd supercommand.go:63 running juju [2.1-beta3 gc go1.6.2]
17:33:44 DEBUG juju.cmd supercommand.go:64 args: []string{"juju", "ssh", "--debug", "5"}
17:33:44 INFO juju.juju api.go:72 connecting to API addresses: [10.245.202.4:17070]
17:33:44 INFO juju.api apiclient.go:570 dialing "wss://10.245.202.4:17070/model/7a97887e-34b0-4d7f-82c2-53e1cba1d8d2/api"
17:33:45 INFO juju.api apiclient.go:501 connection established to "wss://10.245.202.4:17070/model/7a97887e-34b0-4d7f-82c2-53e1cba1d8d2/api"
17:33:45 DEBUG juju.juju api.go:263 API hostnames unchanged - not resolving
17:33:46 DEBUG juju.cmd.juju.commands ssh_common.go:263 proxy-ssh is false
17:33:46 INFO juju.network hostport.go:274 dialed "192.168.122.1:22" successfully
17:33:46 DEBUG juju.cmd.juju.commands ssh_common.go:367 using target "5" address "192.168.122.1"
17:33:46 DEBUG juju.utils.ssh ssh.go:292 using OpenSSH ssh client
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:r34Eo3QVARHyMofRFeWv5ETsgykBh1jJMx2a5A8cCcM.
Please contact your system administrator.
Add correct host key in /tmp/ssh_known_hosts954879309 to get rid of this message.
Offending RSA key in /tmp/ssh_known_hosts954879309:7
  remove with:
  ssh-keygen -f "/tmp/ssh_known_hosts954879309" -R 192.168.122.1
ECDSA host key for 192.168.122.1 has changed and you have requested strict checking.
Host key verification failed.
17:33:46 INFO cmd supercommand.go:465 command finished

Machine State DNS Inst id Series AZ
1 started 10.245.200.40 4y3h8m xenial budapest
1/lxd/0 started 10.245.202.15 juju-a1d8d2-1-lxd-0 xenial
2 started 10.245.200.36 4y3h8k xenial budapest
2/lxd/0 started 10.245.200.41 juju-a1d8d2-2-lxd-0 xenial
5 started 10.245.200.37 4y3ha8 xenial prague

Using 2.1beta3

I have the same problem as above the bug.

I am using juju 2.0.2-xenial-amd64 and deploy openstack with MAAS 2.1.2

At first step, it looked fine to deploy machines and openstack.

But it comes up with ssh bug when I try to connect one of nova-compute machines

juju --debug ssh nova-compute/0
13:39:32 INFO juju.cmd supercommand.go:63 running juju [2.0.2 gc go1.6.2]
13:39:32 DEBUG juju.cmd supercommand.go:64 args: []string{"juju", "--debug", "ssh", "nova-compute/0"}
13:39:32 INFO juju.juju api.go:72 connecting to API addresses: [10.10.1.50:17070]
13:39:32 INFO juju.api apiclient.go:530 dialing "wss://10.10.1.50:17070/model/59543072-caf8-40d4-8bf9-2f7de34aa536/api"
13:39:32 INFO juju.api apiclient.go:466 connection established to "wss://10.10.1.50:17070/model/59543072-caf8-40d4-8bf9-2f7de34aa536/api"
13:39:32 DEBUG juju.juju api.go:263 API hostnames unchanged - not resolving
13:39:32 DEBUG juju.cmd.juju.commands ssh_common.go:263 proxy-ssh is false
13:39:32 INFO juju.network hostport.go:274 dialed "192.168.122.1:22" successfully
13:39:32 DEBUG juju.cmd.juju.commands ssh_common.go:367 using target "nova-compute/0" address "192.168.122.1"
13:39:32 DEBUG juju.utils.ssh ssh.go:292 using OpenSSH ssh client
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:EBOclSl+7NMqLKwKzzq7j3ihwnP8l9Rp5SVMtcyAprk.
Please contact your system administrator.
Add correct host key in /tmp/ssh_known_hosts843183779 to get rid of this message.
Offending RSA key in /tmp/ssh_known_hosts843183779:7
  remove with:
  ssh-keygen -f "/tmp/ssh_known_hosts843183779" -R 192.168.122.1
ECDSA host key for 192.168.122.1 has changed and you have requested strict checking.
Host key verification failed.
13:39:32 DEBUG juju.api monitor.go:35 RPC connection died
13:39:32 INFO cmd supercommand.go:465 command finished

and juju shows status of nova-compute/0

nova-compute/0 unknown lost 5 10.10.1.56

but it is trying to connect my local virbr interface (192.168.122.1)

yeah, juju is incorrectly grabbing the virbr0 IP. virbr0 exists on nova compute hosts nowadays because of the possibility of using lxd as the hypervisor.

We discussed a possible solution today with the Juju team, I want to just
recap here:

Juju could keep doing the same procedure as now to return all IPs,
including the socket connection test to each IP, it just needs to validate
the host keys for each to make sure that more than just the socket is
open. This can be done with

1) ssh <ip> /bin/true

or

2) get host key, verify against what is expected (which is already done as
part of the ssh process, so juju knows this)

On Tue, Jan 10, 2017 at 1:02 PM, Andreas Hasenack <email address hidden>
wrote:

> yeah, juju is incorrectly grabbing the virbr0 IP. virbr0 exists on nova
> compute hosts nowadays because of the possibility of using lxd as the
> hypervisor.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1646329
>
> Title:
> juju ssh/scp proxy ip address selection oddities
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1646329/+subscriptions
>
> Launchpad-Notification-Type: bug
> Launchpad-Bug: product=juju; milestone=2.1-rc1; status=Triaged;
> importance=High; <email address hidden>;
> Launchpad-Bug-Tags: landscape
> Launchpad-Bug-Information-Type: Public
> Launchpad-Bug-Private: no
> Launchpad-Bug-Security-Vulnerability: no
> Launchpad-Bug-Commenters: ahasenack davidpbritton jameinel thkang0
> Launchpad-Bug-Reporter: David Britton (davidpbritton)
> Launchpad-Bug-Modifier: Andreas Hasenack (ahasenack)
> Launchpad-Message-Rationale: Subscriber
> Launchpad-Message-For: davidpbritton
>

--
David Britton <email address hidden>

This is also preventing our jobs to collect logs from compute hosts when a deployment fails, because that relies on juju ssh/scp which is failing because of the wrong ip :(

PR against 2.1: https://github.com/juju/juju/pull/6857

The PR is no longer WIP, I'm looking for reviews and hopefully will land in 2.1 today or tomorrow.

Has landed in 2.1 and 2.2 today.

Where can this released fix be consumed? I don't see 2.1 available in the stable ppa.

@Ryan,
2.1-beta5 went out late last week \o/

Correction: as of today beta5 is only available in the ppa:juju/devel (development repository). When will it be available in proposed and stable?

Will this fix be landed in 2.0.X series? I think it isn't in 2.0.3 (https://launchpad.net/juju/+milestone/2.0.3)

@Junaid Ali (junaidali),
The fix went into Juju 2.1.
We are not planning any further releases of Juju 2.0.x, so I have explicitly marked this bug as Won't Fix for 2.0.

It is unlikely to be in 2.0.x as we're not planning another 2.0 release.
2.1.0 should show up in stable this week.

(flagged Fix Committed per the agreement that Released is only in final stable releases.)