[2.5rc2] Cannot communicate with KVM hosts if stale SSH fingerprint is cached

Hi Florian,

COuld you please provide full installation logs and machine cloud-init logs:

rsyslog: /var/log/maas/rsyslog/<machine-name>/date/messages
machine cloud-init logs: /var/log/cloud-init.log,cloud-init-output.log #from the system itself. (the "deployed" one)

I would assume this is just be an informational message, and the deployment is failing for some other reason, because the `virsh` underprivileged user is intentionally deployed without any SSH keys. MAAS generates a random password to be used to communicate with the pod. (A default user - `ubuntu` - will also be configured with the normal SSH keys that exist in MAAS.)

You might also check the node event log to see if there is anything interesting in there.

Sorry for my lousy grammar. I meant to say, "I assume this is just an informational message", referring to the "no authorized ssh keys" message you observed.

One last thing: I would check /var/log/maas/regiond.log during the time that the deployment was finishing to see if anything interesting has been logged.

I am attaching here the extract from the log files as I was proceeding to deployement.
It seems that it has indeed "Failed talking to pod: Failed to login to virsh console".
Now the node that is deployed seem to be functional again, all packaged deployed, network configured appropriately.

System logs to follow...

It seem to remain at [0/1] of opening 'http://10-80-0-0--16.maas-internal:5248/MAAS/metadata/status/cfppes', but checking manually after deployment, this is reachable by the system.

Would you have any pointer ?
Let me know if I can provide any further element.

I should add that deploying the same node with same configuration and without setting it as a KVM host works and complete successfully

I think the relevant portion of this message is, as you said in a previous comment, the "Failed to login to virsh console" error.

I have a few theories about why that might happen:

(1) A cached SSH key fingerprint for the same IP address (for the `maas` user) caused the KVM driver to fail to log into the newly-deployed KVM host. If this is the cause, we could see this problem when the same IP address is assigned to a host that was previously used as a KVM host in MAAS.

(2) The IP address that MAAS chose to communicate with the newly-deployed machine was unreachable (could be due to firewall issues, etc.)

(3) A race condition occurred; cloud-init reported that it was finished, but the service wasn't yet ready for MAAS to talk to it.

It would be helpful if you could help us narrow down the cause. Note that the deploying KVM host sleeps for 10 seconds before cloud-init finishes, as a precaution to mitigate (3). So I think options (1) or (2) are most likely.

I think you have found our culprit.
It seems to be (1). Indeed doing the following manage to allow the node to deploy with no issue :

root@maas-controller-01:> sudo -u maas -H bash
maas@maas-controller-01:> echo -n > ~/.ssh/known_hosts

The physical nodes on our network are allocated static addresses, so when they get recomissionned/redeployed we could essentially fall into that pitfall again.

Maybe there could be some clever bit of logic added to MAAS to cleanup the known_hosts files appropriately. Or in case there are no obvious way of doing that, maybe add a note in the documentation about this limitation.

Thank you very much for the quick and precise answer !
Very best wishes,

Thanks for your help testing MAAS.

This issue is a little contentious, since by cleaning up the key fingerprints we reduce the security of MAAS by opening up the possibility that an attacker who has infiltrated your network could launch a man-in-the-middle (MitM) attack against virsh-over-SSH.

However, MAAS never provides the option to check the host key to begin with, so the network could already be compromised. And if an attacker has the ability to do that, I guess one's security theater is already bankrupt. ;-)

Therefore, if power drivers in MAAS call `ssh` with the following additional parameters, we believe the user experience will be improved:

    -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null

A full solution to this problem would require users to manually confirm SSH key fingerprints any time they are first cached, or any time they change. And of course, we could also automatically clear out cached fingerprints if a new machine is deployed. But if the goal is to secure MAAS from MitM attacks, that would also require manual confirmation every time a new KVM host is deployed. I don't think that's the user experience MAAS users expect.

Could MAAS proceed with removing the key fingerprint of a virsh pod as soon as the pods gets deleted or the machine gets released based on all known IPs of the node ?

I believe it might be a reasonable approach. Would there be a scenario where an automatic removal for node explicitly decommissioned by the user wouldn't be appropriate ? This wouldn't require further user input.