cloud-init enables ssh password auth in an unexpected config file

Is there a reason cloud-init needs to create an override in the first place, rather than changing the setting in the main file?

Cheers to @jchittum for pointing out that ssh clearly documents sshd_config.d

https://manpages.ubuntu.com/manpages/noble/en/man5/sshd_config.5.html

> /etc/ssh/sshd_config.d/*.conf files are included at the start of the configuration file, so
> options set there will override those in /etc/ssh/sshd_config.

If we continue to use sshd_config.d, how can we help raise awareness?

While the override directory is documented, it is quite unexpected that a default installation will make use of it, which is why this bug exists in the first place.

When using the latest 24.04 server installer, it will ask if you want to opt in to ssh. On opt in, the installer selects "Allow password authentication over SSH" automatically, and can only be turned off if the user provides a key.

So, between sshd_config (5) and subiquity's documentation, I no longer believe this is wholly unexpected. We should still work to resolve this. I like Marc's suggestion of avoiding sshd_config.d all together. If we keep sshd_config.d, a warning in /etc/ssh/sshd_config would be nice.

(Are there other creative solutions to this? sshd_config's conditional Match blocks don't seem to help here.)

> avoiding sshd_config.d all together

Why? Isn't this the entire point of the .d directory? The package provides a default configuration. If another package or service (like cloud-init) wants to change it, they should be dropping in an override file. If cloud-init changes the config file it's less obvious that cloud-init made the change and upgrades are trickier because there are changed values that the user needs to deal with that they never explicitly set.

Perhaps an acceptable solution could be to write the file only if cloud-init needs to overwrite the value to "no", but if the value is "yes", the openssh default, it shouldn't create the file. This would allow continuing to use the .d directory, but would prevent confusion which results in password authentication being enabled contrary to expectations.

> > avoiding sshd_config.d all together

> Why?

You have convinced me that sshd_config.d is the most technically advantageous and correct way to configure ssh. It is not intuitive though. That is why I am for lllf's second recommendation.

I like Marc's suggestion in addition to lllf's. We could implement both together. Similar to what Marc suggests, we could instead change subiquity to only set PasswordAuthentication when the value is No.

> Perhaps an acceptable solution could be to write the file only if cloud-init needs to overwrite the value to "no", but if the value is "yes", the openssh default, it shouldn't create the file.

This would require cloud-init to implement a parser which correctly parses all of sshd's configuration files (sshd_config and sshd_config.d) with exactly the same semantics in order to identify the current value. Sounds easy to implement something that would appear to work, but this suggestion sounds error prone and full of edge cases and risk of implementation drift between sshd and cloud-init. It seems to me that the problem is that users expectations don't align with documented behaviors. I think it would be less risky to fix a user expectations problem with docs / comments rather than potentially introducing vulnerabilities in the process of managing user's perceptions.

If you google "how to disable ssh password authentication", there are pages and pages of instructions that instruct to modify sshd_config. I'm not sure how to correct user expectations. Maybe adding more explicit comments to sshd_config could be okay.

How is cloud-init making sure another file in sshd_config.d isn't superseding its config?

Brett, I think Marc's 'only if cloud-init needs to overwrite the value to "no"' was less about the existing sshd configuration (afterall, if cloud-init is running, did the configuration have meaningful values ten seconds earlier?) and more about the user-data being explicit that passwords should be used in preference to ssh keys for logging in.

Certainly re-implementing sshd's parsing isn't ideal, even if "understand the current configuration" is desired -- sshd -T should be consulted, instead of trying to mimic what sshd -T does.

Thanks

I'm adding the openssh package to this bug, as the default configuration file has a Debian/Ubuntu-specific include directory configured and I think we should add an appropriate comment to inform the user that files included in the directory may override the configuration items in ssd_config. This wouldn't be something added by cloud-init, but something shipped in the openssh package itself.

Perhaps something like:

----
# The files included in the following directory may contain options
# that override the options in the current file. Configuration tools
# such as cloud-init may populate this directory in a default
# installation.
Include /etc/ssh/sshd_config.d/*.conf
----

> the most technically advantageous and correct way to configure ssh

Agreed

> How is cloud-init making sure another file in sshd_config.d isn't superseding its config?

It isn't. I just filed an upstream bug to track this as an issue - and linked it above.

> sshd -T should be consulted, instead of trying to mimic what sshd -T does

Agreed, however ssh -G is more robust since -T is more likely to have failing tests.

> I'm adding the openssh package to this bug

Thanks! That approach sounds good to me.

Since the cloud-init (ubuntu) project exists to track issues in cloud-init on Ubuntu, I'm going to mark it as invalid for now. Feel free to change that back if you believe this to still be an issue with cloud-init.

This bug was fixed in the package openssh - 1:9.9p1-3ubuntu2

---------------
openssh (1:9.9p1-3ubuntu2) plucky; urgency=medium

  * document /etc/ssh/sshd_config.d/*.conf better in sshd_config
    (LP: #2088207)
    - d/p/debian-config.patch: expand comment about configuration options
      and precedence of configuration snippets
    - d/openssh-server.ucf-md5sum: update for new sshd_config comments
    - d/p/sshd-socket-generator.patch: refresh for sshd_config comment
  * d/t/systemd-socket-activation: add wait while unit is reloading
    This avoids a race condition where we sometimes fail the assertion
    that ActiveState=active after systemctl reload.
    (LP: #2089049)

 -- Nick Rosbrook <email address hidden> Tue, 19 Nov 2024 12:06:14 -0500