OpenSSH server config broken on unattended update

Could you please share at least the relevant parts of your sshd_config? Anything that configures Port, ListenAddress, or AddressFamily? Feel free to modify the actual port and listen address values of course; I just need to know where you are using non-default configuration.

# This is the sshd server system-wide configuration file. See
# sshd_config(5) for more information.

# This sshd was compiled with PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented. Uncommented options override the
# default value.

Include /etc/ssh/sshd_config.d/*.conf

# Port and ListenAddress options are not used when sshd is socket-activated,
# which is now the default in Ubuntu. See sshd_config(5) and
# /usr/share/doc/openssh-server/README.Debian.gz for details.
Port <...redacted...>
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key

# Ciphers and keying
#RekeyLimit default none

# Logging
#SyslogFacility AUTH
#LogLevel INFO

# Authentication:

/etc/ssh/sshd_config.d/ is empty

As I've said, my config is working after following the linked steps, but unfortunately that means I don't have the systemd socket configuration files anymore.

I'm hoping the fact that this specific update seemed to stop my ssh service can help narrow this down.

Okay - nothing in /etc/ssh/sshd_config.d/? You just have a non-default port configured?

Was ssh.socket failing to start after the upgrade? E.g. does journalctl -u ssh.socket show errors around that time (or ssh.service for that matter)?

So far I cannot reproduce anything like you are describing. Any other logs from ssh.service and ssh.socket would be helpful.

Since it appears you have another way to gain shell access besides ssh, are you able to test if the problem persists if you attempt to restore the socket activation configuration?

Update was at 6:48 local time today.

journalctl -u ssh.socket shows the socket deactivated and came back, no more activity until I started trying to fix things at 8:13:

Nov 08 06:48:27 www-veltas systemd[1]: ssh.socket: Deactivated successfully.
Nov 08 06:48:27 www-veltas systemd[1]: Closed ssh.socket - OpenBSD Secure Shell server socket.
Nov 08 06:48:30 www-veltas systemd[1]: Listening on ssh.socket - OpenBSD Secure Shell server socket.

This is shown corresponding to that unattended update with -u ssh.service:

Nov 08 06:48:27 www-veltas sshd[1102]: Received signal 15; terminating.
Nov 08 06:48:27 www-veltas systemd[1]: Stopping ssh.service - OpenBSD Secure Shell server...
Nov 08 06:48:27 www-veltas systemd[1]: ssh.service: Deactivated successfully.
Nov 08 06:48:27 www-veltas systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server.
Nov 08 06:48:27 www-veltas systemd[1]: ssh.service: Consumed 2min 19.476s CPU time, 20.1M memory peak, 1.1M memory swap peak.

And in case it was missed: /etc/ssh/sshd_config.d/ is empty.

I can try breaking ssh again to reproduce, but I don't know sockets too well, could you point me to some info on how to enable this? I tried looking and can only find instructions to go the other way, apologies!

No problem. You should be able to restore socket activation with:

# This removes the symlink to mask the generator, if needed.
$ rm -f /etc/systemd/system-generators/sshd-socket-generator
$ systemctl daemon-reload
$ systemctl disable --now ssh.service
$ systemctl enable --now ssh.socket

Thanks.

This has restored socket-based activation, but with a different configuration than I had earlier, and has not reproduced the problem.

I've confirmed that this really is socket-based activation by stopping the ssh service in the recovery console, confirming sshd is dead, and reconnecting, which worked fine.

Unfortunately I didn't save the old socket configuration.

I've reverted the update, 13.7 got removed from noble-updates, 13.5 put back

> but with a different configuration than I had earlier

Different in what way?

I am glad this restored socket-activation for you. I am not sure how to further investigate the bug. If you are able to find some error in the journal, or evidence that ssh.socket was listening on the wrong port etc., please share it.

Did you happen to check systemctl status ssh.socket ssh.service (or something) before making the change to non-socket activation?

> If you are able to find some error in the journal, or evidence that ssh.socket was listening on the wrong port etc., please share it.

Sorry I have nothing, just know that it was refusing connection, and that trying to start sshd manually with `systemctl start ssh.service` didn't work even though it said it was listening on the port in `systemctl status ssh.service` after that.

Sorry I can't be more help.

> Sorry I can't be more help.

No worries. Thanks for all the information you provided.

I just wanted to add to this thread that I experienced the same issue after our server performed the unattended update of openssh-server from 1:9.6p1-3ubuntu13.5 to 1:9.6p1-3ubuntu13.7. We had a custom port for SSH, and our attempts to switch back to default so far have been fruitless.

Kevin - can you provide any additional information about what's not working? If you otherwise have access to the server, are ssh.service or ssh.socket reporting errors?

Nick-
We had the same config as OP on his first report.
We did what OP did by rolling back to the non-socket based SSH. We have our connections back on 22. There were zero errors in the logs to report, and when we were originally trying to ssh with --verbose the only message shown was "connection refused"- We dug around in the logs and nothing was showing as an issue, which led us to this thread.

Okay, thanks for following up.

If anyone finds a way to reliably reproduce this, please let me know. So far I have tried:

- adding a custom port (both directly in /etc/ssh/sshd_config, and using /etc/ssh/sshd_config.d/port.conf)
- upgrading openssh-server (both manually, and by running unattended-upgrades)

In any case, I do not experience any connection refused errors, nor do I see any errors on the server side ssh units (ssh.service, ssh.socket).

I would also throw in some testing around address families, ipv4/ipv6, maybe there is a regression in https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2080216

Thanks for the suggestion Andreas. I have tested some scenarios like that and still cannot reproduce.

Chris and Kevin (or anyone else affected) - Can you please attach the logs in /var/log/unattended-upgrades/? Maybe they will give a clue to anything unusual that happened during the upgrade itself.

I had this happen to me as well as the original poster. I also disabled socket authentication using the steps from the thread attached by the original poster to get access to the server again via SSH. Here is the unattended-upgrades log:

Log started: 2024-11-09 06:52:08
Preconfiguring packages ...
Preconfiguring packages ...
(Reading database ... 121348 files and directories currently installed.)
Preparing to unpack .../openssh-sftp-server_1%3a9.6p1-3ubuntu13.7_amd64.deb ...
Unpacking openssh-sftp-server (1:9.6p1-3ubuntu13.7) over (1:9.6p1-3ubuntu13.5) ...
Preparing to unpack .../openssh-server_1%3a9.6p1-3ubuntu13.7_amd64.deb ...
Unpacking openssh-server (1:9.6p1-3ubuntu13.7) over (1:9.6p1-3ubuntu13.5) ...
Preparing to unpack .../openssh-client_1%3a9.6p1-3ubuntu13.7_amd64.deb ...
Unpacking openssh-client (1:9.6p1-3ubuntu13.7) over (1:9.6p1-3ubuntu13.5) ...
Setting up openssh-client (1:9.6p1-3ubuntu13.7) ...
Setting up openssh-sftp-server (1:9.6p1-3ubuntu13.7) ...
Setting up openssh-server (1:9.6p1-3ubuntu13.7) ...
Processing triggers for man-db (2.12.0-4build2) ...
Processing triggers for ufw (0.36.2-6) ...

Restarting services...

Service restarts being deferred:
 /etc/needrestart/restart.d/dbus.service
 systemctl restart <email address hidden>
 systemctl restart <email address hidden>
 systemctl restart systemd-logind.service
 systemctl restart unattended-upgrades.service

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.
Log ended: 2024-11-09 06:52:14

Thanks, John. That all looks normal.

Can you share the relevant parts of your configuration? I.e.:

sshd -T | grep -E '^port|^listenaddress|^addressfamily'

Again, the specific values are not so important, but I would like to know if they differ from the defaults. Is there any thing else special about your configuration?

Can you also provide any relevant logs from ssh.service and ssh.socket during the time you could not connect? And, what was the exact error you saw on the client side?

Chris, Kevin, John, or anyone else affected:

Is it possible that you changed your sshd_config without actually reloading and restarting ssh.socket, a while (hours, days, etc.) before the upgrade occurred? I am wondering if it's possible that the configuration was already broken by a local change, and the upgrade (which will call systemctl daemon-reload and systemctl restart ssh.socket) just surfaced the problem.

Also, can you all try to restore socket activation, and see if the problem persists? You can restore socket activation by running:

# This removes the symlink to mask the generator, if needed.
$ rm -f /etc/systemd/system-generators/sshd-socket-generator
$ systemctl daemon-reload
$ systemctl disable --now ssh.service
$ systemctl enable --now ssh.socket

Is there any chance that the people affected by this bug perhaps had an upgrade policy such that the new config file shipped with openssh-server would be taken as-is, instead of keeping the local changes? The telltale sign would be a message like "Replacing config file /etc/ssh/sshd_config" in the terminal output during the upgrade. I don't see that in comment #23, so that case was not this.

If you changed the port, and then the upgrade was allowed to install the new config file (because there were changes in the new config file shipped in openssh-server), then the port would be restored to 22.

So when you guys experienced "connection refused", was that on the custom port you had, and you didn't try port 22? Or was port 22 also closed?

Were there any packaging-driven backup files created in /etc/ssh? Like sshd_config.ucf-old, for example

I tried many other things to reproduce this bug:

- looks like the reporter had this happen in a Digital Ocean VM. I tried that too, going through the openssh upgrades all the way to 13.7, changing the port to 2240, and it just worked
- tried ipv4 and ipv6
- then noted I was doing this all via ssh, which could interfere with the troubleshooting. Went back to local lxc and used "lxc console" instead of an ssh connection. It also worked
- then I used unattended-upgrades itself. I configured the system to bump the priority of openssh in noble-proposed, and configured unattended-upgrades to also consider proposed. It upgraded openssh-server without issues, on the different port, and I could ssh in after
- finally, same as above, but I did not restart openssh (or the socket) after changing the port to 2240. I let unattended-upgrades do it, to the version in proposed. It also worked.

I'm out of ideas here. The only case where I could reproduce something similar to what was reported here is if I let the new configuration file from the package overwrite my local changes, but even then, all that would happen is ssh/systemd listening again on port 22 instead of my custom port. If you guys had a firewall on port 22 or something like that, it could explain the system no longer being reachable, but the log from comment #23 disproves that theory for that user at least.

[Expired for openssh (Ubuntu) because there has been no activity for 60 days.]