cloud-init

util.py[WARNING]: Failed generating key type ed25519 to file /etc/ssh/ssh_host_ed25519_key in FIPS enforcing mode

Bug #2017761 reported by Anirban Sinha on 2023-04-26

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	cloud-init	Fix Committed	Undecided	Anirban Sinha

Bug Description

With fips enabled, cloud-init has warning log about "Failed generating key type ed25519 to file /etc/ssh/ssh_host_ed25519_key".
According to doc(https://access.redhat.com/solutions/3643252), ed25519 key is not supported under fips mode, so I am suggesting cloudinit do not try to generate such key type under fips mode.

2023-04-17 03:46:38,665 - util.py[DEBUG]: Restoring selinux mode for /etc/ssh (recursive=True)
2023-04-17 03:46:38,672 - subp.py[DEBUG]: Running command ['ssh-keygen', '-t', 'ed25519', '-N', '', '-f', '/etc/ssh/ssh_host_ed25519_key'] with allowed return codes [0] (shell=False, capture=True)
2023-04-17 03:46:38,721 - util.py[WARNING]: Failed generating key type ed25519 to file /etc/ssh/ssh_host_ed25519_key
2023-04-17 03:46:38,722 - util.py[DEBUG]: Failed generating key type ed25519 to file /etc/ssh/ssh_host_ed25519_key
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/cloudinit/config/cc_ssh.py", line 256, in handle
    out, err = subp.subp(cmd, capture=True, env=lang_c)
  File "/usr/lib/python3.9/site-packages/cloudinit/subp.py", line 332, in subp
    raise ProcessExecutionError(
cloudinit.subp.ProcessExecutionError: Unexpected error while running command.
Command: ['ssh-keygen', '-t', 'ed25519', '-N', '', '-f', '/etc/ssh/ssh_host_ed25519_key']
Exit code: 255
Reason: -
Stdout:
Stderr: ED25519 keys are not allowed in FIPS mode
2023-04-17 03:46:38,723 - util.py[DEBUG]: Restoring selinux mode for /etc/ssh (recursive=True)
2023-04-17 03:46:38,727 - util.py[DEBUG]: Reading from /etc/ssh/ssh_host_rsa_key.pub (quiet=False)

This issue is seen on RHEL 9.1 but can be reproduced on other versions of RHEL. The warning is introduced in ssh-keygen with the following Fedora 26 patch : https://src.fedoraproject.org/rpms/openssh/blob/f26/f/openssh-7.2p1-fips.patch and the following commit:

commit 9dbec70c9c30350a9268be62be4df3c55a93f23e
Author: Jakub Jelen <email address hidden>
Date: Fri Jun 30 12:18:02 2017 +0200

Sync FIPS patch with RHEL

so it has been for a while but it's a valid warning.

Steps to Reproduce:
Manual:
1. Boot into an RHEL-9.1 system with fips enabled
2. Try to clean and init cloud-init again
$ sudo cloud-init clean
$ sudo cloud-init init

cloud-init needs to check fips mode and not generate those keys that are not valid when fips is enabled.

Anirban Sinha (anisinha) on 2023-04-26

Changed in cloud-init:
assignee:	nobody → Anirban Sinha (anisinha)

Revision history for this message

Chad Smith (chad.smith) wrote on 2023-04-28:

Thank you for submitting this bug and the related pull request to cloud-init and improving cloud-init for other downstreams.

Upstream Pull Request in progress https://github.com/canonical/cloud-init/pull/2142.

Thanks for representing this warning from cloud-init.log for ed25519 keys, do you also have a capture of cloud-init.log representing the WARNING for DSA key generation as well?

Given that the related PR is trying to avoid both DSA and ED25519 keys, I just wanted to confirm Redhat is seeing failures for ssh-keygen of both ed25519 and DSA keys.

running the following should give us that confirmation:
egrep 'WARN|ssh-keygen|Stderr:' /var/log/cloud-init.log

One inconsistency I'm seeing on Ubuntu FIPS systems is that ssh-keygen of ED25519 type seems to be allowed. So, it may not be as simple as avoiding ssh-keygen of both DSA and ED25519 key types.

Changed in cloud-init:
status:	New → Triaged

Revision history for this message

Chad Smith (chad.smith) wrote on 2023-04-28:

This is a bug, I'm inclined to marked Triaged. If you can provide additional logs to assert both key types are failing on redhat FIPS it will better inform our accepted upstream fix.
Many thanks,
chad

Revision history for this message

Anirban Sinha (anisinha) wrote on 2023-04-28:

> Given that the related PR is trying to avoid both DSA and ED25519 keys, I just wanted to confirm Redhat is seeing failures for ssh-keygen of both ed25519 and DSA keys.

> running the following should give us that confirmation:
egrep 'WARN|ssh-keygen|Stderr:' /var/log/cloud-init.log

We have this documented here:
https://access.redhat.com/solutions/3643252

On a FIPS-enforcing system, the following key types are only available:
ecdsa | rsa
The following key types are not available on a FIPS-enforcing system:
dsa | ed25519

But also here goes:

[root@ani-rhel9vm cloud]# grep -R ssh_genkeytypes *
cloud.cfg:ssh_genkeytypes: ['rsa', 'dsa', 'ecdsa', 'ed25519']
[root@ani-rhel9vm cloud]# pwd
/etc/cloud
# egrep 'WARN|ssh-keygen|Stderr:' /var/log/cloud-init.log
2023-04-28 04:43:15,393 - subp.py[DEBUG]: Running command ['ssh-keygen', '-t', 'rsa', '-N', '', '-f', '/etc/ssh/ssh_host_rsa_key'] with allowed return codes [0] (shell=False, capture=True)
2023-04-28 04:43:16,651 - subp.py[DEBUG]: Running command ['ssh-keygen', '-t', 'dsa', '-N', '', '-f', '/etc/ssh/ssh_host_dsa_key'] with allowed return codes [0] (shell=False, capture=True)
2023-04-28 04:43:16,682 - util.py[WARNING]: Failed generating key type dsa to file /etc/ssh/ssh_host_dsa_key
Command: ['ssh-keygen', '-t', 'dsa', '-N', '', '-f', '/etc/ssh/ssh_host_dsa_key']
Stderr: DSA keys are not allowed in FIPS mode
2023-04-28 04:43:16,685 - subp.py[DEBUG]: Running command ['ssh-keygen', '-t', 'ecdsa', '-N', '', '-f', '/etc/ssh/ssh_host_ecdsa_key'] with allowed return codes [0] (shell=False, capture=True)
2023-04-28 04:43:16,718 - subp.py[DEBUG]: Running command ['ssh-keygen', '-t', 'ed25519', '-N', '', '-f', '/etc/ssh/ssh_host_ed25519_key'] with allowed return codes [0] (shell=False, capture=True)
2023-04-28 04:43:16,748 - util.py[WARNING]: Failed generating key type ed25519 to file /etc/ssh/ssh_host_ed25519_key
Command: ['ssh-keygen', '-t', 'ed25519', '-N', '', '-f', '/etc/ssh/ssh_host_ed25519_key']
Stderr: ED25519 keys are not allowed in FIPS mode

> One inconsistency I'm seeing on Ubuntu FIPS systems is that ssh-keygen of ED25519 type seems to be allowed. So, it may not be as simple as avoiding ssh-keygen of both DSA and ED25519 key types.

As I have mentioned in the bug report, the error comes from a patch applied in Fedora. So all recent red hat distros are affected. Ubuntu ssh-keygen may not be affected by the warning/error but I believe the core issue is that those keys are not allowed in fips mode in general. Whether cannonical wants to add a similar warning to ssh-keygen is a matter I leave on to the cannonical devs.

> Given that the related PR is trying to avoid both DSA and ED25519 keys, I just wanted to confirm Redhat is seeing failures for ssh-keygen of both ed25519 and DSA keys.

> running the following should give us that confirmation:
  egrep 'WARN|ssh-keygen|Stderr:' /var/log/cloud-init.log

We have this documented here:
https://access.redhat.com/solutions/3643252

On a FIPS-enforcing system, the following key types are only available:
ecdsa | rsa 
The following key types are not available on a FIPS-enforcing system:
dsa | ed25519

But also here goes:

[root@ani-rhel9vm cloud]# grep -R ssh_genkeytypes *
cloud.cfg:ssh_genkeytypes:  ['rsa', 'dsa', 'ecdsa', 'ed25519']
[root@ani-rhel9vm cloud]# pwd
/etc/cloud
# egrep 'WARN|ssh-keygen|Stderr:' /var/log/cloud-init.log
2023-04-28 04:43:15,393 - subp.py[DEBUG]: Running command ['ssh-keygen', '-t', 'rsa', '-N', '', '-f', '/etc/ssh/ssh_host_rsa_key'] with allowed return codes [0] (shell=False, capture=True)
2023-04-28 04:43:16,651 - subp.py[DEBUG]: Running command ['ssh-keygen', '-t', 'dsa', '-N', '', '-f', '/etc/ssh/ssh_host_dsa_key'] with allowed return codes [0] (shell=False, capture=True)
2023-04-28 04:43:16,682 - util.py[WARNING]: Failed generating key type dsa to file /etc/ssh/ssh_host_dsa_key
Command: ['ssh-keygen', '-t', 'dsa', '-N', '', '-f', '/etc/ssh/ssh_host_dsa_key']
Stderr: DSA keys are not allowed in FIPS mode
2023-04-28 04:43:16,685 - subp.py[DEBUG]: Running command ['ssh-keygen', '-t', 'ecdsa', '-N', '', '-f', '/etc/ssh/ssh_host_ecdsa_key'] with allowed return codes [0] (shell=False, capture=True)
2023-04-28 04:43:16,718 - subp.py[DEBUG]: Running command ['ssh-keygen', '-t', 'ed25519', '-N', '', '-f', '/etc/ssh/ssh_host_ed25519_key'] with allowed return codes [0] (shell=False, capture=True)
2023-04-28 04:43:16,748 - util.py[WARNING]: Failed generating key type ed25519 to file /etc/ssh/ssh_host_ed25519_key
Command: ['ssh-keygen', '-t', 'ed25519', '-N', '', '-f', '/etc/ssh/ssh_host_ed25519_key']
Stderr: ED25519 keys are not allowed in FIPS mode

> One inconsistency I'm seeing on Ubuntu FIPS systems is that ssh-keygen of ED25519 type seems to be allowed. So, it may not be as simple as avoiding ssh-keygen of both DSA and ED25519 key types.

Revision history for this message

Anirban Sinha (anisinha) wrote on 2023-05-02:

Fixed with commit https://github.com/canonical/cloud-init/commit/c53f04aeb2acf9526a2ebf3d3320f149ac46caa6