openssl fails with out of memory messages while trying to load the FIPS provider in a non-FIPS container on a FIPS host

Status changed to 'Confirmed' because the bug affects multiple users.

Hi,

I hit the same issue, would like to know if there is an ETA?

Let me narrow down the issue: the problem was resolved when I disabled FIPS on my host. Whatever the underlying cause, it only occurs when FIPS is enabled.

We are seeing this as well when building an image with ubuntu:24.04 as our base. Like the commenter above, we see the issue on our FIPS system, but do not see it on our non-FIPS system. A workaround or any information is certainly appreciated.

Here's a relatively minimal example:

```
$ sudo fips-mode-setup --check
FIPS mode is enabled.
$ uname -a
Linux hostname 4.18.0-553.8.1.el8_10.x86_64 #1 SMP Fri Jun 14 03:19:37 EDT 2024 x86_64 x86_64 x86_64 GNU/Linux
```

```
FROM ubuntu:24.04 AS builder

RUN . /etc/os-release && \
    DEBIAN_FRONTEND=noninteractive apt-get update && apt-get install -y --no-install-recommends \
                curl apt-transport-https ca-certificates xz-utils \
                cmake build-essential pkg-config \
                libboost-all-dev libusb-1.0-0-dev \
                python3-mako python3-numpy python3-setuptools python3-requests
```

Ran with

```
docker build .
```

produces attached output

This issue arises because the container is running on top of a FIPS-enabled kernel which provides the /proc/sys/crypto/fips_enabled = 1 flag.

On Noble, openssl sees this flag and expects therefore to be operating in FIPS mode, but at this point the Noble container is not in FIPS mode - we haven't finished developing the FIPS components and haven't released them - and the result is that it throws some errors.

The workaround is to either not run the container on a FIPS-enabled kernel; or to set the environment variable in the Noble container to disable FIPS mode:

ENV OPENSSL_FORCE_FIPS_MODE=0

Steps to reproduce on any machine on Noble or later:

    cp -r /usr/lib/ssl/certs/ .
    OPENSSL_FORCE_FIPS_MODE=1 openssl rehash certs

NB: we can only SRU that to Noble if it first goes in a development version of Ubuntu which will only be possible when development for 25.04 begins.

My plan is to make the library write the following error message when failing to load a provider (it will apply to any provider):

    While loading "fips" provider: error:07880025:common libcrypto routines::reason(37)

There was already an error but it was silent until it reached the application which could then assume something inappropriate as we've seen here.

There are a couple constraints.

First, I'm not loading error strings because I'm changing the inside of the library and if they haven't been loaded already, the error message will say "37" rather than "ERR_R_DSO_LIB"/"ERR_LIB_DSO". I plan to work around that by expanding the title of this bug report so that it's easy to find with a search engine. (this might also be because I'm relying on an internal static buffer for formatting because I prefer to avoid adding memory management in an error path)

The other constraint is that I need to "fprintf(stderr, ..." which is a bit uncommon for openssl code but I don't want to pop an error from the error stack like `ERR_print_errors{,_fp}()` does since I'm in the library code and not in the application code. This means the only API usable is `ERR_peek_last_error()`. Not a big issue but that makes for some slightly alien-looking code.

All in all, the output for the `openssl rehash certs` from above will contain the following text, once per cert found (i.e. 147 times here):

    While loading "fips" provider: error:07880025:common libcrypto routines::reason(37)
    out of memory

And I hope that one can copy-paste the new error message in a search engine and end directly on this page.

Any thought on this before I integrate this? Rough timeline is that I push that to plucky very soon (tomorrow maybe) and then SRU it in Noble right after (but SRUs take time and this will probably be a staged change until a new openssl security update is released).

For reference, during a build, the testsuite now says the following:

    # Subtest: ../../test/fips_auto_enable_test
    1..1
    While loading "fips" provider: error:07880025:common libcrypto routines::reason(37)
    While loading "fips" provider: error:07880025:common libcrypto routines::reason(37)
    While loading "fips" provider: error:07880025:common libcrypto routines::reason(37)

This makes sense since the testsuite tries to enable the provider and it isn't available. These messages don't appear elsewhere in the build/test log. Therefore things are looking good.

Looks good to me either. This error message is much more reasonable than seeing just `out of memory`. Thanks.

This bug was fixed in the package openssl - 3.4.0-1ubuntu2

---------------
openssl (3.4.0-1ubuntu2) plucky; urgency=medium

  * d/p/fips/crypto-Automatically-use-the-FIPS-provider-when-the-kerne.patch:
    Extend the patch to print the error encounted when a fallback
    provider fails loading, e.g. due to FIPS auto-loading (LP: #2066990)
  * d/p/Revert-When-defining-ossl_ssize_t-ssize_t-remember-t.patch:
    Work-around SWIG using different feature flag defines than GCC and
    parsing sys/select.h differently. (LP: #2091883)

 -- Adrien Nader <email address hidden> Thu, 19 Dec 2024 16:12:42 +0100

What is the plan for the backport to Ubuntu Noble (24.04)?

FIPS users are much more likely to require the use of LTS releases, so we're more likely to experience this issue.

There isn't a plan completely laid out for backporting this. It can be done but as you can see here, it's a very minimal change and it barely passes the criteria for backporting. This means it absolutely must be staged so that it's picked up by a subsequent security update rather than being a standalone update (openssl is installed everywhere so changing the package affects everybody, it's high-impact).

With that said, it's also a change that sits in a weird spot: once you've understood the issue, then you probably don't need the warning anymore and search engines should help find it now. Can you explain your need a bit so I can better gauge the need? Did you encounter the issue or are you rather trying to make sure you don't?

BTW, with plucky being released soon, you'll start seeing the effect of the change if the container you run is on plucky.