add ability to provide an untrusted cert to snapped MAAS

As a suggestion: RBAC snap could be configured in the following way:

sudo snap set canonical-rbac ssl.ca="$(cat /path/to/self/signed/CA.crt cat /path/to/self/signed/intermediate_CA.crt)"

Could we re-use the same approach to inject custom CA certs into the MAAS snap?

MAAS should manage certificates so it can ensure all region and rack controllers have the same set of certificates. LP:1897351 to this as well. I wrote a spec to implemented MAAS managed certificates including optionally using MAAS as a CA[1].

MAAS uses python3-certifi for certificate authentication. python3-certifi only allows you to use the system certificate store for CA certificates. As that is stored in the base Snap we can't easily add to it. As there are other Snaps which will have the same problem I'm adding snapd to add this ability.

[1] https://docs.google.com/document/d/1Dt1U_lSTRg0h1_8RFfNgxcDUF71W_elSWkYMEx83tUc/edit#heading=h.oncww1139tq8

We're aware of the limitation in snaps to provide an additional certificate to a specific snap, and it's generically on our roadmap for some point in time, to be sorted out with other priorities. As such I'm setting the snapd task on this to Low.

subscribed field-high

We landed support for the snaps to see the certs from the hosts /etc/ssl - is this not working? Or is this not what you need?

Just to clarify - with https://github.com/snapcore/snapd/pull/9819 that landed a while ago the mechanism outlined for debs:
"""

In a deb packaged MAAS scenario, a user simply adds the cert to /usr/local/share/ca-certifictes and runs 'dpkg-reconfigure ca-certificates' and the cert will be available system wide. Snapped MAAS cannot access these system-wide certificates.
"""
should just work. I.e. the snap will see the /etc/ssl of the host machine.

This is not working. We have the cert installed on the system.

On Tue, Jul 13, 2021, 12:25 PM Michael Vogt <email address hidden>
wrote:

> Just to clarify - with https://github.com/snapcore/snapd/pull/9819 that
> landed a while ago the mechanism outlined for debs:
> """
>
> In a deb packaged MAAS scenario, a user simply adds the cert to
> /usr/local/share/ca-certifictes and runs 'dpkg-reconfigure ca-certificates'
> and the cert will be available system wide. Snapped MAAS cannot access
> these system-wide certificates.
> """
> should just work. I.e. the snap will see the /etc/ssl of the host machine.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1904605
>
> Title:
> add ability to provide an untrusted cert to snapped MAAS
>
> Status in MAAS:
> Triaged
> Status in snapd:
> Triaged
>
> Bug description:
> MAAS 2.8/stable 2.8.2-8577-g.a3e674063
>
> Currently, there is no way to natively add an untrusted CA to MAAS
> without unpacking/repacking the entire snap and manually adding the
> certificate.
>
> The use case for this is, if a user is attempting to talk to a UCS
> Manager API that is using either self-signed or privately signed
> certificate and has SSL redirect enabled, then MAAS will error
> attempting to talk to this API URL.
>
> In a deb packaged MAAS scenario, a user simply adds the cert to
> /usr/local/share/ca-certifictes and runs 'dpkg-reconfigure ca-
> certificates' and the cert will be available system wide. Snapped
> MAAS cannot access these system-wide certificates.
>
> There should be an ability to "drop in" a certificate somewhere in the
> confined space (/var/snap/maas/) so that there is no need to
> unpack/repack the snap.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1904605/+subscriptions
>
>

Thanks for checking this. Can you please check if you can see your custom ssl cert under /etc/ssl ?

Can you please run the following debug commands and provide the output please?

$ snap version
$ cat /etc/os-release
$ snap list maas
$ snap run --shell maas
[inside this subshell]
$ ls -al /etc/ssl/
$ find /etc/ssl/ -name "*<your-certs-name>*"
$ cat /etc/os-release
$ exit

I tested this by manually adding a test file into /etc/ssl and then using "snap run --shell maas" and I can see my hosts /etc/ssl tree then. My host systems is Ubuntu 20.04.

Any hints how to reproduce would be appreciated.

Here ya go

root@focal:~# snap version
snap 2.51.1
snapd 2.51.1
series 16
ubuntu 20.04
kernel 5.4.0-77-generic
root@focal:~# cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.1 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.1 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

root@focal:~# snap list maas
Name Version Rev Tracking Publisher Notes
maas 2.8.6-8602-g.07cdffcaa 13516 2.8/stable canonical✓ -

root@focal:~# snap run --shell maas

root@focal:/root# ls -al /etc/ssl/
total 39
drwxr-xr-x 4 root root 5 May 3 17:10 .
drwxr-xr-x 97 root root 3840 Jul 13 21:29 ..
drwxr-xr-x 2 root root 262 Jul 13 21:28 certs
-rw-r--r-- 1 root root 10909 Apr 20 2020 openssl.cnf
drwx------ 2 root root 2 Apr 20 2020 private

root@focal:/root# find /etc/ssl -name adp.pem
find: ‘/etc/ssl/private’: Permission denied
/etc/ssl/certs/adp.pem

root@focal:/root# cat /etc/os-release
NAME="Ubuntu Core"
VERSION="18"
ID=ubuntu-core
PRETTY_NAME="Ubuntu Core 18"
VERSION_ID="18"
HOME_URL="https://snapcraft.io/"
BUG_REPORT_URL="http://bugs.launchpad.net/snappy/"

Thanks, the output indicates that the maas snap can see the " /etc/ssl/certs/adp.pem" certificate - I assume you can also just "cat /etc/ssl/certs/adp.pem" it and there is also no permission denied issue? At this point I would love to pull in the MAAS team to hear if there is a way to enable debugging of the certification store reading on their side. AFIACT the cert is availalbe in the right place (this is the feature we added back in January). In parallel we will investigate this some more too.

Python by default reads certificates from a bundle located at /etc/ssl/certs/ca-certificates.crt.

This is generated from certificates in /etc/ssl/certs via `update-ca-certificates`.

Could you please make sure `update-ca-certificates --fresh` is run after adding certs to the dir (and restart maas after that)?

I am having a similar problem in a customer environment.

We've updated the certs with --fresh and restarted snapd and maas. Checking for a random line in our cert, I see it in 3 files in /etc/ssl/certs but when I go into the snap shell, I don't see that line anywhere.

How can /etc/ssl/certs have different content when inside the snap shell?

ubuntu@infra01:~$ sudo update-ca-certificates --fresh
Clearing symlinks in /etc/ssl/certs...
done.
Updating certificates in /etc/ssl/certs...
130 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
ubuntu@infra01:~$ sudo systemctl restart snapd
ubuntu@infra01:~$ sudo snap restart maas
Restarted.
ubuntu@infra01:~$ grep $(head -5 /usr/local/share/ca-certificates/mirror.crt | tail -1) /etc/ssl/certs/*
/etc/ssl/certs/ca-certificates.crt:cnJvci5hdGhlbmFndy5jb20wHhcNMjEwODA2MTYwMzE3WhcNMzEwODA0MTYwMzE3
/etc/ssl/certs/f97749aa.0:cnJvci5hdGhlbmFndy5jb20wHhcNMjEwODA2MTYwMzE3WhcNMzEwODA0MTYwMzE3
/etc/ssl/certs/mirror.pem:cnJvci5hdGhlbmFndy5jb20wHhcNMjEwODA2MTYwMzE3WhcNMzEwODA0MTYwMzE3
ubuntu@infra01:~$ ls -l /etc/ssl/certs/ca-certificates.crt
-rw-r--r-- 1 root root 202391 Aug 13 14:56 /etc/ssl/certs/ca-certificates.crt
ubuntu@infra01:~$ snap run --shell maas
bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

ubuntu@infra01:/home/ubuntu$ grep cnJvci5hdGhlbmFndy5jb20wHhcNMjEwODA2MTYwMzE3WhcNMzEwODA0MTYwMzE3 /etc/ssl/certs/*
ubuntu@infra01:/home/ubuntu$ ls -l /etc/ssl/certs/ca-certificates.crt
-rw-r--r-- 1 root root 200313 Jul 2 07:25 /etc/ssl/certs/ca-certificates.crt

@vern what is `snap version` on this machine?

ubuntu@infra01:~$ snap version
snap 2.49.2+20.04
snapd 2.49.2+20.04
series 16
ubuntu 20.04
kernel 5.4.0-80-generic
ubuntu@infra01:~$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu-Server 20.04.2 LTS STIG customized using Cubic on 2021-08-05 13:34"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
ubuntu@infra01:~$ snap list maas
Name Version Rev Tracking Publisher Notes
maas 2.9.2-9165-g.c3e7848d1 12555 2.9/stable canonical✓ -
ubuntu@infra01:~$ snap run --shell maas
bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

ubuntu@infra01:/home/ubuntu$ ls -al /etc/ssl
total 23
drwxr-xr-x 4 root root 62 Jul 2 07:26 .
drwxr-xr-x 129 root root 12288 Aug 12 17:32 ..
drwxr-xr-x 2 root root 7443 Jul 2 07:26 certs
-rw-r--r-- 1 root root 10909 Apr 28 00:37 openssl.cnf
drwx------ 2 root root 3 Apr 28 00:37 private
ubuntu@infra01:/home/ubuntu$ find /etc/ssl/ -name mirror.crt
find: '/etc/ssl/private': Permission denied
ubuntu@infra01:/home/ubuntu$ sudo find /etc/ssl/ -name mirror.crt
bash: /usr/bin/sudo: Permission denied
ubuntu@infra01:/home/ubuntu$ cat /etc/os-release
NAME="Ubuntu Core"
VERSION="20"
ID=ubuntu-core
PRETTY_NAME="Ubuntu Core 20"
VERSION_ID="20"
HOME_URL="https://snapcraft.io/"
BUG_REPORT_URL="https://bugs.launchpad.net/snappy/"
ubuntu@infra01:/home/ubuntu$

@vern can you do `snap install snapd` and then re-run your test? I don't think the certs stuff was in 2.49.2, I think it was in 2.50

This is an offline deployment. We pushed the latest snapd snap to our snap-store-proxy and did a snap refresh.

Now:

ubuntu@infra01:~$ snap version
snap 2.51.3
snapd 2.51.3
series 16
ubuntu 20.04
kernel 5.4.0-80-generic
ubuntu@infra01:~$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu-Server 20.04.2 LTS STIG customized using Cubic on 2021-08-05 13:34"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
ubuntu@infra01:~$ snap list maas
Name Version Rev Tracking Publisher Notes
maas 2.9.2-9165-g.c3e7848d1 12555 2.9/stable canonical✓ -
ubuntu@infra01:~$ snap run --shell maas
bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

ubuntu@infra01:/home/ubuntu$ ls -al /etc/ssl
total 23
drwxr-xr-x 4 root root 62 Jul 2 07:26 .
drwxr-xr-x 129 root root 12288 Aug 12 17:32 ..
drwxr-xr-x 2 root root 7443 Jul 2 07:26 certs
-rw-r--r-- 1 root root 10909 Apr 28 00:37 openssl.cnf
drwx------ 2 root root 3 Apr 28 00:37 private
ubuntu@infra01:/home/ubuntu$ find /etc/ssl/ -name mirror\*
find: '/etc/ssl/private': Permission denied
ubuntu@infra01:/home/ubuntu$ cat /etc/os-release
NAME="Ubuntu Core"
VERSION="20"
ID=ubuntu-core
PRETTY_NAME="Ubuntu Core 20"
VERSION_ID="20"
HOME_URL="https://snapcraft.io/"
BUG_REPORT_URL="https://bugs.launchpad.net/snappy/"

We ran update-ca-certificates --fresh, restarted snapd and maas, and we still have an old ca-certificates.crt that hasn't changed since July 2.

ubuntu@infra01:/home/ubuntu$ ls -l /etc/ssl/certs/ca-certificates.crt
-rw-r--r-- 1 root root 200313 Jul 2 07:25 /etc/ssl/certs/ca-certificates.crt

After doing the snap refresh, the cert wasn't updating. After rebooting, things were good.

It was suggested that maybe the namespace needed to be remounted so this might have been sufficient:

  sudo /usr/lib/snapd/snap-discard-ns maas

I'm marking this as Invalid for MAAS, since it seems it's working as expected. Except that maybe the snapd upgrade doesn't quite work, but that's nothing MAAS can solve.

I'm marking as Fix Released for snapd, as it sounds like the feature we added to 2.50 for this is working, but perhaps we have the (existing and known) issue of some snap mount namespaces not being updated here coinciding with the lack of this feature.