container management snaps should have Delegate=true in their systemd unit

Bug #1910456 reported by Ian Johnson
290
This bug affects 2 people
Affects Status Importance Assigned to Milestone
snapd
Fix Released
Critical
Ian Johnson

Bug Description

As reported to the ubuntu security team, and discussed in https://github.com/docker-snap/docker-snap/security/advisories/GHSA-798c-v3jq-h646, the docker snap's systemd unit for the dockerd process should have Delegate=true in it to prevent systemd from reaping the control groups for child processes (which in this case are docker containers that are put into control groups which are separate from the the docker snap) and moving the child processes back into the same control group as the parent process that systemd is managing, which is dockerd.

More concretely, not including Delegate=true in the systemd service file means that after a systemctl daemon-reload is executed, docker containers now have the same privilege level as dockerd, which is a privilege escalation.

This was assigned CVE-2020-27352 by Alex Murray.

I have opened this private snapd bug, because this vulnerability affects more than just the docker snap, it also affects any other container management snaps, such as microk8s, Azure IoT Edge, AWS IoT Greengrass and potentially others. The list of snaps affected can be narrowed down to any snap which has been granted privilege to use a "container management" interfaces, which are all super-privileged and require a snap store assertion in order to be uploaded to the Snap Store. The current list of affected interfaces is:

- docker-support (all variants)
- greengrass-support (with the "legacy-containers" flavor or with empty flavor)
- kubernetes-support (with the "kubelet" flavor or with empty flavor)
- lxd-support (though the LXD snap is not affected by this issue, we should fix the interface as if it was for consistencies sake) [1]

[1] This vulnerability specifically does not affect LXD though (or at least it should not), because LXD is already using Delegate=true in it's own generated systemd setup, as shown in this segment of the snap packaging for LXD: https://github.com/lxc/lxd-pkg-snap/blob/latest-edge/snapcraft/commands/daemon.start#L308. See also https://github.com/lxc/lxc/issues/323 which is when the LXD project first encountered the issue with Delegate and systemd.

Changed in snapd:
status: New → Confirmed
importance: Undecided → Critical
Revision history for this message
Ian Johnson (anonymouse67) wrote :

We on the snapd team have a plan for fixing this, by essentially making it such that any snap that plugs these interfaces, will have Delegate=true put into the generated systemd unit.

Specifically, we would like to always include Delegate=true if the snap merely declares a plug for one of these interfaces, regardless of whether the plug is connected to a slot or not. This makes the code implementation much simpler for us, because it means we don't need to modify the systemd service unit for the snap service every time the plug is connected or disconnected, which would be a lot of work codewise for us.

This has the important implication though that if a snap declares the plug for something like docker-support, but does not have docker-support connected to the slot, then it will not have the highly privileged policy that enables something like docker to create containers and confine those containers, _but_ it will still have the setting of Delegate=true which means that systemd will not clean up the control groups for child processes if the child processes were to somehow get into a different control group.

We do not think this is an issue because:
* in order to have Delegate=true in the policy, you still need to declare the plug, and today all snaps that use these interfaces are not allowed to be uploaded to the store unless they are approved/allowed to by store reviewers (this is because the interfaces are super-privileged)
* even if the snap has Delegate=true in it's systemd unit, that alone does not provide the necessary privilege in order to create containers or otherwise create new control groups and place processes into those control groups

However, we would like input on this from the security team about whether this makes sense and is not introducing a vulnerability in this way.

Revision history for this message
Ian Johnson (anonymouse67) wrote :

The other outstanding question we have is whether the "flavor" variants as mentioned in the original description are sufficient or if there are other flavors which need to have Delegate=true generated for services declaring a plug.

Firstly, the greengrass-support interface currently has two flavors, the empty or "legacy-container" flavor, and the "no-container" flavor. The "no-container" flavor is meant as a super minimal interface which does not have the same set of privileges as the other one, really only the privileges necessary for AWS IoT Greengrass's greengrassd to start processes using runC. The question here though is whether the policy that the "no-container" flavor has is actually enough to allow moving into new control groups. I don't think it is, but here is the link to that policy: https://github.com/snapcore/snapd/blob/cc12dae3934eab08fbca39a9b661d8b1da0ba40d/interfaces/builtin/greengrass_support.go#L56-L65. I'm specifically not sure if the "rix" rule for runc executable allows runC to create containers with different control groups, because if it is, then the "no-container" flavor also needs to have the Delegate=true directive added to the generated systemd service unit.

Secondly, the kubernetes-support interface has 4 flavors, which are as follows:

1. the empty flavor, which is used AIUI for a kubernetes daemon which needs all of the flavor policies, i.e. kubelet, kubeproxy, and autobind-unix.

2. the kubelet flavor, which is used only for kubelet, and has this comment in the policy:

# allow managing pods' cgroups
/sys/fs/cgroup/*/kubepods/{,**} rw,

(from https://github.com/snapcore/snapd/blob/cc12dae3934eab08fbca39a9b661d8b1da0ba40d/interfaces/builtin/kubernetes_support.go#L125-L126) which means it needs Delegate=true.

3. the kubeproxy flavor, which from my understanding also manages control groups with similar policy as kubelet in it, and thus should have Delegate=true.

4. the autobind-unix flavor, which does not have any policy associated with managing control groups in it, and thus should _not_ have Delegate=true directive in the generated systemd service unit.

As such, it is my understanding that any kubernetes-support plug declaration, except one with the flavor: autobind-unix attribute, should get Delegate-true generated.

Thirdly, The lxd-support interface is entirely open and as such allows the LXD snap to escape confinement and is not tracked by systemd at all, so adding Delegate=true for the lxd-support interface is largely just educational and not necessary for securing LXD.

Fourthly, all variants of the docker-support interface (with either the privileged: true attribute or the privileged: false or no privileged attribute) are meant for dockerd to manage containers and thus all should get Delegate=true.

I think this is all of the interfaces that we have in snapd for managing contianers, but if someone could double-check this that would be good.

Revision history for this message
Ian Johnson (anonymouse67) wrote :

I have added the original reporters of the vulnerability to this bug report.

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

First off, I want to be clear that with all of the listed interfaces (with the exception of some of the flavors), the policy is considered 'advisory' and there are enough privileges to escape confinement. As such, I'm not convinced this should be granted a CVE because at some point, someone will point out how to break out, a CVE will be assigned and there will be no fix because that is precisely what the interface intends to give. As such I'd be more inclined to call this a normal bug in snapd. Alex, if you agree, we can burn the CVE.

The idea of adding Delegate=true when the affected interfaces are connected makes sense to me.

In terms of flavors, I would be in favor of only adding Delegate=true to affected flavors. I agree with your assessment to include it for all but autobind-unix in kubernetes-support. It should include it for both flavors of docker-support. Adding it for lxd-support makes sense for completeness, especially when we know that it intends to manage containers.

For greengress-support no-container (the legacy-container obviously needs it), rix means that runc will inherit the policies of the invoking process. This on its own is not enough IME to require adding Delegate=true, assuming the invoking process is coming from a snap and not unconfined.

I'll mention that system-files could be used in a manner to require Delegate=true if using 'write' with affected directories.

There is a question of classic snaps and container managers though, like microk8s. Should we add something to per command app yaml for Delegate=true for services? (we could have this flag for manual review in the store). I don't think this should block the proposed fix for strict snaps though.

Revision history for this message
Ian Johnson (anonymouse67) wrote :

> First off, I want to be clear that with all of the listed interfaces (with the exception of some of the flavors), the policy is considered 'advisory' and there are enough privileges to escape confinement.

I think the issue here (and why this deserves a CVE) is not that the snap services can escape confinement due to this issue, but rather that the container management agents like dockerd are not able to effectively confine their own containers without Delegate=true. For example, I want to launch a docker container and there is some reasonable expectation on my part that dockerd is going to confine the container to not have the same privilege level as dockerd.

Regarding classic snaps, I think you are correct in that we should also do this for classic snaps like microk8s; I would need to look into it a bit, but theoretically those classic snaps would also suffer from the same problem since their processes are containers and they are also presumably put into different control groups than the container management agent.

Revision history for this message
Ian Johnson (anonymouse67) wrote :

> I'll mention that system-files could be used in a manner to require Delegate=true if using 'write' with affected directories.

Do you know of any such usages currently in the field? If so, we should think about how to handle those cases, because we can't really add Delegate=true for all system-files usages, and inspecting which system-files are vulnerable is probably rather difficult.

If there are no such usages in the field, perhaps we should just update the docs around granting system-files to not allow snaps to use system-files with those interfaces without first deciding about what snapd should do about using Delegate=true in that case. Perhaps the system-files interface could gain an attribute to reflect this or something like that.

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

I don't, no. I would've rejected them if they came across my desk. I suspect roadmr would've asked us about them as well. Would need a store query to be sure though.

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

> I think the issue here (and why this deserves a CVE) is not that the snap services can escape confinement due to this issue, but rather that the container management agents like dockerd are not able to effectively confine their own containers without Delegate=true.

That's a fair point. Based on your assessment, it surely isn't a bug in the docker snap or any of the others because they aren't in a position to fix the issue. It still seems like a bit of a gray area and feels more like a bug. I defer to Alex and Seth who are more up to date on the current guidelines.

Revision history for this message
Ian Johnson (anonymouse67) wrote :

@jdstrand

> The idea of adding Delegate=true when the affected interfaces are connected makes sense to me.

What do you think about our proposal to make adding Delegate=true a "permanent plug property" such that the interfaces do not need to be connected in order for Delegate=true to be included in the systemd unit? Doing it this way makes the code much simpler for us, and as detailed above I don't think introduces any vulnerabilities because 1) the interfaces doing this are already super-privileged and 2) the line alone doesn't do anything except tell systemd to not move control groups, but in order to take advantage of systemd not moving control groups, the confined process would need to have permissions to create/move control groups which isn't allowed except when the interfaces are connected.

Revision history for this message
Alex Murray (alexmurray) wrote :

Regarding the CVE - I agree this is a gray area, if we assume these interfaces are not expected to provide *any* meaningful confinement then there is no security boundary violated. However, whilst we recognise they are privileged, I think it is reasonable to assume they at least provide the isolation expected for containers which are managed by apps that plug these interfaces - as such I think this is a proper vulnerability and the CVE should stay assigned.

I also feel that making this a permanent plug property doesn't appear to introduce any possible security issue, however I wonder if perhaps it would be better to just make snapd allow any snap to declare for a given daemon that it wants Delegate=True and then leave it up to the snap publisher to do the right thing here - and to then make this require a store assertion to be able to use. This feels more like the correct thing to do here since perhaps some daemons for a given snap would want this and others may not - so this provides a more flexible approach.

Revision history for this message
Ian Johnson (anonymouse67) wrote :

> however I wonder if perhaps it would be better to just make snapd allow any snap to declare for a given daemon that it wants Delegate=True and then leave it up to the snap publisher to do the right thing here - and to then make this require a store assertion to be able to use.

Yes after we have fixed this bug with the automatic adding of Delegate=true for snaps that need it, we want to have a way for snaps to "opt-in" to this behavior in the future. That feature however doesn't need to be developed in private and can be a normal feature request for snapd.

I have gone ahead with preparing the PR to fix this bug, I think that given we are "ok" to go ahead with the permanent plug property (rather than strictly only "connected plug" property), the code is simple as we had hoped and we can probably get this reviewed next week and would be hypothetically ready for a release the week after, so sometime the week of Jan 18th. However, Canonical will be having an internal sprint with many folks who would be instrumental to this release (such as myself and Michael and Samuele from the snapd team) busy that entire week, so I would like to propose the week after, what do folks think of a CRD sometime the week of January 25th ?

Revision history for this message
Alex Murray (alexmurray) wrote :

CRD week of the 25th sounds good to me - I am on leave myself on the 25/26 so perhaps later in the week is better for me but emitorino should be around to help out from the security team if needed on those days.

Revision history for this message
Gilad Reti (giladreti) wrote :

Hi there,

Sorry for the unrelated message, but we would like to confirm with you that the issue we reported may indeed be triggered spontaneously, and does not require an intervention of an admin.

Thanks!

Revision history for this message
Ian Johnson (anonymouse67) wrote :

Alright the private PR I have prepared has been reviewed on our side and is now ready for a security review, @alexmurray, would you like to do the review of the code or is there someone else from the security team who will review the code? I will have to add whomever is reviewing to the private github repo.

Revision history for this message
Gilad Reti (giladreti) wrote :

We believe that there is another issue with this vulnerability that we would like to bring to your attention. Although @anonymouse67 referred to this issue as requiring
systemctl daemon-reload
we have seen this vulnerability triggered spontaneously, periodically, without any user intervention.

Any unit enabling, disabling, editing, or adding a dependency will cause a systemd reload. This means that if any service causes any other service unit to reload, systemd will do a complete reload and the relevant cgroups will be changed. Such a service may be for example the apt unattended-upgrade service. When upgrading some packages, unattended-upgrade may change systemd units and will thus cause it to do a system reload.
Thus, after some unattended-upgrades, systemd will reload and docker cgroups will be deleted which will make the host vulnerable, without any user interaction.

If you have a long running machine, you can use cat /var/log/unattended-upgrades/unattended-upgrades.log | grep "INFO Packages that will be upgraded" to view recent unattended upgrades, and then check journalctl --no-pager | grep "systemd\[1\]: Reloading" to find out if a reload happened immediately after one of the unattended upgrades.

For example, on my machine I have

$ cat /var/log/unattended-upgrades/unattended-upgrades.log
...
2020-12-10 04:55:33,391 INFO Starting unattended upgrades script
2020-12-10 04:55:33,392 INFO Allowed origins are: o=Ubuntu,a=groovy, o=Ubuntu,a=groovy-security, o=UbuntuESMApps,a=groovy-apps-security, o=UbuntuESM,a=groovy-infra-security
2020-12-10 04:55:33,392 INFO Initial blacklist:
2020-12-10 04:55:33,392 INFO Initial whitelist (not strict):
2020-12-10 06:49:05,474 INFO Starting unattended upgrades script
2020-12-10 06:49:05,475 INFO Allowed origins are: o=Ubuntu,a=groovy, o=Ubuntu,a=groovy-security, o=UbuntuESMApps,a=groovy-apps-security, o=UbuntuESM,a=groovy-infra-security
2020-12-10 06:49:05,476 INFO Initial blacklist:
2020-12-10 06:49:05,476 INFO Initial whitelist (not strict):
2020-12-10 06:49:09,571 INFO Packages that will be upgraded: apt apt-transport-https apt-utils curl libapt-pkg6.0 libcurl3-gnutls libcurl4 python-apt-common python3-apt
2020-12-10 06:49:09,571 INFO Writing dpkg log to /var/log/unattended-upgrades/unattended-upgrades-dpkg.log
2020-12-10 06:50:21,609 INFO All upgrades installed
...

and

$ journalctl --no-pager | grep "systemd\[1\]: Reloading"
...
Dec 10 06:49:17 ubuntu systemd[1]: Reloading.
Dec 10 06:49:18 ubuntu systemd[1]: Reloading.
Dec 10 06:49:19 ubuntu systemd[1]: Reloading.
...

Therefore, we believe that this vulnerability has the following CVSS v3.1 Vector - AV:L/AC:L/PR:N/UI:N/S:C/C:H/I:H/A:H, since this vulnerability may be spontaneously triggered and it does not require an admin to trigger it.
Let us know if you want us to supply a PoC.

Revision history for this message
Ian Johnson (anonymouse67) wrote :

Hi Gilad, I will let our security team respond to your request about the CVSS Vector, but note that our proposed solution will always add Delegate=true to the service, so even if it can be triggered via different avenues, the fix will operate the same.

Revision history for this message
Ian Johnson (anonymouse67) wrote :

Hello again Gilad and Nimrod, due to unforeseen circumstances, we will be unable to get a full security review of our proposed fix until next week, so we would like to request to push the CRD out an additional week to sometime during the week of Feb 1st, is a CRD of Feb 3rd okay?

Revision history for this message
Ian Johnson (anonymouse67) wrote :

For the record, I have subscribed some kubernetes folks from Canonical for awareness since this affects snaps using the kubernetes-support interface as well as docker-support.

Revision history for this message
Gilad Reti (giladreti) wrote :

Feb 3rd is okay by us.

Revision history for this message
Gilad Reti (giladreti) wrote :

Hi @anonymouse67, are there any news?

Revision history for this message
Ian Johnson (anonymouse67) wrote : Re: [Bug 1910456] Re: container management snaps should have Delegates=true in their systemd unit
Download full text (3.7 KiB)

Hi, we are waiting for our security team to review the code which should
happen early this week. If you would like, I can build you a version of the
snapd snap that you can install locally to test out the fix. After the fix
has been approved by our security team we will also make the fix available
via a hidden branch that you can install from as well, but we don't want to
publish it in a hidden branch until we have confirmation from our security
team.

On Mon, Jan 25, 2021, 03:10 Gilad Reti <email address hidden> wrote:

> Hi @anonymouse67, are there any news?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1910456
>
> Title:
> container management snaps should have Delegates=true in their systemd
> unit
>
> Status in snapd:
> Confirmed
>
> Bug description:
> As reported to the ubuntu security team, and discussed in
> https://github.com/docker-snap/docker-snap/security/advisories/GHSA-
> 798c-v3jq-h646, the docker snap's systemd unit for the dockerd process
> should have Delegate=true in it to prevent systemd from reaping the
> control groups for child processes (which in this case are docker
> containers that are put into control groups which are separate from
> the the docker snap) and moving the child processes back into the same
> control group as the parent process that systemd is managing, which is
> dockerd.
>
> More concretely, not including Delegate=true in the systemd service
> file means that after a systemctl daemon-reload is executed, docker
> containers now have the same privilege level as dockerd, which is a
> privilege escalation.
>
> This was assigned CVE-2020-27352 by Alex Murray.
>
> I have opened this private snapd bug, because this vulnerability
> affects more than just the docker snap, it also affects any other
> container management snaps, such as microk8s, Azure IoT Edge, AWS IoT
> Greengrass and potentially others. The list of snaps affected can be
> narrowed down to any snap which has been granted privilege to use a
> "container management" interfaces, which are all super-privileged and
> require a snap store assertion in order to be uploaded to the Snap
> Store. The current list of affected interfaces is:
>
> - docker-support (all variants)
> - greengrass-support (with the "legacy-containers" flavor or with empty
> flavor)
> - kubernetes-support (with the "kubelet" flavor or with empty flavor)
> - lxd-support (though the LXD snap is not affected by this issue, we
> should fix the interface as if it was for consistencies sake) [1]
>
> [1] This vulnerability specifically does not affect LXD though (or at
> least it should not), because LXD is already using Delegate=true in
> it's own generated systemd setup, as shown in this segment of the snap
> packaging for LXD: https://github.com/lxc/lxd-pkg-snap/blob/latest-
> edge/snapcraft/commands/daemon.start#L308. See also
> https://github.com/lxc/lxc/issues/323 which is when the LXD project
> first encountered the issue with Delegate and systemd.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad....

Read more...

Revision history for this message
Gilad Reti (giladreti) wrote : Re: container management snaps should have Delegates=true in their systemd unit

Hello all,
We intend to publish a blogpost regarding this bug after your CRD is out.
@alexmurray, can you please confirm our CVSS vector?

Thanks again

summary: - container management snaps should have Delegates=true in their systemd
+ container management snaps should have Delegate=true in their systemd
unit
Revision history for this message
Alex Murray (alexmurray) wrote :

@giladreti - so I have taken another look at the CVSS3.1 vector string you referenced above.

The threat in this case would appear to be from a malicious docker image or similar - would you agree? Given this, then I think the vector makes more sense as follows:

Attack Vector: Network
Attack Complexity: Low
Privileges Required: None
User Interaction: None
Scope: Changed
Confidentiality: High
Integrity: High
Availability: High

The Attack Vector is from the Network since a malicious image has to be downloaded from dockerhub or similar

The Attack Complexity is Low since a malicious image is relatively easy to create

No special Privileges are Required other than what is normally provided to a docker container

There is no User Interaction required since as you state at some point it is quite likely the systemd will reload services and trigger this without any user interaction.

The Scope is changed since this then allows access to devices from the host system that were not originally part of the container's device cgroup.

C/I/A as High is because from what I can tell the snap.docker.dockerd.service devices cgroup appears to allow access to all devices on the system:

$ sudo cat /sys/fs/cgroup/devices/system.slice/snap.docker.dockerd.service/devices.list
a *:* rwm

And so this would appear to allow the container then to have arbitrary read/write access to any device on the system, including the host filesystem etc.

This would then result in a CVSS 3.1 base score of 10.0

@anonymouse67 and others - does this seem to match with your understanding of the vulnerability as well?

Revision history for this message
Alex Murray (alexmurray) wrote :

@giladreti - actually I have thought about this some more and consulted with others and we feel that the most appropriate CVSS vector would be: CVSS:3.1/AV:L/AC:L/PR:L/UI:N/S:C/C:H/I:H/A:H

ie. compared to my previous post above I have changed the following:

Attack Vector: Local since if you have a docker container already executing that you have control of (as the attacker) then this is local access.

Privilege Required: Low - since this does not need to be a privileged docker container but it does need some privileges so that it is actually runnning etc

this gives a CVSS base score of 8.8

Revision history for this message
Gilad Reti (giladreti) wrote :

First, @alexmurray, thanks for your response.
Second, we believe that Privilege Required should be None. The running containerized process does not need any privileges on the host to take over it nor is authorized to perform any action on the vulnerable host. You can see the famous runC cve as a reference - https://nvd.nist.gov/vuln/detail/CVE-2019-5736 - although the exploit requires an attacker to sit inside a container it does not count as a required privilege since the vulnerable component is the host for which the attacker has no initial access.

Revision history for this message
Ian Johnson (anonymouse67) wrote :

Hi Gilad and Alex, I tend to agree with Gilad here in that this bug has the same amount required privilege as the other runC CVE.

Revision history for this message
Ian Johnson (anonymouse67) wrote :

Also, we need to request another extension of the CRD to one week later, to February 10th if no one objects.

Our reason for doing this is that we have actually found that our fix in snapd is not applied immediately to running containers and operators/users will need to restart their containers and snapd should not do this automatically. Instead, affected snaps should be refreshed after snapd has been refreshed, and this can be done by adding a small snippet to the snapcraft.yaml of the affected snaps, depending on whether the snap is strict or classic.

As such, we will need some time to communicate with all affected snap authors and coordinate them to release new versions of their snaps.

The specific change for strict snaps such as the docker snap that are affected is that a new revision should be uploaded to the store and released to the stable risk on all active tracks with no changes except to add this snippet to the snap.yaml (if just repacking the snap) or the snapcraft.yaml (if rebuilding the snap):

assumes: [snapd2.48.3]

This ensures that the snap will be automatically refreshed at the next refresh check after the device has been refreshed to snapd version 2.48.3 (which is the version of snapd which will contain the fix).

For affected classic snaps, the same assumes snippet should be added and additionally, all services which are managing containers, i.e. containerd, etc. should add to their plugs the kubernetes-support interface.

Revision history for this message
Alex Murray (alexmurray) wrote :

Re CVSS I can agree with this argument that Privileges Required: None - as such this leaves the CVSS vector as originally suggested by Gilad:

CVSS:3.1/AV:L/AC:L/PR:N/UI:N/S:C/C:H/I:H/A:H

Revision history for this message
Michael Vogt (mvo) wrote :

Here is my proposed changelog for the snapd source deb:

"""
  * SECURITY UPDATE: sandbox escape vulnerability for containers
    - many: add Delegate=true to generated systemd units for special
      interfaces
    - CVE-2020-27352
    - LP: #1910456
"""

Please double check, I will then attach the debdiffs for xenial,bionic,focal,groovy and prepare the private snapd/core snap builds.

Revision history for this message
Michael Vogt (mvo) wrote :

Ian asked me to also pull in: https://github.com/snapcore/snapd/pull/9764 to 2.48.3 to support the other docker CVE fix. With that the new changelog suggestion would be:

``
+snapd (2.48.3) UNRELEASED; urgency=medium
+
+ * SECURITY UPDATE: sandbox escape vulnerability for containers
+ - many: add Delegate=true to generated systemd units for special
+ interfaces
+ - CVE-2020-27352
+ - LP: #1910456
+ * interfaces/builtin/docker-support: allow /run/containerd/s/...
+ - This is a new path that docker 19.03.14 (with a new version of
+ containerd) uses to avoid containerd CVE issues around the unix
+ socket. See also CVE-2020-15257.
+
+ -- Michael Vogt <email address hidden> Mon, 01 Feb 2021 11:55:18 +0100
```

Please let me know if this looks good, if so I will prepare debdiffs/snaps.

Revision history for this message
Alex Murray (alexmurray) wrote :

@mvo - per https://wiki.ubuntu.com/SecurityTeam/UpdatePreparation this looks pretty good - we usually tend to put the LP bug in parentheses at the end of the SECURITY UPDATE line rather than as a separate sub-item but I don't think this is crucial - if you have already started preparing debdiffs etc then that is fine to leave as is. Thanks.

Revision history for this message
Michael Vogt (mvo) wrote :

This is the xenial debdiff for 2.48.2. Unfortunately 2.48 is the version in *-updates. The 2.48.2 version is in the ~snappy-dev/image PPA here https://launchpad.net/~snappy-dev/+archive/ubuntu/image/+sourcepub/11828035/+listing-archive-extra

I can either create a debdiff against 2.48 or upload the package to a private fileshare or a private PPA, whatever you guys prefer.

Changed in snapd:
status: Confirmed → In Progress
Revision history for this message
Michael Vogt (mvo) wrote :

I uploaded all packages signed and ready to push into the private drive https://drive.google.com/drive/folders/1_PkBI8MwWSBWErXCu66vGIAYcckCzulL?usp=sharing

Revision history for this message
Gilad Reti (giladreti) wrote :

@anonymouse67 another week is fine by us.

Also, will any of you like to read our blog post before we publish it? We intend to publish it when your CRD is out.

Revision history for this message
Alex Murray (alexmurray) wrote :

Gilad, I would definitely be interested in reading your blog post in advance if possible? Thanks.

Revision history for this message
Gilad Reti (giladreti) wrote :

I am attaching latest blog post version. Feel free to make any comments.

Revision history for this message
Alex Murray (alexmurray) wrote :
Download full text (3.6 KiB)

Gilad, thanks for sharing that - there are just a couple changes that would help clarify this:

Replace:

  This issue was a result of bad configurations in Canonical’s snap

With:

  This issue was a result of bad configurations generated by Canonical’s snapd

Also in regards to the following part:

  If you are using snap with any of these packages it is recommended that you upgrade your system immediately.

You may not know, but by default snaps automatically refresh to the latest version every 4 hours. So there is no manual step a user needs to take to 'upgrade their system immediately' as in most cases they will have automatically refreshed to the fixed version by the time they read this blog post - if they want to ensure they have refreshed to this version though they can run `snap refresh` to force it to happen immediately.

Also I think the following part is a bit unclear:

-------

  When installing a service through Snap, Snap forwards the service registration requests to systemd and registers it as a service under the following name structure:

    snap.<package_name>.<service_name>.service

  When snap registered the dockerd service, the Docker Snap package did not request a systemd’s cgroup delegation, leaving the cgroup management in the hands of systemd. Moreover, the Snap daemon (snapd) did not expose this option to packages installed via Snap at all, thus any service installed via Snap had its cgroups managed by systemd. In particular, Docker, when installed via Snap, is vulnerable to this escape.

---

So to clarify this I'll explain how snapd works with systemd - when a snap is installed that contains a daemon, snapd generates a systemd unit file on behalf of the snap daemon. In this case, since snapd had no awareness of/support for the Delegate property of systemd units, there was no way for the docker snap to request this functionality. I just want to ensure it is clear that it is snapd that is at fault here, not the snaps like Docker themselves.

As such you should then also change the following bit:

  Due to a bug in Snap’s Docker package, the Docker daemon is no longer allowed to manage cgroups for its containers, while systemd is taking ownership of the containers’ cgroups, to cause a misconfiguration.

To instead be something like:

  Due to this missing feature in snapd, the docker snap is not able to indicate to systemd that it will manage cgroups on behalf of its containers itself, and so systemd takes ownership of these cgroups, resulting in this misconfiguration.

Also there are easier ways to trigger apport to run on the host than having to write a program which causes a segmentation fault - instead you can just send the SIGSEGV (or SIGABORT) to an existing process - ie running the following from a terminal will cause Apport to be spawned:

 bash -c 'kill -SIGABRT $$'

Finally, I think the conclusion is a bit sensationalist - given that snaps auto-refresh (including snapd itself), the majority of users will have their snapd updated and hence patched against this vulnerabilility automatically. Also this only affects users of either the docker snap or other snaps which are container managers like the ...

Read more...

Revision history for this message
Gilad Reti (giladreti) wrote :

Thanks @alexmurray for your helpful comments. We would be glad if you will be able to clarify some things:
1. Regarding the numbers of affected systems - we wouldn't want to overestimate the numbers of course. The numbers that appear in the blogpost are rough estimates done by us, based on the numbers of ubuntu users etc. If you have better numbers or estimates (including the other products (greengrass, microk8s, etc) too that would be great and we would like to replace our estimates with it.
2. Speaking of aws greengrass, we contacted aws too before publishing the blogpost. They insist that only the v1 package is affected which is deprecated, and that greengrass v2 is not vulnerable to this CVE. Is that true?

Thanks.

Revision history for this message
Alex Murray (alexmurray) wrote :

Yes the aws-iot-greengrass snap has 2 different tracks:

$ snap info aws-iot-greengrass
...
channels:
  latest/stable: 1.11.0 2020-12-30 (44) 111MB -
  latest/candidate: 1.11.0 2020-12-30 (44) 111MB -
  latest/beta: 1.11.0 2020-12-30 (44) 111MB -
  latest/edge: 1.11.0 2020-12-30 (44) 111MB -
  1.8.x/stable: 1.8.4 2020-12-30 (4) 98MB -
  1.8.x/candidate: 1.8.4 2020-12-30 (4) 98MB -
  1.8.x/beta: ↑
  1.8.x/edge: ↑

Only the 1.8.x snap uses one of the snapd interfaces which is affected by this bug - and this is deprecated. If a user goes to install this snap now, it will install from the the 'latest' channel and this version does not use an affected interface and so those users are not affected by this bug.

Revision history for this message
Gilad Reti (giladreti) wrote :

Hey all.
We would like to have your assistance in to things:
1. We would be glad if any if you can credit CyberArk Labs too, something like "Gilad Reti and Nimrod Stoler from CyberArk Labs".
2. Are you aware of any other products affected by this bug? Other than microk8s, Azure IoT Edge, AWS IoT Greengrass. We will need those before posting our blogpost, for legal reasons.

Thank you very much and sorry for the inconvenience.

Revision history for this message
Alex Murray (alexmurray) wrote :

Hi Gilad, our policy is to credit individuals, not companies, so I have updated the CVE and USN to add Nimrod as well.

As far as other affected products, the Azure IoT Edge snap is not publicly available to my knowledge so I don't think this is as relevant. The affected snaps from the public https://snapcraft.io store are docker, kubernetes-worker, aws-iot-greengrass (1.8.x only) and microk8s.

Revision history for this message
Gilad Reti (giladreti) wrote :

Hi @alexmurray, sorry for bothering you again.
Is the "priority" that appear on your site is the CVSS priority, or some inhouse scale?
Thanks, and sorry again. We will publish our blogpost in the next couple of days, currently we are facing some legal issues.

Revision history for this message
Ian Johnson (anonymouse67) wrote :

This was technically "Fix Released" with 2.48.3, but I can't seem to create that milestone in LP so I used 2.49 instead (which does exist)

Changed in snapd:
status: In Progress → Fix Committed
milestone: none → 2.49
status: Fix Committed → Fix Released
Revision history for this message
Alex Murray (alexmurray) wrote :

Gilad, the priority you see on https://ubuntu.com/security/CVE-2020-27352 as 'high' is our own teams priority - we take CVSS into account when assigning this but this is not the only factor we consider - see https://people.canonical.com/~ubuntu-security/priority.html for more technical detail on this priority score and https://ubuntu.com/blog/securing-open-source-through-cve-prioritisation for some wider discussion on how we assign priority etc.

Revision history for this message
Seth Arnold (seth-arnold) wrote :

Is there a reason to keep this bug private? I'm curious if this is related to https://bugs.launchpad.net/charm-kubernetes-master/+bug/1928806 and this bug being private makes it awkward to suggest the connection in that bug.

Thanks

Revision history for this message
Ian Johnson (anonymouse67) wrote :

Seth, personally I was waiting for the security team to make this bug public after the CRD, now that that has happened, this can surely be made public.

information type: Private Security → Public Security
To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.