apparmor unconfined profile blocks pivot_root

Bug #2067900 reported by Aleksandr Mikhalitsyn
50
This bug affects 9 people
Affects Status Importance Assigned to Milestone
AppArmor
Confirmed
Undecided
Unassigned
apparmor (Ubuntu)
Confirmed
Undecided
Unassigned

Bug Description

SRU Justification:

[Impact]

Ubuntu sauce commit "apparmor: convert easy uses of unconfined() to label_mediates()" was applied to Noble and Oracular respectively as dc757a645cfa ("UBUNTU: SAUCE: apparmor4.0.0 [81/90]: apparmor: convert easy uses of unconfined() to label_mediates()") and 621bcec8dae4 ("UBUNTU: SAUCE: apparmor4.0.0 [80/99]: apparmor: convert easy uses of unconfined() to label_mediates()"). This commit prevents the launching of Docker containers inside a LXC container because apparmor unconfined profile blocks pivot_root. It also blocks containers that uses an old apparmor version (e.g. 2.7) to get an IPV4 address through DHCP.

[Fix]

Noble:
 - Backport a revert of commit dc757a645cfa ("UBUNTU: SAUCE: apparmor4.0.0 [81/90]: apparmor: convert easy uses of unconfined() to label_mediates()")
Oracular:
 - Backport a revert of commit 621bcec8dae4 ("UBUNTU: SAUCE: apparmor4.0.0 [80/99]: apparmor: convert easy uses of unconfined() to label_mediates()")

[Test Plan]

This fix can be tested in Noble and Oracular by running docker in LXC and checking how they behave, as below:

 1/ Install LXD on a 24.04 machine
 2/ Run a LXD container with support for security.nesting
 3/ In the LXD container install docker.io
 4/ Run a Docker container

With this patch applied, the docker container will work instead of failing with the following error:

```
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error jailing process inside rootfs: pivot_root .: permission denied: unknown.
ERRO[0000] error waiting for container:
```

The other issue related to old apparmor versions not supporting ABIs can be tested by running:

```
$ lxc launch ubuntu:12.04
$ lxc list status=running
```

and checking that the IPV4 field is non-null in the newly-started container

[Where problems could occur]

This revert backport is small and returns to the old tested behavior. Hence, this SRU should not cause problems.

[Changes between v2 and v3]

 - Create separate patches for Noble and Oracular.
 - Fix patch corruption in v2.

[Other Info]

External links:
 - https://github.com/canonical/lxd/issues/13389
 - https://discourse.ubuntu.com/t/containers-with-ubuntu-12-04-5-lts-are-not-getting-ipv4s-anymore/47371

----------------------------
Original description:

LXD team have got a report (https://github.com/canonical/lxd/issues/13389) from our user that on the Ubuntu Noble host it's not possible to run Docker containers inside a LXC container.

After some investigation, it was discovered that problem connected with AppArmor profile which is shipped by default /etc/apparmor.d/runc (comes from https://git.launchpad.net/ubuntu/+source/apparmor/commit/profiles/apparmor.d/runc?h=ubuntu/noble-devel&id=997aea8111bfa1e03960ae3a40321da73f0a6d96 )

This profile is unconfined and should give all permissions to the runc daemon. But it does not work.

Manual adding of "pivot_root," line and executing "systemctl reload apparmor.service" makes it work.

After some further investigation it was found that on upstream Linux kernel problem is not reproducible.

Our team was able to find a problematic commit:
https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/noble/commit/?id=dc757a645cfa82f6ac252365df20a36a9ff82760

The following (partial) revert helps to solve the issue on Ubuntu kernel:

diff --git a/security/apparmor/mount.c b/security/apparmor/mount.c
index 74b7293ab971..b12e6bdfefb2 100644
--- a/security/apparmor/mount.c
+++ b/security/apparmor/mount.c
@@ -678,7 +678,7 @@ static struct aa_label *build_pivotroot(const struct cred *subj_cred,
        AA_BUG(!new_path);
        AA_BUG(!old_path);

- if (!RULE_MEDIATES(rules, AA_CLASS_MOUNT))
+ if (profile_unconfined(profile) || !RULE_MEDIATES(rules, AA_CLASS_MOUNT))
                return aa_get_newest_label(&profile->label);

        error = aa_path_name(old_path, path_flags(profile, old_path),

System info:

# uname -a
Linux ubuntu 6.8.0-31-generic #31-Ubuntu SMP PREEMPT_DYNAMIC Sat Apr 20 00:40:06 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

# cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04 LTS"
<CUT>

Revision history for this message
John Johansen (jjohansen) wrote :

This requires a v4.0 apparmor parser and Ubuntu not upstream kernel.

The ubuntu kernel carries a patch that is work toward splitting unconfined and making so it can replaced and only cause mediation overhead for the classes being mediated.

The 4.0 parser is setting mediated classes in unconfined profiles when it shouldn't, causing pivot root to fail.

Changed in apparmor (Ubuntu):
status: New → Confirmed
Changed in apparmor:
status: New → Confirmed
Revision history for this message
Aleksandr Mikhalitsyn (mihalicyn) wrote :

It looks like the same issue happens with "kill" syscall:

Jul 01 15:52:45 kernel: audit: type=1400 audit(1719849165.951:291): apparmor="DENIED" operation="signal" class="signal" profile="lxd-v1_</var/snap/lxd/common/lxd>" pid=15369 comm="lxd" requested_mask="receive" denied_mask="receive" signal=kill peer="snap.lxd.daemon"

this started to appear after LXD was enabled to use unconfined profile mode.

Revision history for this message
Thomas Parrott (tomparrott) wrote :

This issue is now occuring in lxd latest/edge builds after we merged initial support for restricted user namespaces.

Is there an eta on a fix?

Revision history for this message
Aleksandr Mikhalitsyn (mihalicyn) wrote :
Revision history for this message
Thomas Parrott (tomparrott) wrote :

Hi all, what is the latest on this? It appears to not have been fixed in Ubuntu 24.04.1

https://github.com/canonical/lxd/issues/13389#issuecomment-2319129052

Revision history for this message
Aleksandr Mikhalitsyn (mihalicyn) wrote :
Revision history for this message
Georgia Garcia (georgiag) wrote :

Sorry for the delay. The fix had landed but it was reverted due to a regression. We have a 4.0.1really4.0.1-0ubuntu0.24.04.3 update but it is still sitting in noble-proposed https://people.canonical.com/~ubuntu-archive/pending-sru.html

Revision history for this message
Aleksandr Mikhalitsyn (mihalicyn) wrote (last edit ):

We have another problem which disappears when I revert dc757a645cfa82f6ac252365df20a36a9ff82760 ("UBUNTU: SAUCE: apparmor4.0.0 [81/90]: apparmor: convert easy uses of unconfined() to label_mediates()") commit.

Now it is not connected with unconfined profiles at all, it involves Ubuntu Noble (host) + LXD (any version) + Ubuntu 12.04 container. And that container fails to get an IPv4 address using dhcp client with the following error:
dhclient3 eth0
RTNETLINK answers: Operation not permitted
RTNETLINK answers: Operation not permitted

On the host side we can see a following AppArmor denial:
Sep 05 12:01:09 kernel: audit: type=1400 audit(1725534069.603:228): apparmor="DENIED" operation="capable" class="cap" namespace="root//lxd-c1_<var-lib-lxd>" profile="/sbin/dhclient" pid=28122 comm="ip" capability=12 capname="net_admin"

Precisely the same user space works well with upstream kernels 6.8.12 and 6.11.0-rc7. But fails on 6.8.12-based Ubuntu Noble's kernel. Reverting of dc757a645cfa82f6ac252365df20a36a9ff82760 makes things to work again.

Reproducer is as simple as lxc launch ubuntu:12.04 myct and check if myct gets an IPv4 address (it won't).

External link: https://discourse.ubuntu.com/t/containers-with-ubuntu-12-04-5-lts-are-not-getting-ipv4s-anymore

Revision history for this message
Georgia Garcia (georgiag) wrote :

Hi, mihalicyn, sorry for the delay answering.

That's unfortunately right. Ubuntu 12.04 ships apparmor 2.7 which didn't have support for ABIs yet, so dc757a645cfa82f6ac252365df20a36a9ff82760 causes a regression on those early versions. I talked to @jjohansen and we have agreed that this patch needs to be reverted, or partially reverted so it doesn't affect older versions of the apparmor_parser - even though are not currently supported, shouldn't break on newer kernels.

This partial revert makes it work in my tests:

--- a/security/apparmor/domain.c
+++ b/security/apparmor/domain.c
@@ -665,7 +665,7 @@ static struct aa_label *profile_transition(const struct cred *subj_cred,
                goto audit;
        }

- if (!profile_mediates(profile, AA_CLASS_FILE)) {
+ if (profile_unconfined(profile)) {
                new = find_attach(bprm, profile->ns,
                                  &profile->ns->base.profiles, name, &info);
                if (new) {

Revision history for this message
Aleksandr Mikhalitsyn (mihalicyn) wrote :

Hi Georgia,

thanks a lot for looking into this issue!

Kind regards,
Alex

Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux/6.8.0-56.58 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux' to 'verification-done-noble-linux'. If the problem still exists, change the tag 'verification-needed-noble-linux' to 'verification-failed-noble-linux'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-noble-linux-v2 verification-needed-noble-linux
Revision history for this message
Maxime Bélair (mbelair) wrote :

Verification completed on noble kernel 6.8.0-56.58:

$ lxc launch ubuntu:24.04 test -c security.nesting=true
Launching test
$ lxc exec test bash
root@test:~# uname -a
Linux test 6.8.0-56-generic #58-Ubuntu SMP PREEMPT_DYNAMIC Fri Feb 14 15:33:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
root@test:~# apt update; apt install -y docker.io
[...]
root@test:~# docker run hello-world
Hello from Docker!
This message shows that your installation appears to be working correctly.
[...]

Maxime Bélair (mbelair)
tags: added: verification-done-noble-linux
removed: verification-needed-noble-linux
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux/6.11.0-21.21 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-oracular-linux' to 'verification-done-oracular-linux'. If the problem still exists, change the tag 'verification-needed-oracular-linux' to 'verification-failed-oracular-linux'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-oracular-linux-v2 verification-needed-oracular-linux
Revision history for this message
Georgia Garcia (georgiag) wrote :

Verification completed in oracular linux/6.11.0-21.21. Works as expected.

georgia@sec-oracular-amd64:~$ uname -a
Linux sec-oracular-amd64 6.11.0-21-generic #21-Ubuntu SMP PREEMPT_DYNAMIC Wed Feb 19 16:50:40 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

georgia@sec-oracular-amd64:~$ sudo lxc launch ubuntu:12.04
[sudo] password for georgia:
Launching the instance
Instance name is: grateful-muskrat
georgia@sec-oracular-amd64:~$ sudo lxc ls name=grateful-muskrat
+------------------+---------+-----------------------+-----------------------------------------------+-----------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+------------------+---------+-----------------------+-----------------------------------------------+-----------+-----------+
| grateful-muskrat | RUNNING | 10.212.214.210 (eth0) | fd42:fbca:f64c:406c:216:3eff:fe36:f101 (eth0) | CONTAINER | 0 |
+------------------+---------+-----------------------+-----------------------------------------------+-----------+-----------+

georgia@sec-oracular-amd64:~$ sudo apt install docker.io docker-compose-v2
georgia@sec-oracular-amd64:~$ sudo docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
e6590344b1a5: Pull complete
Digest: sha256:e0b569a5163a5e6be84e210a2587e7d447e08f87a0e90798363fa44a0464a1e8
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

tags: added: verification-done-oracular-linux
removed: verification-needed-oracular-linux
Revision history for this message
The Owl (the-real-owl) wrote :

Dear Georgia,
can you please test that docker works *within* an LXC container (better if provisioned via LXD) with the same OS version of the host?
The bug would not show when the 2 would run alongside on the host, but instead when docker runs within an LXC container.

Revision history for this message
Georgia Garcia (georgiag) wrote :

Hi The Owl, my apologies. I updated the description containing the SRU justification with the thorough testing steps.
Here's the correct verification:

root@sec-oracular-amd64:~# lxc launch ubuntu:24.10 test -c security.nesting=true
Launching test
root@sec-oracular-amd64:~# lxc exec test bash
root@test:~# uname -a
Linux test 6.11.0-21-generic #21-Ubuntu SMP PREEMPT_DYNAMIC Wed Feb 19 16:50:40 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
root@test:~# apt update
root@test:~# apt install docker.io
root@test:~# docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
e6590344b1a5: Pull complete
Digest: sha256:e0b569a5163a5e6be84e210a2587e7d447e08f87a0e90798363fa44a0464a1e8
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

description: updated
Revision history for this message
The Owl (the-real-owl) wrote :

Hello Georgia, no need to apologize, you have done what I wanted to do "as soon as I have 10 minutes" :).

Thanks!

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.