lxc-attach needs to mount a /proc for apparmor's use

Hi,

could you please show a simple script (doing lxc-create && lxc-start && lxc-wait && lxc-attach) that I could reproduce with?

Just to be sure - you get this error running this by hand, right? If it happens at boot time, then you should make sure that the 'lxc' init job has completed before your job begins, since that is the one which will load the apparmor profiles.

Hi Serge,

This does not happen when doing this by hand, only in a script. And I can't seem to reproduce on a container created using a template. So far, I've only been able to reproduce this when creating the container using the procedure in lp:unity8-preview-lxc. This is a project that was started by stgraber and taken over by me.

Basically, it's a python script that creates a specialized config and uses an unpacked Ubuntu Desktop Next (Unity 8) ISO to start up the Unity 8 session inside the container. If you'd like to try it, you can add ppa:unity8-desktop-session-team/unity8-preview-lxc.

I'll also attach the config and the output of the logs when this issue occurs.

Thanks!

I've been trying to capture a log of the failure when running lxc-attach and it seems to cause just enough delay to get by the race. Also, it does not always occur, but more time than not, the error happens leading more credence to this being a race. I've attached the config I use for the container.

I assume that bin/unity8-lxc-compositor is the script you are using. But
how does it get run? Do I just install that package in a container to
reproduce this?

Actually, running "sudo unity8-lxc-setup" has a start/wait/attach sequence that is causing failures. Another way I try to reproduce after the whole container is setup is to create a little script with the following:

lxc-start -n unity8-lxc
lxc-wait -t 5 -s RUNNING -n unity8-lxc
lxc-attach -n unity8-lxc -- uname -a

Run that as sudo since it's privileged and I can sometimes get the apparmor error. Keep in mind that after the container starts for the first time, I have never been able to reproduce this issue unless I reboot. Just doing a lxc-stop is not enough.

Also, if you still need some help, you can ping me on irc @ ChrisTownsend.

Which distribution is in the rootfs?

The rootfs used Vivid and now Wily, but the issue remains with both.

I cannot reproduce this.

Would it be possible to coordinate your handing off a broken system to me? Unfortunately I am out tomorrow and Friday, but perhaps sometime next week?

Dang, I was afraid you might not be able to reproduce, being a race and all. I'm not sure it's possible to hand off the system as it's my primary development machine.

I think I will take a stab in trying to debug this. A quick looks shows that it is failing in this block in lsm/apparmor.c:

if (aa_change_profile(label) < 0)

Looking in the apparmor source code shows a few places this may fail, especially with the "No such file or directory" errno value.

The call(s) in setprocattr(), particularly procattr_path(). I'm thinking that perhaps the /proc path is not really fully up before lxc_wait says that the container is in a RUNNING state. I'm using SSD's which may or may not make a difference, but they can sometimes add some races due to their fast nature.

Next, I will look into what exactly lxc_wait is querying to know when it is RUNNING...

So, if I remove the lxc-wait out of the start/wait/attach sequence, then I always get the failure. This really points to a race where RUNNING is being reported before it is really fully started.

It looks like the RUNNING state is set in start.c, so perhaps it is being set a bit too early where some asynchronous activity isn't really complete yet???? I'll continue to look in that area.

This happens if the container config doe snot specify mounting /proc, because there is no (or a wrong) /proc in the container until the container's init mounts it.

So lxc-attach needs to mount a temporary /proc (after switching namespaces) for the sake of setting the lsm label.

Thanks again for opening this bug. A candidate fix has been sent to the lxc-devel mailing list. The workaround is to make sure that the lxc configuration mounts a procfs for the container, using either a lxc.mount.entry or lxc.mount.auto.