Apparmor uses rsyslogd profile for different processes - utopic HWE

This errors started appearing in the log since Feb 21 when I've rebooted to complete Utopic HWE installation:
# dpkg -l | grep utopic
ii linux-generic-lts-utopic 3.16.0.31.24 amd64 Complete Generic Linux kernel and headers
ii linux-headers-generic-lts-utopic 3.16.0.31.24 amd64 Generic Linux kernel headers
ii linux-image-generic-lts-utopic 3.16.0.31.24 amd64 Generic Linux kernel image

I can confirm that the audit message is gone from /var/log/syslog after I got back to the original 3.13.0-46 Ubuntu 14.04.2 kernel.

There's a chance this is the AppArmor "cross profile" IPC check; when one process performs an IPC operation with a second process, in different profiles, both profiles have to allow the operation.

If my guess is right, these ought to be silenced by changing:

  /dev/log wl,
to
  /dev/log rwl,

in the /etc/apparmor.d/usr.sbin.rsyslogd file and reloading the policy with apparmor_parser --replace /etc/apparmor.d/usr.sbin.rsyslogd

Thanks

Seth is correct,

when a message is sent across a unix based file socket their is a cross check (one check for the sender, and another for the receiver). In this case the receiving end of the message (rsyslogd) does not have the correct permission (r perm).

The reason this is reported with the pid and comm of other processes is that the check (for both sender and receiver) is done in the kernel context of the sending process.

First of all, Trusty kernel worked somehow differently. It didn't complain for /dev/log.

Yes, my /etc/apparmor.d/usr.sbin.rsyslogd does not have 'r' for /dev/log:
/dev/log wl,

So does it mean that the rsyslog profile has to be patched for trusty?
Or at least some kind of Utopic HWE release notes has to be updated?

I've updated /etc/apparmor.d/local/usr.sbin.rsyslogd and can confirm that message is gone with Utopic kernel.

Are you using the apparmor user space from trusty or the utopic or backport apparmor userspace?

When using the trusty userspace the HWE kernel should behave like the trusty kernel, however when using a newer userspace with the HWE kernel policy will need to be updated.

Ubuntu have chosen to not update the policy and userspace for trusty even when the HWE kernel is used.

John,

I'm using Ubuntu Trusty with Utopic HWE, so my apparmor is from Trusty.
I didn't install anything from backports or Utopic by hand to not mess package system and LTS state of distribution.
But somehow the system does not behave like trusty.

Note that I've updated this system from 12.04 w/ Trusty HWE to 14.04 with do-release-upgrade few months ago.

$ LANG=C apt-cache policy apparmor apparmor-profiles apparmor-utils
apparmor:
  Installed: 2.8.95~2430-0ubuntu5.1
  Candidate: 2.8.95~2430-0ubuntu5.1
  Version table:
 *** 2.8.95~2430-0ubuntu5.1 0
        500 http://ru.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
        100 /var/lib/dpkg/status
     2.8.95~2430-0ubuntu5 0
        500 http://ru.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
apparmor-profiles:
  Installed: 2.8.95~2430-0ubuntu5.1
  Candidate: 2.8.95~2430-0ubuntu5.1
  Version table:
 *** 2.8.95~2430-0ubuntu5.1 0
        500 http://ru.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
        100 /var/lib/dpkg/status
     2.8.95~2430-0ubuntu5 0
        500 http://ru.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
apparmor-utils:
  Installed: 2.8.95~2430-0ubuntu5.1
  Candidate: 2.8.95~2430-0ubuntu5.1
  Version table:
 *** 2.8.95~2430-0ubuntu5.1 0
        500 http://ru.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
        100 /var/lib/dpkg/status
     2.8.95~2430-0ubuntu5 0
        500 http://ru.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages

Pavel,

okay thanks, this does sound like a bug in the kernel. I will look into it more

So after looking into this more it looks like this is actually a bug in the trusty kernel, that has been fixed on utopic.

Trusty needs a kernel patch and an update to its policy.

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1425398

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Status changed to 'Confirmed' because the bug affects multiple users.

Same problem here (Trusty+HWE kernel) and adding "/dev/log r," to /etc/apparmor.d/local/usr.sbin.rsyslogd does not help.

The rsyslog apparmor policy is shipped in the rsyslog package, not apparmor. Opening a task against rsyslog, closing the apparmor userspace task.

This is fixed in the rsyslog policy in vivid; closing there.

As I said before, I don't use Vivid, I use Trusty which is stated to be supported until 2019!

The bug is against trusty, please don't close it as fixed in an non-LTS release

Also if you read comments of other Ubuntu _developers_, they are rather confident that the bug is in Utopic HWE for Trusty

Pavel, note that Steve closed the rsyslog/vivid task because that has been fixed; the rsyslog/trusty task is still open, as are the kernel portions of the bug.

Thanks

John, what changes are needed kernel side to address this issue?

Attached is a debdiff to address the rsyslog policy side of things.

The issue here is that one end of the socket is an fs socket and the other end is anonymous. When the check is done from the socket at the anonymous end, the check is being dropped.

the patch is a backport of what is in utopic/vivid but is currently untested. I am building a test kernel

It turns out that there is a small bit of the AppArmor userspace that needs to be addressed, the regression tests need to be slightly adjusted to take the permissions change into account.

ACK on the debdiff in comment #20. Uploaded for processing by the SRU team. Thanks!

Hello Pavel, or anyone else affected,

Accepted rsyslog into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/rsyslog/7.4.4-1ubuntu2.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

It seems that issue is gone with this fix.

1. Installed the package from trusty-proposed
# LANG=C apt-cache policy rsyslog
rsyslog:
  Installed: 7.4.4-1ubuntu2.6
  Candidate: 7.4.4-1ubuntu2.6
  Version table:
 *** 7.4.4-1ubuntu2.6 0
        400 http://archive.ubuntu.com/ubuntu/ trusty-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     7.4.4-1ubuntu2.5 0
        500 http://ru.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
     7.4.4-1ubuntu2.3 0
        500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
     7.4.4-1ubuntu2 0
        500 http://ru.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages

2. commented out the workaround "/dev/log rwl," from /etc/apparmor.d/local/usr.sbin.rsyslogd

3. Rebooted to the latest Utopic HWE kernel
# uname -a
Linux emma 3.16.0-37-generic #51~14.04.1-Ubuntu SMP Wed May 6 15:23:14 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

4. The issue is gone:
# dmesg | grep DENIED
#

Works here too, thanks!

This bug was fixed in the package rsyslog - 7.4.4-1ubuntu2.6

---------------
rsyslog (7.4.4-1ubuntu2.6) trusty-proposed; urgency=medium

  * debian/usr.sbin.rsyslog: grant read access to /dev/log to cope with
    behaviorial change of apparmor in utopic HWE kernel (LP: #1425398)

 -- Steve Beattie <email address hidden> Thu, 30 Apr 2015 12:20:51 -0700

The verification of the Stable Release Update for rsyslog has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Here is the patch to address the apparmor userspace component of this bug as part of a trusty SRU. It's already been addressed in utopic and later.

For the apparmor trusty SRU, I reproduced the failure with the source for apparmor 2.8.95~2430-0ubuntu5.1 from trusty-updates with a hardware enablement kernel and confirmed that apparmor 2.8.95~2430-0ubuntu5.2 addresses the issue. I also verified that the rest of the apparmor regression tests passed.

Oddly enough, I'm still seeing some variation of this error:

May 21 12:27:28 xeon kernel: [95104.918686] audit: type=1400 audit(1432225648.230:57): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/rsyslogd" name="dev/log" pid=3444 comm="logger" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

$ dpkg -l| grep rsyslog
ii rsyslog 7.4.4-1ubuntu2.6 amd64 reliable system and kernel logging daemon
$ uname -a
Linux xeon 3.16.0-38-generic #52~14.04.1-Ubuntu SMP Fri May 8 09:43:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Simon, that is a different issue unrelated to this update. It is being tracked in bug #1373070.

This bug was fixed in the package apparmor - 2.8.95~2430-0ubuntu5.2

---------------
apparmor (2.8.95~2430-0ubuntu5.2) trusty-proposed; urgency=medium

  * debian/patches/php5-Zend_semaphore-lp1401084.patch: allow php5
    abstraction access to Zend opcache files (LP: #1401084)
  * debian/patches/dnsmasq-lxc_networking-lp1403468.patch: update
    profile for lxc support (LP: #1403468)
  * debian/patches/profiles-texlive_font_generation-lp1010909.patch:
    allow generation of texlive fonts by sanitized-helpers
    (LP: #1010909)
  * debian/apport/source_apparmor.py: fix the apparmor apport hook
    so it does not raise an exception if a non-unicode character is
    found in /var/log/kern.log or in /var/log/syslog. This should
    work under python3 or python2.7 (LP: #1304447)
  * debian/patches/profiles-dovecot-updates-lp1296667.patch: update
    dovecot profiles to address several missing permissions.
    (LP: #1296667)
  * debian/patches/profiles-adjust_X_for_lightdm-lp1339727.patch:
    adjust X abstraction for LightDM xauthority location (LP: #1339727)
  * debian/patches/libapparmor-fix_memory_leaks-lp1340927.patch; fix
    memory leaks in log parsing component of libapparmor (LP: #1340927)
  * debian/patches/libapparmor-another_audit_format-lp1399027.patch:
    add support for another log format style (LP: #1399027)
  * debian/patches/tests-workaround_for_unix_socket_change-lp1425398.patch:
    work around apparmor kernel behavioral change in regression tests
    (LP: #1425398)
  * debian/control: add breaks on python3-apparmor against older
    apparmor-utils that used to be where python bits lived
    (LP: #1373259)
  * debian/patches/utils-update_to_2.9.2.patch: update the python
    utilities to the upstream 2.9.2 (LP: #1449769, incorporating a
    large number of fixes and improvements, including:
    - fix aa-genprof traceback with apparmor 2.8.95 (LP: #1294797)
    - fix aa-genprof crashing when selecting scan on Ubuntu 14.04 server
      (LP: #1319829)
    - make aa-logprof read profile instead of program binary
      (LP: #1317176, LP: #1324154)
    - aa-complain: don't traceback when marking multiple profiles
      (LP: #1378095)
    - make python tools able to parse mounts with UTF-8 non-ascii
      characters (LP: #1310598)

 -- Steve Beattie <email address hidden> Thu, 30 Apr 2015 12:18:08 -0700