apparmor profile needs entries to support NUMA

Status changed to 'Confirmed' because the bug affects multiple users.

Hi Mathias,
thank you for your report - it is an interesting one.

I think (but I subscribe Lars and Robie for more expertise) that you are right that it is not working but wrong on the current assumption why. Let me outline why I think so.

[1]: defines it needs WITH_NUMA, but since MySQL 5.7.17 it is set by CMAKE "based on whether the current platform has NUMA support."
Now with that in mind what it states: " For the innodb_numa_interleave option to be available".
So what happens in your case is NOT "innodb_numa_interleave - unknown option", instead it understands and try to follow and only later then fails to actually execute it:
"Failed to set NUMA memory policy of buffer pool page frames to MPOL_INTERLEAVE (error: Invalid argument)."
I think the availability of libnuma-dev in [2] is enough to make CMake enable WITH_NUMA.
And the issue you face is that the system you are on is not numa aware/capable and/or it might be a permission issue.

So far that is only a theory - so please if you know better correct me :-)

Next I thought it might be a platform thing, so I tried with numactl and set up a KVM guest with fake numa to work for sure.

Container (no numa)
# sudo numactl --interleave 0-1 echo foo
libnuma: Warning: node argument 1 out of range

Guest (fake numa)
$ sudo numactl --interleave 0-1 echo foo
foo

So on the latter I re-ran the test with mysql that you reported (thanks for the steps to reproduce). And it still triggers.
So far +1 on confirmed for the actual issue.

I went into root as I thought it is a permission issue, but it isn't (usually user mysql).

But then I saw there is a set of apparmor denials, none
[ 792.298983] audit: type=1400 audit(1512141533.009:33): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=3699 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=113 ouid=0

That is numa related, maybe it does some checks up front.
So I disabled apparmor for mysqld
$ sudo aa-disable /usr/sbin/mysqld
And e voila:

2017-12-01T15:16:13.126125Z 0 [Note] InnoDB: Setting NUMA memory policy to MPOL_INTERLEAVE
2017-12-01T15:16:13.132921Z 0 [Note] InnoDB: Setting NUMA memory policy to MPOL_DEFAULT
2017-12-01T15:16:13.132949Z 0 [Note] InnoDB: Completed initialization of buffer pool

I checked the Denies and wrote rules for them.
If you add the following to the "/etc/apparmor.d/usr.sbin.mysqld" profile you will be good.

# numa enabling
  /sys/devices/system/node/ r,
  /sys/devices/system/node/** r,
  @{PROC}/@{pid}/status r,

Then reload with:
$ sudo apparmor_parser -r /etc/apparmor.d/usr.sbin.mysqld

So it is not a build issue, but instead an imperfect apparmor profile.
I leave the inclusion of the fix to our mysql packagers (subscribing them).

[1]: https://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_numa_interleave
[2]: https://launchpadlibrarian.net/341325097/buildlog_ubuntu-artful-amd64.mysql-5.7_5.7.20-0ubuntu0.17.10.1_BUILDING.txt.gz

Since the solution is known this is set to triaged.
Also updated the title, I think my comment explains all that is needed as well as being a workaround until then.

I see the same, plus one more, apparmor denials on a fresh up-to-date bionic system where I just install "mysql-server":
[ter nov 20 15:30:58 2018] audit: type=1400 audit(1542741161.387:263): apparmor="DENIED" operation="open" namespace="root//lxd-bionic-mysql_<var-lib-lxd>" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=15939 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=165536 ouid=0
[ter nov 20 15:30:58 2018] audit: type=1400 audit(1542741161.391:264): apparmor="DENIED" operation="capable" namespace="root//lxd-bionic-mysql_<var-lib-lxd>" profile="/usr/sbin/mysqld" pid=15939 comm="mysqld" capability=2 capname="dac_read_search"
[ter nov 20 15:30:58 2018] audit: type=1400 audit(1542741161.403:265): apparmor="DENIED" operation="open" namespace="root//lxd-bionic-mysql_<var-lib-lxd>" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=15943 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=165647 ouid=0

Trying to find dormant bugs that could/should be resolved ...

To summarize (for mysql8.0) as of today:

Build:
-- Looking for include file numa.h
-- Looking for include file numa.h - found
-- Looking for include file numaif.h
-- Looking for include file numaif.h - found
-- Performing Test HAVE_LIBNUMA
-- Performing Test HAVE_LIBNUMA - Success

it is enabled, libnuma shows in dependencies, ...
All fine feature wise.

The apparmor rule was fixed in 5.7.17

commit ba7bd14c0eff8c68664c66a0e0d241fafc142a5d
Author: Bjoern Boschman <email address hidden>
Date: Wed Dec 14 13:36:55 2016 +0100

    Imported Upstream version 5.7.17

diff --git a/packaging/deb-in/extra/apparmor-profile b/packaging/deb-in/extra/apparmor-profile
index 83ed6ec1e..8004069fc 100644
--- a/packaging/deb-in/extra/apparmor-profile
+++ b/packaging/deb-in/extra/apparmor-profile
@@ -9,6 +9,9 @@

 # Allow system resource access
   /sys/devices/system/cpu/ r,
+ /sys/devices/system/node/ r,
+ /sys/devices/system/node/** r,
+ /proc/*/status r,
   capability sys_resource,
   capability dac_override,
   capability setuid,

But the report and tests were with later versions e.g. reported with 5.7.20.

Clearly on a Kinetic system I see the right rules being present.

@Lena / Robie / Larse - maybe that is an easy test to recheck if at least in recent mysql-8.0 this can be enabled and works without any apparmor issues.
And if there are, that sounds like an easy but helpful fix for this cycles merge time.

I ran the reproducer on Kinetic with mysql-server 8.0.29-0ubuntu4. error.log reports no relevant errors. So I think this is either fixed or needs the reproducer steps adjusted. For now, let's assume it's fixed in Kinetic. But if that's wrong, please comment and explain how to reproduce.

For older releases we might be able to make adjustments. Is this still impacting someone in an older release, and if so, please let us know which release, which package version, and could you confirm the reproduction steps on that release please?