"sudo" calls try to connect to host DBus from within a container

/var/log/audit/audit.log:type=USER_ACCT msg=audit(1553621646.085:3730): pid=71808 uid=42436 auid=4294967295 ses=4294967295 subj=system_u:system_r:container_t:s0:c493,c684 msg='op=PAM:accounting grantors=pam_unix acct="nova" exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success'UID="unknown(42436)" AUID="unset"
/var/log/audit/audit.log:type=USER_CMD msg=audit(1553621646.085:3731): pid=71808 uid=42436 auid=4294967295 ses=4294967295 subj=system_u:system_r:container_t:s0:c493,c684 msg='cwd="/" cmd="kolla_set_configs" terminal=? res=success'UID="unknown(42436)" AUID="unset"
/var/log/audit/audit.log:type=CRED_REFR msg=audit(1553621646.085:3732): pid=71808 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:container_t:s0:c493,c684 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset"
/var/log/audit/audit.log:type=AVC msg=audit(1553621646.085:3733): avc: denied { connectto } for pid=71808 comm="sudo" path="/run/dbus/system_bus_socket" scontext=system_u:system_r:container_t:s0:c493,c684 tcontext=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 tclass=unix_stream_socket permissive=0
/var/log/audit/audit.log:type=SYSCALL msg=audit(1553621646.085:3733): arch=c000003e syscall=42 success=no exit=-13 a0=4 a1=56057de3b810 a2=1d a3=7ffcbd665f90 items=0 ppid=71791 pid=71808 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="sudo" exe="/usr/bin/sudo" subj=system_u:system_r:container_t:s0:c493,c684 key=(null)ARCH=x86_64 SYSCALL=connect AUID="unset" UID="root" GID="root" EUID="root" SUID="root" FSUID="root" EGID="root" SGID="root" FSGID="root"
/var/log/audit/audit.log:type=USER_START msg=audit(1553621646.085:3734): pid=71808 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:container_t:s0:c493,c684 msg='op=PAM:session_open grantors=pam_limits,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset"
/var/log/audit/audit.log:type=USER_END msg=audit(1553621646.180:3735): pid=71808 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:container_t:s0:c493,c684 msg='op=PAM:session_close grantors=pam_limits,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset"
/var/log/audit/audit.log:type=CRED_DISP msg=audit(1553621646.180:3736): pid=71808 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:container_t:s0:c493,c684 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset"
/var/log/secure:Mar 26 17:34:06 undercloud-0 sudo[71808]: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/kolla_set_configs
/var/log/secure:Mar 26 17:34:06 undercloud-0 sudo[71808]: pam_systemd(sudo:session): Failed to connect to system bus: Permission denied
/var/log/secure:Mar 26 17:34:06 undercloud-0 sudo[71808]: pam_unix(sudo:session): session opened for user root by (uid=0)
/var/log/secure:Mar 26 17:34:06 undercloud-0 sudo[71808]: pam_unix(sudo:session): session closed for user root

So it seems this is the nova user:...

So I looked at this a bit more and it is not specific to nova. This is how kolla really works.

When a container calls '/usr/local/bin/kolla_start' the following line gets invoked:
sudo -E kolla_set_configs

This will make sure that we invoke kolla_set_config via sudo (in case we're running as a non-root user). From /etc/sudoers
%kolla ALL=(root) NOPASSWD: /usr/local/bin/kolla_set_configs

So now, given that this is how sudo/kolla work to set the configs, we need to find a way to avoid the denials.

hmmmm. the right question is: "is -E option really needed? What if we drop it?" imho.

I'm pretty sure `sudo' wants to connect to dbus in order to fetch the user env or something, in order to preserve it (-E: the user wishes to preserve their existing environment variables.)

This is especially true since the sudo call isn't denied, else we would have some more failures. Sooo.. what about dropping that -E thingy? :)

I wish :). The -E is needed because we need to pass down the KOLLA_ env variables. Kolla fails if it does not have the KOLLA_COPY_STRATEGY var, for example.

The reason we talk to dbus is probably more to do with pam than anything else (I quickly checked the sudo code and did not see explicit dbus calls, so I am tending to think that we can prolly fix this with either a selinux policy change and/or a pam change inside the containers.

We actually did something similar for rabbit via Ic0789da4645a4ee186d82ad7d943de78d4d5c443
I am wondering if we should try something like that but more global.

So, while I have not tried this personally yet, I am moderately sure this is due to pam+sudo (I say this because in https://review.opendev.org/#/c/629200/ we did something potentially similar).

Let's explore the pam config theory for a moment:
In the nova_api rhel8 container:
()[root@undercloud-0 /etc/pam.d]$ cat /etc/pam.d/sudo
#%PAM-1.0
auth include system-auth
account include system-auth
password include system-auth
session include system-auth

()[root@undercloud-0 /etc/pam.d]$ cat system-auth
#%PAM-1.0
# This file is auto-generated.
# User changes will be destroyed the next time authselect is run.
auth required pam_env.so
auth sufficient pam_unix.so try_first_pass nullok
auth required pam_deny.so

account required pam_unix.so

password requisite pam_pwquality.so try_first_pass local_users_only retry=3 authtok_type=
password sufficient pam_unix.so try_first_pass use_authtok nullok sha512 shadow
password required pam_deny.so

session optional pam_keyinit.so revoke
session required pam_limits.so
-session optional pam_systemd.so
session [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session required pam_unix.so

Note that the minus sign in front of pam_systemd.so is:
"""
If the type value from the list above is prepended with a -
character the PAM library will not log to the system log if it is not
possible to load the module because it is missing in the system. This
can be useful especially for modules which are not always installed on
the system and are not required for correct authentication and
authorization of the login session.
"""

I think it is worth exploring if a kolla change in docker/base/Dockerfile.j2 like the following will fix this for us:
{% if base_distro == 'centos' or if base_distro == 'rhel' %}
RUN sed -ri 's/-session(\s+)optional(\s+)pam_systemd.so//' /etc/pam.d/system-auth
{% endif %}

Now the reason we do not need that line in containers anyway is because it is useless inside a
container: pam_systemd registers user sessions with the systemd login manager systemd-logind.service(8), and hence the systemd control group hierarchy.

for info, the patch has been integrated to kolla for a long time now:
commit c4b62490714ea849cedfe07d4b9a6aebf1f815c0
Author: Cédric Jeanneret <email address hidden>
Date: Tue Apr 30 16:11:46 2019 +0200

    Drop systemd support from PAM

    The line being removed is the cause of audit.log spam with a
    SELinux enforcing (and permissive) system. They are due to the
    fact we're calling "sudo" from within some containers, and PAM
    wants to connect to the host "dbus" socket (mounted in /run).
    This action is obviously denied by SELinux.

    The "-" prefix makes this module optional anyway, but apparently
    PAM does see it and tries to use it.

    Dropping this module has no consequences other than preventing
    log spamming.

    Change-Id: I3e0e86026f5a4a78473bed824cd1682d3a020cd5
    Co-Authored-By: Michele Baldessari <email address hidden>
    Resolves-Bug: #1819461

BUT. apparently we still see some of those lines from time to time.. Needs some more investigations.

Related fix proposed to branch: master
Review: https://review.opendev.org/704808

Reviewed: https://review.opendev.org/704808
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=2426780600c6d2f610e066111947d0655c325fd9
Submitter: Zuul
Branch: master

commit 2426780600c6d2f610e066111947d0655c325fd9
Author: Cédric Jeanneret <email address hidden>
Date: Wed Jan 29 15:03:58 2020 +0100

    Remove /run bind-mount from mistral containers

    Apparently, there isn't any real use for this bind-mount. Removing it
    might also make things better for the content of audit.log, especially
    if we can avoid exposing DBus socket in the container (sudo does some
    weird things with it).

    Change-Id: I3683820bf6c547f5151fc4f5c9696e58b50dac75
    Related-Bug: #1819461