The Setup

Let's paint a picture of a fun scenario. In my home lab I run a mix of CentOS and Windows. Windows for Active Directory and those Windows specific apps, CentOS for most everything else. For CentOS systems I use Spacewalk as an inventory and patch distribution system. While Spacewalk is pretty dang heavy for such a simple workflow it at least keeps me tuned into what is likely to be found in production environments. Rather than using local accounts in Spacewalk I configured it to use PAM, which in turn uses pam_krb5 connected to my lab's Windows domain.

One day during sign-in I noticed the pam_krb5 connection had broken. This being a lab setup I don't sign into the back-end services very often, which means I have no idea when the service stopped working. The web interface throws a generic "Invalid Username/Password" error and the system throws these two messages:

Dec  6 19:22:27 node1.localdomain server[1186]: 2017-12-06 19:22:27,669 [ajp-bio-0:0:0:0:0:0:0:1-8009-exec-7] WARN  com.redhat.rhn.domain.user.legacy.UserImpl - PAM login for user User spack (id 2, org_id 1) failed with error System error.
Dec  6 19:22:29 node1.localdomain server[1186]: 2017-12-06 19:22:29,670 [ajp-bio-0:0:0:0:0:0:0:1-8009-exec-7] INFO  com.redhat.rhn.frontend.action.LoginAction - LOCAL AUTH FAILURE: [spack]

I've been dealing with SELinux long enough that it is the first place I check, but alas no AVC messages to be found in the audit log. After some period of time bumbling around, and verifying pam_krb5 works for other services, I decide to disable SELinux and see what happens. Lo and behold, I can sign-in just fine. Clearly there is an AVC event but it is not getting logged.

The Diagnosis

Further reading indicates that it is, in fact, perfectly normal and expected for denies to be silent. There are a number of reasons this might be the case, but there is an expectation that most denies are cosmetic noise and should not be seen. Since the decision to hide a denial is determined by the policy author we may occasionally find scenarios where a denial is, in fact, not cosmetic and is instead a real problem.

The critical tool here is seinfo, which is shipped as part of the setools-console package. It spits out a bunch of nice statistical information about the running configuration, but what's really relevant here can be found with

[root@node1 ~]# seinfo --stats | grep audit
   Auditallow:        155    Dontaudit:        8855

This is showing that of all the loaded policies we have 155 explicit allows configured and a grand total of 8855 rules set to block but not log. That's quite a lot of denies we normally don't get to see, so presumably one of those is causing our problem.

The dontaudit statement is a keyword that disables logging on a particular rule. The easiest approach for us right now is to disable the dontaudit keyword. By doing so it won't take affect and, more importantly, we won't be required to modify each shipped policy. We can do this by running:

semodule --disable_dontaudit --build

After a few minutes the rules will be recompiled. Now if we go back to the webpage and try signing in again we get receive the failed sign-in error, as expected, but this time there's an audit entry:

type=SYSCALL msg=audit(1513616321.280:10659): arch=c000003e syscall=44 success=no exit=-13 a0=41 a1=7f1fe06b8ef0 a2=c0 a3=0 items=0 ppid=1 pid=1186 auid=4294967295 uid=91 gid=91 euid=91 suid=91 fsuid=91 egid=91 sgid=91 fsgid=91 tty=(none) ses=4294967295 comm="java" exe="/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.151-1.b12.el7_4.x86_64/jre/bin/java" subj=system_u:system_r:tomcat_t:s0 key=(null)
type=AVC msg=audit(1513616321.280:10659): avc:  denied  { write } for  pid=1186 comm="java" scontext=system_u:system_r:tomcat_t:s0 tcontext=system_u:system_r:tomcat_t:s0 tclass=netlink_audit_socket

It looks like the existing policies likely didn't handle an upgrade very well.

The Repair

In addition to the normal search and modification tools there's a great utility called audit2allow in policycoreutils-python that can be used to automagically generate policy files from audit logs. Using the excellent SELinux HowTo we can build a local policy and push it out. In my case there were two permits we needed to add. This is, of course, just a policy snippet of the specific changes I made. Your new policy may look very different.

module spacewalklocalpam 1.0;

require {
        type tomcat_t;
        class netlink_audit_socket { write read };
}

allow tomcat_t self:netlink_audit_socket { write read };

Once loaded and tested make sure to re-enable the dontaudit option. (Unless you don't want to, that's your log volume not mine) by running

semodule --build