ServerFault user ewwhite describes a rather interesting situation regarding application distribution wherein code must be compiled in production. In short he wants to keep track of changes to a specific directory path and send alerts via email.

Let's assume that there already exists some basic form of auditd policy in play, so we'll be building out a snippet to be inserted into your existing /etc/audit/audit.rules. Ed was sparse on some of the specifics related to the application, understandably so, so let's make some additional assumptions. Let's assume that the source code directory in question is /opt/application/src and that all binaries are installed into /opt/application/bin. With those assumptions in place let's add these rules.

-w /opt/application/src -p w -k app-policy
-w /opt/application/bin -p wa -k app-policy

I've decided to process each directory, source and binaries, separately. The commonality between the two are the -w and the -k options. The -w option says to watch that directory recursively. The -k option says to add the text app-policy, called a key, to the output log message, this is just to make log reviews easier. The -p option is actually where the magic happens, and is the real reason to separate these two rules out.

As we discussed in the introductory post -p w instructs the kernel to log on file writes. One would assume this is accomplished by attaching to the POSIX system call write(), I certainly did. In truth that syscall gets called quite a lot when files are actually saved. So as to not overwhelm the logging system auditd instead attaches to the open() system call. By using the (w)rite argument we look for any instance of open() that uses the O_WRONLY or O_RDWR flags. It's worth noting that this does not mean a file was actually modified, only that it was opened in such a way that would allow for modification. For example, if a user opened /opt/application/src/app.h in a text editor a log would be generated, however if it was written to the terminal using cat or read using less then no log would be generated. This is pretty important to remember as many people will read a file using a text editor and simply exit without saving changes (hopefully).

We also want to watch for file writes in the binary directory except here we would expect them to be more reliable. It would be rather unusual, but not out of the question, for someone to attempt to use a text editor to open an executable. In addition we added the (a)ttribute option. This will alert us if any of the ownership or access permissions change, most importantly if a file is changed to be executable or the ownership is changed. This will not catch SELinux context changes but since SELinux uses the auditd logging engine then those changes will still be logged and saved into the same log file.

Now that we have the rules constructed we can move on to the alerting. Ed wanted the events to be emailed out. This is actually quite a bit more complicated. By default auditd uses its own built in logging engine instead of relying on something external like syslogd or `rsyslog. By not relying on an external logger it is better able to withstand misconfigurations. However, it also means that making modifications can be trickier because the custom engine means mind-share is significantly smaller and will require yet another expertise requirement on your team.

There does exist a subsystem called audispd that acts as a log multiplexer. There are a number of output plugins available, such as syslog, UNIX socket, prelude IDS, etc. None of them really do what Ed wants, so I think our best bet would run reports. Auditd is, after all, an auditing tool and not an enforcement tool. So let's look at something a little different.

Remember how we tacked on -k app-policy to those rules above? Now we get to the why. Let's try running the command:

aureport -k -ts yesterday 00:00:00 -te yesterday 23:59:59

We should now see a list of all of the logs that contain any keys and occurred yesterday. Let's look at a concrete example of me editing a file in that directory and the subsequent logs.

root@ node1:~> mkdir -p /opt/application/src
root@ node1:~> vim /opt/application/src/app.h
root@ node1:~> aureport -k

Key Report
===============================================
# date time key success exe auid event
===============================================
1. 09/24/2013 11:41:29 app-policy yes /usr/bin/vim 1000 13446
2. 09/24/2013 11:41:29 app-policy yes /usr/bin/vim 1000 13445
3. 09/24/2013 11:41:29 app-policy yes /usr/bin/vim 1000 13447
4. 09/24/2013 11:41:29 app-policy yes /usr/bin/vim 1000 13448
5. 09/24/2013 11:41:29 app-policy yes /usr/bin/vim 1000 13449
6. 09/24/2013 11:41:35 app-policy yes /usr/bin/vim 1000 13451
7. 09/24/2013 11:41:35 app-policy yes /usr/bin/vim 1000 13450

The report tells us that at 11:41:29 on September the 24th a user ran the command /usr/bin/vim and triggered a rule labeled app-policy. It's all good so far, but not very detailed. The last two fields, however, are quite useful. The first, 1000, is the UID of my personal account. That is important because notice I was actually running as root. Since I had originally used sudo -i to gain a root shell my original UID was still preserved, this is good! The last field is a unique event ID generated by auditd. Let's look at that first event, numbered 13446.

root@ node1:~> grep :13446 /var/log/audit/audit.log
type=SYSCALL msg=audit(1380037289.364:13446): arch=c000003e syscall=2 success=yes exit=4 a0=bffa20 a1=c2 a2=180 a3=0 items=2 ppid=21950 pid=22277 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=1226 tty=pts0 comm="vim" exe="/usr/bin/vim" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="app-policy"
type=CWD msg=audit(1380037289.364:13446):  cwd="/root"
type=PATH msg=audit(1380037289.364:13446): item=0 name="/opt/application/src/" inode=2747242 dev=fd:01 mode=040755 ouid=0 ogid=0 rdev=00:00 obj=unconfined_u:object_r:usr_t:s0
type=PATH msg=audit(1380037289.364:13446): item=1 name="/opt/application/src/.app.h.swx" inode=2747244 dev=fd:01 mode=0100600 ouid=0 ogid=0 rdev=00:00 obj=unconfined_u:object_r:usr_t:s0

This is what we mean when we say audit logs are verbose. In the introductory blog post we discussed some of those fields so I'll save us the pain of going over it again. What we can see, however, is that the user with uid 1000 (see auid=1000) ran the command vim as root (see euid=0) and that the command resulted in a change to both /opt/application/src/ and /opt/application/src/.app.h.swx.

What we should be able to see here is that the report generated by aureport doesn't contain everything we need to see what happened, but it does tell us something happened and gives us the necessary data to find the information. In an ideal world you would have some kind of log aggregation system, like Splunk or Graylog or a SIEM, and send the raw logs there. That system would then have all the alerting functionality built in to alert an admin to the potential policy violation. However, we don't live in a perfect world and Ed's request for email alerts implies he doesn't have access to such a system. What I would do is set up a daily cron job to run that report for the previous day. Every morning the log reviewer can check their mailbox and see if any of those files changed when they weren't supposed to. If daily isn't reactive enough then we can simply change the values passed to -ts and -te and run the job more frequently.

Pulling it all together we should have something that looks like this.

#/etc/audit/audit.rules
# This file contains the auditctl rules that are loaded
# whenever the audit daemon is started via the initscripts.
# The rules are simply the parameters that would be passed
# to auditctl.

# First rule - delete all
-D

# Increase the buffers to survive stress events.
# Make this bigger for busy systems
-b 320

# Feel free to add below this line. See auditctl man page
-w /opt/application/src -p w -k app-policy
-w /opt/application/bin -p wa -k app-policy

#/etc/cron.d/audit-report
MAILTO=ewwhite@example.com

1 0   * * *     root  /sbin/aureport -k -ts yesterday 00:00:00 -te yesterday 23:59:59