A fellow Brakeing Down Security slacker, Ceafin asked a fun question question to the group at large.

He further went on to explain that he was investigating a potentially compromised system and didn't trust the standard tools or logs.

While there were a few good answers that other users brought up my first instinct was good ole fashioned auditd. Since the auditing subsystem operates as a series of kernel hooks this should¹ give low enough level insight to bypass most simple evasion.

The high level workflow here would be to let auditd log process execution events, then once logged we can review the events at our leisure. Given the IR style setup we have going on we have to be a little suspicious of any information we generate or view. If log forwarding were configured then we'd be in a little better place, however we should allow for the possibility of kernel level injection that interferes with our data collection. In my opinion interference at this level is unlikely, however I don't do DFIR much so I could easily be wrong.

In the end we're looking at a surprisingly simple ruleset. Let's create a new rules snippet called /etc/audit/rules.d/10-procmon.rules and add the lines

-a exit,always -F arch=b64 -S execve -k procmon
-a exit,always -F arch=b32 -S execve -k procmon

By using the -a we are creating a syscall rule. This rule should always create an event when a system call exits². Instead of triggering an event on every system call we instead only want to log when a program call execve(2). Lastly let's append the key value "procmon" to each log for easier searching. Since 64-bit kernels will sometimes have a different syscall entry path for each architecture, auditd wants us to explicitly define which architecture we want to monitor system calls for. If this was a 32-bit system I'd simply remove one of the lines and strip out the arch filter. Lastly, we use the -k argument to tag each log entry with a specific keyword for easier searching.

Once we've added the rules (either manually with auditctl or by restarting the auditd service) we can check up on our work by reviewing the log file. As an example I decided to simply ran the command cat 10-procmon.rules to dump the rules out to the screen. Looking through /var/log/auditd.log for our logs we can see the following.

type=SYSCALL msg=audit(1532489108.216:3721): arch=c000003e syscall=59 success=yes exit=0 a0=16169e0 a1=16116f0 a2=161ab60 a3=7ffd940300a0 items=2 ppid=10627 pid=20240 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=3 comm="cat" exe="/usr/bin/cat" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="procmon"
type=EXECVE msg=audit(1532489108.216:3721): argc=2 a0="cat" a1="10-procmon.rules"
type=SYSCALL msg=audit(1532489109.283:3722): arch=c000003e syscall=59 success=yes exit=0 a0=1f14e30 a1=1f1a110 a2=1f21d20 a3=7ffc85e6eee0 items=2 ppid=10382 pid=20241 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="grep" exe="/usr/bin/grep" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="procmon"
type=EXECVE msg=audit(1532489109.283:3722): argc=4 a0="grep" a1="--color=auto" a2="procmon" a3="/var/log/audit/audit.log"

As discussed in the introductory blog post the logs are...somewhat verbose. The key points are to notice the format of the entries. First we have a type SYSCALL with the "exe" entry of /usr/bin/cat followed immediately by a log of type EXECVE. Normally writing an event on a syscall results in a single entry that contains the process name and not much else. However, the execve call is different. Monitoring for this particular system call automatically triggers an additional log of type EXECVE. This log type also includes the string values of the function arguments as opposed to the value of the pointer (see the a0 discussion under Fields in the man page). We can see that the first entry says I ran the command cat 10-procmon.rules. As a fun aside we can also see in the second log entry where I searched for this information using the command grep procmon /var/log/audit/audit.log.