sudo provides users with temporary elevated privileges to perform operations. No matter what your security philosophy, sudo is more than likely enabled on your system if even for a limited number of users.
And if it is enabled, creating an audit log of exactly what was run (and who ran it) is essential to reporting. The auditd service provides this capability.
On top of this, we will add forwarding of these events to a remote syslog host which in addition to archiving, could also be used to detect suspect behavior and intrusion detection.
Install
sudo apt-get update sudo apt-get install auditd audispd-plugins -y # list current rules sudo auditctl -l
Create test user/scripts
Create a test user ‘alice’ (password=alicepass), that has full sudo privileges. Then create a simple BASH script named “/tmp/quicktest.sh” that can only be run as root/sudo.
I have a ‘usersetup.sh’ script that sets all this up in github.
wget https://raw.githubusercontent.com/fabianlee/blogcode/master/vagrant/auditd/usersetup.sh chmod ug+r+x usersetup.sh sudo usersetup.sh
Here is a full Vagrantfile if you want all this run automatically.
Add rules
Now we will add auditing rules that capture the execution events. There are many events that can be captured including: file modifications, kernel module modification, time changes, network events, etc…but we will focus just on command execution with “execve”.
sudo auditctl -a exit,always -F arch=b64 -F euid=0 -S execve -k rootcmd sudo auditctl -a exit,always -F arch=b32 -F euid=0 -S execve -k rootcmd
Adding these rules with ‘auditctl’ means they will persist only until the server is restarted, if we were creating persist rules, we would use this same format and append them to the file “/etc/audit/audit.rules”.
Get a listing of the rules now applied:
sudo auditctl -l
Which will show the two execve rules above.
Test audit of sudo on command
Let’s test these auditing rules by opening a new console and logging in as the user ‘alice’ and running a simple command using sudo.
# logged in as alice sudo grep alice /etc/passwd
Now in the other console logged in as root/sudo, run an audit search using Alice’s name or id.
ausearch -ua alice | grep passwd type=EXECVE msg=audit(1543761872.247:198): argc=3 a0="grep" a1="alice" a2="/etc/passwd"
You can see the sudo command in the a* arguments.
Test audit of sudo on script
Now let’s test the execution of a shell script using sudo. If you ran my usersetup.sh script earlier it is already there, but otherwise you should place quicktest.sh into the /tmp folder and make it executable.
This bash script reports back if the invoker is running in sudo mode and does a grep against the /etc/shadow file.
# logged in as alice sudo /tmp/quicktest.sh foo bar
Now in the other console logged in as root/sudo, run an audit search using Alice’s name or id.
ausearch -ua alice | grep shadow type=EXECVE msg=audit(1543762771.033:235): argc=4 a0="/bin/bash" a1="/tmp/quicktest.sh" a2="foo" a3="bar" type=SYSCALL msg=audit(1543762771.033:235): arch=c000003e syscall=59 success=yes exit=0 a0=5595138a2948 a1=559513890068 a2=55951389dc70 a3=5595138af000 items=3 ppid=3140 pid=3141 auid=20001 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=4 comm="quicktest.sh" exe="/bin/bash" key="rootcmd"
Several audit records of different type will be found, notice that like before the EXECVE shows the script name and parameters used. The SYSCALL shows information like parent process id and process id (3140 and 3141), uid of alice in the auid field (20001).
Notice the metadata value of ‘rootcmd’ on ‘key’ which comes from us setting “-k rootcmd” on the audit rule.
Reporting
By now you have seen that the audit records coming from ausearch are very detailed, which is great for computer consumption but not as helpful if you are looking at them manually trying to track down activity.
As a quick utility, I wrote a Python script named replayUserAudit.py available on github that will parse out the records from ausearch and produce a brief report showing what commands were run and the parameters.
./replaceUserAudit.py alice | tail -n 8
Forwarding audit events with syslog
Beyond capturing these audit events locally, security policies often dictate they be sent to a central server either for collection or analysis.
If you have not enabled syslog on Ubuntu, follow the instructions in the article I wrote here first.
By default the audit events go to the file, “/var/log/audit/audit.log”. You can forward audit events to syslog by modifying “/etc/audisp/plugins.d/syslog.conf” and setting the “active” key to “yes”. Then reload the auditctl service.
# enable syslog forwarding sudo sed -i 's/^active.*/active = yes/g' /etc/audisp/plugins.d/syslog.conf # reload service so this change is reflected sudo service auditctl reload # watch for events coming to local syslog file tail -f /var/log/syslog
Note that the non-persistent rules will survive a reload, but not a service restart. You can validate the rules using “sudo auditctl -l”.
If you tail the “/var/log/syslog” file now while executing “sudo /tmp/quicktest.sh” as the user alice, you will see the audit records sent to the local syslog file as shown below.
Dec 2 16:48:44 auditd1604 audispd: node=auditd1604 type=EXECVE msg=audit(1543769324.489:571): argc=4 a0="/bin/bash" a1="/tmp/quicktest.sh" a2="foo" a3="bar"
Now the task is to forward these events remotely. Create a file named “/etc/rsyslog.d/40-audispd.conf” with the content below.
if $programname == 'audispd' or $syslogtag == 'audispd' then {
# action(type="omfile" file="/var/log/audispd.log")
action(type="omfwd" Target="targetHost.com" Port="514" Protocol="udp")
stop
}
And do a restart of the rsyslog service.
# xenial sudo systemctl restart rsyslog # trusty sudo service rsyslog restart
From a remote host (e.g. ‘targetHost.com’) you can either enable rsyslog or use netcat to listen on 514 (sudo nc -vul 514), and you can watch the audit records come in from the source host.
Locking rules
In a real-world scenario, you want to create an immutable set of rules that cannot be tampered with or modified, or the service disabled. This can be done using the temporary rule:
sudo auditctl -e 0
Where immutability is set until the host is restarted. Or appending this rule to the “/etc/audit/audit.rules” file and restarting the service. But beware this has to be the very last rule applied.
REFERENCES
https://linux.die.net/man/8/ausearch (man page ausearch)
http://www.tothenew.com/blog/auditing-your-ubuntu-servers/
https://www.scip.ch/en/?labs.20150604 (execve for root and non-root)
https://github.com/EricGershman/auditd-examples (auditd explanations and examples)
https://blog.shichao.io/2015/04/22/auditing_user_tty_and_root_commands_with_auditd_on_ubuntu.html
https://unix.stackexchange.com/questions/108577/how-to-log-commands-within-a-sudo-su
https://serverfault.com/questions/470755/log-all-commands-run-by-admins-on-production-servers/475134#475134 (rules for 32 and 64 bit code)
https://serverfault.com/questions/473601/getting-auditd-to-record-the-original-user (original user id, pam_loginuid.so)
https://linux-audit.com/configuring-and-auditing-linux-systems-with-audit-daemon/ (audit rule watching file)
https://gist.github.com/Neo23x0/9fe88c0c5979e017a389b90fd19ddfee (example audit rules)
https://www.thegeekdiary.com/understanding-system-auditing-with-auditd/ (put watch on event with ausearch, shows different types of aureport)
https://serverfault.com/questions/473601/getting-auditd-to-record-the-original-user (auid is where original id of user is stored)
https://www.digitalocean.com/community/tutorials/how-to-write-custom-system-audit-rules-on-centos-7 (explanation of audit rule format and examples)
http://mcs.une.edu.au/doc/audit/rules/30-pci-dss-v31.rules (big list of example rules)
https://serverfault.com/questions/673279/auditd-execve-arguments-that-looks-like-encoded-data (decode hex given by ausearch with -i)
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/security_guide/sec-audit_record_types (types of audit records)
http://whmcr.com/2011/10/14/auditd-logging-all-commands/ (see comments, forward to syslog /etc/audisp/plugins.d/syslog.conf)
https://unix.stackexchange.com/questions/111460/log-every-command-typed-in-any-shell-output-from-logger-function-to-syslog-ng (/var/log/audit/audit.log or journald)
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=560281 (bug where auid is not set unless pam_loginuid.so used)
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=741546 (pam misconfig losing auid)
https://www.rsyslog.com/doc/v8-stable/configuration/modules/omfwd.html (action to forward syslog events)
NOTES
reload rules from file
sudo auditctl -R /etc/audit/audit.rules
delete all rules
sudo auditctl -D
sudo command sent to auth.log even without auditd
cat /var/log/auth.log | grep sudo | grep alice
Python3 on Xenial
python3 --version /usr/bin/python3
Python2 on Xenial
sudo apt-get install python2.7-dev python2.7 --version /usr/bin/python2.7
Private key for vagrant
cat .vagrant/machines/default/virtualbox/private_key ssh -i .vagrant/machines/default/virtualbox/private_key -p 2222 vagrant@localhost
Enabling ‘alice’ user for direct login using pki
# from inside VM sudo su - alice ssh-keygen -t rsa (creates ~/.ssh/id_rsa, id_rsa.pub) cd .ssh cp id_rsa.pub authorized_keys (public key needed on host side) chmod 700 authorized_keys cat id_rsa (will paste pvt key to remote host) # on remote host vi alice.key (paste in private key) chmod 700 alice.key (ssh will not work if file perm not set) ssh -i alice.key -p 22 alice@<host> (if vagrant look at port forwarding, port 22??+ at localhost)
vagrant show port forwarding
vboxmanage showvminfo <virtualBoxName> | grep "host port"
sudo commands can be seen in auth.log also, but can be tampered with
grep -E " sudo:\s*alice" /var/log/auth.log | grep -v "pam_unix"
remove audit logs
truncate /var/log/audit/audit.log --size 0