Today I will talk about auditd, a very powerfull tool to debug anything you need in a linux platform. In this very case, I had a problem with a mail server using all IO available. ‘iostat’ command told me it was reads ops that saturate it. A mail server has many process, so using « strace » is not helpful here. I choose to use auditd.

auditd is using watch expressions to know what to report. By default, it will write to /var/log/audit/audit.log (at least on RHEL / CentOS). Take care to file size, it will grow soon 😉

For my specific problem, I wanted to know all read ops to folder /home/vpopmail/domains. Syntax to add the watch is :

auditctl -w /home/vpopmail/domains -p r

Let it play for a couple of minutes. To stop auditing, just replace -w by -W like this :

 auditctl -W /home/vpopmail/domains -p r

Then, inspect log file produced. A sample look like this :

type=CWD msg=audit(1409925299.701:2195329): cwd="/home/vpopmail/domains/0/redacted.com/sarah.vardon/Maildir"
type=PATH msg=audit(1409925299.701:2195329): item=0 name="./courierimapkeywords/.4668285.1400358789.18892.mail.redacted.com,S=2782" inode=2725095 dev=00:13 mode=0100644 ouid=89 ogid=89 rdev=00:00 nametype=NORMAL
type=SYSCALL msg=audit(1409925299.701:2195330): arch=40000003 syscall=5 success=yes exit=5 a0=90916b0 a1=0 a2=1b6 a3=0 items=1 ppid=13531 pid=7369 auid=0 uid=89 gid=89 euid=89 suid=89 fsuid=89 egid=89 sgid=89 fsgid=89 ses=229253 tty=(none) comm="imapd" exe="/usr/local/courier-imap/bin/imapd" key=(null)

You can report to this documentation for fields explanations : http://doc.opensuse.org/products/draft/SLES/SLES-security_sd_draft/cha.audit.comp.html

I wanted to see TOP user accounts used by imapd process (it was a QMail install). By default, this process chdir to userbox/Maildir. So with command line tools, I can create a TOP of these accounts :

$ cat /var/log/audit/audit.log |fgrep CWD |awk '{print $3}' |sort |uniq -c |sort -rn |head
213 cwd="/home/vpopmail/domains/0/redacted.com/sarah.dupont/Maildir"
12 cwd="/home/vpopmail/domains/redacted2.net/vincent/Maildir"
6 cwd="/home/vpopmail/domains/0/redacted.com/rose.la/Maildir"

So now I do have the top mailboxes accessed with operations done inside. If a specific account is doing too much pressure, I can inspect a specific mailbox a little further.

Conclusion : One day, I’ll move this server from Qmail to Postfix, I swear 🙂