Alerting on sudo events using syslog-ng

Why use syslog-ng to alert on sudo events? At the moment, alerting in sudo is limited to E-mail. Using syslog-ng, however, you can send alerts (more precisely, selected logs) to a wide variety of destinations. Logs from sudo are automatically parsed by recent (3.13+) syslog-ng releases, enabling fine-grained alerting. There is a lot of hype around our new Slack destination, so that is what I’ll show here. Naturally, there are many others available as well, including Telegram and, of course, good old E-mail. If something is not yet directly supported by syslog-ng, you can often utilize an HTTP API or write some glue code in Python.

From this blog post you can learn how to build up a syslog-ng configuration step by step and how to use different filters to make sure that you only receive logs (i.e. alerts) that are truly relevant for you.

Before you begin

You need:

• A syslog-ng version 3.19 or later: If your platform carries an earlier version, check https://syslog-ng.com/3rd-party-binaries for pointers to bring packages up to date.

• JSON and HTTP support enabled in syslog-ng: As a rule, JSON and HTTP support is not part of the main syslog-ng package, but available in sub-packages (like syslog-ng-http or syslog-ng-curl).

• SCL enabled in syslog-ng: The syslog-ng Slack destination is a ready-to-use configuration block in the syslog-ng configuration library (SCL). Make sure that it is enabled in your syslog-ng.conf (it is there in the default configuration):

@include "scl.conf"

• A Slack workspace and a channel in that workspace. If you do not have Slack configured yet, check https://www.syslog-ng.com/community/b/blog/posts/send-your-log-messages-to-slack for information about how to set it up.

Getting started

When I build a configuration, I usually do multiple iterations. First, I log everything locally, then I add filtering. Finally, I also add network destinations. Here you can see the iterations I did while creating Slack-based alerting for sudo.

First, I simply collect all sudo related logs into a separate log file. The only fancy thing here is the destination file, which is JSON-formatted. All sudo logs are parsed by syslog-ng by default, and this way you can see the parsed values in the log file:

filter f_sudo {program(sudo)};

destination d_test {
  file("/var/log/sudo.json"
  template("$(format-json --scope nv_pairs --scope dot_nv_pairs --scope rfc5424)\n\n"));
};

log {
    source(s_sys);
    filter(f_sudo);
    destination(d_test);
};

If you take a look at the log path, you can see a source not defined above. Here I use the local log source as defined in the syslog-ng configuration on Fedora / RHEL / CentOS. You might need to change this value to match your environment.

The filter named f_sudo matches all messages where the program name is sudo.

The destination called d_test saves log messages in JSON format. Since name-value pairs are not saved by the default file destination template, this template is necessary.

You will see similar lines in your /var/log/sudo.json file:

{"_sudo":{"USER":"root","TTY":"pts/0","SUBJECT":"czanik","PWD":"/home/czanik","COMMAND":"/bin/joe"},"_journald":{"_UID":"0","_TRANSPORT":"syslog","_SYSTEMD_UNIT":"session-1.scope","_SYSTEMD_SLICE":"user-0.slice","_SYSTEMD_SESSION":"1","_SYSTEMD_OWNER_UID":"0","_SYSTEMD_CGROUP":"/user.slice/user-0.slice/session-1.scope","_SOURCE_REALTIME_TIMESTAMP":"1556038664933703","_PID":"10642","_MACHINE_ID":"0bfc080f67eb45d5add6422ef393b389","_HOSTNAME":"centos7.localdomain","_GID":"1000","_EXE":"/usr/bin/sudo","_COMM":"sudo","_CMDLINE":"sudo joe","_CAP_EFFECTIVE":"1fffffffff","_BOOT_ID":"f2f86de3faa0436fa8b947bc6176b853","_AUDIT_SESSION":"1","_AUDIT_LOGINUID":"0","SYSLOG_IDENTIFIER":"sudo","SYSLOG_FACILITY":"10","PRIORITY":"5","MESSAGE":"  czanik : TTY=pts/0 ; PWD=/home/czanik ; USER=root ; COMMAND=/bin/joe"},"_app":{"name":"sudo"},"SOURCE":"s_sys","PROGRAM":"sudo","PRIORITY":"notice","PID":"10642","MESSAGE":"  czanik : TTY=pts/0 ; PWD=/home/czanik ; USER=root ; COMMAND=/bin/joe","HOST_FROM":"centos7","HOST":"centos7.localdomain","FACILITY":"authpriv","DATE":"Apr 23 18:57:44","0":"\"czanik :\""}

Filtering sudo logs

As a next step, we add some filtering. With if/else statements introduced, it became a lot simpler in recent syslog-ng versions. Here we filter on the user name and if there is a match, we store the results into a separate file. This file will contain what later will become alerts sent to Slack.

filter f_sudo {program(sudo)};

destination d_test {
  file("/var/log/sudo.json"
  template("$(format-json --scope nv_pairs --scope dot_nv_pairs --scope rfc5424)\n\n"));
};

log {
    source(s_sys);
    filter(f_sudo);
    if (match("czanik" value(".sudo.SUBJECT"))) {
        destination { file("/var/log/sudo_filtered"); };
    };
    destination(d_test);
};

This way, any commands ran through sudo by user czanik will be logged into the /var/log/sudo_filtered file.

Sending logs to Slack

As a last step, we add the parts sending log messages (i.e. alerts) to Slack. The minimal configuration for Slack is simply the URL receiving the messages.

filter f_sudo {program(sudo)};

destination d_test {
  file("/var/log/sudo.json"
  template("$(format-json --scope nv_pairs --scope dot_nv_pairs --scope rfc5424)\n\n"));
};

destination d_slack {
  slack(
    hook-url("https://hooks.slack.com/services/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")
  );
};

log {
    source(s_sys);
    filter(f_sudo);
    if (match("czanik" value(".sudo.SUBJECT"))) {
        destination { file("/var/log/sudo_filtered"); };
        destination(d_slack);
    };
    destination(d_test);
};

Once receiving alerts to Slack works as expected, you can safely comment out the file destinations in your log path.

If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or you can even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/balabit/syslog-ng. On Twitter, I am available as @PCzanik.