Version 3.27 of syslog-ng has brought many smaller, but useful features to us. The new Sumo Logic destination was already covered in an earlier blog. You can now also check exactly where a message arrived on a network source (IP address, port and protocol). Rewriting the facility of a syslog message was also made easy. For a complete list of new features and changes, check the release notes at https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-3.27.1
Before you begin
To test these features, you need to have syslog-ng 3.27 or later. There is a good chance that this version is not yet available as an official package for your Linux distribution. Luckily, there are third party repositories available for the most popular distros that might already carry 3.27. Check https://syslog-ng.com/3rd-party-binaries for further details. I did my tests on FreeBSD as a syslog-ng server and Linux as a client, but the examples below should work everywhere else with minimal modifications.
Where did the message arrive?
Many people stick to the KISS principle (https://en.wikipedia.org/wiki/KISS_principle) when it comes to configuring syslog-ng. In essence: have a network source combining all incoming messages from TCP and UDP connections into a single source, and process them together. While it works perfectly well in most situations, sometimes you might need to know exactly where a message arrived and process it accordingly.
Using the following source in your configuration, when log messages arrive from the same host using both TCP and UDP, they will look exactly the same:
source src { system();
udp(); tcp(port(514)); internal();
};
Here are the commands I used to generate test messages:
logger -n 172.16.167.151 -P 514 -d --rfc3164 this is a UDP test logger -n 172.16.167.151 -P 514 -T --rfc3164 this is a TCP test
When you check the logs, you can see that the only difference is the text I entered, but otherwise the logs would look identical:
May 19 12:32:21 172.16.167.141 root: this is a UDP test May 19 12:32:40 172.16.167.141 root: this is a TCP test
This is where the new DESTIP/DESTPORT/PROTO macros can come in handy. These can show you where the log messages actually arrived. Here is a configuration snippet that uses the above defined source, does some minimal filtering to lessen the noise, and stores messages in a file using a template that utilizes the new macros:
destination d_bla {
file("/var/log/bla" template("destip=$DESTIP destport=$DESTPORT proto=$PROTO message=$MSG\n"));
};
log { source(src); filter(f_notice); destination(d_bla); };
As you can see from the logs below, even if the IP address and the port are the same, the protocol is different:
destip=172.16.167.151 destport=514 proto=17 message=this is a UDP test destip=172.16.167.151 destport=514 proto=6 message=this is a TCP test
Using the new if/else syntax of syslog-ng, you can keep the convenience of a single source and still easily treat part of the logs differently when necessary. You can find a number of examples in my blog about analyzing Suricata log messages, and also a simple example below.
Rewriting the syslog facility
If you filter based on the syslog facility associated with a log message, sometimes you might need to change the facility of a log message. It can be easily done now, using the new set-facility() rewrite function of syslog-ng 3.27. The example below does not make much sense, but at least it is easy to re-create it in your own environment and use it as a starting point. In this log, statement logs from “sudo” are set to facility “mail” and stored together with the rest of your mail logs.
log {
source(src);
if (program("sudo")) {
rewrite { set-facility("mail"); };
};
filter { facility("mail"); };
destination { file("/var/log/myemail"); };
};
In the configuration snippet above, we use the default local log source, called “src” in case of the default syslog-ng configuration on FreeBSD. Next, we filter on the program name “sudo”, and rewrite the facility to “mail” when there is a match. Before writing the logs to disk, we filter on the “mail” facility. Let’s take a look at the logs of an unsuccessful sudo attempt:
# tail /var/log/myemail May 19 14:02:01 fb121 sudo[1649]: czanik : user NOT in sudoers ; TTY=pts/0 ; PWD=/usr/home/czanik ; USER=root ; COMMAND=/bin/sh May 19 14:02:01 fb121 sendmail[1652]: 04JC21JJ001652: from=root, size=223, class=0, nrcpts=1, msgid=<202005191202.04JC21JJ001652@fb121>, relay=root@localhost May 19 14:02:01 fb121 sendmail[1652]: STARTTLS=client, relay=[127.0.0.1], version=TLSv1.3, verify=FAIL, cipher=TLS_AES_256_GCM_SHA384, bits=256/256 May 19 14:02:01 fb121 sm-mta[1653]: STARTTLS=server, relay=localhost [127.0.0.1], version=TLSv1.3, verify=NO, cipher=TLS_AES_256_GCM_SHA384, bits=256/256 May 19 14:02:01 fb121 sm-mta[1653]: 04JC213F001653: from=<root@fb121>, size=474, class=0, nrcpts=1, msgid=<202005191202.04JC21JJ001652@fb121>, proto=ESMTPS, daemon=Daemon0, relay=localhost [127.0.0.1] May 19 14:02:01 fb121 sendmail[1652]: 04JC21JJ001652: to=root, ctladdr=root (0/0), delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=30223, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (04JC213F001653 Message accepted for delivery) May 19 14:02:01 fb121 sm-mta[1654]: 04JC213F001653: to=<root@fb121>, ctladdr=<root@fb121> (0/0), delay=00:00:00, xdelay=00:00:00, mailer=local, pri=30762, relay=local, dsn=2.0.0, stat=Sent
As you can see, right after the unsuccessful sudo attempt, there is also an e-mail alert about the event. Log messages from sudo are now stored together with e-mail logs.
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik.