No matter how awkward you feel when you hear about UDP syslog in the age of encrypted TCP connections, UDP syslog is here to stay in some special cases. The scalability issues of UDP log collection were first addressed in syslog-ng Open Source Edition (OSE) (the so-reuseport() parameter), and later a more advanced solution arrived to syslog-ng Premium Edition (PE) (the udp-balancer() source). The good news is that a new, open-source implementation is now available as part of syslog-ng 4.2.0.
Background
Two years ago, I had a longer post on UDP log collection: https://www.syslog-ng.com/community/b/blog/posts/using-the-udp-balancer-source-of-syslog-ng-pe
Here I’m providing you a quick recap, read the original blog for more details.
As much as we want to forget UDP syslog, it is here to stay. In some cases, there are well-defined technical reasons for it: networking devices or industrial automation cannot risk being slowed down, or blocked by TCP syslog. Logging is important, but the availability of service is more important. The lossy but non-blocking UDP protocol is perfect for the job. Of course, sometimes the reasons for using UDP syslog are not technical: large enterprises are slow to change.
Collecting UDP syslog messages has some scalability problems. By default, syslog-ng collects log messages from a single UDP port using a single thread. If you have a multi-core machine, it means that only a single core is used.
With a low message rate, of course, there are no problems. If there are occasional pikes in message rate, you can lower message loss by increasing memory buffers both in kernel and in syslog-ng. If the message rate is higher, but if messages come in a relatively even distribution from a larger number of hosts, you can use su-reuseport(). This way, syslog-ng uses multiple threads, and “connections” (source-destination pairs) are distributed among threads.
The real problems start when you have one or just a few high message rate UDP log sources, as even with so-reuseport(), they are directed to the same thread. The udp-balancer() source of PE distributes incoming messages randomly among syslog-ng threads. It uses the eBPF infrastructure of the Linux kernel and works on all platforms supported by PE, except for RHEL 7.
What is new?
Version 4.2.0 of syslog-ng OSE now includes an alternative eBPF support implementation. See the release notes at: https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-4.2.0 The end result is the same: syslog-ng OSE and PE can now both handle high UDP syslog load from a single log source. However, both the code and configuration are different.
As the Release Notes says, this is advanced functionality:
NOTE: The ebpf() plugin is considered advanced usage so its compilation is disabled by default. Please don't use it unless all other avenues of configuration solutions are already tried. You will need a special toolchain and a recent kernel version to compile and run eBPF programs.
Of course, I took this as a challenge :-) My openSUSE and Fedora 38 packages now support eBPF, and I gave it a quick try in a test environment. Everything worked as expected: message loss became minimal and load was distributed evenly among CPU cores.
What are the differences?
From the user’s point of view, there are two main differences between the two implementations. The first difference is configuration. As you could see in my blog about the udp-balancer() source, syslog-ng PE hides implementation details from the user. You use the udp-balancer() source instead of udp() and everything is configured in the background. The implementation in syslog-ng OSE extends the udp() source when you use the so-reuseport() option:
source s_udp {
udp(so-reuseport(yes) port(2000) persist-name("udp1")
ebpf(reuseport(sockets(4)))
);
udp(so-reuseport(yes) port(2000) persist-name("udp2"));
udp(so-reuseport(yes) port(2000) persist-name("udp3"));
udp(so-reuseport(yes) port(2000) persist-name("udp4"));
};
The other difference is in requirements. Previously, I mentioned that I compiled eBPF support for the latest Fedora and openSUSE versions. RHEL (and compatible) is the main platform for syslog-ng, but even RHEL 9 proved to be too “old” for the eBPF implementation in syslog-ng 4.2.0. Which means that for the coming years you have to use syslog-ng PE if you need to collect UDP syslog at a high message rate on an enterprise Linux distribution.
How can you try it?
If you want to try eBPF support in syslog-ng OSE, you either have to compile syslog-ng yourself, as described in the syslog-ng 4.2.0 Release Notes, or use one of my RPM packages. They are available for openSUSE 15.4 and 15.5, and Fedora 38.
Based on the configuration sample available in the release notes, I used this simple configuration for testing:
source s_udp {
udp(so-reuseport(yes) port(2000) persist-name("udp1")
ebpf(reuseport(sockets(4)))
);
udp(so-reuseport(yes) port(2000) persist-name("udp2"));
udp(so-reuseport(yes) port(2000) persist-name("udp3"));
udp(so-reuseport(yes) port(2000) persist-name("udp4"));
};
destination d_fromudp {
file("/var/log/fromudp");
};
log {source(s_udp); destination(d_fromudp); };
Of course, in a production environment, you also want to increase buffer sizes, as described in my previous blog about UDP syslog.
Note that to run syslog-ng with ebpf() support, you have to run syslog-ng as root and disable Linux capabilities support (if it was enabled at compilation time). You can disable capabilities support using the following syslog-ng command line option:
--no-caps
For generating UDP syslog messages I used loggen, the bundled testing and benchmarking tool of syslog-ng:
loggen -r 30000000 -i -D localhost 2000
When the configuration line about ebpf() was commented out, syslog-ng only used a single core. When ebpf() was enabled, then the load was distributed among multiple CPU cores. Message loss also practically disappeared. Without ebpf(), message loss was close to 50%, with ebpf() support enabled, it was well below 1%.
What is next?
If you need to collect high message rate UDP syslog messages, you can use the new ebpf() plugin in cutting-edge Linux distributions. Or you can use syslog-ng PE (https://www.syslog-ng.com/products/log-management-software/) on one of the supported enterprise Linux distributions.
-
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik, on Mastodon as @Pczanik@fosstodon.org.