A few weeks ago, I posted about sngbench, a shell script to measure syslog-ng performance. The performance of syslog-ng is influenced by many factors, including the hardware and OS it runs on, and syslog-ng itself. This blog summarizes some of my findings using the script.
Before you begin
The sngbench.sh script was born several years ago, but we made it public only last month. This is because it was ugly as hell and difficult to use, but now, after a refactor, it is ready for public use. It is a synthetic benchmark, so while it is not fully realistic, it is good to measure peak performance, and compare how your choice of hardware, OS or syslog-ng version and configuration influences syslog-ng performance.
You can learn more about the script here:
In this blog, I focus on what I found while using the script.
Hardware
The performance of syslog-ng is mostly affected by hardware, so do not expect the same performance from a Raspberry Pi and an AMD EPYC server. While syslog-ng scales well on multi-core systems, it still prefers fewer, stronger CPU cores. The CPU cache size also matters, with more cache resulting in better performance. Entry level AMD Ryzen CPUs with large caches can easily beat just a few years older server CPUs with smaller caches.
Turning off SMT (Simultaneous Multithreading, which is called Hyper-Threading in Intel terminology) can have a significant effect. Turning off HT on an Intel CPU resulted in a 5-15% performance increase depending on the test. Moreover, on POWER, it could even result in a 300% performance increase in some cases.
I did not do any in-depth testing, but virtualization can significantly slow down syslog-ng. Depending on the technology, it could result even in a 20-40% performance loss.
Most of my tests focused on the CPU, so I did short test runs where logs were mostly stored in the RAM instead of a real storage. Needless to say, in the latter case, an SSD is faster than an HDD. Or, in case of a Raspberry Pi, a USB stick is faster than storing logs on an SD card.
Operating system
Another important factor is the operating system, as the OS and its many components can have a significant performance impact. Different operating systems or different versions of the same operating system can produce completely different performance results on the same hardware. And the tuning possibilities of an OS are nearly endless. Just think of the scheduler used, the various cache sizes configured for networking, the various file system types, and so on.
I did not experiment much with OS tuning, as there are too many possibilities. Using the same hardware and syslog-ng configuration, configuration changes in the OS could result in a performance difference of up to 50% in some of the tests. Interestingly enough, latest does not always mean greatest. Often, the more recent the OS, the more optimizations glibc, glib2 and other dependencies have. However, in some cases, new OS versions can introduce system tuning changes from server tasks to interactive desktop use that can suddenly drop syslog-ng performance significantly.
Also, some optimizations are only available on Linux, like speeding up UDP as discussed last week.
Syslog-ng version
The version of syslog-ng can also have an impact, but compared to other factors, it is usually small. Of course newer versions can sometimes introduce significant performance improvements in various parts of syslog-ng: just think of making the http() driver threaded or adding bulk support to the MongoDB destination. However, depending on the tests, there was also a minor but consistent performance decrease between syslog-ng 3.35 and 4.2 when using the tcp() source. So the performance differences also depend on the part of syslog-ng you utilize.
Syslog-ng configuration
This is probably the most underestimated part of performance differences. The sngbench script comes with two different configurations. One is almost empty, configured for performance, while the other is a more realistic one.
Even if you have a top-of-the-line hardware and a well-tuned configuration, syslog-ng does not scale much with a single log source (except for the UDP case I mentioned earlier). With multiple CPU cores and multiple clients, most parts of syslog-ng scale really well, but not all of them. This is why it is important to test not just with a single client but multiple clients as well (or with multiple threads, in case of loggen). You can check which parts of your configuration slows syslog-ng down. With a bit of luck, you can reorganize your configuration so that it is used only by a small proportion of log messages.
You can easily extend sngbench with your own configurations. The readme of sngbench describes how you can add / replace configurations and how you can change loggen parameters:
https://github.com/czanik/sngbench
This way, you can check UDP performance, do longer test runs to take disk speed into account, add an http() or other destination instead of a file, parse messages with PatternDB or other parsers, and so on. The possibilities are endless.
What is next?
In this blog, I shared some of the experiences I gained while using sngbench. However, these are just some generic guidelines. Hardware and hardware settings, operating system and OS settings, syslog-ng version and syslog-ng configurations can all have significant effects on syslog-ng performance. As combinations are practically endless, you should check your own environment. Sngbench can help you in the process.
-
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik, on Mastodon as @Pczanik@fosstodon.org.