The syslog-ng disk buffer is one of the most often used syslog-ng options to ensure message delivery. However, it is not always necessary and using the safest variant has serious performance impacts. If you utilize disk-buffer in your syslog-ng configuration, it is worth to make sure that you use a recent syslog-ng version.
From this blog, you can learn when to use the disk-buffer option, the main differences between reliable and non-reliable disk-buffer, and why is it worth to use the latest syslog-ng version.
Before you begin
The disk-buffer option was introduced many years ago, in syslog-ng version 3.8.1. If you have an older syslog-ng version, you should upgrade to this (or rather to a more recent) version. Pointers to third party repositories containing up-to-date syslog-ng packages can be found at https://www.syslog-ng.com/products/open-source-log-management/3rd-party-binaries.aspx
When (not) to use a disk-buffer
You should NOT use disk-buffer when you have a slow but otherwise reliable destination. In such cases, using any disk-buffer option will just slow the whole pipeline even more. Instead, you should rather use a larger memory-buffer, as it is much faster than a disk-buffer. You can also throttle the destination to make sure that the destination is not overloaded, and use flow-control to make sure that syslog-ng does not receive more log messages than it can forward to the slow destination. A typical example could be an Elasticsearch cluster or a database server on the localhost. Another option is to use filters to distribute logs among multiple destinations, or use load-balancing in case of Elasticsearch or Splunk.
So, when should you use disk-buffer?
-
When the destination is sometimes unavailable due to maintenance, network or other problems.
-
When there are some sudden large pikes on the source side that the destination cannot handle.
-
When using flow-control is not an option, for example when using a UDP source.
-
When you need guarantees that unsent messages are not lost.
As you can see, using disk-buffer is a lot slower than using memory-buffers. Using the disk-buffer option cannot solve performance problems. What it can do is providing guarantees against message loss when you have a sudden pike in incoming message rates and when you cannot use flow control.
Which disk-buffer to use?
There are two variants of the disk-buffer option: reliable and non-reliable. Which one to use really depends on your use-case. But no matter how scary it sounds, you will most likely need the non-reliable variant. And why?
Because non-reliable disk-buffer covers most use-cases and it is fast. The drawback is that it can lose some log messages when syslog-ng is abnormally terminated (because of a crash, power outage, sigkill, and so on). However, there is no message loss when syslog-ng is properly shut down.
Reliable disk-buffer, on the other hand, provides extra guarantees to make sure that no messages get lost even if syslog-ng is terminated abnormally. This is achieved by writing all messages to disk after processing but before sending it off to the destination. Of course, the extra guarantees come with a cost: reliable disk-buffer is slower. That said, you rarely need this level of protection – you most likely need to use reliable disk-buffer to meet some compliance requirements.
Why the latest?
The disk-buffer code in syslog-ng is constantly improving. There are regular smaller performance and reliability improvements. However, recently there was also a marked improvement in performance: about 50% for the non-reliable and about 80% for the reliable disk-buffer. Recovery from abnormal termination is also improved considerably. In short, if you utilize the disk-buffer option, you should update to one of the latest syslog-ng releases.
What is next?
In the next disk-buffer blogs, I will explain some of the disk-buffer parameters and why disk-buffer queue files can stay huge even when they are empty.
-
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik.