The syslog-ng disk buffer is one of the most often used syslog-ng options to ensure message delivery. However, it is not always necessary and using the safest variant has serious performance impacts. If you utilize disk-buffer in your syslog-ng configuration, it is worth to make sure that you use a recent syslog-ng version.
From this blog, you can learn when to use the disk-buffer option, the main differences between reliable and non-reliable disk-buffer, and why is it worth to use the latest syslog-ng version.
Before you begin
The disk-buffer option was introduced many years ago, in syslog-ng version 3.8.1. If you have an older syslog-ng version, you should upgrade to this (or rather to a more recent) version. Pointers to third party repositories containing up-to-date syslog-ng packages can be found at https://www.syslog-ng.com/products/open-source-log-management/3rd-party-binaries.aspx
When (not) to use a disk-buffer
You should NOT use disk-buffer when you have a slow but otherwise reliable destination. In such cases, using any disk-buffer option will just slow the whole pipeline even more. Instead, you should rather use a larger memory-buffer, as it is much faster than a disk-buffer. You can also throttle the destination to make sure that the destination is not overloaded, and use flow-control to make sure that syslog-ng does not receive more log messages than it can forward to the slow destination. A typical example could be an Elasticsearch cluster or a database server on the localhost. Another option is to use filters to distribute logs among multiple destinations, or use load-balancing in case of Elasticsearch or Splunk.
So, when should you use disk-buffer?
When the destination is sometimes unavailable due to maintenance, network or other problems.
When there are some sudden large pikes on the source side that the destination cannot handle.
When using flow-control is not an option, for example when using a UDP source.
When you need guarantees that unsent messages are not lost.
As you can see, using disk-buffer is a lot slower than using memory-buffers. Using the disk-buffer option cannot solve performance problems. What it can do is providing guarantees against message loss when you have a sudden pike in incoming message rates and when you cannot use flow control.
Which disk-buffer to use?
There are two variants of the disk-buffer option: reliable and non-reliable. Which one to use really depends on your use-case. But no matter how scary it sounds, you will most likely need the non-reliable variant. And why?
Because non-reliable disk-buffer covers most use-cases and it is fast. The drawback is that it can lose some log messages when syslog-ng is abnormally terminated (because of a crash, power outage, sigkill, and so on). However, there is no message loss when syslog-ng is properly shut down.
Reliable disk-buffer, on the other hand, provides extra guarantees to make sure that no messages get lost even if syslog-ng is terminated abnormally. This is achieved by writing all messages to disk after processing but before sending it off to the destination. Of course, the extra guarantees come with a cost: reliable disk-buffer is slower. That said, you rarely need this level of protection – you most likely need to use reliable disk-buffer to meet some compliance requirements.
Why the latest?
The disk-buffer code in syslog-ng is constantly improving. There are regular smaller performance and reliability improvements. However, recently there was also a marked improvement in performance: about 50% for the non-reliable and about 80% for the reliable disk-buffer. Recovery from abnormal termination is also improved considerably. In short, if you utilize the disk-buffer option, you should update to one of the latest syslog-ng releases.
What is next?
In the next disk-buffer blogs, I will explain some of the disk-buffer parameters and why disk-buffer queue files can stay huge even when they are empty.
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik.