Version 4.10 of syslog-ng introduced file size-based log rotation. Thanks to this, storage space is no longer filled with logs with the risk that you might not see older logs if the message rate is higher than expected.
Before you begin
You need at least version 4.10 of syslog-ng to use this feature. If it is not yet available in your operating system of choice, check https://www.syslog-ng.com/products/open-source-log-management/3rd-party-binaries.aspx if there are third-party syslog-ng repositories available for your OS.
Personally, I love this feature. I already had my /var partition filling up due to some applications sending too many logs. However, file size-based log rotation comes with a risk: your older logs might be already deleted by the time you want to check them. Turning on debug logging for an app can easily generate 10x more logs than normally. So, you must decide what is the greater risk in your environment: losing new messages (due to disk fill-up) or losing old messages (due to rotation).
Configuring syslog-ng
Append this to syslog-ng.conf or create a new configuration snippet under the /etc/syslog-ng/conf.d/ directory, if your syslog-ng configuration is set to include this directory:
source s_net {
tcp(port(514));
};
destination d_file1 {
file("/tmp/logfile_$HOST.log", logrotate(enable(yes), size(5MB), rotations(10)));
};
destination d_file2 {
file("/tmp/logfile_simple.log", logrotate(enable(yes), size(1GB), rotations(2)));
};
log {
source(s_net);
destination(d_file1);
destination(d_file2);
};
This configuration expects logs to arrive using the legacy syslog protocol and saves them to two sets of rotated log files.
File size-based log rotation is enabled using the logrotate() option in the file() destination. You can set the file size limit in KB, MB or even GB. Note that the actual number of files, and thus the maximum space used, is the number of rotations set plus the actual log file. So, for example, if you keep 9 log files, each with a size of 1 GB, then the maximum size will be 10 GB.
Danger: The first configuration uses the HOST macro in the file name. This works perfectly well, but it also multiplies the number of rotated files. Because of this, this works well if there is a limited number of values for the macro, but it can cause surprises if the macro values are unpredictable.
Testing
You can use loggen to generate some logs quickly. This will also remind you about your preferred message loss method. Unless you set some extremely large file sizes and rotation numbers, your first log messages will rotate away quickly. I used loggen from localhost and another host to generate log messages. Replace localhost with the name or IP address of your syslog-ng host when sending logs remotely:
loggen -i -S -r 1000000 --active-connections=5 -I 120 localhost 514
Note that firewall or SELinux rules might prevent remote loggen instances from delivering log messages to your syslog-ng server.
You can see a listing of the resulting log files below. Logs from the two hosts are rotated separately.
leap156:~ # ls -l /tmp/logfile_* -rw-r----- 1 root root 3175903 Sep 25 13:08 /tmp/logfile_172.16.167.1.log -rw-r----- 1 root root 5000171 Sep 25 13:08 /tmp/logfile_172.16.167.1.log.1 -rw-r----- 1 root root 5000171 Sep 25 13:08 /tmp/logfile_172.16.167.1.log.10 -rw-r----- 1 root root 5000171 Sep 25 13:08 /tmp/logfile_172.16.167.1.log.2 -rw-r----- 1 root root 5000171 Sep 25 13:08 /tmp/logfile_172.16.167.1.log.3 -rw-r----- 1 root root 5000171 Sep 25 13:08 /tmp/logfile_172.16.167.1.log.4 -rw-r----- 1 root root 5000171 Sep 25 13:08 /tmp/logfile_172.16.167.1.log.5 -rw-r----- 1 root root 5000171 Sep 25 13:08 /tmp/logfile_172.16.167.1.log.6 -rw-r----- 1 root root 5000171 Sep 25 13:08 /tmp/logfile_172.16.167.1.log.7 -rw-r----- 1 root root 5000171 Sep 25 13:08 /tmp/logfile_172.16.167.1.log.8 -rw-r----- 1 root root 5000171 Sep 25 13:08 /tmp/logfile_172.16.167.1.log.9 -rw-r----- 1 root root 1491968 Sep 25 13:08 /tmp/logfile_localhost.log -rw-r----- 1 root root 5000176 Sep 25 13:08 /tmp/logfile_localhost.log.1 -rw-r----- 1 root root 5000176 Sep 25 13:08 /tmp/logfile_localhost.log.10 -rw-r----- 1 root root 5000176 Sep 25 13:08 /tmp/logfile_localhost.log.2 -rw-r----- 1 root root 5000176 Sep 25 13:08 /tmp/logfile_localhost.log.3 -rw-r----- 1 root root 5000176 Sep 25 13:08 /tmp/logfile_localhost.log.4 -rw-r----- 1 root root 5000176 Sep 25 13:08 /tmp/logfile_localhost.log.5 -rw-r----- 1 root root 5000176 Sep 25 13:08 /tmp/logfile_localhost.log.6 -rw-r----- 1 root root 5000176 Sep 25 13:08 /tmp/logfile_localhost.log.7 -rw-r----- 1 root root 5000176 Sep 25 13:08 /tmp/logfile_localhost.log.8 -rw-r----- 1 root root 5000176 Sep 25 13:08 /tmp/logfile_localhost.log.9 -rw-r----- 1 root root 294816448 Sep 25 13:08 /tmp/logfile_simple.log -rw-r----- 1 root root 1000000105 Sep 25 13:08 /tmp/logfile_simple.log.1 -rw-r----- 1 root root 1000000048 Sep 25 13:08 /tmp/logfile_simple.log.2
What is next?
We would like to hear your feedback! Share your experiences with this new feature at https://github.com/syslog-ng/syslog-ng/discussions
-
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik, on Mastodon as @Pczanik@fosstodon.org.