A long-waited feature for syslog-ng, the Kafka source, is getting ready soon. The development is still in progress, but you can already try it, and it is worth the effort. How? Using the very same tool the syslog-ng testing and release process relies on.
From this blog you can learn how to download and patch syslog-ng git sources and build packages for popular RPM and DEB Linux distributions. Once you have installable packages, comes the fun part: getting the Kafka source working.
Before you begin
I have built and installed the syslog-ng packages on a Rocky Linux 9 machine. However, package building using DBLD should work on most Linux distros where Docker, git, and make are available. DBLD stands for “Docker BuiLD”, but it works just as fine using Podman. You also need to have Kafka up and running. In my case I used an existing Kafka installation running on a FreeBSD machine.
You can postpone Kafka installation until configuring syslog-ng, but I suggest reading at least the overview page of the syslog-ng Kafka source documentation. Right now it is available at https://hofione.github.io/syslog-ng.github.io/admin-guide/060_Sources/038_Kafka/README. Once Kafka support is merged, this page will be under the source section of the official documentation at https://syslog-ng.github.io/.
Downloading and patching syslog-ng git sources
-
The code for the syslog-ng Kafka source is built on top of the very latest syslog-ng source on GitHub. Download it using git:
[root@localhost ~]# git clone https://github.com/syslog-ng/syslog-ng/ Cloning into 'syslog-ng'... remote: Enumerating objects: 131015, done. remote: Counting objects: 100% (544/544), done. remote: Compressing objects: 100% (227/227), done. remote: Total 131015 (delta 398), reused 320 (delta 317), pack-reused 130471 (from 4) Receiving objects: 100% (131015/131015), 39.69 MiB | 22.72 MiB/s, done. Resolving deltas: 100% (96685/96685), done.
-
As the development of the Kafka source is still in progress, it is not yet merged into the syslog-ng source code. It is available as a pull request at https://github.com/syslog-ng/syslog-ng/pull/5564. Luckily, if you append .patch to this URL, GitHub turns this into a patch, which you can easily apply.
Note: The base URL of .patch file will be different. Use the wget command to download it:
wget https://patch-diff.githubusercontent.com/raw/syslog-ng/syslog-ng/pull/5564.patch
-
When you changed to the syslog-ng directory, use the git command to apply this patch:
git am ../5564.patch
Building syslog-ng using DBLD
-
Running the following command without any parameters gives you some information how to use DBLD and which Linux distributions on which platforms are supported:
dbld/rules
If a required container is missing from your system, it is downloaded automatically. However, you can also build it yourself, locally.
-
It is not strictly necessary to do this as a separate step, but you better know that source tgz generation from git sources is available as a separate step. If you want to build syslog-ng from patched sources but on a target system not supported by DBLD, you can still use it to generate a proper source release tgz for you:
dbld/rules tarball
-
This step is also done automatically, if you build syslog-ng for a given Linux distro, and the source tgz is not yet available. At the end of the process, you should see the source tgz in the dbld/build directory:
[root@localhost syslog-ng]# ls -l dbld/build total 6740 drwxr-xr-x 19 root root 161 Dec 1 12:59 ccache drwxr-xr-x 12 root root 4096 Dec 1 13:00 dist-build -rw-r--r-- 1 root root 6895622 Dec 1 13:00 syslog-ng-4.10.2.194.gc132bc7.tar.gz
-
Once the source tgz is built, you can build syslog-ng packages from the patched sources for your Linux distro of choice. Look for the “supported OSs” line in the dbld/build output.
I chose rocky-9 as platform, but choosing almalinux-9 should work just as fine. Regardless of the base platform, the resulting binaries run just fine on any compatible Linux distros.
dbld/rules rpm-rocky-9
or
dbld/rules rpm-almalinux-9
The resulting packages are available under the dbld/build/rocky-9 and dbld/build/almalinux-9 directories.
Installing the resulting packages
Installing the resulting syslog-ng packages will be slightly different, depending on the Linux distribution you use. Make sure that necessary dependencies are available. In case of the RHEL and compatible distros, make sure that the EPEL repository and its dependencies are available.
-
Install the necessary packages with the following command, once I changed to the dbld/build/almalinux-9 directory:
[root@localhost almalinux-9]# dnf install syslog-ng-4.10.2.194.gc132bc7-2+20251201T120932.x86_64.rpm syslog-ng-kafka-4.10.2.194.gc132bc7-2+20251201T120932.x86_64.rpm syslog-ng-logrotate-4.10.2.194.gc132bc7-2+20251201T120932.x86_64.rpm
Last metadata expiration check: 1:02:23 ago on Mon 01 Dec 2025 12:35:50 PM CET.
Dependencies resolved.
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
syslog-ng x86_64 4.10.2.194.gc132bc7-2+20251201T120932 @commandline 1.0 M
syslog-ng-kafka
x86_64 4.10.2.194.gc132bc7-2+20251201T120932 @commandline 50 k
syslog-ng-logrotate
x86_64 4.10.2.194.gc132bc7-2+20251201T120932 @commandline 8.4 k
Installing dependencies:
ivykis x86_64 0.42.4-7.el9 epel 46 k
librdkafka x86_64 1.6.1-102.el9 appstream 662 k
Transaction Summary
================================================================================
Install 5 Packages
Total size: 1.8 M
Total download size: 707 k
Installed size: 6.4 M
Is this ok [y/N]:
Note:The exact package names and the installed dependencies might differ on your system.
-
Check that Kafka support is enabled by running the syslog-ng version check command. You should be able to see Kafka listed among the available modules, just like here:
[root@localhost almalinux-9]# syslog-ng -V syslog-ng 4 (4.10.2.194.gc132bc7) Config version: 4.2 Installer-Version: 4.10.2.194.gc132bc7 Revision: Compile-Date: Oct 14 2025 00:00:00 Module-Directory: /usr/lib64/syslog-ng Module-Path: /usr/lib64/syslog-ng Include-Path: /usr/share/syslog-ng/include Available-Modules: system-source,tags-parser,tfgetent,timestamp,xml,kafka,add-contextual-data,affile,afprog,afsocket,afstomp,afuser,appmodel,basicfuncs,cef,confgen,correlation,cryptofuncs,csvparser,disk-buffer,examples,graphite,hook-commands,json-plugin,kvformat,linux-kmsg-format,map-value-pairs,metrics-probe,pacctformat,pseudofile,rate-limit-filter,regexp-parser,sdjournal,stardate,syslogformat Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Stackdump: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: off Enable-Linux-Caps: on Enable-Systemd: on
Testing
Note: Kafka heavily relies on host names. If you do not have a working DNS, make sure that at least the hosts files have proper entries pointing to your Kafka instance.
-
Depending on your syslog-ng configuration, append the following configuration snippet to your syslog-ng.conf, or save it with a .conf extension under the /etc/syslog-ng/conf.d directory:
source s_kafka {
kafka(
bootstrap-servers("172.16.167.162:9092")
topic("test-topic" => "-1")
);
};
destination d_file {
file("/var/log/fromkafka");
};
log {
source(s_kafka);
destination(d_file);
};
Make sure that the bootstrap-servers() option points at your Kafka instance, and that the topic within topic() exists. The -1 parameter here means that syslog-ng is reading from all partitions. Read the Kafka source documentation for more details.
Note: It is recommended to start syslog-ng with debug options in the foreground while doing your first steps with the syslog-ng Kafka source. This way you can easily spot connection problems, failures related to DNS / host names, and so on.
syslog-ng -Fvde
-
Once the connection works, post data to the topic configured in syslog-ng. You can use the kafka-console-producer.sh for this. If you started syslog-ng in a terminal window, and sent “this is a test” as test message, you should see something similar:
[2025-12-01T15:06:15.240367] Incoming log message; input='is a test', driver='s_kafka#0', worker_index='1', msg='0x7f8580005150', rcptid='0' [2025-12-01T15:06:24.525387] Initializing destination file writer; template='/var/log/fromkafka', filename='/var/log/fromkafka', symlink_as='(null)' [2025-12-01T15:06:24.526275] Outgoing message; message='Dec 1 15:06:15 localhost this is a test\x0a'
What is next?
We plan to add more features, which add even more flexibility and scalability to the syslog-ng Kafka source when integrating syslog-ng Kafka on a large scale. However, the code is already stable enough to get started with experimenting, and initial performance results are also very promising.
If you test the syslog-ng Kafka source, let us know your experiences! Posting issues on GitHub helps us to ensure that syslog-ng works with the broadest possible versions of Kafka, librdkafka (the client used by syslog-ng to reach Kafka) and configuration combinations. But of course, we are also very happy about success reports!
-
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik, on Mastodon as @Pczanik@fosstodon.org.