Even if most people ask me to compare systemd-journald vs. syslog-ng, I would say that they complement each other. Systemd-journald excels at collecting local log messages, including those of various system services. The focus of syslog-ng is on central log collection and forwarding the logs to a wide variety of destinations after processing and filtering. Combining the two gives you the most flexibility.
Systemd-journald
The focus of systemd-journald (from now on: journald) is local log collection. It can collect not just logs sent by applications, but it also collects service logs: messages emitted by various services started by systemd. Previously there was no standardized and easy way of collecting these messages.
Another major difference is that instead of text files, logs are collected into a message journal, a database file for log files. It stores not just a date and a message, but many name-value pairs to provide extra context for the log message. These databases can be queried using the journalctl command.
Recently central log collection became available for journald. Central log collection is not a built-in feature of journald, you have to install some extra utilities for journal forwarding. The collected logs can be searched and viewed using journalctl.
Syslog-ng
If you are a regular reader of the syslog-ng blog, then I do not really have to introduce you to syslog-ng. For the rest: syslog-ng is one of the available syslog implementations. It has its own configuration language, allowing you to create complex message processing and filtering rules. The focus of syslog-ng is central log collection. For the last decade it has not only meant storing logs to text files at a central location, but also forwarding logs to various other destinations: SQL databases; document stores, like Elasticsearch; message queuing, like Kafka, Big Data, like Hadoop; various LaaS (logging as a service) destinations, like Sumo Logic; instant messaging systems for alerting, like Slack; and so on.
Originally, syslog-ng did all local log collection too. Nowadays on systemd Linux systems (the majority of Linux systems today) it is journald, which collects log system logs. Syslog-ng just reads the log messages from the journal.
The road to the current situation was a bumpy one. Originally, there was just a compatibility socket provided by systemd for syslog implementations to read from. Later reading from the journal directly became possible, gaining access to all the name-value pairs stored by journald. There were all kinds of stability problems and API changes. I still often hear complaints, but a lot less than even just a few years ago.
Almost all syslog-ng users utilize the system() source. Using the system() source, syslog-ng can automatically configure local system log sources. This is also a byproduct of the SysV → Systemd change. It made the change to Systemd systems easier, as syslog-ng could detect if it needs to use /dev/log or the journal as a log source, without needing to edit the configuration file.
Syslog-ng vs. journald
After a bit of history lesson, we are ready to answer the question: which software should be used, syslog-ng or journald? The short answer is both.
Mentioning journald brings up some traumatic memories for many sysadmins who used it in its first few years. However, nowadays it works a lot more reliably and can collect log messages that were next to impossible to collect before journald became available. Just think about the logs of various background services, or the additional name-value pairs journald can collect as a part of systemd.
While journald has tools to forward journal entries among hosts, they still stay journal entries. Most log analytics software expect syslog formatted messages, or some other specialized formatting. They expect to receive log entries using the syslog protocol, message queuing, various HTTP-based protocols, like Elasticsearch, and so on. This is where syslog-ng can help. Syslog-ng can read journal entries, including all the name-value pairs stored by journald. Once journal entries are read by syslog-ng, they can be further parsed, enriched, rewritten (like anonymization) and reformatted. Message parsing allows you precise filtering and message routing: discarding surplus log messages and making sure that the right messages reach the right destinations.
Syslog-ng supports many different kinds of destinations, and can forward log messages to multiple destinations at the same time. Of course you can still save logs to text files, like many decades ago, when syslog was born. You can also forward messages over the network using the syslog protocol, or message queuing (like Kafka), to Hadoop, to MongoDB, Elasticsearch, and so on. If there is no ready-to-use destination driver for something, you can extend syslog-ng not just in C, but also in Java and Python.
What is next?
Unless you use a single host, you will quickly realize that central logging is important. If you have multiple hosts and still no central log collection, check out this part of my syslog-ng tutorial about basic logging concepts: https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-101-part-2-basic-concepts
TL;DR: use journald to collect local logs, and syslog-ng to process, filter and collect logs centrally, and forward them to various destinations for further analysis.
-
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik, on Mastodon as @Pczanik@fosstodon.org.