Logging is not just syslog anymore. Still, many syslog-ng users stick to using one of the syslog protocols for log transport and flat files for log storage. While most SIEMs and log analytics tools can receive syslog messages or read them using their own agents, in most cases, you can use the http() destination of syslog-ng as well to send logs to them. You gain extreme performance and an architecture that is easier to maintain.
Some of the drivers in syslog-ng built on top of the http() destination include elasticsearch-http() destination for sending logs to Elasticsearch / OpenSearch. Many other services utilizing the Elasticsearch Bulk API exist, and there are also others using Sumo Logic, Splunk, and so on. Of course, there are some destinations where performance is not a major concern (or can even be a drawback, in fact). For example,various instant messaging services used for alerting (like Telegram or Slack) belong in this category. You can also read the API docs and write a new destination based on http() yourself.
Before you begin
Unless you use an ancient operating system, like SLES 12 or RHEL 7, the syslog-ng version bundled with your OS is good enough for everything when it comes to http() features. You need at least syslog-ng version 3.23. This version is available in the RHEL 8 EPEL repository. Other operating systems usually have more recent versions. If your OS has an older version, you can still check if there is a 3rd party repo with up-to-date packages at https://www.syslog-ng.com/products/open-source-log-management/3rd-party-binaries.aspx. You can also use these repositories if you want to get access to the very latest syslog-ng features (for example, type support or a fast MongoDB destination).
Simplifying your architecture
While many SIEMs and other log analysis tools provide their own log forwarding tools, you can greatly simplify your architecture if you use syslog-ng not only for central log collection, but also for forwarding logs to analytics tools. In most cases, you want to save all (or most) incoming log messages using syslog-ng for long-term archiving. However, you want to send only a smaller subset of log messages for further analysis. Also, each log analysis tool needs a different subset of log messages.
Of course, you can simply let the various log forwarding tools read the logs you save for long-term archiving. However, consider that in this case, you are wasting lots of resources: sending all logs using multiple applications over the network. Also, on the analytics part, you have more logs to process. Furthermore, in case of commercial applications, you have higher licensing costs, as they are often licensed based on the number of logs. You can also save various subsets of logs to files and read them, but in this case, you still waste disk space and run extra applications.
Using syslog-ng to forward logs to various SIEMs and analysis tools, you can make sure that recipients receive only the subset of logs they really need. All this without the need to use extra disk space or extra applications.
Boosting performance
Elasticsearch, Sumo Logic, Splunk all provide an HTTP-based API for log collection. Sending logs to this APIs directly doesn’t only simplify your logging architecture, but can also provide a significant performance boost. The http() destination has built-in support for multiple workers and load-balancing. What does this mean for you? First, syslog-ng can utilize multiple CPU cores (and thus, multiple network connections) to send log messages. Second, it also means that syslog-ng can send logs to multiple ingest nodes in parallel, balancing the load among multiple nodes.
These two features of the http() destination also mean simplification: a single syslog-ng node can easily feed multiple Splunk or Elasticsearch nodes without using a dedicated load-balancing application or appliance.
Note that while the tcp() destination has similar possibilities, those are more limited and not widely available. No wonder the Cribl documentation also recommends using the elasticsearch-http() destination (one of the drivers based on the http() destination) instead of using the syslog protocol: https://docs.cribl.io/stream/usecase-logging-agents/
What is next?
As HTTP is becoming a commonly used protocol for log transport, logging is a lot more than just the syslog protocol. Of course, there are some APIs that work over HTTP, but cannot be implemented using the http() destination of syslog-ng. However, more often than not, you do not even need to read API docs and create an http()-based destination yourself from scratch. Many applications and services are available through the Elasticsearch Bulk API or the Splunk HEC API. In most cases, all you have to do is to figure out the right URL to use.
-
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik, on Mastodon as @Pczanik@fosstodon.org.