While testing the latest Elasticsearch release with syslog-ng, I realized that there was already a not fully documented elasticsearch-datastream() driver. Instead of fixing the docs, I reworked the elasticsearch-http() destination to support data streams.
So, what was the problem? The driver follows a different logic in multiple places than the base elasticsearch-http() destination driver. Some of the descriptions were too general, others were missing completely. You had to read the configuration file in the syslog-ng configuration library (SCL) to configure the destination properly.
While preparing for syslog-ng 4.11.0, the OpenSearch destination received a change that allows support for data streams. I applied these changes to the elasticsearch-http() destination, and did a small compatibility change along the way, so old configurations and samples from blogs work.
Before you begin
These changes will be part of the syslog-ng 4.11.0 release. If you want to test it, install one of our nightly syslog-ng builds or wait until it is officially released.
What changed?
-
The elasticsearch-datastream() destination is deprecated. It has not been removed yet, as some people still might use it, but it receives no updates or proper documentation.
-
The last Elasticsearch release requiring the type() parameter is now end of life. This parameter is still accepted by the elasticsearch-http() destination, so old configs and examples work, but it is silently discarded. If you really need Elasticsearch 6 support, then use an older syslog-ng version too...
-
The elasticsearch-http() destination now supports data streams. You can configure everything on the syslog-ng side, as with regular indices, the logic is the same. All you have to do is to add a single line to the configuration:
op_type("create")
What is next?
These changes were only tested using Elasticsearch. However, there is a growing number of applications providing an Elasticsearch compatible API for data collection. Let us know if you run into any problems! Also, let us know if you use the elasticsearch-datastream() destination.
-
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik, on Mastodon as @Pczanik@fosstodon.org.