Recently, one of our power users contributed OpenSearch data streams support to syslog-ng, which reminded me to also do some minimal testing on the latest OpenSearch release with syslog-ng. TL;DR: both worked just fine.
Before you begin
Adding data streams support only requires a minor change to the related configuration snippet in the syslog-ng configuration library (SCL). We normally recommend not to touch anything in those files, but it is possible to add support in earlier syslog-ng releases just by editing a file there, based on https://github.com/syslog-ng/syslog-ng/pull/5578 Of course, this will confuse your package manager unless you revert the changes before a package upgrade.
Once the syslog-ng 4.11 release is out, you just need to find an up-to-date package for your platform. Before that, you can use the nightly RPM or DEB syslog-ng builds, the nightly container image or a package built yourself for testing. Make sure that http support is enabled / installed. It is usually available in a sub-package, called syslog-ng-http or syslog-ng-mod-http. It is enabled by default in FreeBSD ports.
For my tests, I used OpenSearch 3.3.2 installed on Rocky Linux. Installing OpenSearch was a little challenging: finding a password accepted by the installer proved difficult. And of course, you have to install an 1.2GB RPM over and over again. But in the end, it worked.
Note that there are many other OpenSearch installation possibilities: containers and even a minimal download with just core features.
Testing the latest OpenSearch
Once OpenSearch and OpenSearch Dashboards were up and running, I configured syslog-ng. I used the simplest configuration possible to send all parsed name-value pairs from syslog-ng to OpenSearch. Depending on your syslog-ng configuration, either append the following configuration snippet to syslog-ng.conf or save to /etc/syslog-ng/conf.d/ with a .conf extension.
destination d_opensearch {
opensearch(
index("syslog-ng")
user("admin")
password("h#n9s3R5zF")
url("https://localhost:9200/_bulk")
template("$(format-json --scope rfc5424 --scope dot-nv-pairs
--rekey .* --shift 1 --scope nv-pairs
--exclude DATE @timestamp=${ISODATE})")
tls(peer-verify(no))
);
};
log {
source(s_sys);
destination(d_opensearch);
};
Before reloading the syslog-ng configuration, make sure that it matches your environment. The index name, password, URL or the source name for local logs all might be different.
Send a few log messages and check if they show up in OpenSearch Dashboards. If there are any problems, try starting syslog-ng in debug mode in the foreground to figure out what went wrong: wrong IP address, incorrect password, firewall issues, and so on - the possibilities are endless…
Testing OpenSearch data streams
As I do not use OpenSearch regularly, I used the web interface instead of the API to create a data stream. You can do that under Index Management. I used “myindex” as index name, and “@timestamp” as time field.
Why “@timestamp”? Check the format-json() template above or below. By default, syslog-ng uses a date format in the DATE macro, as expected by RFC3164. With default parameters OpenSearch expects ISO date format, and this configuration provides it in a name-value pair called @timestamp. The original DATE name-value pair is not stored.
Once the data stream was ready, I configured syslog-ng. As you can see, the configuration is almost the same. However, other than the name of the index, there is a small, but important difference:
destination d_opensearch {
opensearch(
index("myindex")
op_type("create")
user("admin")
password("h#n9s3R5zF")
url("https://localhost:9200/_bulk")
template("$(format-json --scope rfc5424 --scope dot-nv-pairs
--rekey .* --shift 1 --scope nv-pairs
--exclude DATE @timestamp=${ISODATE})")
tls(peer-verify(no))
);
};
log {
source(s_sys);
destination(d_opensearch);
};
There is a new parameter:
op_type("create")
By default, its value is “index”, and works perfectly when storing data to regular indexes. However, when storing logs to data streams, you must set it to “create”.
Once you reloaded the configuration and sent a few log messages, you should see them appearing on the OpenSearch dashboard.
What is next
Note that this blog only scratches the surface. It shows you that data streams and the latest OpenSearch work fine with syslog-ng. For a production environment, you need multiple OpenSearch nodes in a cluster, proper key management, user management instead of doing everything as “admin”, and so on. However, these topics are beyond the scope of the syslog-ng blog.
-
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik, on Mastodon as @Pczanik@fosstodon.org.