Actions

Tags

Peter Czanik
Using syslog-ng with Elasticsearch 9.5

Recently, I heard complaints within the syslog-ng community that using Elasticsearch is not that easy anymore. I installed Elasticsearch 9.5 with Kibana to verify these claims.

Before you begin

For my tests, I installed Elasticsearch 9.5, Kibana 9.5 and syslog-ng 4.12 in a Rocky Linux 10 virtual machine. I followed the lazy approach and disabled the firewall instead of opening port 5601 to Kibana. After seeing some strange errors, I also disabled SELinux. And while this was not the root cause, I left it disabled anyway. Another lazy thing I did was using the “elastic” superuser both for Kibana and for sending logs.

Installing Elasticsearch & Kibana

I used the rpm repo method, as described at https://www.elastic.co/docs/deploy-manage/deploy/self-managed/install-elasticsearch-with-rpm#rpm-repo After installing the packages, I carefully noted the printed information, as they include useful commands and the password of the “elastic” superuser:

--------------------------- Security autoconfiguration information ------------------------------

Authentication and authorization are enabled.
TLS for the transport and HTTP layers is enabled and configured.

The generated password for the elastic built-in superuser is : ssA657Ut1aA=GtdpzA9P

If this node should join an existing cluster, you can reconfigure this with
'/usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token <token-here>'
after creating an enrollment token on your existing cluster.

You can complete the following actions at any time:

Reset the password of the elastic built-in superuser with
'/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic'.

Generate an enrollment token for Kibana instances with
 '/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana'.

Generate an enrollment token for Elasticsearch nodes with
'/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node'.

-------------------------------------------------------------------------------------------------
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable elasticsearch.service
### You can start elasticsearch service by executing
 sudo systemctl start elasticsearch.service

Once I started Elasticsearch, I also started Kibana. When I opened Kibana, it asked me to create an enrollment token, and paste the result in a form. Unfortunately, that attempt failed: running the command to create an enrollment token for Kibana resulted in an error message:

/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana

Error:

ERROR: Failed to determine the health of the cluster. Unexpected http status [503], with exit code 65

Disabling the firewall (and later SELinux as well) did not solve the problem. I checked the documentation again, and even though it said that no further steps were needed for single node installations, I did all the changes necessary for clusters in the elasticsearch.yml file. And then, suddenly, token generation started working...

Configuring syslog-ng

Next, I configured syslog-ng. I created a new configuration snippet in the /etc/syslog-ng/conf.d/ directory with the following content:

destination d_elasticsearch_https {
    elasticsearch-http(
        url("https://localhost:9200/_bulk")
        index("syslog-ng")
        user("elastic")
        password("ssA657Ut1aA=GtdpzA9P")
        template("$(format-json --scope rfc5424 --scope dot-nv-pairs
        --rekey .* --shift 1 --scope nv-pairs
        --exclude DATE @timestamp=${ISODATE})")
        tls(
            peer-verify(no)
        )
    );
};

log {
  source(s_sys);
  destination(d_elasticsearch_https);
};

It adds an Elasticsearch destination and sends all local log messages to this destination. Here is a short explanation for the Elasticsearch destination:

  • the URL points to localhost and uses HTTPS to reach Elasticsearch

  • the index name is syslog-ng, but you can add macros, like ${YEAR}${MONTH}${DAY} to have index names based on the current date

  • I used the “elastic” superuser, but this is not recommended for production

  • defining a template is not necessary, if you only store syslog messages without the parsed values

  • peer verification for TLS connections is turned off, which means that connections are encrypted, but peers are not verified.

Restart syslog-ng for the configuration to take effect.

Configuring Kibana

When opening “Discover” in Kibana, there is already a pre-defined Data view for logs. However, it is not much help, as its index names are fixed to LogStash and FileBeat and they cannot be changed. The solution is to create a new data view, where you can already add “syslog-ng” as an index name. After that, you are ready to explore your log messages.

Testing

If you can see the syslog-ng index, it already means that syslog-ng and Elasticsearch are working together. You can also check the support for extra name-value pairs without much effort. Just run something using sudo, and syslog-ng will automatically parse its log messages and send name-value pairs to Elasticsearch.

What is next?

Each time I test Elasticsearch, I feel that support for third-party tools is getting more and more difficult to configure. It still works though, and I expect that it will keep working. However, in the coming weeks, I also plan to revisit OpenSearch, and see how open they really are :-)

-

If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik, on Mastodon as @Pczanik@fosstodon.org.

Related Content