There is a new drop-in replacement for Elasticsearch, at least if you don’t mind the limitations and the alpha status. However, it definitely lives up to the promise that it provides an Elasticsearch-compatible API for data ingestion. I tested it with the elasticsearch-http() destination of syslog-ng, and it worked perfectly after I modified the URL in the configuration example I found.
So, what is Zinc? It is a search engine written in Go that provides an Elasticsearch-compatible API for data ingestion. You cannot use Kibana with it, only its own web interface. If you are not into graphs and dashboards, and want to search text messages, then it is perfect. The application itself is a single binary and it does not have any external dependencies. It is lightweight and easy to configure, as practically there are no configuration options at all.
Note: Zinc is still in alpha state. There are no guarantees that later versions will be compatible at any level. Error messages can sometimes be cryptic and you might run into unexpected behavior.
Before you begin
On the syslog-ng side, you need syslog-ng 3.23 or later. This is where the elasticsearch-http() destination was introduced.
You also need Zinc. I downloaded the binary from the project’s GitHub page: https://github.com/prabhatsharma/zinc/ You can set up the initial password using environmental variables:
FIRST_ADMIN_USER=admin FIRST_ADMIN_PASSWORD=admin ./zinc
Of course, you should use a password that is harder to guess :-)
Configuring syslog-ng
Any syslog-ng sample configuration should work with minimal modifications. You most likely need to change the URL and add user/password authentication. Depending on how syslog-ng is configured on your system, create a zinc.conf under the /etc/syslog-ng/conf.d/ directory or append it to syslog-ng.conf:
destination d_elasticsearch_http {
elasticsearch-http(
index("syslog-ng")
type("")
user("admin")
password("admin")
url("http://localhost:4080/api/_bulk")
template("$(format-json --scope rfc5424 --scope dot-nv-pairs
--rekey .* --shift 1 --scope nv-pairs
--exclude DATE --key ISODATE @timestamp=${ISODATE})")
);
};
log {
source(src);
destination(d_elasticsearch_http);
flags(flow-control);
};
The main difference is the URL. While Elasticsearch normally uses port 9200 and the path part in the URL is /_bulk, Zinc uses port 4080 and /api/_bulk as the path. I have not seen a possibility to configure encryption, so you need to use external software for that. However, user/password authentication is mandatory both for data ingestion and the web interface.
The destination in the above configuration sends logs to Zinc, all syslog-ng fields, all name-value pairs syslog-ng found and the date using ISO formatting. The source simply connects the local source to the destination. The name on your system might be different.
Testing
Once you saved and reloaded the configuration, you are ready for testing. You can use the “logger” command to generate some log messages. You can also use “sudo”, as syslog-ng automatically parses log messages from sudo. You can see the generated name-value pairs in the Zinc web interface.
Here is the mandatory screenshot:
What is next?
Zinc is still under development, so you might not want to use it in production. However, I think it is worth to follow its development, as it is really compact and runs in environments where Elasticsearch would not even start due to resource constraints.
-
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik.