elasticsearch: Sending messages directly to Elasticsearch version 1.x

elasticsearch: Sending messages directly to Elasticsearch version 1.x

Starting with version 3.7 of syslog-ng OSE can directly send log messages to Elasticsearch, allowing you to search and analyze your data in real time, and visualize it with Kibana.

Note the following limitations when using the syslog-ng OSE elasticsearch destination:

• This destination is only supported on the Linux platform.

• Since syslog-ng OSE uses the official Java Elasticsearch libraries, the elasticsearch destination has significant memory usage.

• Sending messages over the HTTP REST API is supported only using the elastic2() destination. Note that in HTTP mode, the elasticsearch2 destination can send log messages to Elasticsearch version 1.x and newer. For details, see elasticsearch2: Sending logs directly to Elasticsearch and Kibana 2.0 or higher.

• The log messages of the underlying client libraries are available in the internal() source of syslog-ng OSE.

@module mod-java
@include "scl.conf"

elasticsearch(
    index("syslog-ng_${YEAR}.${MONTH}.${DAY}")
    type("test")
    cluster("syslog-ng")
);

@module mod-java
@include "scl.conf"

destination d_elastic {
  elasticsearch(
    index("syslog-ng_${YEAR}.${MONTH}.${DAY}")
    type("test")
  );
};

@module mod-java
@include "scl.conf"

options {
  threaded(yes);
  use-uniqid(yes);
};

source s_syslog {
  syslog();
};

destination d_elastic {
  elasticsearch(
    index("syslog-ng_${YEAR}.${MONTH}.${DAY}")
    type("test")
    cluster("syslog-ng")
    client-mode("transport")
    custom-id("${UNIQID}")
    flush-limit("10000")
  );
};

log {
  source(s_syslog);
  destination(d_elastic);
  flags(flow-control);
};

• To install the software required for the elasticsearch destination, see Prerequisites.

• For details on how the elasticsearch destination works, see How syslog-ng OSE interacts with Elasticsearch.

• For the list of options, see Elasticsearch destination options.

The elasticsearch() driver is actually a reusable configuration snippet configured to receive log messages using the Java language-binding of syslog-ng OSE. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of the elasticsearch configuration snippet on GitHub. For details on extending syslog-ng OSE in Java, see the Getting started with syslog-ng development guide.