syslog-ng Open Source Edition 3.16 - Administration Guide

Preface Introduction to syslog-ng The concepts of syslog-ng Installing syslog-ng The syslog-ng OSE quick-start guide The syslog-ng OSE configuration file source: Read, receive, and collect log messages
How sources work default-network-drivers: Receive and parse common syslog messages internal: Collecting internal messages file: Collecting messages from text files wildcard-file: Collecting messages from multiple text files network: Collecting messages using the RFC3164 protocol (network() driver) nodejs: Receiving JSON messages from nodejs applications mbox: Converting local e-mail messages to log messages osquery: Collect and parse osquery result logs pipe: Collecting messages from named pipes pacct: Collecting process accounting logs on Linux program: Receiving messages from external applications snmptrap: Read Net-SNMP traps sun-streams: Collecting messages on Sun Solaris syslog: Collecting messages using the IETF syslog protocol (syslog() driver) system: Collecting the system-specific log messages of a platform systemd-journal: Collecting messages from the systemd-journal system log storage systemd-syslog: Collecting systemd messages using a socket tcp, tcp6, udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol— OBSOLETE unix-stream, unix-dgram: Collecting messages from UNIX domain sockets stdin: Collecting messages from the standard input stream
destination: Forward, send, and store log messages
amqp: Publishing messages using AMQP elasticsearch: Sending messages directly to Elasticsearch version 1.x elasticsearch2: Sending logs directly to Elasticsearch and Kibana 2.0 or higher file: Storing messages in plain-text files graphite: Sending metrics to Graphite Sending logs to Graylog hdfs: Storing messages on the Hadoop Distributed File System (HDFS) Posting messages over HTTP http: Posting messages over HTTP without Java kafka: Publishing messages to Apache Kafka loggly: Using Loggly logmatic: Using Logmatic.io mongodb: Storing messages in a MongoDB database network: Sending messages to a remote log server using the RFC3164 protocol (network() driver) osquery: Sending log messages to osquery's syslog table pipe: Sending messages to named pipes program: Sending messages to external applications pseudofile() redis: Storing name-value pairs in Redis riemann: Monitoring your data with Riemann smtp: Generating SMTP messages (e-mail) from logs Splunk: Sending log messages to Splunk sql: Storing messages in an SQL database stomp: Publishing messages using STOMP syslog: Sending messages to a remote logserver using the IETF-syslog protocol syslog-ng: Forwarding messages and tags to another syslog-ng node tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers) Telegram: Sending messages to Telegram unix-stream, unix-dgram: Sending messages to UNIX domain sockets usertty: Sending messages to a user terminal: usertty() destination Write your own custom destination in Java or Python
log: Filter and route log messages using log paths, flags, and filters Global options of syslog-ng OSE TLS-encrypted message transfer template and rewrite: Format, modify, and manipulate log messages parser: Parse and segment structured messages db-parser: Process message content with a pattern database (patterndb) Correlating log messages Enriching log messages with external data Statistics of syslog-ng Multithreading and scaling in syslog-ng OSE Troubleshooting syslog-ng Best practices and examples The syslog-ng manual pages Third-party contributions Creative Commons Attribution Non-commercial No Derivatives (by-nc-nd) License About us

Message representation in syslog-ng OSE

When the syslog-ng OSE application receives a message, it automatically parses the message. The syslog-ng OSE application can automatically parse log messages that conform to the RFC3164 (BSD or legacy-syslog) or the RFC5424 (IETF-syslog) message formats. If syslog-ng OSE cannot parse a message, it results in an error.

TIP:

In case you need to relay messages that cannot be parsed without any modifications or changes, use the flags(no-parse) option in the source definition, and a template containing only the ${MESSAGE} macro in the destination definition.

To parse non-syslog messages, for example, JSON, CSV, or other messages, you can use the built-in parsers of syslog-ng OSE. For details, see parser: Parse and segment structured messages.

A parsed syslog message has the following parts.

  • Timestamps

    Two timestamps are associated with every message: one is the timestamp contained within the message (that is, when the sender sent the message), the other is the time when syslog-ng OSE has actually received the message.

  • Severity

    The severity of the message.

  • Facility

    The facility that sent the message.

  • Tags

    Custom text labels added to the message that are mainly used for filtering. None of the current message transport protocols adds tags to the log messages. Tags can be added to the log message only within syslog-ng OSE. The syslog-ng OSE application automatically adds the id of the source as a tag to the incoming messages. Other tags can be added to the message by the pattern database, or using the tags() option of the source.

  • IP address of the sender

    The IP address of the host that sent the message. Note that the IP address of the sender is a hard macro and cannot be modified within syslog-ng OSE but the associated hostname can be modified, for example, using rewrite rules.

  • Hard macros

    Hard macros contain data that is directly derived from the log message, for example, the ${MONTH} macro derives its value from the timestamp. The most important consideration with hard macros is that they are read-only, meaning they cannot be modified using rewrite rules or other means.

  • Soft macros

    Soft macros (sometimes also called name-value pairs) are either built-in macros automatically generated from the log message (for example, ${HOST}), or custom user-created macros generated by using the syslog-ng pattern database or a CSV-parser. The SDATA fields of RFC5424-formatted log messages become soft macros as well. In contrast with hard macros, soft macros are writable and can be modified within syslog-ng OSE, for example, using rewrite rules.

    NOTE:

    It is also possible to set the value of built-in soft macros using parsers, for example, to set the ${HOST} macro from the message using a column of a CSV-parser.

    The data extracted from the log messages using named pattern parsers in the pattern database are also soft macros.

    TIP:

    For the list of hard and soft macros, see Hard vs. soft macros.

Message size and encoding

Internally, syslog-ng OSE represents every message as UTF-8. The maximal length of the log messages is limited by the log-msg-size() option: if a message is longer than this value, syslog-ng OSE truncates the message at the location it reaches the log-msg-size() value, and discards the rest of the message.

When encoding is set in a source (using the encoding() option) and the message is longer (in bytes) than log-msg-size() in UTF-8 representation, syslog-ng OSE splits the message at an undefined location (because the conversion between different encodings is not trivial).


Was this topic helpful?

[Select Rating]



Structuring macros, metadata, and other value-pairs

Available in syslog-ng OSE 3.3 and later.

The syslog-ng OSE application allows you to select and construct name-value pairs from any information already available about the log message, or extracted from the message itself. You can directly use this structured information, for example, in the following places:

When using value-pairs, there are three ways to specify which information (that is, macros or other name-value pairs) to include in the selection.

  • Select groups of macros using the scope() parameter, and optionally remove certain macros from the group using the exclude() parameter.

  • List specific macros to include using the key() parameter.

  • Define new name-value pairs to include using the pair() parameter.

These parameters are detailed in value-pairs().


Was this topic helpful?

[Select Rating]



Specifying data types in value-pairs

By default, syslog-ng OSE handles every data as strings. However, certain destinations and data formats (for example, SQL, MongoDB, JSON, AMQP) support other types of data as well, for example, numbers or dates. The syslog-ng OSE application allows you to specify the data type in templates (this is also called type-hinting). If the destination driver supports data types, it converts the incoming data to the specified data type. For example, this allows you to store integer numbers as numbers in MongoDB, instead of strings.

Caution:

Hazard of data loss! If syslog-ng OSE cannot convert the data into the specified type, an error occurs, and syslog-ng OSE drops the message by default. To change how syslog-ng OSE handles data-conversion errors, see on-error().

To use type-hinting, enclose the macro or template containing the data with the type: <datatype>("<macro>"), for example: int("$PID").

Currently the mongodb() destination and the format-json template function supports data types.

Example: Using type-hinting

The following example stores the MESSAGE, PID, DATE, and PROGRAM fields of a log message in a MongoDB database. The DATE and PID parts are stored as numbers instead of strings.

mongodb(
    value-pairs(pair("date", datetime("$UNIXTIME"))
        pair("pid", int64("$PID"))
        pair("program", "$PROGRAM"))
        pair("message", "$MESSAGE"))
    )
);

The following example formats the same fields into JSON.

$(format-json date=datetime($UNIXTIME) pid=int64($PID) program=$PROGRAM message=$MESSAGE)

The syslog-ng OSE application currently supports the following data-types.

  • boolean: Converts the data to a boolean value. Anything that begins with a t or 1 is converted to true, anything that begins with an f or 0 is converted to false.

  • datetime: Use it only with UNIX timestamps, anything else will likely result in an error. This means that currently you can use only the $UNIXTIME macro for this purpose.

  • double: A floating-point number.

  • literal: The data as a literal string, without adding any quotes or escape characters.

  • int or int32: 32-bit integer.

  • int64: 64-bit integer.

  • string: The data as a string.


Was this topic helpful?

[Select Rating]



value-pairs()

Type: parameter list of the value-pairs() option
Default:
empty string

Description: The value-pairs() option allows you to select specific information about a message easily using predefined macro groups. The selected information is represented as name-value pairs and can be used formatted to JSON format, or directly used in a mongodb() destination.

Example: Using the value-pairs() option

The following example selects every available information about the log message, except for the date-related macros (R_* and S_*), selects the .SDATA.meta.sequenceId macro, and defines a new value-pair called MSGHDR that contains the program name and PID of the application that sent the log message.

value-pairs(
    scope(nv_pairs core syslog all_macros selected_macros everything)
    exclude("R_*")
    exclude("S_*")
    key(".SDATA.meta.sequenceId")
    pair("MSGHDR" "$PROGRAM[$PID]: ")
)

The following example selects the same information as the previous example, but converts it into JSON format.

$(format-json --scope nv_pairs,core,syslog,all_macros,selected_macros,everything \
    --exclude R_* --exclude S_* --key .SDATA.meta.sequenceId \
    --pair MSGHDR="$PROGRAM[$PID]: ")

NOTE:

Every macro is included in the selection only once, but redundant information may appear if multiple macros include the same information (for example, including several date-related macros in the selection).

The value-pairs() option has the following parameters. The parameters are evaluated in the following order:

  1. scope()

  2. exclude()

  3. key()

  4. pair()

exclude()
Type: Space-separated list of macros to remove from the selection created using the scope() option.
Default: empty string

Description: This option removes the specified macros from the selection. Use it to remove unneeded macros selected using the scope() parameter.

For example, the following example removes the SDATA macros from the selection.

value-pairs(
    scope(rfc5424 selected_macros)
    exclude(".SDATA*")
)

The name of the macro to remove can include wildcards (*, ?). Regular expressions are not supported.

key()
Type: Space-separated list of macros to be included in selection
Default: empty string

Description: This option selects the specified macros. The selected macros will be included as MACRONAME = MACROVALUE, that is using key("HOST") will result in HOST = $HOST. You can use wildcards (*, ?) to select multiple macros. For example:

value-pairs(
    scope(rfc3164)
    key("HOST")
)
value-pairs(
    scope(rfc3164)
    key("HOST", "PROGRAM")
)
pair()
Type: name value pairs in "<NAME>" "<VALUE>" format
Default: empty string

Description: This option defines a new name-value pair to be included in the message. The value part can include macros, templates, and template functions as well. For example:

value-pairs(
    scope(rfc3164)
    pair("TIME" "$HOUR:$MIN")
    pair("MSGHDR" "$PROGRAM[$PID]: ")
)
rekey()
Type: <pattern-to-select-names>, <list of transformations>
Default: empty string

Description: This option allows you to manipulate and modify the name of the value-pairs. You can define transformations, which are are applied to the selected name-value pairs. The first parameter of the rekey() option is a glob pattern that selects the name-value pairs to modify. If you omit the pattern, the transformations are applied to every key of the scope. For details on globs, see glob.

If you want to modify the names of several message fields, see also map-value-pairs: Rename value-pairs to normalize logs.

  • If rekey() is used within a key() option, the name-value pairs specified in the glob of the key() option are transformed.

  • If rekey() is used outside the key() option, every name-value pair of the scope() is transformed.

The following transformations are available:

  • add-prefix("<my-prefix>")
  • Adds the specified prefix to every name. For example, rekey( add-prefix("my-prefix."))

  • replace-prefix("<prefix-to-replace>", "<new-prefix>")
  • Replaces a substring at the beginning of the key with another string. Only prefixes can be replaced. For example, replace-prefix(".class", ".patterndb") changes the beginning tag .class to .patterndb

    This option was called replace() in syslog-ng OSE version 3.4.

  • shift("<number>")
  • Cuts the specified number of characters from the beginning of the name.

Example: Using the rekey() option

The following sample selects every value-pair that begins with .cee., deletes this prefix by cutting 4 characters from the names, and adds a new prefix (events.).

value-pairs(
    key(".cee.*"
        rekey(
            shift(4)
            add-prefix("events.")
        )
    )
)

The rekey() option can be used with the format-json template-function as well, using the following syntax:

$(format-json --rekey .cee.* --add-prefix events.)
scope()
Type: space-separated list of macro groups to include in selection
Default: empty string

Description: This option selects predefined groups of macros. The following groups are available:

  • nv-pairs: Every soft macro (name-value pair) associated with the message, except the ones that start with a dot (.) character. Macros starting with a dot character are generated within syslog-ng OSE and are not originally part of the message, therefore are not included in this group.

  • dot-nv-pairs: Every soft macro (name-value pair) associated with the message which starts with a dot (.) character. For example, .classifier.rule_id and .sdata.*. Macros starting with a dot character are generated within syslog-ng OSE and are not originally part of the message.

  • all-nv-pairs: Include every soft macro (name-value pair). Equivalent to using both nv-pairs and dot-nv-pairs.

  • rfc3164: The macros that correspond to the RFC3164 (legacy or BSD-syslog) message format: $FACILITY, $PRIORITY, $HOST, $PROGRAM, $PID, $MESSAGE, and $DATE.

  • rfc5424: The macros that correspond to the RFC5424 (IETF-syslog) message format: $FACILITY, $PRIORITY, $HOST, $PROGRAM, $PID, $MESSAGE, $MSGID, $R_DATE, and the metadata from the structured-data (SDATA) part of RFC5424-formatted messages, that is, every macro that starts with .SDATA..

    The rfc5424 group also has the following alias: syslog-proto. Note that the value of $R_DATE will be listed under the DATE key.

    The rfc5424 group does not contain any metadata about the message, only information that was present in the original message. To include the most commonly used metadata (for example, the $SOURCEIP macro), use the selected-macros group instead.

  • all-macros: Include every hard macro. This group is mainly useful for debugging, as it contains redundant information (for example, the date-related macros include the date-related information several times in various formats).

  • selected-macros: Include the macros of the rfc3164 groups, and the most commonly used metadata about the log message: the $TAGS, $SOURCEIP, and $SEQNUM macros.

  • sdata: The metadata from the structured-data (SDATA) part of RFC5424-formatted messages, that is, every macro that starts with .SDATA.

  • everything: Include every hard and soft macros. This group is mainly useful for debugging, as it contains redundant information (for example, the date-related macros include the date-related information several times in various formats).

For example:

value-pairs(
    scope(rfc3164 selected-macros)
)

Was this topic helpful?

[Select Rating]



Related Documents