syslog-ng Open Source Edition 3.33 - Administration Guide

Preface Introduction to syslog-ng The concepts of syslog-ng Installing syslog-ng The syslog-ng OSE quick-start guide The syslog-ng OSE configuration file source: Read, receive, and collect log messages
How sources work default-network-drivers: Receive and parse common syslog messages internal: Collecting internal messages file: Collecting messages from text files wildcard-file: Collecting messages from multiple text files linux-audit: Collecting messages from Linux audit logs network: Collecting messages using the RFC3164 protocol (network() driver) nodejs: Receiving JSON messages from nodejs applications mbox: Converting local email messages to log messages osquery: Collect and parse osquery result logs pipe: Collecting messages from named pipes pacct: Collecting process accounting logs on Linux program: Receiving messages from external applications python: writing server-style Python sources python-fetcher: writing fetcher-style Python sources snmptrap: Read Net-SNMP traps sun-streams: Collecting messages on Sun Solaris syslog: Collecting messages using the IETF syslog protocol (syslog() driver) system: Collecting the system-specific log messages of a platform systemd-journal: Collecting messages from the systemd-journal system log storage systemd-syslog: Collecting systemd messages using a socket tcp, tcp6, udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol— OBSOLETE unix-stream, unix-dgram: Collecting messages from UNIX domain sockets stdin: Collecting messages from the standard input stream
destination: Forward, send, and store log messages
amqp: Publishing messages using AMQP collectd: sending metrics to collectd elasticsearch2: Sending messages directly to Elasticsearch version 2.0 or higher (DEPRECATED) elasticsearch-http: Sending messages to Elasticsearch HTTP Bulk API file: Storing messages in plain-text files graphite: Sending metrics to Graphite Sending logs to Graylog hdfs: Storing messages on the Hadoop Distributed File System (HDFS) Posting messages over HTTP http: Posting messages over HTTP without Java kafka: Publishing messages to Apache Kafka (Java implementation) kafka(): Publishing messages to Apache Kafka (C implementation, using the librdkafka client) loggly: Using Loggly logmatic: Using Logmatic.io mongodb: Storing messages in a MongoDB database mqtt() destination: sending messages from a local network to an MQTT broker network: Sending messages to a remote log server using the RFC3164 protocol (network() driver) osquery: Sending log messages to osquery's syslog table pipe: Sending messages to named pipes program: Sending messages to external applications pseudofile() python: writing custom Python destinations redis: Storing name-value pairs in Redis riemann: Monitoring your data with Riemann slack: Sending alerts and notifications to a Slack channel smtp: Generating SMTP messages (email) from logs snmp: Sending SNMP traps Splunk: Sending log messages to Splunk sql: Storing messages in an SQL database stomp: Publishing messages using STOMP Sumo Logic destinations: sumologic-http() and sumologic-syslog() syslog: Sending messages to a remote logserver using the IETF-syslog protocol syslog-ng(): Forward logs to another syslog-ng node tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers) Telegram: Sending messages to Telegram unix-stream, unix-dgram: Sending messages to UNIX domain sockets usertty: Sending messages to a user terminal: usertty() destination Write your own custom destination in Java or Python Client-side failover
log: Filter and route log messages using log paths, flags, and filters Global options of syslog-ng OSE TLS-encrypted message transfer template and rewrite: Format, modify, and manipulate log messages parser: Parse and segment structured messages db-parser: Process message content with a pattern database (patterndb) Correlating log messages Enriching log messages with external data Statistics of syslog-ng Multithreading and scaling in syslog-ng OSE Troubleshooting syslog-ng Best practices and examples The syslog-ng manual pages Creative Commons Attribution Non-commercial No Derivatives (by-nc-nd) License Glossary

value-pairs()

Type: parameter list of the value-pairs() option
Default:
empty string

Description: The value-pairs() option allows you to select specific information about a message easily using predefined macro groups. The selected information is represented as name-value pairs and can be used formatted to JSON format, or directly used in a mongodb() destination.

Example: Using the value-pairs() option

The following example selects every available information about the log message, except for the date-related macros (R_* and S_*), selects the .SDATA.meta.sequenceId macro, and defines a new value-pair called MSGHDR that contains the program name and PID of the application that sent the log message.

value-pairs(
    scope(nv_pairs core syslog all_macros selected_macros everything)
    exclude("R_*")
    exclude("S_*")
    key(".SDATA.meta.sequenceId")
    pair("MSGHDR" "$PROGRAM[$PID]: ")
)

The following example selects the same information as the previous example, but converts it into JSON format.

$(format-json --scope nv_pairs,core,syslog,all_macros,selected_macros,everything \
    --exclude R_* --exclude S_* --key .SDATA.meta.sequenceId \
    --pair MSGHDR="$PROGRAM[$PID]: ")

NOTE: Every macro is included in the selection only once, but redundant information may appear if multiple macros include the same information (for example, including several date-related macros in the selection).

The value-pairs() option has the following parameters. The parameters are evaluated in the following order:

  1. scope()

  2. exclude()

  3. key()

  4. pair()

exclude()
Type: Space-separated list of macros to remove from the selection created using the scope() option.
Default: empty string

Description: This option removes the specified macros from the selection. Use it to remove unneeded macros selected using the scope() parameter.

For example, the following example removes the SDATA macros from the selection.

value-pairs(
    scope(rfc5424 selected_macros)
    exclude(".SDATA*")
)

The name of the macro to remove can include wildcards (*, ?). Regular expressions are not supported.

key()
Type: Space-separated list of macros to be included in selection
Default: empty string

Description: This option selects the specified macros. The selected macros will be included as MACRONAME = MACROVALUE, that is using key("HOST") will result in HOST = $HOST. You can use wildcards (*, ?) to select multiple macros. For example:

value-pairs(
    scope(rfc3164)
    key("HOST")
)
value-pairs(
    scope(rfc3164)
    key("HOST", "PROGRAM")
)
omit-empty-values()

 

Type: flag
Default: N/A

Description: If this option is specified, syslog-ng OSE does not include value-pairs with empty values in the output. For example: $(format-json --scope none --omit-empty-values) or

value-pairs(
    scope(rfc3164 selected-macros)
    omit-empty-values()
)

Available in syslog-ng OSE version 3.21 and later.

 

pair()
Type: name value pairs in "<NAME>" "<VALUE>" format
Default: empty string

Description: This option defines a new name-value pair to be included in the message. The value part can include macros, templates, and template functions as well. For example:

value-pairs(
    scope(rfc3164)
    pair("TIME" "$HOUR:$MIN")
    pair("MSGHDR" "$PROGRAM[$PID]: ")
)
rekey()
Type: <pattern-to-select-names>, <list of transformations>
Default: empty string

Description: This option allows you to manipulate and modify the name of the value-pairs. You can define transformations, which are are applied to the selected name-value pairs. The first parameter of the rekey() option is a glob pattern that selects the name-value pairs to modify. If you omit the pattern, the transformations are applied to every key of the scope. For details on globs, see Options of regular expressions.

If you want to modify the names of several message fields, see also map-value-pairs: Rename value-pairs to normalize logs.

  • If rekey() is used within a key() option, the name-value pairs specified in the glob of the key() option are transformed.

  • If rekey() is used outside the key() option, every name-value pair of the scope() is transformed.

The following transformations are available:

  • add-prefix("<my-prefix>")

    Adds the specified prefix to every name. For example, rekey( add-prefix("my-prefix."))

  • replace-prefix("<prefix-to-replace>", "<new-prefix>")

    Replaces a substring at the beginning of the key with another string. Only prefixes can be replaced. For example, replace-prefix(".class", ".patterndb") changes the beginning tag .class to .patterndb

    This option was called replace() in syslog-ng OSE version 3.4.

  • shift("<number>")

    Cuts the specified number of characters from the beginning of the name.

  • shift-levels("<number>")

    Similar to --shift, but instead of cutting characters, it cuts dot-delimited "levels" in the name (including the initial dot). For example, --shift-levels 2 deletes the prefix up to the second dot in the name of the key: .iptables.SRC becomes SRC

Example: Using the rekey() option

The following sample selects every value-pair that begins with .cee., deletes this prefix by cutting 4 characters from the names, and adds a new prefix (events.).

value-pairs(
    key(".cee.*"
        rekey(
            shift(4)
            add-prefix("events.")
        )
    )
)

The rekey() option can be used with the format-json template-function as well, using the following syntax:

$(format-json --rekey .cee.* --add-prefix events.)
scope()
Type: space-separated list of macro groups to include in selection
Default: empty string

Description: This option selects predefined groups of macros. The following groups are available:

  • nv-pairs: Every soft macro (name-value pair) associated with the message, except the ones that start with a dot (.) character. Macros starting with a dot character are generated within syslog-ng OSE and are not originally part of the message, therefore are not included in this group.

  • dot-nv-pairs: Every soft macro (name-value pair) associated with the message which starts with a dot (.) character. For example, .classifier.rule_id and .sdata.*. Macros starting with a dot character are generated within syslog-ng OSE and are not originally part of the message.

  • all-nv-pairs: Include every soft macro (name-value pair). Equivalent to using both nv-pairs and dot-nv-pairs.

  • rfc3164: The macros that correspond to the RFC3164 (legacy or BSD-syslog) message format: $FACILITY, $PRIORITY, $HOST, $PROGRAM, $PID, $MESSAGE, and $DATE.

  • rfc5424: The macros that correspond to the RFC5424 (IETF-syslog) message format: $FACILITY, $PRIORITY, $HOST, $PROGRAM, $PID, $MESSAGE, $MSGID, $R_DATE, and the metadata from the structured-data (SDATA) part of RFC5424-formatted messages, that is, every macro that starts with .SDATA..

    The rfc5424 group also has the following alias: syslog-proto. Note that the value of $R_DATE will be listed under the DATE key.

    The rfc5424 group does not contain any metadata about the message, only information that was present in the original message. To include the most commonly used metadata (for example, the $SOURCEIP macro), use the selected-macros group instead.

  • all-macros: Include every hard macro. This group is mainly useful for debugging, as it contains redundant information (for example, the date-related macros include the date-related information several times in various formats).

  • selected-macros: Include the macros of the rfc3164 groups, and the most commonly used metadata about the log message: the $TAGS, $SOURCEIP, and $SEQNUM macros.

  • sdata: The metadata from the structured-data (SDATA) part of RFC5424-formatted messages, that is, every macro that starts with .SDATA.

  • everything: Include every hard and soft macros. This group is mainly useful for debugging, as it contains redundant information (for example, the date-related macros include the date-related information several times in various formats).

  • none: Reset previously added scopes, for example, to delete automatically-added name-value pairs. The following example deletes every value-pair from the scope, and adds only the ones starting with iptables: $(format-welf --scope none .iptables.*)

For example:

value-pairs(
    scope(rfc3164 selected-macros)
)

Was this topic helpful?

[Select Rating]



Things to consider when forwarding messages between syslog-ng OSE hosts

When you send your log messages from a syslog-ng OSE client through the network to a syslog-ng OSE server, you can use different protocols and options. Every combination has its advantages and disadvantages. The most important thing is to use matching protocols and options, so the server handles the incoming log messages properly.

In syslog-ng OSE you can change many aspects of the network communication. First of all, there is the structure of the messages itself. Currently, syslog-ng OSE supports two standard syslog protocols: the BSD (RFC3164) and the syslog (RFC5424) message format.

These RFCs describe the format and the structure of the log message, and add a (lightweight) framing around the messages. You can set this framing/structure by selecting the appropriate driver in syslog-ng OSE. There are two drivers you can use: the network() driver and the syslog() driver. The syslog() driver is for the syslog (RFC5424) protocol and the network() driver is for the BSD (RFC3164) protocol.

The tcp() and udp() drivers are now deprecated, they are essentially equivalent with the network(transport(tcp)) and network(transport(udp)) drivers.

In addition to selecting the driver to use, both drivers allow you to use different transport-layer protocols: TCP and UDP, and optionally also higher-level transport protocols: TLS (over TCP. To complicate things a bit more, you can configure the network() driver (corresponding to the BSD (RFC3164) protocol) to send the messages in the syslog (RFC5424) format (but without the framing used in RFC5424) using the flag(syslog-protocol) option.

Because some combination of drivers and options are invalid, you can use the following drivers and options as sources and as destinations:

  1. syslog(transport(tcp))

  2. syslog(transport(udp))

  3. syslog(transport(rltp))

  4. syslog(transport(tls))

  5. syslog(transport(rltp(tls-required(yes)))

  6. network(transport(tcp))

  7. network(transport(udp))

  8. network(transport(rltp))

  9. network(transport(tls))

  10. network(transport(rltp(tls-required(yes)))

  11. network(transport(tcp) flag(syslog-protocol))

  12. network(transport(udp) flag(syslog-protocol))

  13. network(transport(rltp)flag(syslog-protocol))

  14. network(transport(tls) flag(syslog-protocol))

  15. network(transport(rltp(tls-required(yes)) flag(syslog-protocol))

If you use the same driver and options in the destination of your syslog-ng OSE client and the source of your syslog-ng OSE server, everything should work as expected. Unfortunately there are some other combinations, that seem to work, but result in losing parts of the messages. The following table show the combinations:

Table 5: Source-destination driver combinations
Source \ Destination syslog/tcp syslog/udp syslog/tls network/tcp network/udp network/tls network/tcp/flag network/udp/flag network/tls/flag
syslog/tcp - - ! - - ! - -
syslog/udp - - - ! - - ! -
syslog/tls - - - - ! - - !
network/tcp - - - - - ✔? - -
network/udp - ✔? - - - - ✔? -
network/tls - - - - - - - ✔?
network/tcp/flag ! - - ! - - - -
network/udp/flag - ! - - ! - - -
network/tls/flag - - ! - - ! - -
  • - This method does not work. The logs will not get to the server.

  • ✔ This method works.

  • ! This method has some visible drawbacks. The logs go through, but some of the values are missing/misplaced/and so on.

  • ✔? This method seems to work, but it is not recommended because this can change in a future release.


Was this topic helpful?

[Select Rating]



Commercial version of syslog-ng

The syslog-ng application has a commercial version available, called syslog-ng Premium Edition (syslog-ng PE). The commercial version comes with well-tested features from its open source foundation, a number of extra features, enterprise-level support, as well as a ready-to-use log management appliance built on the strengths of syslog-ng Premium Edition.

Exclusive features related to compliance

Collecting and analyzing log messages is required directly or indirectly by several regulations, frameworks, and standards, including the Sarbanes-Oxley Act (SOX), the Health Insurance and Portability Act (HIPAA), and the Payment Card Industry Data Security Standard (PCI-DSS). syslog-ng PE provides a set of features that help you comply with regulations that require the central collection of log messages in a tamperproof way:

  • Logstore files enable you to store log messages securely in encrypted, compressed and timestamped binary files. From a compliance point of view, this serves a double purpose. Encryption guarantees the integrity of log messages so you can be sure that they have not been manipulated. Timestamping provides verifiable proof about the exact time when log messages arrived.

  • Reliable Log Transfer Protocol (RLTP) is a proprietary transport protocol that prevents message loss during connection breaks. When using this protocol, the sender detects which messages the receiver has successfully received (based on the acknowledgments returned by the receiver after having processed messages). If messages are lost during transfer, the sender resends the missing messages, starting from the last successfully received message. Therefore, messages are not duplicated at the receiving end in case of a connection break.

Wide range of supported platforms with binary installers

syslog-ng Premium Edition comes with tested binary files that are available for a wide array of server platforms, reducing the time required for installation and maintenance. Support for a wide range of operating system and hardware platforms also make syslog-ng PE an ideal choice to collect logs in massively heterogeneous environments.

Enterprise-level support services

As all commercial software, syslog-ng PE also comes with various enterprise-level support packages, which means that you get immediate and pro-active assistance (24x7 if you choose a top-tier package), dedicated to resolving your issue as soon as possible when you experience problems.

For more information about syslog-ng Premium Edition, see The syslog-ng Premium Edition Administrator Guide.

syslog-ng Store Box, ready-to-use log management appliance

syslog-ng Store Box (SSB) is a log management appliance that is built on syslog-ng Premium Edition. It is a turnkey solution to manage your log data, meaning that no software installation is necessary. As SSB is available both as a virtual machine and a physical appliance, it is also easily scalable.

SSB provides a number of features that can add value for your use cases:

  • A web GUI that makes searching logs, as well as configuring and managing SSB itself easy:

    • The search interface allows you to use wildcards and Boolean operators to perform complex searches, and drill down on the results. You can gain a quick overview and pinpoint problems fast by generating ad-hoc charts from the distribution of the log messages.

      In addition, you can easily create customized reports from the charts and statistics you create on the search interface to demonstrate compliance with standards and regulations such as PCI-DSS, ISO 27001, SOX and HIPAA.

    • Configuring SSB is done through the user interface. All of the flexible filtering, classification and routing features in the syslog-ng Open Source Edition and syslog-ng Premium Edition can be configured with it. Access and authentication policies can be set to integrate with Microsoft Active Directory, LDAP and Radius servers. The web interface is accessible through a network interface dedicated to management traffic. This management interface is also used for backups, sending alerts, and other administrative traffic.

  • High availability support to ensure continuous log collection in business-critical environments.

For further details about syslog-ng Store Box, see The syslog-ng Store Box Administrator Guide.

Upgrading from syslog-ng OSE to syslog-ng PE

If you wish to upgrade from syslog-ng OSE to syslog-ng PE, read the blog post Upgrading from syslog-ng OSE to syslog-ng PE for instructions and tips.


Was this topic helpful?

[Select Rating]



Installing syslog-ng

This chapter explains how to install syslog-ng Open Source Edition on various platforms.

  • You can install syslog-ng OSE on many platforms using the package manager and official repositories of the platform. For a list of third-party packages available for various Linux, UNIX, and other platforms, see the Downloads page.

  • For instructions on compiling syslog-ng Open Source Edition from the source code, see Compiling syslog-ng from source.

  • You can use a syslog-ng docker image.

    For detailed information on how to run your central log server in Docker and other Docker-related syslog-ng use cases, see the Logging in Docker using syslog-ng white paper.


Was this topic helpful?

[Select Rating]



Related Documents