syslog-ng Open Source Edition 3.25 - Administration Guide

Preface Introduction to syslog-ng The concepts of syslog-ng Installing syslog-ng The syslog-ng OSE quick-start guide The syslog-ng OSE configuration file source: Read, receive, and collect log messages
How sources work default-network-drivers: Receive and parse common syslog messages internal: Collecting internal messages file: Collecting messages from text files wildcard-file: Collecting messages from multiple text files linux-audit: Collecting messages from Linux audit logs network: Collecting messages using the RFC3164 protocol (network() driver) nodejs: Receiving JSON messages from nodejs applications mbox: Converting local email messages to log messages osquery: Collect and parse osquery result logs pipe: Collecting messages from named pipes pacct: Collecting process accounting logs on Linux program: Receiving messages from external applications python: writing server-style Python sources python-fetcher: writing fetcher-style Python sources snmptrap: Read Net-SNMP traps sun-streams: Collecting messages on Sun Solaris syslog: Collecting messages using the IETF syslog protocol (syslog() driver) system: Collecting the system-specific log messages of a platform systemd-journal: Collecting messages from the systemd-journal system log storage systemd-syslog: Collecting systemd messages using a socket tcp, tcp6, udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol— OBSOLETE unix-stream, unix-dgram: Collecting messages from UNIX domain sockets stdin: Collecting messages from the standard input stream
destination: Forward, send, and store log messages
amqp: Publishing messages using AMQP collectd: sending metrics to collectd elasticsearch2: Sending messages directly to Elasticsearch version 2.0 or higher (DEPRECATED) elasticsearch-http: Sending messages to Elasticsearch HTTP Bulk API file: Storing messages in plain-text files graphite: Sending metrics to Graphite Sending logs to Graylog hdfs: Storing messages on the Hadoop Distributed File System (HDFS) Posting messages over HTTP http: Posting messages over HTTP without Java kafka: Publishing messages to Apache Kafka (Java implementation) kafka: Publishing messages to Apache Kafka (C implementation, using the librdkafka client) loggly: Using Loggly logmatic: Using Logmatic.io mongodb: Storing messages in a MongoDB database network: Sending messages to a remote log server using the RFC3164 protocol (network() driver) osquery: Sending log messages to osquery's syslog table pipe: Sending messages to named pipes program: Sending messages to external applications pseudofile() python: writing custom Python destinations redis: Storing name-value pairs in Redis riemann: Monitoring your data with Riemann slack: Sending alerts and notifications to a Slack channel smtp: Generating SMTP messages (email) from logs snmp: Sending SNMP traps Splunk: Sending log messages to Splunk sql: Storing messages in an SQL database stomp: Publishing messages using STOMP syslog: Sending messages to a remote logserver using the IETF-syslog protocol syslog-ng(): Forward logs to another syslog-ng node tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers) Telegram: Sending messages to Telegram unix-stream, unix-dgram: Sending messages to UNIX domain sockets usertty: Sending messages to a user terminal: usertty() destination Write your own custom destination in Java or Python Client-side failover
log: Filter and route log messages using log paths, flags, and filters Global options of syslog-ng OSE TLS-encrypted message transfer template and rewrite: Format, modify, and manipulate log messages parser: Parse and segment structured messages db-parser: Process message content with a pattern database (patterndb) Correlating log messages Enriching log messages with external data Statistics of syslog-ng Multithreading and scaling in syslog-ng OSE Troubleshooting syslog-ng Best practices and examples The syslog-ng manual pages Creative Commons Attribution Non-commercial No Derivatives (by-nc-nd) License Glossary

Managing and checking syslog-ng OSE service on Linux

This section describes how to start, stop and check the status of syslog-ng Open Source Edition (syslog-ng OSE) service on Linux.

Starting syslog-ng OSE

To start syslog-ng OSE, execute the following command as root.

Example: starting syslog-ng OSE

systemctl start syslog-ng

If the service starts successfully, no output will be displayed.

The following message indicates that syslog-ng OSE can not start (see Checking syslog-ng OSE status):

Job for syslog-ng.service failed because the control process exited with error code. See systemctl status syslog-ng.service and journalctl -xe for details.

Stopping syslog-ng OSE

To stop syslog-ng OSE

  1. Execute the following command as root.
    Example: command for stopping syslog-ng OSE

    systemctl stop syslog-ng

  2. Check the status of syslog-ng OSE service (see Checking syslog-ng OSE status).

Restarting syslog-ng OSE

To restart syslog-ng OSE, execute the following command as root.

Example: command for restarting syslog-ng OSE

systemctl restart syslog-ng

Reloading configuration file without restarting syslog-ng OSE

To reload the configuration file without restarting syslog-ng OSE, execute the following command as root.

Example: command for reloading the configuration file without restarting syslog-ng OSE

systemctl reload syslog-ng

Checking syslog-ng OSE status

To check the following status-related components, observe the suggestions below.

  • Checking the status of syslog-ng OSE service

    To check the status of syslog-ng OSE service

    1. Execute the following command as root.

      Example: command for checking the status of syslog-ng OSE service

      systemctl --no-pager status syslog-ng

    2. Check the Active: field, which shows the status of syslog-ng OSE service. The following statuses are possible:

      • active (running) - syslog-ng OSE service is up and running

        Example: syslog-ng OSE service active
        syslog-ng.service - System Logger Daemon
        Loaded: loaded (/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled)
        Active: active (running) since Tue 2019-06-25 08:58:09 CEST; 5s ago
        Main PID: 6575 (syslog-ng)
        Tasks: 3
        Memory: 13.3M
        CPU: 268ms
        CGroup: /system.slice/syslog-ng.service
        6575 /opt/syslog-ng/libexec/syslog-ng -F --no-caps --enable-core
      • inactive (dead) - syslog-ng service is stopped

        Example: syslog-ng OSE status inactive
        syslog-ng.service - System Logger Daemon
        Loaded: loaded (/lib/systemd/system/syslog-ng.service; enabled; vendor preset: enabled)
        Active: inactive (dead) since Tue 2019-06-25 09:14:16 CEST; 2min 18s ago
        Process: 6575 ExecStart=/opt/syslog-ng/sbin/syslog-ng -F --no-caps --enable-core $SYSLOGNG_OPTIONS (code=exited, status=0/SUCCESS)
        Main PID: 6575 (code=exited, status=0/SUCCESS)
        Status: "Shutting down... Tue Jun 25 09:14:16 2019"
        Jun 25 09:14:31 as-syslog-srv systemd: Stopped System Logger Daemon.
  • Checking the process of syslog-ng OSE

    To check the process of syslog-ng OSE, execute one of the following commands.

    • Example: command ps u `pidof syslog-ng`

      ps u `pidof syslog-ng`

      Expected output example:

      USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND

      syslogng 6709 0.0 0.6 308680 13432 ? Ss 09:17 0:00 /opt/syslog-ng/libexec/syslog-ng -F --no-caps --enable-core

    • Example: command ps axu | grep syslog-ng | grep -v grep

      ps axu | grep syslog-ng | grep -v grep

      Expected output example:

      syslogng 6709 0.0 0.6 308680 13432 ? Ss 09:17 0:00 /opt/syslog-ng/libexec/syslog-ng -F --no-caps --enable-core

  • Checking the internal logs of syslog-ng OSE

    The internal logs of syslog-ng OSE contains informal, warning and error messages.

    By default, syslog-ng OSE log messages (generated on the internal() source) are written to /var/log/messages.

    Check the internal logs of syslog-ng OSE for any issue.

  • Message processing

    The syslog-ng OSE application collects statistics about the number of processed messages on the different sources and destinations.

    • Central statistics

      To check the central statistics, execute the following command to see the number of received and queued (sent) messages by syslog-ng OSE.

      Example: command for checking central statistics

      watch "/opt/syslog-ng/sbin/syslog-ng-ctl stats | grep ^center"

      The output will be updated in every 2 seconds.

      If the numbers are changing, syslog-ng OSE is processing the messages.

      Example: output example
      Every 2.0s: /opt/syslog-ng/sbin/syslog-ng-ctl stats | grep ^center       Tue Jun 25 10:33:25 2019
      center;;queued;a;processed;112
      center;;received;a;processed;28
    • Source statistics

      To check the source statistics, execute the following command to see the number of received messages on the configured sources.

      Example: command for checking central statistics

      watch "/opt/syslog-ng/sbin/syslog-ng-ctl stats | grep ^source"

      The output will be updated in every 2 seconds.

      If the numbers are changing, syslog-ng OSE is receiving messages on the sources.

      Example: output example
      Every 2.0s: /opt/syslog-ng/sbin/syslog-ng-ctl stats | grep ^source      Tue Jun 25 10:40:50 2019
      source;s_null;;a;processed;0
      source;s_net;;a;processed;0
      source;s_local;;a;processed;90
    • Destination statistics

      To check the source statistics, execute the following command to see the number of received messages on the configured sources.

      Example: command for checking destination statistics

      watch "/opt/syslog-ng/sbin/syslog-ng-ctl stats | grep ^source"

      The output will be updated in every 2 seconds.

      If the numbers are changing, syslog-ng OSE is receiving messages on the sources.

      Example: output example
      Every 2.0s: /opt/syslog-ng/sbin/syslog-ng-ctl stats | grep ^destination      Tue Jun 25 10:41:02 2019
      destination;d_logserver2;;a;processed;90
      destination;d_messages;;a;processed;180
      destination;d_logserver;;a;processed;90
      destination;d_null;;a;processed;0

NOTE:If you find error messages in the internal logs, messages are not processed by syslog-ng OSE or you encounter any issue, you have the following options:


Was this topic helpful?

[Select Rating]



The syslog-ng OSE configuration file


Was this topic helpful?

[Select Rating]



Location of the syslog-ng configuration file

To configure syslog-ng OSE, edit the syslog-ng.conf file with any regular text editor application. The location of the configuration file depends on how you installed syslog-ng OSE. Native packages of a platform (like the ones downloaded from Linux repositories) typically place the configuration file under the /etc/syslog-ng/ directory.


Was this topic helpful?

[Select Rating]



The configuration syntax in detail

Every syslog-ng configuration file must begin with a line containing the version information of syslog-ng. For syslog-ng version 3.25, this line looks like:

@version: 3.25

Versioning the configuration file was introduced in syslog-ng 3.0. If the configuration file does not contain the version information, syslog-ng assumes that the file is for syslog-ng version 2.x. In this case it interprets the configuration and sends warnings about the parts of the configuration that should be updated. Version 3.0 and later will correctly operate with configuration files of version 2.x, but the default values of certain parameters have changed since 3.0.

Example: A simple configuration file

The following is a very simple configuration file for syslog-ng: it collects the internal messages of syslog-ng and the messages from /dev/log into the /var/log/messages_syslog-ng.log file.

@version: 3.25
source s_local {
    unix-dgram("/dev/log"); internal();
};
destination d_file {
    file("/var/log/messages_syslog-ng.log");
};
log {
    source(s_local); destination(d_file);
};

As a syslog-ng user described on a mailing list:

Alan McKinnon

The syslog-ng's config file format was written by programmers for programmers to be understood by programmers. That may not have been the stated intent, but it is how things turned out. The syntax is exactly that of C, all the way down to braces and statement terminators.

  • The main body of the configuration file consists of object definitions: sources, destinations, logpaths define which log message are received and where they are sent. All identifiers, option names and attributes, and any other strings used in the syslog-ng configuration file are case sensitive. Object definitions (also called statements) have the following syntax:

    type-of-the-object identifier-of-the-object {<parameters>};
    • Type of the object: One of source, destination, log, filter, parser, rewrite rule, or template.

    • Identifier of the object: A unique name identifying the object. When using a reserved word as an identifier, enclose the identifier in quotation marks.

      All identifiers, attributes, and any other strings used in the syslog-ng configuration file are case sensitive.

      TIP:

      Use identifiers that refer to the type of the object they identify. For example, prefix source objects with s_, destinations with d_, and so on.

      NOTE:

      Repeating a definition of an object (that is, defining the same object with the same id more than once) is not allowed, unless you use the @define allow-config-dups 1 definition in the configuration file.

    • Parameters: The parameters of the object, enclosed in braces {parameters}.

    • Semicolon: Object definitions end with a semicolon (;).

    For example, the following line defines a source and calls it s_internal.

    source s_internal {
        internal();
    };

    The object can be later referenced in other statements using its ID, for example, the previous source is used as a parameter of the following log statement:

    log {
        source(s_internal); destination(d_file);
    };
  • The parameters and options within a statement are similar to function calls of the C programming language: the name of the option followed by a list of its parameters enclosed within brackets and terminated with a semicolon.

    option(parameter1, parameter2); option2(parameter1, parameter2);

    For example, the file() driver in the following source statement has three options: the filename (/var/log/apache/access.log), follow-freq(), and flags(). The follow-freq() option also has a parameter, while the flags() option has two parameters.

    source s_tail {
        file("/var/log/apache/access.log" follow-freq(1) flags(no-parse, validate-utf8));
    };

    Objects may have required and optional parameters. Required parameters are positional, meaning that they must be specified in a defined order. Optional parameters can be specified in any order using the option(value) format. If a parameter (optional or required) is not specified, its default value is used. The parameters and their default values are listed in the reference section of the particular object.

    Example: Using required and optional parameters

    The unix-stream() source driver has a single required argument: the name of the socket to listen on. Optional parameters follow the socket name in any order, so the following source definitions have the same effect:

    source s_demo_stream1 {
        unix-stream("<path-to-socket>" max-connections(10) group(log));
    };
    source s_demo_stream2 {
        unix-stream("<path-to-socket>" group(log) max-connections(10));
    };
  • Some options are global options, or can be set globally, for example, whether syslog-ng OSE should use DNS resolution to resolve IP addresses. Global options are detailed in Global options of syslog-ng OSE.

    options {
        use-dns(no);
    };
  • Objects can be used before definition.

  • Objects can be defined inline as well. This is useful if you use the object only once (for example, a filter). For details, see Defining configuration objects inline.

  • To add comments to the configuration file, start a line with # and write your comments. These lines are ignored by syslog-ng.

    # Comment: This is a stream source
    source s_demo_stream {
        unix-stream("<path-to-socket>" max-connections(10) group(log));
    };
TIP:

Before activating a new configuration, check that your configuration file is syntactically correct using the syslog-ng --syntax-only command.

To activate the configuration, reload the configuration of syslog-ng using the /etc/init.d/syslog-ng reload command.


Was this topic helpful?

[Select Rating]



Related Documents