We are currently preforming website maintenance, any feature requiring sign-in is temporarily unavailable, if you have an issue requiring immediate assistance please call Technical Support.

syslog-ng Open Source Edition 3.19 - Administration Guide

Preface Introduction to syslog-ng The concepts of syslog-ng Installing syslog-ng The syslog-ng OSE quick-start guide The syslog-ng OSE configuration file source: Read, receive, and collect log messages
How sources work default-network-drivers: Receive and parse common syslog messages internal: Collecting internal messages file: Collecting messages from text files wildcard-file: Collecting messages from multiple text files linux-audit: Collecting messages from Linux audit logs network: Collecting messages using the RFC3164 protocol (network() driver) nodejs: Receiving JSON messages from nodejs applications mbox: Converting local e-mail messages to log messages osquery: Collect and parse osquery result logs pipe: Collecting messages from named pipes pacct: Collecting process accounting logs on Linux program: Receiving messages from external applications python: writing server-style Python sources python-fetcher: writing fetcher-style Python sources snmptrap: Read Net-SNMP traps sun-streams: Collecting messages on Sun Solaris syslog: Collecting messages using the IETF syslog protocol (syslog() driver) system: Collecting the system-specific log messages of a platform systemd-journal: Collecting messages from the systemd-journal system log storage systemd-syslog: Collecting systemd messages using a socket tcp, tcp6, udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol— OBSOLETE unix-stream, unix-dgram: Collecting messages from UNIX domain sockets stdin: Collecting messages from the standard input stream
destination: Forward, send, and store log messages
amqp: Publishing messages using AMQP elasticsearch: Sending messages directly to Elasticsearch version 1.x (DEPRECATED) elasticsearch2: Sending logs directly to Elasticsearch and Kibana 2.0 or higher file: Storing messages in plain-text files graphite: Sending metrics to Graphite Sending logs to Graylog hdfs: Storing messages on the Hadoop Distributed File System (HDFS) Posting messages over HTTP http: Posting messages over HTTP without Java kafka: Publishing messages to Apache Kafka loggly: Using Loggly logmatic: Using Logmatic.io mongodb: Storing messages in a MongoDB database network: Sending messages to a remote log server using the RFC3164 protocol (network() driver) osquery: Sending log messages to osquery's syslog table pipe: Sending messages to named pipes program: Sending messages to external applications pseudofile() python: writing custom Python destinations redis: Storing name-value pairs in Redis riemann: Monitoring your data with Riemann slack: Sending alerts and notifications to a Slack channel smtp: Generating SMTP messages (e-mail) from logs Splunk: Sending log messages to Splunk sql: Storing messages in an SQL database stomp: Publishing messages using STOMP syslog: Sending messages to a remote logserver using the IETF-syslog protocol syslog-ng: Forwarding messages and tags to another syslog-ng node tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers) Telegram: Sending messages to Telegram unix-stream, unix-dgram: Sending messages to UNIX domain sockets usertty: Sending messages to a user terminal: usertty() destination Write your own custom destination in Java or Python Client-side failover
log: Filter and route log messages using log paths, flags, and filters Global options of syslog-ng OSE TLS-encrypted message transfer template and rewrite: Format, modify, and manipulate log messages parser: Parse and segment structured messages db-parser: Process message content with a pattern database (patterndb) Correlating log messages Enriching log messages with external data Statistics of syslog-ng Multithreading and scaling in syslog-ng OSE Troubleshooting syslog-ng Best practices and examples The syslog-ng manual pages Third-party contributions Creative Commons Attribution Non-commercial No Derivatives (by-nc-nd) License About us

syslog-ng.8


Table of Contents

syslog-ng— syslog-ng system logger application
Name

syslog-ng — syslog-ng system logger application

Synopsis

syslog-ng [options]

Description

This manual page is only an abstract, for the complete documentation of syslog-ng, see the syslog-ng Documentation page or the syslog-ng page.

The syslog-ng OSE application is a flexible and highly scalable system logging application. Typically, syslog-ng is used to manage log messages and implement centralized logging, where the aim is to collect the log messages of several devices on a single, central log server. The different devices - called syslog-ng clients - all run syslog-ng, and collect the log messages from the various applications, files, and other sources. The clients send all important log messages to the remote syslog-ng server, where the server sorts and stores them.

Options
--caps

Run syslog-ng OSE process with the specified POSIX capability flags.

  • If the --no-caps option is not set,syslog-ng OSE has been compiled with the --enable-linux-caps compile option, and the host supports CAP_SYSLOG, syslog-ng OSE uses the following capabilities: "cap_net_bind_service, cap_net_broadcast, cap_net_raw, cap_dac_read_search, cap_dac_override, cap_chown, cap_fowner=p cap_syslog=ep"

  • If the --no-caps option is not set, and the host does not support CAP_SYSLOG, syslog-ng OSE uses the following capabilities: "cap_net_bind_service, cap_net_broadcast, cap_net_raw,cap_dac_read_search, cap_dac_override, cap_chown, cap_fowner=p cap_sys_admin=ep"

For example:

              /opt/syslog-ng/sbin/syslog-ng -Fv --caps cap_sys_admin,cap_chown,cap_dac_override,cap_net_bind_service,cap_fowner=pi

Note that the capabilities are not case sensitive, the following command is also good: /opt/syslog-ng/sbin/syslog-ng -Fv --caps CAP_SYS_ADMIN,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_NET_BIND_SERVICE,CAP_FOWNER=pi

For details on the capability flags, see the following man pages: cap_from_text(3) and capabilities(7)

--cfgfile <file> or -f <file>

Use the specified configuration file.

--chroot <dir> or -C <dir>

Change root to the specified directory. The configuration file is read after chrooting so, the configuration file must be available within the chroot. That way it is also possible to reload the syslog-ng configuration after chrooting. However, note that the --user and --group options are resolved before chrooting.

--control <file> or -c <file>

Set the location of the syslog-ng control socket. Default value: /var/run/syslog-ng.ctl

--debug or -d

Start syslog-ng in debug mode.

--default-modules

A comma-separated list of the modules that are loaded automatically. Modules not loaded automatically can be loaded by including the @module <modulename> statement in the syslog-ng OSE configuration file. The following modules are loaded by default: affile, afprog, afsocket, afuser, basicfuncs, csvparser, dbparser, syslogformat, afsql, system-source. Available only in syslog-ng Open Source Edition 3.3 and later.

--enable-core

Enable syslog-ng to write core files in case of a crash to help support and debugging.

--fd-limit <number>

Set the minimal number of required file descriptors (fd-s). This sets how many files syslog-ng can keep open simultaneously. Default value: 4096. Note that this does not override the global ulimit setting of the host.

--foreground or -F

Do not daemonize, run in the foreground. When running in the foreground, syslog-ng OSE starts from the current directory ($CWD) so it can create core files (normally, syslog-ng OSE starts from $PREFIX/var).

--group <group> or -g <group>

Switch to the specified group after initializing the configuration file.

--help or -h

Display a brief help message.

--module-registry

Display the list and description of the available modules. Note that not all of these modules are loaded automatically, only the ones specified in the --default-modules option. Available only in syslog-ng Open Source Edition 3.3 and later.

--no-caps

Run syslog-ng as root, without capability-support. This is the default behavior. On Linux, it is possible to run syslog-ng as non-root with capability-support if syslog-ng was compiled with the --enable-linux-caps option enabled. (Execute syslog-ng --version to display the list of enabled build parameters.)

To run syslog-ng OSE with specific capabilities, use the --caps option.

--persist-file <persist-file> or -R <persist-file>

Set the path and name of the syslog-ng.persist file where the persistent options and data are stored.

--pidfile <pidfile> or -p <pidfile>

Set path to the PID file where the pid of the main process is stored.

--preprocess-into <output-file>

After processing the configuration file and resolving included files and variables, write the resulting configuration into the specified output file. Available only in syslog-ng Open Source Edition 3.3 and later.

--process-mode <mode>

Sets how to run syslog-ng: in the foreground (mainly used for debugging), in the background as a daemon, or in safe-background mode. By default, syslog-ng runs in safe-background mode. This mode creates a supervisor process called supervising syslog-ng , that restarts syslog-ng if it crashes.

--stderr or -e

Log internal messages of syslog-ng to stderr. Mainly used for debugging purposes in conjunction with the --foreground option. If not specified, syslog-ng will log such messages to its internal source.

--syntax-only or -s

Verify that the configuration file is syntactically correct and exit.

--user <user> or -u <user>

Switch to the specified user after initializing the configuration file (and optionally chrooting). Note that it is not possible to reload the syslog-ng configuration if the specified user has no privilege to create the /dev/log file.

--verbose or -v

Enable verbose logging used to troubleshoot syslog-ng.

--version or -V

Display version number and compilation information, and also the list and short description of the available modules. For detailed description of the available modules, see the --module-registry option. Note that not all of these modules are loaded automatically, only the ones specified in the --default-modules option. When including configuration snippets in the configuration files, the default path where syslog-ng looks for the snippets is displayed as Include-Path.

--worker-threads

Sets the number of worker threads syslog-ng OSE can use, including the main syslog-ng OSE thread. Note that certain operations in syslog-ng OSE can use threads that are not limited by this option. This setting has effect only when syslog-ng OSE is running in multithreaded mode. Available only in syslog-ng Open Source Edition 3.3 and later. See The syslog-ng Open Source Edition 3.15 Administrator Guide for details.

Files

/opt/syslog-ng/

/opt/syslog-ng/etc/syslog-ng.conf

See also

syslog-ng.conf(5)

Note

For the detailed documentation of syslog-ng OSE see the syslog-ng Documentation page

If you experience any problems or need help with syslog-ng, visit the syslog-ng mailing list.

For news and notifications about of syslog-ng, visit the syslog-ng blogs.

Author

This manual page was written by the One Identity Documentation Team.

Copyright

The authors grant permission to copy, distribute and/or modify this manual page under the terms of the GNU General Public License Version 2 or newer (GPL v2+).


Was this topic helpful?

[Select Rating]



syslog-ng.conf.5


Table of Contents

syslog-ng.conf— syslog-ng configuration file
Name

syslog-ng.conf — syslog-ng configuration file

Synopsis

syslog-ng.conf

Description

This manual page is only an abstract, for the complete documentation of syslog-ng, see the syslog-ng Documentation page or the syslog-ng page.

The syslog-ng OSE application is a flexible and highly scalable system logging application. Typically, syslog-ng is used to manage log messages and implement centralized logging, where the aim is to collect the log messages of several devices on a single, central log server. The different devices - called syslog-ng clients - all run syslog-ng, and collect the log messages from the various applications, files, and other sources. The clients send all important log messages to the remote syslog-ng server, where the server sorts and stores them.

Basic concepts of syslog-ng OSE

The syslog-ng application reads incoming messages and forwards them to the selected destinations. The syslog-ng application can receive messages from files, remote hosts, and other sources.

Log messages enter syslog-ng in one of the defined sources, and are sent to one or more destinations.

Sources and destinations are independent objects, log paths define what syslog-ng does with a message, connecting the sources to the destinations. A log path consists of one or more sources and one or more destinations: messages arriving from a source are sent to every destination listed in the log path. A log path defined in syslog-ng is called a log statement.

Optionally, log paths can include filters. Filters are rules that select only certain messages, for example, selecting only messages sent by a specific application. If a log path includes filters, syslog-ng sends only the messages satisfying the filter rules to the destinations set in the log path.

Other optional elements that can appear in log statements are parsers and rewriting rules. Parsers segment messages into different fields to help processing the messages, while rewrite rules modify the messages by adding, replacing, or removing parts of the messages.

Configuring syslog-ng
  • The main body of the configuration file consists of object definitions: sources, destinations, logpaths define which log message are received and where they are sent. All identifiers, option names and attributes, and any other strings used in the syslog-ng configuration file are case sensitive. Object definitions (also called statements) have the following syntax:

                  type-of-the-object identifier-of-the-object {<parameters>};
    • Type of the object: One of source, destination, log, filter, parser, rewrite rule, or template.

    • Identifier of the object: A unique name identifying the object. When using a reserved word as an identifier, enclose the identifier in quotation marks.

      All identifiers, attributes, and any other strings used in the syslog-ng configuration file are case sensitive.

      Tip:

      Use identifiers that refer to the type of the object they identify. For example, prefix source objects with s_, destinations with d_, and so on.

      Note

      Repeating a definition of an object (that is, defining the same object with the same id more than once) is not allowed, unless you use the @define allow-config-dups 1 definition in the configuration file.

    • Parameters: The parameters of the object, enclosed in braces {parameters}.

    • Semicolon: Object definitions end with a semicolon (;).

    For example, the following line defines a source and calls it s_internal.

                  source s_internal { internal(); };

    The object can be later referenced in other statements using its ID, for example, the previous source is used as a parameter of the following log statement:

                  log { source(s_internal); destination(d_file); };
  • The parameters and options within a statement are similar to function calls of the C programming language: the name of the option followed by a list of its parameters enclosed within brackets and terminated with a semicolon.

                  option(parameter1, parameter2); option2(parameter1, parameter2);

    For example, the file() driver in the following source statement has three options: the filename (/var/log/apache/access.log), follow-freq(), and flags(). The follow-freq() option also has a parameter, while the flags() option has two parameters.

                  source s_tail { file("/var/log/apache/access.log"
        follow-freq(1) flags(no-parse, validate-utf8)); };

    Objects may have required and optional parameters. Required parameters are positional, meaning that they must be specified in a defined order. Optional parameters can be specified in any order using the option(value) format. If a parameter (optional or required) is not specified, its default value is used. The parameters and their default values are listed in the reference section of the particular object.

    Example�1.�Using required and optional parameters

    The unix-stream() source driver has a single required argument: the name of the socket to listen on. Optional parameters follow the socket name in any order, so the following source definitions have the same effect:

    source s_demo_stream1 {
            unix-stream("<path-to-socket>" max-connections(10) group(log)); };
    source s_demo_stream2 {
            unix-stream("<path-to-socket>" group(log) max-connections(10)); };

  • Some options are global options, or can be set globally, for example, whether syslog-ng OSE should use DNS resolution to resolve IP addresses. Global options are detailed in ???.

    options { use-dns(no); };
  • Objects can be used before definition.

  • Objects can be defined inline as well. This is useful if you use the object only once (for example, a filter). For details, see ???.

  • To add comments to the configuration file, start a line with # and write your comments. These lines are ignored by syslog-ng.

                  # Comment: This is a stream source
    source s_demo_stream {
            unix-stream("<path-to-socket>" max-connections(10) group(log)); };

The syntax of log statements is as follows:

log {
    source(s1); source(s2); ...
    optional_element(filter1|parser1|rewrite1);
    optional_element(filter2|parser2|rewrite2);
    ...
    destination(d1); destination(d2); ...
    flags(flag1[, flag2...]);
};

The following log statement sends all messages arriving to the localhost to a remote server.

        source s_localhost { network(ip(127.0.0.1) port(1999)); };
destination d_tcp { network("10.1.2.3" port(1999) localport(999)); };
log { source(s_localhost); destination(d_tcp); };

The syslog-ng application has a number of global options governing DNS usage, the timestamp format used, and other general points. Each option may have parameters, similarly to driver specifications. To set global options, add an option statement to the syslog-ng configuration file using the following syntax:

        options { option1(params); option2(params); ... };

Example�2.�Using global options

To disable domain name resolving, add the following line to the syslog-ng configuration file:

options { use-dns(no); };

The sources, destinations, and filters available in syslog-ng are listed below. For details, see the syslog-ng Documentation page.

Table�1.�Source drivers available in syslog-ng

Name Description
file() Opens the specified file and reads messages.
wildcard-file() Reads messages from multiple files and directories.
internal() Messages generated internally in syslog-ng.
network() Receives messages from remote hosts using the BSD-syslog protocol over IPv4 and IPv6. Supports the TCP, UDP, and TLS network protocols.
nodejs() Receives JSON messages from nodejs applications.
mbox() Read e-mail messages from local mbox files, and convert them to multiline log messages.
osquery() Run osquery queries, and convert their results into log messages.
pacct() Reads messages from the process accounting logs on Linux.
pipe() Opens the specified named pipe and reads messages.
program() Opens the specified application and reads messages from its standard output.
snmptrap() Read and parse the SNMP traps of the Net-SNMP's snmptrapd application.
sun-stream(), sun-streams() Opens the specified STREAMS device on Solaris systems and reads incoming messages.
syslog() Listens for incoming messages using the new IETF-standard syslog protocol.
system() Automatically detects which platform syslog-ng OSE is running on, and collects the native log messages of that platform.
systemd-journal() Collects messages directly from the journal of platforms that use systemd.
systemd-syslog() Collects messages from the journal using a socket on platforms that use systemd.
unix-dgram() Opens the specified unix socket in SOCK_DGRAM mode and listens for incoming messages.
unix-stream() Opens the specified unix socket in SOCK_STREAM mode and listens for incoming messages.
stdin() Collects messages from the standard input stream.

Table�2.�Destination drivers available in syslog-ng

Name Description
amqp() Publishes messages using the AMQP (Advanced Message Queuing Protocol).
elasticsearch and elasticsearch6 Sends messages to an Elasticsearch server. The elasticsearch6 driver supports Elasticsearch version 2 and newer.
file() Writes messages to the specified file.
graphite() Sends metrics to a Graphite server to store numeric time-series data.
graylog2() Sends syslog messages to Graylog.
hdfs() Sends messages into a file on a Hadoop Distributed File System (HDFS) node.
http() Sends messages over the HTTP protocol. There are two different implementations of this driver: a Java-based http driver, and an http driver without Java.
kafka() Publishes log messages to the Apache Kafka message bus, where subscribers can access them.
loggly() Sends log messages to the Loggly Logging-as-a-Service provider.
logmatic() Sends log messages to the Logmatic.io Logging-as-a-Service provider.
mongodb() Sends messages to a MongoDB database.
network() Sends messages to a remote host using the BSD-syslog protocol over IPv4 and IPv6. Supports the TCP, UDP, and TLS network protocols.
pipe() Writes messages to the specified named pipe.
program() Forks and launches the specified program, and sends messages to its standard input.
redis() Sends messages as name-value pairs to a Redis key-value store.
riemann() Sends metrics or events to a Riemann monitoring system.
smtp() Sends e-mail messages to the specified recipients.
sql() Sends messages into an SQL database. In addition to the standard syslog-ng packages, the sql() destination requires database-specific packages to be installed. Refer to the section appropriate for your platform in ???.
stomp() Sends messages to a STOMP server.
syslog() Sends messages to the specified remote host using the IETF-syslog protocol. The IETF standard supports message transport using the UDP, TCP, and TLS networking protocols.
unix-dgram() Sends messages to the specified unix socket in SOCK_DGRAM style (BSD).
unix-stream() Sends messages to the specified unix socket in SOCK_STREAM style (Linux).
usertty() Sends messages to the terminal of the specified user, if the user is logged in.

Table�3.�Filter functions available in syslog-ng OSE

Name Description
facility() Filter messages based on the sending facility.
filter() Call another filter function.
host() Filter messages based on the sending host.
inlist() File-based whitelisting and blacklisting.
level() or priority() Filter messages based on their priority.
match() Use a regular expression to filter messages based on a specified header or content field.
message() Use a regular expression to filter messages based on their content.
netmask() Filter messages based on the IP address of the sending host.
program() Filter messages based on the sending application.
source() Select messages of the specified syslog-ng OSE source statement.
tags() Select messages having the specified tag.

Files

/opt/syslog-ng/

/opt/syslog-ng/etc/syslog-ng.conf

See also

syslog-ng(8)

Note

For the detailed documentation of syslog-ng OSE see the syslog-ng Documentation page

If you experience any problems or need help with syslog-ng, visit the syslog-ng mailing list.

For news and notifications about of syslog-ng, visit the syslog-ng blogs.

Author

This manual page was written by the One Identity Documentation Team.

Copyright

The authors grant permission to copy, distribute and/or modify this manual page under the terms of the GNU General Public License Version 2 or newer (GPL v2+).


Was this topic helpful?

[Select Rating]



Third-party contributions

This appendix includes the open source licenses and attributions applicable to syslog-ng Open Source Edition.


Was this topic helpful?

[Select Rating]



GNU General Public License

Version 2, June 1991

1989, 1991 Free Software Foundation, Inc.

Free Software Foundation, Inc. 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA

Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.

Version 2, June 1991


Was this topic helpful?

[Select Rating]



Related Documents