When you send your log messages from a syslog-ng OSE client through the network to a syslog-ng OSE server, you can use different protocols and options. Every combination has its advantages and disadvantages. The most important thing is to use matching protocols and options, so the server handles the incoming log messages properly.
In syslog-ng OSE you can change many aspects of the network communication. First of all, there is the structure of the messages itself. Currently, syslog-ng OSE supports two standard syslog protocols: the BSD (RFC3164) and the syslog (RFC5424) message format.
These RFCs describe the format and the structure of the log message, and add a (lightweight) framing around the messages. You can set this framing/structure by selecting the appropriate driver in syslog-ng OSE. There are two drivers you can use: the network() driver and the syslog() driver. The syslog() driver is for the syslog (RFC5424) protocol and the network() driver is for the BSD (RFC3164) protocol.
The tcp() and udp() drivers are now deprecated, they are essentially equivalent with the network(transport(tcp)) and network(transport(udp)) drivers.
In addition to selecting the driver to use, both drivers allow you to use different transport-layer protocols: TCP and UDP, and optionally also higher-level transport protocols: TLS (over TCP. To complicate things a bit more, you can configure the network() driver (corresponding to the BSD (RFC3164) protocol) to send the messages in the syslog (RFC5424) format (but without the framing used in RFC5424) using the flag(syslog-protocol) option.
Because some combination of drivers and options are invalid, you can use the following drivers and options as sources and as destinations:
If you use the same driver and options in the destination of your syslog-ng OSE client and the source of your syslog-ng OSE server, everything should work as expected. Unfortunately there are some other combinations, that seem to work, but result in losing parts of the messages. The following table show the combinations:
|Source \ Destination||syslog/tcp||syslog/udp||syslog/tls||network/tcp||network/udp||network/tls||network/tcp/flag||network/udp/flag||network/tls/flag|
- This method does not work. The logs will not get to the server.
✔ This method works.
! This method has some visible drawbacks. The logs go through, but some of the values are missing/misplaced/and so on.
✔? This method seems to work, but it is not recommended because this can change in a future release.
The syslog-ng application has a commercial version available, called syslog-ng Premium Edition (syslog-ng PE). The commercial version comes with well-tested features from its open source foundation, a number of extra features, enterprise-level support, as well as a ready-to-use log management appliance built on the strengths of syslog-ng Premium Edition.
Collecting and analyzing log messages is required directly or indirectly by several regulations, frameworks, and standards, including the Sarbanes-Oxley Act (SOX), the Health Insurance and Portability Act (HIPAA), and the Payment Card Industry Data Security Standard (PCI-DSS). syslog-ng PE provides a set of features that help you comply with regulations that require the central collection of log messages in a tamperproof way:
Logstore files enable you to store log messages securely in encrypted, compressed and timestamped binary files. From a compliance point of view, this serves a double purpose. Encryption guarantees the integrity of log messages so you can be sure that they have not been manipulated. Timestamping provides verifiable proof about the exact time when log messages arrived.
syslog-ng Premium Edition comes with tested binary files that are available for a wide array of server platforms, reducing the time required for installation and maintenance. Support for a wide range of operating system and hardware platforms also make syslog-ng PE an ideal choice to collect logs in massively heterogeneous environments.
As all commercial software, syslog-ng PE also comes with various enterprise-level support packages, which means that you get immediate and pro-active assistance (24x7 if you choose a top-tier package), dedicated to resolving your issue as soon as possible when you experience problems.
For more information about syslog-ng Premium Edition, see The syslog-ng Premium Edition Administrator Guide.
syslog-ng Store Box (SSB) is a log management appliance that is built on syslog-ng Premium Edition. It is a turnkey solution to manage your log data, meaning that no software installation is necessary. As SSB is available both as a virtual machine and a physical appliance, it is also easily scalable.
SSB provides a number of features that can add value for your use cases:
A web GUI that makes searching logs, as well as configuring and managing SSB itself easy:
The search interface allows you to use wildcards and Boolean operators to perform complex searches, and drill down on the results. You can gain a quick overview and pinpoint problems fast by generating ad-hoc charts from the distribution of the log messages.
In addition, you can easily create customized reports from the charts and statistics you create on the search interface to demonstrate compliance with standards and regulations such as PCI-DSS, ISO 27001, SOX and HIPAA.
Configuring SSB is done through the user interface. All of the flexible filtering, classification and routing features in the syslog-ng Open Source Edition and syslog-ng Premium Edition can be configured with it. Access and authentication policies can be set to integrate with Microsoft Active Directory, LDAP and Radius servers. The web interface is accessible through a network interface dedicated to management traffic. This management interface is also used for backups, sending alerts, and other administrative traffic.
High availability support to ensure continuous log collection in business-critical environments.
For further details about syslog-ng Store Box, see The syslog-ng Store Box Administrator Guide.
If you wish to upgrade from syslog-ng OSE to syslog-ng PE, read the blog post Upgrading from syslog-ng OSE to syslog-ng PE for instructions and tips.
This chapter explains how to install syslog-ng Open Source Edition on various platforms.
You can install syslog-ng OSE on many platforms using the package manager and official repositories of the platform. For a list of third-party packages available for various Linux, UNIX, and other platforms, see the syslog-ng OSE third-party binaries page.
For instructions on compiling syslog-ng Open Source Edition from the source code, see Compiling syslog-ng from source.
You can use a syslog-ng docker image.
For detailed information on how to run your central log server in Docker and other Docker-related syslog-ng use cases, see the Logging in Docker using syslog-ng white paper.
To compile syslog-ng Open Source Edition (OSE) from the source code, complete the following steps. Alternatively, you can use precompiled binary packages on several platforms. For a list of third-party packages available for various Linux, UNIX, and other platforms, see the syslog-ng OSE third-party binaries page.
Download the latest version of syslog-ng OSE from GitHub. The source code is available as a tar.gz archive file.
Install the following packages that are required to compile syslog-ng. These packages are available for most UNIX/Linux systems. Alternatively, you can also download the sources and compile them.
A version of the gcc C compiler that properly supports Thread Local Storage (TLS), for example, version 4.5.
The GNU flex lexical analyser generator, available here.
The bison parser generator, available here.
The development files of the glib library, available here.
The development files of the Autoconf Archive package, available here.
The syslog-ng OSE application now uses PCRE-type regular expressions by default. It requires the libpcre library package, available here.
If you want to use the Java-based modules of syslog-ng OSE (for example, the Elasticsearch, HDFS, or Kafka destinations), you must compile syslog-ng OSE with Java support.
Download and install the Java Runtime Environment (JRE), 1.7 (or newer).
Install gradle version 2.2.1 or newer.
Set LD_LIBRARY_PATH to include the libjvm.so file, for example:LD_LIBRARY_PATH=/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server:$LD_LIBRARY_PATH
Note that many platforms have a simplified links for Java libraries. Use the simplified path if available. If you use a startup script to start syslog-ng OSE set LD_LIBRARY_PATH in the script as well.
If you are behind an HTTP proxy, create a gradle.properties under the modules/java-modules/ directory. Set the proxy parameters in the file. For details, see The Gradle User Guide.
If you want to post log messages as HTTP requests using the http() destination, install the development files of the libcurl library. This library is not needed if you use the --disable-http compile option. Alternatively, you can use a Java-based implementation of the HTTP destination.
If you want to use the spoof-source function of syslog-ng, install the development files of the libnet library, available here.
If you want to send e-mails using the smtp() destination, install the development files of the libesmtp library. This library is not needed if you use the --disable-smtp compile option.
If you want to use the /etc/hosts.deny and /etc/hosts.allow for TCP access, install the development files of the libwrap (also called TCP-wrappers) library, available here.
Enter the new directory and issue the following commands. (If the ./configure file does not exist, for example, because you cloned the repository from GitHub instead of using a release tarball, execute the ./autogen.sh command.)
$ ./configure $ make $ make install
Uncompress the syslog-ng archive using the
tar xvfz syslog-ng-x.xx.tar.gz
unzip -c syslog-ng-x.xx.tar.gz | tar xvf -
command. A new directory containing the source code of syslog-ng will be created.
Enter the new directory and issue the following commands:
$ ./configure $ make $ make install
These commands will build syslog-ng using its default options.
If needed, use the following options to change how syslog-ng is compiled using the following command syntax:
$ ./configure --compile-time-option-name
You can also use --disable options, to explicitly disable a feature and override autodetection. For example, to disable the TCP-wrapper support, use the --disable-tcp-wrapper option. For the list of available compiling options, see Compiling options of syslog-ng OSE.
The default linking mode of syslog-ng is dynamic. This means that syslog-ng might not be able to start up if the /usr directory is on NFS. On platforms where syslog-ng is used as a system logger, the --enable-mixed-linking is preferred.