Optimize your SIEM with syslog-ng
Whether it's user activity, performance metrics, Windows events logs, network traffic or any other log data, syslog-ng can collect and centralize log data. You can remove data silos and gain full-stack visibility of your IT environment. Depending on its configuration, one syslog-ng server can collect more than half a million log message per second from thousands of log sources.
Have confidence in the data underlying your analytics, forensics and compliance efforts.
Using local disk buffering, client-side failover and application layer acknowledgement, syslog-ng can transfer logs with zero message loss. Encrypted transfer and storage ensure logs cannot be tampered with, preserving the digital chain of custody.
syslog-ng Premium Edition can send and receive log messages in a reliable way over the TCP transport layer using the Advanced Log Transfer Protocol™ (ALTP™).
ALTP™ is a new transport protocol that prevents message loss during connection breaks.
Log messages may contain sensitive information that should not be accessed by third parties. Therefore, syslog-ng Premium Edition uses the Transport Layer Security (TLS) protocol to encrypt the communication.
TLS also allows the mutual authentication of the host and the server using X.509 certificates.
syslog-ng Premium Edition can store log messages securely in encrypted, compressed, indexed and timestamped binary files, so any sensitive data is available only for authorized personnel who have the appropriate encryption key.
Timestamps can be requested from external timestamping authorities.
Depending on its configuration, one syslog-ng server can collect more than half a million log messages per second from thousands of log sources.
A single central server can collect log messages from more than 5,000 log source hosts. When deployed in a client relay configuration, a single syslog-ng log server can collect logs from tens of thousands of log sources.
The syslog-ng application is optimized for performance, and can handle an enormous amount of messages.
Depending on its exact configuration, it can process over half a million messages per second in real-time, and over 24 GB of raw logs per hour on standard server hardware.
With the syslog-ng client-relay architecture, IT organizations can collect log messages from more than 10,000 log sources across a geographically distributed environment on one central log server.
syslog-ng allows you to granularly select which statistics of syslog-ng you want to monitor. The statistics are available as structured name-value pairs, so you can format the output similarly to other log messages.
That way, you can easily convert the statistics and metrics and send the results into your enterprise monitoring solution (for example, IBM Tivoli Netcool, Riemann, Redis, or Graphite).
This short video will show you how syslog-ng scales to the largest IT environments, ensuring your log infrastructure can reliably and securely collect and manage log data.
syslog-ng can be deployed as an agent on a wide variety of hosts and flexibly route logs to multiple analytic tools or databases, eliminating the need to deploy multiple agents on servers.
Tested binary files for the syslog-ng Premium Edition are available for more than 50 server platforms, reducing the time required for installation and maintenance.
syslog-ng Premium Edition can natively collect and process log messages from SQL databases, enabling users to easily manage log messages from a wide variety of enterprise software and custom applications.
The syslog-ng Agent for Windows is an event log collector and forwarder application for Microsoft Windows platforms.
Some applications use many different log files, and sometimes these files are not even located in the same folder. Automatically generated file and folder names are also often a problem.
To solve these issues, the filenames and paths specifying the log files read by syslog-ng can include wildcards, and syslog-ng can automatically scan entire subfolder trees for the specified files.
The syslog-ng Premium Edition application is also able to process multi-line log messages, for example, Apache Tomcat messages.
Many large organizations need to send their logs to multiple log analysis tools. Different groups, including IT operations, IT security and corporate risk and governance, need access to the same log data but have different log analysis goals and tools.
The syslog-ng application can send logs directly to SQL databases, MongoDB and Hadoop Distributed File System (HDFS) nodes, or use the Standard Network Management Protocol (SNMP) and Simple Mail Transfer Protocol (SMTP) for other destinations.
With powerful filtering, parsing, re-writing and classification options, syslog-ng can transform logs on remote hosts, reducing the amount and complexity of log data forwarded to analytic tools like SIEM or APM, reducing their total cost of ownership.
The flexible configuration language allows users to construct powerful, complex log processing systems on remote hosts with simple rules.
syslog-ng can sort the incoming log messages based on their content and various parameters like the source host, application, and priority. Directories, files, and database tables can be created dynamically using macros.
Complex filtering using regular expressions and boolean operators offers almost unlimited flexibility to forward only the important log messages to the selected destinations.
By comparing log messages to known patterns, syslog-ng is able to identify the exact type of the messages, and sort them into message classes. The message classes can then be used to classify the type of the event described in the log message.
The message classes can be customized, and for example can label the messages as user login, application crash, file transfer, etc. events.
syslog-ng can use an external database file to append custom name-value pairs to incoming logs, thus extending, enriching, and complementing the data found in the log message.
You can also correlate and aggregate information from log messages using a few simple filters that are similar to SQL GROUPBY statements.
syslog-ng stores messages on the local hard disk if the central log server or the network connection becomes unavailable. The syslog-ng application automatically sends the stored messages to the server when the connection is re-established, in the same order the messages were received. The disk buffer is persistent - no messages are lost even if syslog-ng is restarted.
Flow control uses a control window to determine if there is free space in the output buffer of syslog-ng for new messages. If the output buffer is full and the destination cannot accept new messages for some reason, for example it's overloaded or the network connection has become unavailable. In such cases, syslog-ng stops reading messages from the source until some messages have been successfully sent to the destination.
By comparing log messages to known patterns, syslog-ng is able to identify the exact type of the messages, and sort them into message classes. The message classes can be used to classify the type of the event described in the log message. The message classes can be customized, and, for example, can label the messages as user login, application crash, file transfer, etc.
The Python log parser allows you to write your own parsers in Python. Practically, that way you can process the log message (or parts of the log message) any way you need. You can also write your own template functions in Python.
syslog-ng can compare the contents of the log messages to a database of predefined message patterns.
syslog-ng PE can read these traps from a log file, and extract their content into name-value pairs, making it easy to forward them as a structured log message (for example, in JSON format).
In addition to classifying messages, you can also add different tags which can be used later for filtering messages. For example, to collect messages tagged as user_login to a separate file or to perform conditional post processing on the tagged messages.
syslog-ng also makes real time event correlation possible. This can be useful in many different situations, for example important data for a single event is often scattered into multiple syslog messages. Also login and logout events are often logged far away from each other, even in different log files, making log analysis difficult. Using correlation, these can be collected into a single new message.
The Windows Event Collector (WEC) is a stand-alone log-collector and-forwarder tool for the Microsoft Windows platform. It collects log messages from Windows hosts and forwards them – by source-initiated push subscriptions and WinRM protocol - to a syslog-ng Premium Edition server (7.0 or later). Because it’s a remote solution and doesn’t require installation on the host itself, WEC is an ideal log solution for IT environments that want to minimize the use of agent software.
Collect, process, store, search and audit your logs with enterprise log management software
syslog-ng is a single, high-performance log collector for all of your device-, system-, and application logs, giving you the ability to greatly simplify your logging architecture.
syslog-ng is a log management application that enables you to collect logs from multiple platforms in a central space. Whether it's user activity, performance metrics, network traffic or any other log data, syslog-ng can collect and centralize log data
Most log messages traditionally come from three types: system logs of servers, network devices, and applications.
Operating systems, applications, and network devices generate text messages of the events that happen to them: a user logs in, a file is created, a network connection is opened to a remote host.
Collect, process, and deliver logs to a wide variety of destinations with a flexible log management
Collect, process, store, search and audit your logs with enterprise log management software
Collect, process, store, search and audit log data with a enterprise log management appliance