syslog-ng Open Source Edition 3.37 - Administration Guide

Preface Introduction to syslog-ng The concepts of syslog-ng Installing syslog-ng The syslog-ng OSE quick-start guide The syslog-ng OSE configuration file source: Read, receive, and collect log messages
How sources work default-network-drivers: Receive and parse common syslog messages internal: Collecting internal messages file: Collecting messages from text files wildcard-file: Collecting messages from multiple text files kubernetes: Collecting and parsing the Kubernetes CRI (Container Runtime Interface) format linux-audit: Collecting messages from Linux audit logs mqtt: receiving messages from an MQTT broker network: Collecting messages using the RFC3164 protocol (network() driver) nodejs: Receiving JSON messages from nodejs applications mbox: Converting local email messages to log messages osquery: Collect and parse osquery result logs pipe: Collecting messages from named pipes pacct: Collecting process accounting logs on Linux program: Receiving messages from external applications python: writing server-style Python sources python-fetcher: writing fetcher-style Python sources snmptrap: Read Net-SNMP traps sun-streams: Collecting messages on Sun Solaris syslog: Collecting messages using the IETF syslog protocol (syslog() driver) system: Collecting the system-specific log messages of a platform systemd-journal: Collecting messages from the systemd-journal system log storage systemd-syslog: Collecting systemd messages using a socket tcp, tcp6, udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol— OBSOLETE unix-stream, unix-dgram: Collecting messages from UNIX domain sockets stdin: Collecting messages from the standard input stream
destination: Forward, send, and store log messages
amqp: Publishing messages using AMQP collectd: sending metrics to collectd discord: Sending alerts and notifications to Discord elasticsearch2: Sending messages directly to Elasticsearch version 2.0 or higher (DEPRECATED) elasticsearch-http: Sending messages to Elasticsearch HTTP Bulk API file: Storing messages in plain-text files graphite: Sending metrics to Graphite Sending logs to Graylog hdfs: Storing messages on the Hadoop Distributed File System (HDFS) Posting messages over HTTP http: Posting messages over HTTP without Java kafka: Publishing messages to Apache Kafka (Java implementation) kafka-c(): Publishing messages to Apache Kafka using the librdkafka client (C implementation) loggly: Using Loggly logmatic: Using Logmatic.io mongodb(): Storing messages in a MongoDB database mqtt() destination: sending messages from a local network to an MQTT broker network: Sending messages to a remote log server using the RFC3164 protocol (network() driver) osquery: Sending log messages to osquery's syslog table pipe: Sending messages to named pipes program: Sending messages to external applications pseudofile() python: writing custom Python destinations redis: Storing name-value pairs in Redis riemann: Monitoring your data with Riemann slack: Sending alerts and notifications to a Slack channel smtp: Generating SMTP messages (email) from logs snmp: Sending SNMP traps Splunk: Sending log messages to Splunk sql: Storing messages in an SQL database stomp: Publishing messages using STOMP Sumo Logic destinations: sumologic-http() and sumologic-syslog() syslog: Sending messages to a remote logserver using the IETF-syslog protocol syslog-ng(): Forward logs to another syslog-ng node tcp, tcp6, udp, udp6: Sending messages to a remote log server using the legacy BSD-syslog protocol (tcp(), udp() drivers) Telegram: Sending messages to Telegram unix-stream, unix-dgram: Sending messages to UNIX domain sockets usertty: Sending messages to a user terminal: usertty() destination Write your own custom destination in Java or Python Client-side failover
log: Filter and route log messages using log paths, flags, and filters Global options of syslog-ng OSE TLS-encrypted message transfer template and rewrite: Format, modify, and manipulate log messages parser: Parse and segment structured messages
Parsing syslog messages Parsing messages with comma-separated and similar values Parsing key=value pairs JSON parser XML parser Parsing dates and timestamps Python parser Parsing tags Apache access log parser Linux audit parser Cisco parser Parsing enterprise-wide message model (EWMM) messages iptables parser Netskope parser panos-parser(): parsing PAN-OS log messages Sudo parser MariaDB parser Websense parser Fortigate parser Check Point Log Exporter parser Regular expression (regexp) parser db-parser: Process message content with a pattern database (patterndb)
Correlating log messages Enriching log messages with external data Statistics of syslog-ng Multithreading and scaling in syslog-ng OSE Troubleshooting syslog-ng Best practices and examples The syslog-ng manual pages Creative Commons Attribution Non-commercial No Derivatives (by-nc-nd) License Glossary

Configuring Microsoft SQL Server to accept logs from syslog-ng

Complete the following steps to configure your Microsoft SQL Server to enable remote logins and accept log messages from syslog-ng.

  1. Start the SQL Server Management Studio application. Select Start > Programs > Microsoft SQL Server 2005 > SQL Server Management Studio.

  2. Create a new database.

    1. Figure 5: Creating a new MSSQL database 1.

      In the Object Explorer, right-click on the Databases entry and select New Database.

    2. Figure 6: Creating a new MSSQL database 2.

      Enter the name of the new database (for example, syslogng) into the Database name field and click OK.

  3. Create a new database user and associate it with the new database.

    1. Figure 7: Creating a new MSSQL user 1.

      In the Object Explorer, select Security, right-click on the Logins entry, then select New Login.

    2. Figure 8: Creating a new MSSQL user 2.

      Enter a name (for example, syslog-ng) for the user into the Login name field.

    3. Select the SQL Server Authentication option and enter a password for the user.

    4. In the Default database field, select the database created in Step 2 (for example, syslogng).

    5. In the Default language field, select the language of log messages that you want to store in the database, then click OK.

      Caution:

      Incorrect language settings may result in the database converting the messages to a different character-encoding format. That way the log messages may become unreadable, causing information loss.

    6. In the Object Explorer, select Security > Logins, then right-click on the new login created in the previous step, and select Properties.

    7. Figure 9: Associating database with the new user

      Select User Mapping. In the Users mapped to this login option, check the line corresponding to the new login (for example, syslogng). In the Database role membership field, check the db_owner and public options.

  4. Figure 10: Associating database with the new user

    Enable remote logins for SQL users.

    In the Object Explorer right-click on your database server, and select Properties > Security, and set the Server Authentication option to SQL Server and Windows Authentication mode.


Was this topic helpful?

[Select Rating]



The syslog-ng OSE quick-start guide

This chapter provides a very brief introduction into configuring the syslog-ng OSE application. For details on the format of the configuration file and how to configure sources, destinations, and other features, refer to the subsequent chapters.


Was this topic helpful?

[Select Rating]



Configuring syslog-ng on client hosts

Purpose:

To configure syslog-ng on a client host, complete the following steps.

Steps:
  1. Install the syslog-ng application on the host. For details installing syslog-ng on specific operating systems, see Installing syslog-ng.

  2. Configure the local sources to collect the log messages of the host. Starting with version 3.2, syslog-ng OSE automatically collects the log messages that use the native system logging method of the platform, for example, messages from /dev/log on Linux, or /dev/klog on FreeBSD. For a complete list of messages that are collected automatically, see system: Collecting the system-specific log messages of a platform.

    To configure syslog-ng OSE, edit the syslog-ng.conf file with any regular text editor application. The location of the configuration file depends on how you installed syslog-ng OSE. Native packages of a platform (like the ones downloaded from Linux repositories) typically place the configuration file under the /etc/syslog-ng/ directory.

    Add sources to collect the messages from your log files. File sources look like this:

    source s_myfilesource {
        file("/var/log/myapplication.log" follow-freq(1));
    };

    Name every source uniquely. For details on configuring file sources, see file: Collecting messages from text files.

    TIP: Many applications send log messages to logfiles by default (for example, the Roundcube webmail client, or the ProFTPD FTP server), but can be configured to send them to syslog instead. If possible, it is recommended to reconfigure the application that way.

    NOTE: The default configuration file of syslog-ng OSE collects platform-specific log messages and the internal log messages of syslog-ng OSE.

    source s_local {
        system();
        internal();
    };
  3. Create a network destination that points directly to the syslog-ng server, or to a local relay. The network destination greatly depends on the protocol that your log server or relay accepts messages. Many systems still use the legacy BSD-syslog protocol (RFC3162) over the unreliable UDP transport:

    destination d_network { network("10.1.2.3" transport("udp")); };

    However, if possible, use the much more reliable IETF-syslog protocol over TCP transport:

    destination d_network {
        syslog("10.1.2.3" transport("tcp"));
    };
  4. Create a log statement connecting the local sources to the syslog-ng server or relay. For example:

    log {
        source(s_local); destination(d_network);
    };
  5. If the logs will also be stored locally on the host, create local file destinations.

    NOTE: The default configuration of syslog-ng OSE places the collected messages into the /var/log/messages file:

    destination d_local {
        file("/var/log/messages");
    };
  6. Create a log statement connecting the local sources to the file destination.

    NOTE: The default configuration of syslog-ng OSE has only one log statement:

    log {
        source(s_local); destination(d_local);
    };
  7. Set filters, macros and other features and options (for example, TLS encryption) as necessary.

    Example: The default configuration file of syslog-ng OSE

    The following is the default configuration file of syslog-ng OSE3.37. It collects local log messages and the log messages of syslog-ng OSE and saves them in the /var/log/messages file.

    @version: 3.37
    @include "scl.conf"
    source s_local {
        system(); internal();
    };
    destination d_local {
        file("/var/log/messages");
    };
    log {
        source(s_local); destination(d_local);
    };
    Example: A simple configuration for clients

    The following is a simple configuration file that collects local log messages and forwards them to a logserver using the IETF-syslog protocol.

    @version: 3.37
    @include "scl.conf"
    source s_local {
        system(); internal();
    };
    destination d_syslog_tcp {
        syslog("192.168.1.1" transport("tcp") port(2010));
    };
    log {
        source(s_local);destination(d_syslog_tcp);
    };

Was this topic helpful?

[Select Rating]



Configuring syslog-ng on server hosts

Purpose:

To configure syslog-ng on a server host, complete the following steps.

Steps:
  1. Install the syslog-ng application on the host. For details installing syslog-ng on specific operating systems, see Installing syslog-ng.

  2. Starting with version 3.2, syslog-ng OSE automatically collects the log messages that use the native system logging method of the platform, for example, messages from /dev/log on Linux, or /dev/klog on FreeBSD. For a complete list of messages that are collected automatically, see system: Collecting the system-specific log messages of a platform.

  3. To configure syslog-ng OSE, edit the syslog-ng.conf file with any regular text editor application. The location of the configuration file depends on how you installed syslog-ng OSE. Native packages of a platform (like the ones downloaded from Linux repositories) typically place the configuration file under the /etc/syslog-ng/ directory.

    Configure the network sources that collect the log messages sent by the clients and relays. How the network sources should be configured depends also on the capabilities of your client hosts: many older networking devices support only the legacy BSD-syslog protocol (RFC3164) using UDP transport:

    source s_network {
        syslog(ip(10.1.2.3) transport("udp"));
    };

    However, if possible, use the much more reliable TCP transport:

    source s_network {
        syslog(ip(10.1.2.3) transport("tcp"));
    };

    For other options, see syslog: Collecting messages using the IETF syslog protocol (syslog() driver) and tcp, tcp6, udp, udp6: Collecting messages from remote hosts using the BSD syslog protocol— OBSOLETE.

    NOTE: Starting with syslog-ng OSE version 3.2, the syslog() source driver can handle both BSD-syslog (RFC 3164) and IETF-syslog (RFC 5424-26) messages.

  4. Create local destinations that will store the log messages, for example, file- or program destinations. The default configuration of syslog-ng OSE places the collected messages into the /var/log/messages file:

    destination d_local {
        file("/var/log/messages");
    };

    If you want to create separate logfiles for every client host, use the ${HOST} macro when specifying the filename, for example:

    destination d_local {
        file("/var/log/messages_${HOST}");
    };

    For details on further macros and how to use them, see template and rewrite: Format, modify, and manipulate log messages.

  5. Create a log statement connecting the sources to the local destinations.

    log {
        source(s_local); source(s_network); destination(d_local);
    };
  6. Set filters, options (for example, TLS encryption) and other advanced features as necessary.

    NOTE: By default, the syslog-ng server will treat the relayed messages as if they were created by the relay host, not the host that originally sent them to the relay. In order to use the original hostname on the syslog-ng server, use the keep-hostname(yes) option both on the syslog-ng relay and the syslog-ng server. This option can be set individually for every source if needed.

    If you are relaying log messages and want to resolve IP addresses to hostnames, configure the first relay to do the name resolution.

    Example: A simple configuration for servers

    The following is a simple configuration file for syslog-ng Open Source Edition that collects incoming log messages and stores them in a text file.

    @version: 3.37
    @include "scl.conf"
    options {
        time-reap(30);
        mark-freq(10);
        keep-hostname(yes);
    };
    source s_local {
        system(); internal();
    };
    source s_network {
        syslog(transport(tcp));
    };
    destination d_logs {
        file(
            "/var/log/syslog-ng/logs.txt"
            owner("root")
            group("root")
            perm(0777)
            );
        };
    log {
        source(s_local); source(s_network); destination(d_logs);
    };

Was this topic helpful?

[Select Rating]



Related Documents