Python source in syslog-ng

Using syslog-ng 3.18 and newer releases, you can write new source drivers for syslog-ng in Python. While performance is not as good as C, you gain flexibility and ease of implementation. There are quite a few log sources without a ready to use C API, but with a Python API. Using the Python source of syslog-ng you can leverage these.

From this blog you can learn the very basics of the Python source. The example code is good enough to fill up your hard drive quickly. For a complete list of options, check the documentation of the syslog-ng Python source http://support.oneidentity.com/technical-documents/syslog-ng-open-source-edition/3.18/administration-guide/source-read-receive-and-collect-log-messages/python-writing-server-style-python-sources

Before you begin

The Python source was released with version 3.18.1 of syslog-ng. There is a good chance that your Linux distribution of choice does not have this version yet, or does not have Python support enabled at all. Check https://www.syslog-ng.com/products/open-source-log-management/3rd-party-binaries.aspx for a list of 3^rd party package sources for up to date packages. Python support is usually not included in the core package, but available as a sub-package. Usually it is called syslog-ng-python, syslog-ng-mod-python, or something similar, depending on the distribution you use.

There are two ways to store the code of your Python source . For short code you can store the Python code inline in your syslog-ng configuration. In this blog I use this simple method. While for short code is is much easier to include it inline in the syslog-ng configuration, there are advantages of storing Python code separately, not just for long or complex code. For example you can easily validate code with pylint, have syntax highlighting, or use a dedicated Python IDE, like IDLE or Spyder3. You can learn how to do that from my Python destination blog at https://www.syslog-ng.com/community/b/blog/posts/python-destination-getting-into-details

My example code was only tested with Python 3, but most likely works with Python 2 with small modifications.

Configuration / code

source s_python {
    python(
        class("MySource")
        options(
            "option1" "value1",
            "option2" "value2"
        )
    );
};

python {
from syslogng import LogSource
from syslogng import LogMessage
 
class MySource(LogSource):
    def init(self, options): # optional
        print("init")
        print(options)
        self.exit = False
        return True
 
    def run(self): # mandatory
        print("run")
        while not self.exit:
            msg = LogMessage("this is a log message")
            self.post_message(msg)
 
    def request_exit(self): # mandatory
        print("exit")
        self.exit = True
};

destination d_file {
    file("/var/log/python.txt");
};

log { source(s_python); destination(d_file); };

Note: the above code generated two and a half million log messages in less than ten seconds in a resource constrained virtual machine on my laptop. Use with care!

As you can see (and might remember from the Python destination or parser), there are two major parts of the configuration: the actual syslog-ng configuration, and the Python source code.

Syslog-ng configuration

As with any other Python bindings in syslog-ng, you have to refer to the Python code from the syslog-ng configuration. In this case you have a source called s_python. The only mandatory option of the Python source is class() where you name the Python class of your source driver. You can also pass options from the syslog-ng configuration to the Python code. This is optional, but helps to make your Python code more generic and reusable.

In order to see the collected log messages from the Python source, you also need a destination and a log statement connecting the Python source with that destination. In the above configuration the destination is a file called python.txt under the /var/log/ directory.

Python source code

You can store the Python source code either inline in the syslog-ng configuration or in external files. Here it is included in the syslog-ng configuration enclosed in a python {} block.

The Python block starts with a number of imports:

• The source driver needs to be inherited from the syslogng.LogSource class.
• The message you pass to syslog-ng from your Python code needs to use the LogMessage API for log messages.

The init() method is optional. It initializes your source driver using the options() defined in the syslog-ng configuration. In the example code it only prints the values set in options() and sets up an attribute used later to stop the source driver.

If init() returns with False, syslog-ng stops. This can be used to check if options passed to init() are correct, and prevent syslog-ng from starting if there is any problem.

The run() method is mandatory. This is where your event loop runs. In the above code it is a simple while cycle, running as long as the exit attribute is false. It creates a very simple log message using the LogMessage API. You can learn more about it at http://support.oneidentity.com/technical-documents/syslog-ng-open-source-edition/3.18/administration-guide/source-read-receive-and-collect-log-messages/python-writing-server-style-python-sources/python-logmessage-api

Finally, you also have to implement the mandatory request_exit method. This is called when syslog-ng is reloaded or stopped and used to stop the run() method. In the sample code it changes the exit attribute to True, which controls the event loop in the run() method.

Testing

Before testing, make sure that you have enough disk space on the partition hosting your /var/log/ directory.

As soon as you saved your configuration, you are ready for testing. You do not need any external commands, as syslog-ng generates the test messages itself. Once you started syslog-ng, you should see similar lines when running tail:

# tail /var/log/python.txt 
Nov  7 12:20:31 linux-6chp this is a log message
Nov  7 12:20:31 linux-6chp this is a log message
Nov  7 12:20:31 linux-6chp this is a log message
Nov  7 12:20:31 linux-6chp this is a log message
Nov  7 12:20:31 linux-6chp this is a log message
Nov  7 12:20:31 linux-6chp this is a log message
Nov  7 12:20:31 linux-6chp this is a log message
Nov  7 12:20:31 linux-6chp this is a log message
Nov  7 12:20:31 linux-6chp this is a log message
Nov  7 12:20:31 linux-6chp this is a log message

Stop syslog-ng before it fills up your disk.

If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or you can even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/balabit/syslog-ng. On Twitter, I am available as @PCzanik.