Syslog-ng on the edge

After many years of pushing all computing from on-site to the cloud or huge data centers, there is a new trend: edge computing. There can be many reasons, legal or practical, why data should be processed locally instead of being sent to a central location as soon as it is created. Edge computing was a central theme of last week’s Red Hat Summit. Luckily syslog-ng is well prepared for this use case right from the beginning. While most people only know that syslog-ng can act as a client or a server, it can also collect, process and forward log messages. In syslog-ng terminology it is called a relay, but on the edge you might want to combine server and a relay functionality into one.

Why edge?

Everyone is talking about centralization and economy of scale. So, why people are making a U-turn and work on decentralization and infrastructure on-premise? It is not really a U-turn. It is an acknowledgment, that you cannot centralize everything. Management is still done centrally, but some or most of the processing and storage of data happens on-premise instead of the cloud or data center.

It might be a legal requirement. Some examples:

• No personal data might leave a given country, so any log processing and storage needs to happen in the local office instead of your data center in another country.

• Due to GDPR you want to keep European data in European offices, where privacy of data is better ensured.

• In an office dealing with sensitive data, no log data might leave the building at all.

It also might be a practical reason. For example:

• The amount of log messages created in your branch offices would be too expensive to transfer to a central location.

• Logs collected on a ship, car (yes, syslog-ng runs on the BMW i3 electric car) or an airplane (a security feature by Airbus was just merged into syslog-ng 3.27) often cannot be forwarded immediately or at all to a central location.

How?

Earlier I’ve mentioned the syslog-ng relay functionality. It is just part of the story. A syslog-ng relay collects log messages, optionally processes them and then forwards the logs to a central server or the next relay in the chain. From the syslog-ng Premium Edition (PE) point of view, a relay does not store incoming log messages, only forwards them. You can find more information why you should introduce relays in your infrastructure in my earlier blog.

When using syslog-ng on the edge, in most situations you not just forward log messages but also store them. The disk-buffer of syslog-ng is good for short term storage, but when the next syslog-ng is unavailable for an extended period of time, it is more practical to save logs locally. And in many situations you want to forward only a fraction of your log messages anyway and store the rest locally. From the syslog-ng PE point of view it means a separate server license.

As you can see, running syslog-ng on the edge has the same challenges as running your central syslog-ng server. You collect log messages, parse them, filter them and finally store most of them locally and forward some of them to another server. You also need to rotate logs and solve archival of log messages. You do not have to learn new skills, only apply your existing syslog-ng knowledge to the new situation.

If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik.