Hello, and thank you for joining me for this segment at Virtual Unite. My name is James Bonamico, and I'm a senior sales engineer here at One Identity. So today we're going to talk about one of the newer custom destination drivers that's now available in syslog-ng premium edition. This allows you to take logs that have been processed by syslog-ng and send them by API to an Azure Sentinel workspace.
So as you know, syslog-ng is a universal enterprise log management solution designed from the bottom up for performance, scalability, and flexibility. It is truly the foundation of log management when it comes to being able to ingest nearly any type of log, from any platform, over any protocol, and parse, filter, process, and route them at unmatched speed and scale.
And by utilizing these capabilities to filter out less important logs and normalize critical parsed data into those formats required by SIEM systems in other log consumers, you're reducing complexity and increasing the efficiency of those other systems-- and ultimately, that results in the reduction of IT expenditure.
So part of the value of syslog-ng are all out of the box connectivity options. The latest push has been to accommodate the explosive growth of cloud platforms, so we've implemented several new cloud connectors-- such as Google Stackdriver, Google Pub/Sub, Microsoft Office 365, and of course, Azure Sentinel.
So what you see here is a very high-level architecture overview of syslog-ng deployment environment. So on the left, you have your various log sources-- such as cloud services, applications, security and network devices, of course, servers. And they'll be sending logs in all different types of formats over various protocols-- like UDP, TCP, encrypted via TLS-- into syslog-ng.
In many cases, you want to have a source like energy relay as the first ingest point. Now, if you have a relay installed at, say, at every data center, or a typical site, those relays are close to the sources, and they can ingest reliably. They can serve as aggregators. They can pre-process those logs if necessary, pre-filter them before sending them on to syslog-ng's server. And then that connection between the relay and server can be reliable as well as encrypted.
And then from there, syslog-ng server will then further parse and process and filter that data, and then send it out to the various log consumers, wherever you need those logs. Maybe those logs need to go into multiple different destinations, in different formats. Well, syslog-ng can do that. So whether you're writing to file, writing into a database-- Elastic, Kafka, Splunk, Hadoop, Google Cloud, and then, of course, Azure Sentinel-- those options, and others, are available to you.
So to get started with Sentinel, the first thing you need to do is create a log analytics workspace. Once you do this, you'll need to take note of two pieces of information. So that's the workspace ID and the primary key. With these credentials, we'll be able to connect the syslog-ng Sentinel destination driver with Azure Sentinel itself.
And to find these pieces of information, you can go to your workspace, then under Settings, you'll see all the Agent managements tab. However, with syslog-ng, we won't actually be needing Sentinel agents at all, since we'll be sending logs directly via the API. So that's one less thing you'll need to install and manage. But here's where the keys are listed, and can be regenerated if need be. So with this information from Microsoft in hand, it's time to edit the syslog-ng configuration file.
So here you can see that the basic configuration is extremely simple. You just name the destination, declare the Sentinel destination driver, and then plug in the workspace ID and auth secret that you've obtained from Azure. This is all you need to then create a log path and start sending logs from syslog-ng into Sentinel. However, there are a few other options available if you'd like to customize the data or connection, and I'll go over some of them here.
So first are the batching parameters, which are disabled by default. These all tell syslog-ng when to fire off a post command to the Sentinel API. If not specified, we'll send each log message as it comes in, but for efficiency and speed, you can tune this behavior. You can set the total amount of data accumulated as the threshold.
So for example, we'll send a bunch of messages in one go every, say, five megabytes. Or you can do the same when a certain number of lines are reached, like 10 or 100. Lastly, the batch timeout regulates when we'll send a batch after there's a break in the incoming stream, and the unsent data doesn't yet reach either of the other thresholds. So for example, 100 messages come in, but then there's no activity on the endpoint for a few minutes. We'll wait until the batch timeout to send any unsent messages, regardless of if they fill a batch or not.
So now for the actual data. So the fields parameter is the basic set of key value pairs included in the payload by default. If not specified, this is what is sent, but again, you don't need to add this line if you're not changing it. So you can customize this to include or exclude any set of elements. So you can see, we have a few pieces of information here. You can assign the macros however you like, but these are the defaults.
And then syslog-ng also gives you the ability to override the body of the message itself. Again, here are the defaults, which simply take all the key value elements in the fields parameter and arrange them in a JSON data structure. So it's JSON that we're actually sending into Azure.
And then, here are some examples of the various ways you can customize this. You can add scopes, which in syslog-ng energy are predefined sets of structured data elements, which can be inherently part of the message, or information that's extracted by built-in in or custom syslog-ng parsers. So you have full granular control over the data that is sent to Sentinel, even after filtering of the messages themselves.
Then lastly, there's the log type parameter. So this is mandatory for the Sentinel API, and its default value is Syslog_CL.
So one thing to note about Sentinel log types is that there are two categories. The first are built-in in, and the second are external. So to differentiate these, and prevent users from overloading the built-in type definitions, Sentinel will automatically apply the CL suffix to log events that originate from external sources. So this way, it knows all events arriving via the API are custom log types. And then when you want to store messages of different log types that all connect to the same workspace ID, you can simply create multiple Sentinel destinations that have a common workspace ID, but different log type parameters.
So now I'll show you the actual destination driver in action. So I'm going to switch over to my terminal running syslog-ng.
So here it is. Normally syslog-ng runs as a service in the background, but here I have it in the foreground so you can see what it's doing. And we've got messages coming in, and you can see that the message payload, how it's-- the message-- is being parsed, and then HTTP request and response as we meet the API call to Sentinel. So you can see these messages are just streaming in through syslog-ng. Here we have an HTTP request, and then the acknowledgments from Azure.
And then as more messages come in, you may be able to see the way syslog-ng actually parses that data. There you go. Setting and resetting value is where syslog-ng is actually taking elements of the message and assigning them to macros, so those are customizable bits of data, and then we're repackaging that in a JSON structure and then shipping that off to Azure.
So now that's working. I can actually show you what it looks like in Azure itself. So here is my Sentinel dashboard here. I'm going to run this again. This is just a query, this is going to be pulling the last batch of logs. And there you go. You can see I'm also reformatting just the display so I can show you the specific bits of information that I'm interested in.
So you can see the time generated, you can see the hosts, the program-- like Microsoft Windows Security Auditing-- that's generating the message, the user name, the authority system-- I have a column for event IDs, so these are all parsed out-- the SIDs. If I scroll over here, I can see the message itself, and I can expand on any one of these to see the entire message content. Yeah, you can include these specific macros or parameters, or any other ones that you want to see fit to send to Azure.
And I have two other screens here I just want to quickly show you. So you can see I have some recent incidents that I've triggered. So for example, on my Linux server, I have too many authentication failures. So these are all alerts and incidences that have been triggered from activity that I've sent into Sentinel. So here's a custom alert that I've just triggered by logging on too many times and failing that password. So mistyped password, too many failed logins, I have triggered this alert.
So you can see, I have this data in Azure my demo system here. And it's readable, it's passable, and you can take actions based on it in Azure.
And that the Azure Sentinel destination driver. Again, thank you for coming, virtually, to this session. If you have any questions, or you're interested in setting this up, please reach out to us. We'll be very happy to work with you on this.