From my previous Active Roles blogs, you could learn how to forward regular Active Roles logs from Windows Event Log to a central syslog-ng server, where it parses, filters, stores and forwards the logs. In this blog, I show you how to work with Active Roles debug logs, that is reading them using syslog-ng Agent for Windows and forwarding them to a central syslog-ng server for long(er) term storage.
Debug logs are typically huge and the Active Roles debug logs are no exceptions, so you must make sure that you collect them only when really necessary. However, there can be situations when you need to collect these logs centrally for easier access or even long-term storage. Active Roles might generate gigabytes of debug logs even in just a few hours, so make sure that you collect these logs separately from the rest of your logs. This way, you can easily discard these logs when they are no longer needed.
Before you begin
If you want to test what you read in this blog, you need 2-4 hosts. Installing Active Roles needs 1-3 Windows Server machines, while installing the syslog-ng PE server needs a supported Linux host. Describing the system requirements or the installation processes are outside the scope of this blog. Contact One Identity at https://www.syslog-ng.com/register/115582/ to receive a trial version of syslog-ng Premium Edition and help to get started. A free trial of Active Roles can be found at https://www.oneidentity.com/register/62175/.
Collecting Active Roles debug logs
You can enable debug logging from the Active Roles Configuration Center. There are five different debug logs, all disabled by default, as they can quickly produce an insane amount of logs. You can see a list of log sources together with paths and file names. For the purpose of this blog, I chose the first one from the list, which stores logs to the C:\Program Files\One Identity\Active Roles\<active-roles-version>\Service directory, with the ds.log file name.
Note that on the syslog-ng Agent for Windows side, we extend the configuration we did in the first blog of the series: https://www.syslog-ng.com/community/b/blog/posts/collecting-active-roles-logs-centrally-using-the-syslog-ng-windows-agent As such, we do not discuss here how to add a destination.
In the syslog-ng Agent for Windows MMC, go to “File Sources”, and make sure that it is set to “Enable”. You can now add a new file source. First, enter the directory name from above (might be slightly different on your host) and the file name. As Active Roles sometimes produces multi-line log messages, we also add a Prefix for the date format. Set “Application” to “Custom”, then enter the following expression:
\[[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}\.[0-9]{3}\]
Yes, it is ugly and all Active Roles debug logs have a slightly different format. Having this properly configured makes sure that multi-line log messages are forwarded as a single message. If you are not comfortable enough with writing regular expressions, you should use websites like https://regexr.com/ or similar services to create one.
Once you have restarted syslog-ng Agent for Windows, your central syslog-ng server is flooded with debug logs from Active Roles.
Configuring the syslog-ng PE server
In my previous blog, we built a syslog-ng configuration, which collected log messages from Windows using the RFC5424 syslog protocol, parsed and filtered the logs, and finally stored them to various destinations. We do not explain the various destinations here again, only the changes. You can find the second Active Roles + syslog-ng PE blog at https://www.syslog-ng.com/community/b/blog/posts/working-with-parsed-active-roles-logs-in-syslog-ng for more details.
You should append this configuration to your syslog-ng.conf:
# source for Windows clients, RFC5424
source s_win {
syslog(port(601));
};
# xml parser for Windows XML logs
parser p_xml {
xml(prefix('winxml.'));
};
# destination for Windows logs
destination d_fromwin {
file("/var/log/fromwin");
file("/var/log/fromwin.json" template("$(format-flat-json --scope rfc5424 --scope dot-nv-pairs
--rekey .* --shift 1 --scope nv-pairs --exclude MESSAGE)\n") );
};
# destination for debug logs
destination d_fromdebug {
file("/var/log/fromdebug");
file("/var/log/fromdebug.json" template("$(format-flat-json --scope rfc5424 --scope dot-nv-pairs
--rekey .* --shift 1 --scope nv-pairs)\n") );
};
# destination for syslog-ng Agent for windows internal logs
destination d_frominternal {
file("/var/log/frominternal");
file("/var/log/frominternal.json" template("$(format-flat-json --scope rfc5424 --scope dot-nv-pairs
--rekey .* --shift 1 --scope nv-pairs)\n") );
};
# Elasticsearch destination for easy searching and reporting
destination d_elasticsearch_http {
elasticsearch-http(
index("syslog-ng")
type("")
user("elastic")
password("-w35jWfY_syp577m-UfS")
url("https://localhost:9200/_bulk")
template("$(format-json --omit-empty-values --scope rfc5424 --scope dot-nv-pairs
--rekey .* --shift 1 --scope nv-pairs
--exclude DATE --exclude MESSAGE
--exclude winxml.Event.EventData.*
@timestamp=${ISODATE})")
tls(peer-verify(no))
);
};
# long term storage for regular Active Roles logs
destination d_logstore {
logstore("/var/log/ars.lgs" compress(9) );
};
# long term storage Active Roles debug logs
destination d_debugstore {
logstore("/var/log/arsdebug.lgs" compress(9) );
};
# alerts to file and Slack
destination d_alerts {
file("/var/log/alerts.txt");
slack(
hook-url("https://hooks.slack.com/services/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")
template("User ${.SDATA.win@18372.4.EVENT_USERNAME} deleted a ${.SDATA.win@18372.4.EVENT_SID_TYPE} on ${HOST} using ARS")
);
};
# log path for logs from syslog-ng Agent for Windows
log {
source(s_win);
# syslog-ng Agent for Windows and logs read from files come with
# "syslog-ng-agent" as program name
if ("${PROGRAM}" eq "syslog-ng-agent") {
# if there is a file name
if ("${.SDATA.file@18372.4.name}" ne "") {
destination(d_fromdebug);
destination(d_debugstore);
flags(final);
# the rest are internal logs
} else {
destination(d_frominternal);
};
};
# all other logs come in XML format from the Agent
parser(p_xml);
destination(d_fromwin);
destination(d_elasticsearch_http);
# regular Active Roles logs are saved to LogStore
if ("${PROGRAM}" eq "ARAdminSvc") {
destination(d_logstore);
};
# send an alert if someone deletes an object using Active Roles
if ("${.SDATA.win@18372.4.EVENT_TASK}" eq "ObjectDelete") {
destination(d_alerts);
};
};
Let’s check this configuration in more detail, focusing on the new elements and changes in the log path.
There are two new file destinations added. The first one is called “d_fromdebug” and this is where debug logs from Active Roles are stored in plain text and in JSON format. Unlike in “d_fromwin”, the content of the MESSAGE macro is not parsed, so we keep it in the logs as-is. The second one is for internal syslog-ng Agent for Windows log messages, which would be otherwise hard to find if logged together with Active Roles debug logs.
There is also a new LogStore destination for debug logs, which enables compression.
The heart of the configuration is the log path. This was rewritten almost from scratch. I tried to include plenty of comments in this part for clarity, but I will also explain it below in more detail.
The log path starts with a block dedicated to logs where the program name is “syslog-ng-agent”. Why? Because all Active Roles debug logs also belong to this category. The Agent reads the log files and forwards them to the syslog-ng PE server. As there is no application name associated with text files, the Agent uses its own name as PROGRAM. Processing the busiest log source first and storing it with the “final” flag helps to lower the load on the rest of the processing pipeline.
The rest of the logs from the Agent are parsed with an XML parser and saved to various destinations, as it was done previously.
What is next?
Even if this configuration looks complex, it is simplified in many ways. For example, while you should use encryption in a production environment, for the sake of this test, we simply send messages in clear text. Using date macros in file names could also help you delete old debug logs, either by hand or using a script.
Also, did I mention that debug logs are huge? In my environment, collecting logs just for half an hour resulted in about 10 megabytes of Windows and regular Active Roles logs, and close to half a gigabyte of debug logs. However, the good news is that using the LogStore destination with compression can store the same amount of logs in approximately 20 megabytes.
-
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik, on Mastodon as @Pczanik@fosstodon.org.