Parsing Cisco logs in syslog-ng

Log messages generated by Cisco devices look like syslog messages at first glance, but on a closer inspection you will see that there are many smaller differences. By default, syslog-ng treats all incoming messages as syslog messages, however, Cisco logs do not conform. Log messages collected over the network from Cisco devices and saved to a file look broken. There are many Cisco log variants but luckily a good part of them are covered by the cisco-parser() of syslog-ng.

From this blog you can learn how the Cisco parser in syslog-ng works and how you can check if it really works with your Cisco log messages.

Why bother at all

Here is how a typical syslog message received over the network looks when saved into a plain text file:

Aug 29 16:03:03 localhost root: this is a regular syslog message

A date, a time, a host name, a username and the text of the log message itself. Below you can see how Cisco log messages look like when they hit an unsuspecting syslog-ng server:

Aug 28 16:04:01 localhost foo: *Apr 29 13:58:40.411: %SYS-5-CONFIG_I: Configured from console by console
Aug 28 16:04:01 localhost foo: *Apr 29 13:58:46.411: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 192.168.1.239 stopped - CLI initiated
Aug 28 16:04:01 localhost foo: *Apr 29 13:58:46.411: %SYSMGR-STANDBY-3-SHUTDOWN_START: The System Manager has started the shutdown procedure.
Aug 28 16:04:01 localhost foo: *Apr 29 13:58:46.411: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 192.168.1.239 started - CLI initiated
Aug 28 16:04:01 localhost 0.0.0.0: *Apr 29 13:59:12.491: %SYS-5-CONFIG_I: Configured from console by console

There are multiple host names, two different dates in different formats and finally the text of the log message. This is not what we would like to see in a syslog message. Using the cisco-parser() of syslog-ng we can fix these problems and also save regular looking syslog messages. Or we can do even better: convert messages into JSON format with name-value pairs parsed from the message included (for example, the Cisco MNEMONIC)

Of course, JSON is not too practical in text files, but at least you can see how syslog-ng parses name-value pairs from the message. You can then use the name-value pairs for alerting (filtering) in syslog-ng or reporting in Kibana if using Elasticsearch as a destination.

Before you begin

If you are reading this, then you most likely have a Cisco device around. But you can test the cisco-parser() of syslog-ng even without a Cisco device. In this blog I used some sample logs from the syslog-ng documentation, but obviously the goal is to collect and parse logs from real Cisco devices. That will be most likely covered in a follow-up blog.

On the syslog-ng side, the cisco-parser() was introduced quite some time ago, but it has been improved regularly a lot over the years. So, I recommend you have at least version 3.27 of syslog-ng. If your distro of choice does not provide it out of box check at https://www.syslog-ng.com/3rd-party-binaries if up-to-date 3^rd party syslog-ng binaries are available for your system.

Note, that even 3.27 is old, and 3.30 will have some improvements related to date parsing. It was merged right after 3.29 was released last week: https://github.com/syslog-ng/syslog-ng/pull/3395 If you have syslog-ng 3.27-29, you can easily patch your cisco-parser() as it is actually a configuration snippet in the syslog-ng configuration library. Just download and apply the patch (assuming that SCL files are under the /usr/share/syslog-ng/include/ directory):

wget https://patch-diff.githubusercontent.com/raw/syslog-ng/syslog-ng/pull/3395.patch
cd /usr/share/syslog-ng/include/
patch -p1 < /root/3395.patch

And of course, do not forget to reload or restart syslog-ng.

Configuring syslog-ng

In most Linux distributions syslog-ng is configured in a way that you can extend it by dropping a configuration file with a .conf extension into the /etc/syslog-ng/conf.d/ directory. In other cases, simply append the below configuration to syslog-ng.conf.

source s_regular { tcp(port(5141)); };
source s_net {
    default-network-drivers(flags(store-raw-message));
};
source s_ciscoonly { tcp(port(5140) flags(no-parse,store-raw-message)); };

template t_jsonfile {
    template("$(format-json --scope rfc5424 --scope dot-nv-pairs
        --rekey .* --shift 1 --scope nv-pairs --key ISODATE)\n\n");
};

parser p_cisco { cisco-parser(); };

destination d_fromcisco {
    file("/var/log/fromcisco" template(t_jsonfile));
};
destination d_other {
    file("/var/log/other");
};
destination d_raw {
    file("/var/log/raw" template("${RAWMSG}\n"));
};

log {
    source(s_regular);
    destination(d_other);
};
log {
    source(s_net);
    destination(d_raw);
    if ("${.app.name}" eq "cisco") {
        destination(d_fromcisco);
    } else {
        destination(d_other);
    };
};
log {
    source(s_ciscoonly);
    destination(d_raw);
    parser(p_cisco);
    destination(d_fromcisco);
};

Let’s go over this configuration in detail, in the order the statements appear in the configuration. To make your life easier I copied the relevant snippets below before explaining them.

Sources

source s_regular { tcp(port(5141)); };

This is a pretty regular TCP legacy syslog source on a random high port. This was used to create the logs in the “Why bother at all” section. By default, the tcp() source handles all incoming log messages as if they were legacy (RFC3164) formatted. When it receives non-standard log messages, it results in broken log messages written to disk.

source s_net {
    default-network-drivers(flags(store-raw-message));
};

The default-network-drivers() source driver is a kind of a wild card. It opens a number of different UDP and TCP ports using both the legacy and the new RFC5424 syslog protocols. Instead of just expecting everything to be regular syslog, it tries a number of different parsers on incoming logs, including the cisco-parser(). Of course, the extra parsing has some overhead, but it is not a problem unless you have a very high message rate.

The store-raw-message flag means that syslog-ng preserves the original log message as is. It might be useful for debugging or if a log analysis software expects unmodified Cisco messages.

source s_ciscoonly { tcp(port(5140) flags(no-parse,store-raw-message)); };

The third source, as its name also implies is only for Cisco log messages. Use this when you have a high message rate, you only send Cisco log messages at the given port and you are sure that the cisco-parser() of syslog-ng can process all of your Cisco logs correctly. The no-parse flag here means that incoming messages are not parsed automatically as they arrive. As you will see later, parsing is done in the next step by cisco-parser().

Templates

template t_jsonfile {
    template("$(format-json --scope rfc5424 --scope dot-nv-pairs
        --rekey .* --shift 1 --scope nv-pairs --key ISODATE)\n\n");
};

This is the template, which is also often used together with Elasticsearch (without the line breaks at the end). This blog does not cover Elasticsearch as there are many other blogs, which cover this topic. But the same template using the JSON template function is also useful here so that we can see the name-value pairs parsed from Cisco log messages. These JSON formatted logs include all syslog-related fields, any name-value pairs parsed from the message and the date using the ISO standardized format. There is one little trick, which might confuse you: the rekey and shift part removes the leading dots from name-value pair names. The leading dot is used by syslog-ng for name-value pairs created by parsers in the syslog-ng configuration library (SCL).

Parsers

parser p_cisco { cisco-parser(); };

This parser can deal with the log messages of Cisco devices. It cancorrectly parse the date and also extract facility, severity and mnemonic from the message. By default, these are stored in name-value pairs starting with .cisco., but you can change the prefix using the prefix() parameter. Note that the cisco-parser() drops the message if it does not match the rules. This can be a problem when you use it directly instead of using the default-network-drivers() where messages go through a long list of parsers.

Destinations

destination d_fromcisco {
    file("/var/log/fromcisco" template(t_jsonfile));
};

This is a file destination where logs are JSON formatted using the template from above. This way you can see all the name-value pairs created by syslog-ng. We use it for Cisco log messages.

destination d_other {
    file("/var/log/other");
};

This is a flat file destination using regular syslog formatting. We will use it to store non-Cisco log messages.

destination d_raw {
    file("/var/log/raw" template("${RAWMSG}\n"));
};

This is yet another file destination. The difference here is that it is using a special template defined in-line. Using the RAWMSG macro, syslog-ng stores the log message without any modifications. This possibility is enabled by utilizing the store-raw-message flag on the source side and it is useful for debugging or when a SIEM or other analysis software needs the original log message. Note that by default syslog-ng re-formats log messages to be standards compliant.

Log statements

Previously we defined the building blocks of the configuration. Using the log statements below we connect these building blocks together. The different building blocks in the cases below can be used multiple times in the same configuration.

log {
    source(s_regular);
    destination(d_other);
};

This is the simplest one: connecting the regular tcp() source with a flat file destination. You can see the results from this in the “Why bother at all” section. Most people when they see these logs start researching what was wrong, and that is how they get to the cisco-parser() of syslog-ng.

log {
    source(s_net);
    destination(d_raw);
    if ("${.app.name}" eq "cisco") {
        destination(d_fromcisco);
    } else {
        destination(d_other);
    };
};

Unless you have a high message rate, above 50,000 to 200,000 events per second (EPS), or not enough hardware capacity, the recommended way of receiving Cisco messages is using the default-network-drivers(). It uses a bit more resources, but if a message does not match the expectations of the cisco-parser(), it is still kept. The above log statement receives the message and then stores it immediately in raw. It can be used for debugging and disabled later.

The if statement below checks if the message is recognized as a Cisco log message. If it is, the log is saved as a JSON formatted file. If not, it is saved in a flat file.

Note that the name-value pair is originally called .app.name but in the output you will see it as app.name, as the template removes the leading dot.

log {
    source(s_ciscoonly);
    destination(d_raw);
    parser(p_cisco);
    destination(d_fromcisco);
};

If you have a high message rate and you are sure that the cisco-parser() recognizes all of your logs, you can use this solution to collect logs from your Cisco devices. For safety and debug purposes I inserted the raw file destination before the parser. This way you can compare the number of lines in the two different file destinations. If they are not equal, take a closer look and check the logs that were discarded by the cisco-parser().

Testing

For testing I used a couple of sample log messages from the syslog-ng documentation saved into a file in /root/ciscologs.txt. The regular syslog message was generated by logger, the Cisco messages were submitted using netcat. Here is the content of /root/ciscologs.txt:

<189>29: foo: *Apr 29 13:58:40.411: %SYS-5-CONFIG_I: Configured from console by console
<190>30: foo: *Apr 29 13:58:46.411: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 192.168.1.239 stopped - CLI initiated
<189>32: foo: *Apr 29 13:58:46.411: %SYSMGR-STANDBY-3-SHUTDOWN_START: The System Manager has started the shutdown procedure.
<190>31: foo: *Apr 29 13:58:46.411: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 192.168.1.239 started - CLI initiated
<189>32: 0.0.0.0: *Apr 29 13:59:12.491: %SYS-5-CONFIG_I: Configured from console by console

Depending on to which port you send the logs you will get different results. You have already seen what happens when you send logs to port 5141. It does not look nice except for the regular syslog message.

Now let’s send some logs to port 514:

logger -T --rfc3164 -n 127.0.0.1 -P 514 this is a regular syslog message
cat /root/ciscologs.txt | netcat -4 -n -N -v 127.0.0.1 514

In this case you should see in the files:

localhost:~ # cat /var/log/fromcisco
{"cisco":{"severity":"5","mnemonic":"CONFIG_I","facility":"SYS"},"app":{"name":"cisco"},"SOURCE":"s_net","RAWMSG":"<189>29: foo: *Apr 29 13:58:40.411: %SYS-5-CONFIG_I: Configured from console by console","PRIORITY":"notice","MESSAGE":"%SYS-5-CONFIG_I: Configured from console by console","ISODATE":"2020-04-29T13:58:40+02:00","HOST_FROM":"localhost","HOST":"foo","FACILITY":"user","DATE":"Apr 29 13:58:40","3":".411","2":"*Apr 29 13:58:40.411","1":"Apr 29 13:58:40.411","0":"*Apr 29 13:58:40.411"}

{"cisco":{"severity":"6","mnemonic":"LOGGINGHOST_STARTSTOP","facility":"SYS"},"app":{"name":"cisco"},"SOURCE":"s_net","RAWMSG":"<190>30: foo: *Apr 29 13:58:46.411: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 192.168.1.239 stopped - CLI initiated","PRIORITY":"info","MESSAGE":"%SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 192.168.1.239 stopped - CLI initiated","ISODATE":"2020-04-29T13:58:46+02:00","HOST_FROM":"localhost","HOST":"foo","FACILITY":"user","DATE":"Apr 29 13:58:46","3":".411","2":"*Apr 29 13:58:46.411","1":"Apr 29 13:58:46.411","0":"*Apr 29 13:58:46.411"}

{"cisco":{"severity":"3","mnemonic":"SHUTDOWN_START","facility":"SYSMGR-STANDBY"},"app":{"name":"cisco"},"SOURCE":"s_net","RAWMSG":"<189>32: foo: *Apr 29 13:58:46.411: %SYSMGR-STANDBY-3-SHUTDOWN_START: The System Manager has started the shutdown procedure.","PRIORITY":"err","MESSAGE":"%SYSMGR-STANDBY-3-SHUTDOWN_START: The System Manager has started the shutdown procedure.","ISODATE":"2020-04-29T13:58:46+02:00","HOST_FROM":"localhost","HOST":"foo","FACILITY":"user","DATE":"Apr 29 13:58:46","4":"STANDBY","3":".411","2":"*Apr 29 13:58:46.411","1":"Apr 29 13:58:46.411","0":"*Apr 29 13:58:46.411"}

{"cisco":{"severity":"6","mnemonic":"LOGGINGHOST_STARTSTOP","facility":"SYS"},"app":{"name":"cisco"},"SOURCE":"s_net","RAWMSG":"<190>31: foo: *Apr 29 13:58:46.411: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 192.168.1.239 started - CLI initiated","PRIORITY":"info","MESSAGE":"%SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 192.168.1.239 started - CLI initiated","ISODATE":"2020-04-29T13:58:46+02:00","HOST_FROM":"localhost","HOST":"foo","FACILITY":"user","DATE":"Apr 29 13:58:46","3":".411","2":"*Apr 29 13:58:46.411","1":"Apr 29 13:58:46.411","0":"*Apr 29 13:58:46.411"}

{"cisco":{"severity":"5","mnemonic":"CONFIG_I","facility":"SYS"},"app":{"name":"cisco"},"SOURCE":"s_net","RAWMSG":"<189>32: 0.0.0.0: *Apr 29 13:59:12.491: %SYS-5-CONFIG_I: Configured from console by console","PRIORITY":"notice","MESSAGE":"%SYS-5-CONFIG_I: Configured from console by console","ISODATE":"2020-04-29T13:59:12+02:00","HOST_FROM":"localhost","HOST":"0.0.0.0","FACILITY":"user","DATE":"Apr 29 13:59:12","3":".491","2":"*Apr 29 13:59:12.491","1":"Apr 29 13:59:12.491","0":"*Apr 29 13:59:12.491"}

localhost:~ # cat /var/log/other
Aug 31 18:00:28 localhost root: this is a regular syslog message
localhost:~ # cat /var/log/raw
<13>Aug 31 18:00:28 localhost root: this is a regular syslog message
<189>29: foo: *Apr 29 13:58:40.411: %SYS-5-CONFIG_I: Configured from console by console
<190>30: foo: *Apr 29 13:58:46.411: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 192.168.1.239 stopped - CLI initiated
<189>32: foo: *Apr 29 13:58:46.411: %SYSMGR-STANDBY-3-SHUTDOWN_START: The System Manager has started the shutdown procedure.
<190>31: foo: *Apr 29 13:58:46.411: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 192.168.1.239 started - CLI initiated
<189>32: 0.0.0.0: *Apr 29 13:59:12.491: %SYS-5-CONFIG_I: Configured from console by console

When you send logs to port 5140 where the cisco-parser() is the only parser, the results should be pretty similar. The only difference is that the regular syslog message is only saved to the raw file used for debugging, as it is discarded by the cisco-parser().

If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik.