Loggly released the second generation of their Logging as a Service (LaaS) recently. As a technology enthusiast, and someone who already liked the first generation of their LaaS, I could not resist trying it.
Basic message forwarding from syslog-ng on Linux is very well documented on the Loggly website. There is a python based configurator script, but as usual, I chose the hard way of editing configuration by hand. A configuration sample ready for copy and paste is available when you click on the “Source Setup” link in your browser after logging in to Loggly. Using that, your local syslog-ng will send your logs to Loggly servers in no time. There is also a dedicated syslog-ng page, which also includes, how to send logs in a more secure way, encrypted.
Sending logs from the syslog-ng Windows Agent directly
Using the syslog-ng Agent for Windows with Loggly is also easy. It requires the same “Source Setup” page, which was mentioned previously. Adding the Loggly destination can be done either during installation or later on in the configuration application. It is the same in either case. To add a new server, enter “logs-01.loggly.com” as server, set server port to “514” and change to the “messages tab” for further settings. There, set protocol to “Legacy BSD” and copy & paste the template from the “Source Setup” page to the “Template” field. Use only the part between the quotation marks. It should look like this, but with a different Loggly “customer token”:
<${PRI}>1 ${ISODATE} ${HOST} ${PROGRAM} ${PID} ${MSGID} [abcd1234-aaaa-bbbb-1234-1234abcd1234@41058] $MSG\n
Once ready, click “OK”, restart the Agent and your logs will appear in Loggly. A logout / login will definitely generate some logs, that’s what I use for testing.
Sending logs from the syslog-ng Windows Agent through a central server
Using the above setup is easy, but has some shortcomings. Logs are sent directly off-site. If there is a longer networking outage then there is no central log collection at all. Also, using the legacy BSD syslog protocol instead of IETF has the disadvantage that structural data is lost. Both of these problems can be resolved by using a local central log collector, where logs are sent using the IETF protocol. Loggly Gen2 can turn JSON based messages into easy to search name value pairs and syslog-ng can turn SDATA from IETF log messages into JSON.
On the syslog-ng Windows Agent side, setup is even easier than the previous one. Just add a new server in the configuration interface and point it to port 601 of your local central syslog-ng server.
On the syslog-ng side, the following configuration will collect IETF logs and forward it to Loggly in JSON format. Obviously you need to replace the “customer token” with your own. If in doubt, you can check it once you logged in to the Loggly web interface. It can be found under “Source Setup”, and listed under “Customer Tokens”.
source s_windows {
syslog();
};
template LogglyFormat {
template("<${PRI}>1 ${ISODATE} ${HOST} ${PROGRAM} ${PID}
${MSGID} [abcd1234-aaaa-bbbb-1234-1234abcd1234@41058] $(format_json
--scope sdata --scope selected_macros)\n");};
destination d_loggly {
tcp("logs-01.loggly.com" port(514) template(LogglyFormat));
};
log { source(s_windows);
destination(d_loggly); };
“s_windows” adds an IETF syslog source, “LogglyFormat” is a slightly modified Loggly template, which replaces the message part with JSON data, “d_loggly” is the Loggly destination, and at the end the log statement glues all of these together.
Sending name value pairs from syslog-ng on Linux
On Linux, one can use patterndb to parse log messages and generate name value pairs from them. These can also be forwarded to Loggly using the JSON output. In the following example, I use the ssh pattern from the BalaBit pattern repository available at https://github.com/balabit/syslog-ng-patterndb It parses username, authentication method, source host, etc. information out from sshd syslog messages and even adds a few more extra information.
Here is the configuration snippet, as usual, please use your own “Customer Token”:
parser p_db {
db-parser(file("/etc/syslog-ng/ssh.xml"));
};
template LogglyFormat {
template("<${PRI}>1 ${ISODATE} ${HOST} ${PROGRAM} ${PID}
${MSGID} [abcd1234-aaaa-bbbb-1234-1234abcd1234@41058] $MSG
$(format_json --scope nv_pairs)\n");};
destination d_loggly {
tcp("logs-01.loggly.com" port(514) template(LogglyFormat));
};
log { source(src); parser(p_db);
destination(d_loggly); };
As not all logs are known by patterndb, $MSG is left in the template and JSON output is appended to the end. Once syslog-ng is restarted, it will parse SSH messages and the extra information will be available in the Loggly web interface.
Enter the following query in the search field to find root logins:
json.usracct.username:root
If you want to receive an alert on root logins, first save this query by clicking on the star on the left hand side and choose “Save this search as…”. Once ready, click on the “Alerts” menu at the top of the page, and choose “Add new”. Here you can set different parameters, including which saved search to run.
Tips and tricks
If your machines are behind a firewall, port 514 might be closed due to the bad memories of the r* commands. Even if Loggly does not provide another unencrypted syslog port, there is a workaround: use encrypted syslog, which can be sent to port 6514. Use of encryption is well documented on the Loggly syslog-ng page and easy to apply to any of the above configurations. And it has the added bonus, that nobody can read your logs on the way to Loggly.
