The syslog-ng stats-exporter() allows you to collect syslog-ng statistics using Prometheus. Version 4.10 of syslog-ng added options to ensure that all related options of syslog-ng-ctl are now available in stats-exporter() too.
Before you begin
The stats-exporter() feature of syslog-ng arrived just recently, in version 4.9. It is already part of several larger Linux distributions. In most cases, it is good enough for exporting statistics to Prometheus. Version 4.10 added two minor options to make stats-exporter() on pair with syslog-ng-ctl:
-
stats-without-orphaned() ensures that values from abandoned counters are not included.
-
stats-with-legacy() is similar to the --with-legacy-metrics option of syslog-ng-ctl: when using the Prometheus output format, legacy metrics are only included if set to yes.
If you need these options, then you need syslog-ng 4.10 or later. Check https://www.syslog-ng.com/products/open-source-log-management/3rd-party-binaries.aspx if this version is not yet part of your operating system of choice.
Configuring syslog-ng
It is easier to test stats-with-legacy(), so the sample configuration uses that. Append this to syslog-ng.conf or create a new snippet under /etc/syslog-ng/conf.d/ if your configuration expects configuration snippets there.
source s_net {
tcp(port(514));
};
destination d_fromnet {
file("/var/log/fromnet");
};
log {
source(s_net); destination(d_fromnet);
};
source s_prometheus_stat {
stats-exporter(
ip("0.0.0.0")
port(8080)
stat-type("stats")
scrape-freq-limit(1)
single-instance(no)
# stats-with-legacy(yes)
# stats-without-orphaned(yes)
);
};
log {
source(s_prometheus_stat);
destination { file(/var/log/scraper.log); };
};
If you want to test only the bare minimum, then the second half of the configuration is enough, starting with source s_prometheus_stat. It defines a stats-exporter() and a log statement to insert it into the configuration. Note that the new options are currently commented out.
The first half of the configuration is not compulsory. It is a network source and a file destination. It can be used to send a large amount of log messages to syslog-ng quickly.
Testing
Once you reloaded syslog-ng with the new configuration, you are ready for testing. You can check the output of stats-exporter() even without Prometheus. The next command uses curl:
curl -s http://localhost:8080/metrics/ | sort
The -s option removes any extra output from curl. Sorting the result is optional but makes the results easier to read.
And syslog-ng will send back an output similar to this:
syslogng_input_events_total{driver="journal",id="src#0"} 2036
syslogng_input_events_total{id="src#1"} 5
syslogng_internal_events_queue_capacity 10000
syslogng_internal_events_total{result="dropped"} 0
syslogng_internal_events_total{result="processed"} 5
syslogng_internal_events_total{result="queued"} 0
syslogng_last_config_file_modification_timestamp_seconds 1759233524
syslogng_last_config_reload_timestamp_seconds 1759905079
syslogng_last_successful_config_reload_timestamp_seconds 1759905079
syslogng_scratch_buffers_bytes 0
syslogng_scratch_buffers_count 14
syslogng_socket_connections{id="s_net#0",driver="afsocket",transport="stream",address="AF_INET(0.0.0.0:514)",direction="input"} 0
syslogng_socket_connections{id="s_prometheus_stat#0",driver="afsocket",transport="stream",address="AF_INET(0.0.0.0:8080)",direction="input"} 1
syslogng_socket_max_connections{id="s_net#0",driver="afsocket",transport="stream",address="AF_INET(0.0.0.0:514)",direction="input"} 10
syslogng_socket_max_connections{id="s_prometheus_stat#0",driver="afsocket",transport="stream",address="AF_INET(0.0.0.0:8080)",direction="input"} 10
As you can see, several lines of the “syslog-ng-ctl stats” output is missing. Now open the syslog-ng configuration and make sure that stats-with-legacy(yes) is not commented out. For this, use:
syslog-ng-ctl reload
And now, if you check the exporter, you will see much more data:
syslogng_center_processed{stat_instance="queued"} 879087
syslogng_center_processed{stat_instance="received"} 879042
syslogng_destination_processed{id="acpid"} 0
syslogng_destination_processed{id="#anon-destination0"} 3
syslogng_destination_processed{id="console"} 10
syslogng_destination_processed{id="d_fromnet"} 877003
syslogng_destination_processed{id="firewall"} 0
syslogng_destination_processed{id="localmessages"} 2
syslogng_destination_processed{id="mail"} 8
syslogng_destination_processed{id="mailerr"} 0
syslogng_destination_processed{id="mailinfo"} 8
syslogng_destination_processed{id="mailwarn"} 0
syslogng_destination_processed{id="messages"} 2028
syslogng_destination_processed{id="netmgm"} 0
syslogng_destination_processed{id="warn"} 15
syslogng_destination_processed{id="xconsole"} 10
syslogng_global_msg_clones_processed 0
syslogng_global_payload_reallocs_processed 3747
syslogng_global_sdata_updates_processed 0
syslogng_input_events_total{driver="journal",id="src#0"} 2035
syslogng_input_events_total{id="src#1"} 1
syslogng_internal_events_queue_capacity 10000
syslogng_internal_events_total{result="dropped"} 0
syslogng_internal_events_total{result="processed"} 1
syslogng_internal_events_total{result="queued"} 0
syslogng_last_config_file_modification_timestamp_seconds 1759233524
syslogng_last_config_reload_timestamp_seconds 1759904447
syslogng_last_successful_config_reload_timestamp_seconds 1759904447
syslogng_scratch_buffers_bytes 0
syslogng_scratch_buffers_count 14
syslogng_socket_connections{id="s_net#0",driver="afsocket",transport="stream",address="AF_INET(0.0.0.0:514)",direction="input"} 0
syslogng_socket_connections{id="s_prometheus_stat#0",driver="afsocket",transport="stream",address="AF_INET(0.0.0.0:8080)",direction="input"} 1
syslogng_socket_max_connections{id="s_net#0",driver="afsocket",transport="stream",address="AF_INET(0.0.0.0:514)",direction="input"} 10
syslogng_socket_max_connections{id="s_prometheus_stat#0",driver="afsocket",transport="stream",address="AF_INET(0.0.0.0:8080)",direction="input"} 10
syslogng_source_processed{id="s_net"} 877003
syslogng_source_processed{id="s_prometheus_stat"} 3
syslogng_source_processed{id="src"} 2036
What is next?
If you used --remove-orphans option of syslog-ng-ctl, you will also appreciate the stats-without-orphaned(). It is up to you and your requirements which options you use. Share your experiences with us at https://github.com/syslog-ng/syslog-ng/discussions.
-
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik, on Mastodon as @Pczanik@fosstodon.org.