On Search Optimization for Log Management

Centralized log management certainly is a great solution when you are trying to tackle issues related to handling huge amounts of logs. With the right tool, it can be also beneficial on collection and search performance – areas that syslog-ng Store Box (SSB) traditionally excels at. Sure, both collection and search performance depend on quite a few factors, but with well thought-out test environments, we can get some reference numbers that show the performance value in a proper central log management system.

This is why we decided to publish our performance guidelines for syslog-ng Store Box. Since it is an appliance, installation and configuration is usually a breeze, thus its positive impact on the log management layer’s performance can be felt right from the get-go. Of course, these results are not etched in stone, and as mentioned above, depend on many different factors like available processing power, storage subsystem, network architecture and so on. Still, they give you an idea of the possible benefits when using a high-performance central log management tool.

The whitepaper has all the details on:

  • Factors with significant effect on performance
  • Factors without significant effect on performance
  • Introduction to SSB search algorithms
  • The difference between simple and complex search expressions and their impact
  • Collection and search performance and response times

 

Overall, the test measurements show that the processing capabilities and search performance of syslog-ng Store Box have increased significantly since version 4 LTS, and that SSB is capable of receiving and processing high-volume log traffic. The largest SSB appliance is capable of scaling up to 100,000 event per second (100k EPS).

For techies like us who love their numbers as much as we do, the in-depth whitepaper is certainly worth reading.

More than just performance

Of course, central log management is so much more than “just” improving performance, although that is certainly a very welcome improvement. A recently published report by Gartner details how organizations can benefit from implementing the centralized approach to log management, and includes recommendations that security and risk management leaders responsible for security monitoring and operations should pay attention to, like:

  • Use a CLM tool to address security monitoring and compliance use cases where there are insufficient resources or budget for a SIEM or for managed security services.
  • For midsize organizations, look to use existing IT and network operations log management tools to collect and manage security event logs.
  • Consider a multitier approach using a CLM tool when planning a SIEM deployment to avoid overutilization, and overlicensing, from the start.
  • Use a CLM tool to better manage your existing SIEM tool investment if your organization has an existing SIEM solution that cannot scale its collection and analysis capabilities due to budget constraints.
Related Content