A couple weeks ago our CTO, Balázs Scheidler reflected on the importance of central log management, an approach that we have been advocating for a long time and was also underpinned by a recent report by Gartner.
Having spent the past week at the RSA Conference in San Francisco and talking to numerous IT operations leaders, security managers and consultants, one thing became clear. On the surface, the ongoing discussion seemed to be all about automation, analytics and intelligence. Nevertheless, the people involved in day-to-day operation of SIEMs and other log-consuming systems all agreed that the basics in log management are still missing at many companies – be those SMBs or Fortune 1000 ones.
Free the logs!
Since IT environments are very heterogenous, log collection and management can become a real pain. Without thoughtful planning and highly capable tools to execute it, logs from different sources usually end up in hard-to-use silos, spread across the infrastructure, many times represented in their own format. This makes life hard when experts try to gain insight on what’s really happening in their environment, to do forensics after a malicious insider activity or data breach, or just trying to feed the same log data to different destinations.
Most point products, even sometimes teams operate with the assumption that they are the sole consumer of logs in their raw format and that all downstream applications use them as a platform. But log data is critical to operations, security, development, compliance and business intelligence as well – so you need to decouple log transport and management from analytics and build an independent log management layer. A layer that is flexible enough to:
- Securely and reliably collect, consolidate and store logs from a wide variety of sources
- Distribute real time feeds downstream
- Store raw data securely and efficiently
- Parse, transform, normalize and enrich data at ingestion time
- Scale on demand to hundreds of thousands of events per second
- Integrate with the most popular log consumers and big data destinations like HDFS and Elasticsearch
Analytics is only as good as the data feeding it
You cannot let any of the systems monopolize log data. Central log management gives you the opportunity to establish with a centralized log data lake, instead of silos that are difficult to use and maintain. It gives you the flexibility that modern companies and enterprises need in a data-driven world, while still providing that much-needed solid foundation to build upon.
Of course, logs are not the be-all and end-all of IT environments, and centralized log management is not an approach that will magically solve all your log-related issues companies might be facing. But it is a very effective way to fight fragmented log collection and rising SIEM licensing costs, scale out analytics tools, build great auditing solutions and improve the overall quality of log data that might be the base of serious business decisions.