Sumo Logic Introduces Anomaly Detection and a Taste of Learning Systems to Come

This week sees some interesting news from log management and analytics vendor Sumo Logic that points to the future of IT intelligence systems in particular and, more generally,the future of a world of connected devices and closed loop intelligence. First an introduction to the company. Founded in 2010, Sumo Logic’s founders point to the direction this company is going – one of the co-founders, Kumar Saurabh, was formerly in charge of designing and running the data analysis infrastructure for Mint.com. Mint is the personal financial management company that was acquired by Intuit and led the market for tools deriving insights from disparate data.

It should come as no surprise to readers conversant in this space that the quantity of machine data is increasing exponentially – a couple of data points  to note: firstly from the Wall Street Journal which reported that there is as much data generated in 10 minutes today as was produced in the entire year of 2003. Secondly from Cisco who estimated that there was 30 billion GB of data in existence in 2005 and that this will be 20* greater this year – big data indeed!

Anyway – back to Sumo Logic who are taking advantage of that flood of data and has 130 customers across service providers, telcos and software companies and most of its customers are using the product to ensure compliance with SLAs. Existing customers fall mainly into this traditional area of application availability and performance. As Sanjay Sarathy from Sumo Logic put it, these areas at the low hanging fruit for log management. However over time there is an opportunity to move to higher value performance opportunities – using log analytics to reduce development cycle time and the like.

Having a handle on log data helps to ensure application availability and performance remain within usual bounds. But as I’ve written previously when covering the log management space, the real opportunity is to close the loop and deliver solutions that offer the full spectrum of measure-analyze-react. As I said previously:

In the same way that cloud monitoring is a very useful tool, but less useful than a combined monitoring/management solution, I’d like to see log vendors either build or partner with others to create highly specific vertical offerings that combine monitoring and action-taking functions for specific use cases. While it’s de rigueur for cloud vendors to point to a broad ecosystem of integration partners, the enterprises I speak to are keen to find solutions that offer monitoring and management from one vendor – an integration is good, but not as good as broad out of the box functionality.

This higher-value opportunity is where Sumo Logic wants to be going and we’re seeing that start of this with the release today of its Anomaly Detection product. Essentially the offering aims to automate what is generally a manual process with current offerings – instead of needing human inputs to ask questions of the log data (whether by ad hoc search, scheduled search or alert notifications for example), the new offering follows the existing Sumo Logic product, Log Reduce, by ingesting a customer’s log data but then takes the patterns from the machine data and automatically identifies the baseline patterns that exists and sets up anomaly detection based on those baselines.

Picture1

The interesting thing is that Sumo Logic is adamant that only 24 hours of log data is required to infer baselines and patterns and that after this initial period, meaningful anomalies can be detected and notified. Obviously over time the product learns from both source data and user feedback on both the source data and the anomalies being presented – the two of these feedback mechanisms allow Sumo Logic to continually refine the detection algorithms and hence become more accurate the longer the product runs over log data.

Anomaly detection looks like a really interesting offering but I see it as only the first step in the journey and I suspect we’ll see Sumo Logic or other vendors of its ilk start to offer highly specific vertical offerings for particular workloads and IT architectures. IT organizations will love the automated nature of this offering but will look towards complete systems that not only ingest data and derive insights from it, but allow actions to be taken from that data – this is the next bastion of IT management and as enterprise IT becomes ever more complex, a complete end to end solution becomes ever more critical.