After many years of stasis in the analytics market, we are beginning to see change. For a long time, relatively small data stores and insufficient horsepower left many analytics users wondering if and when things would improve.
In fact, the recent focus on "big data" turned that wonder into something approaching desperation. Social big data is being dumped into CRM systems and elsewhere in the enterprise, causing everyone in the front office to wonder what to do with it. The short answer is that social and CRM data is full of information, yet you still need analytics to extract it.
But what kind of analytics will do the trick? Well, a curious thing happens when you collect big data: The relationships between each datum and all the others increase even faster than the data itself, and it's the relationships we really care about.
Last week, Ayasdi -- a Palo Alto, Calif.-based company that came out of a Stanford lab and uses DARPA capital -- brought itself out of the shadows. Its mission is exactly what the doctor ordered for big data everywhere, including CRM.
Ayasdi does analytics through a new mathematical concept called topological data analysis (TDA). It is a fundamentally different approach to extracting information. TDA analyzes the shape of the data, officials there told me, meaning that the tool looks for clusters of connections rather than sifting through the whole mass in a more or less rote fashion.
The approach is important for many reasons. As data gets truly large, our ability to sift through it, even with the fastest computers, will lose steam.
More on social big data:
Social media calls for more listening and less marketing
ROI halting companies' embrace of social media analytics
That's because of the persistent connections problem. Think of blowing up a balloon. As you pump air into it, the surface area expands by the square but the volume is cubed, which means the volume doesn't just increase faster than the surface, it accelerates. When people maxed out on the number of relationships they could maintain, social media was born to enable more connections. You can think of TDA that way, as a tool that lets us manage more connections.
That's the condition we find in analytics today. If you think of the connections, which are where interesting things happen, our ability to analyze the connections will only recede. To prove a point, while Ayasdi was in beta, it worked with pharmaceutical companies on some of their big data problems related to new drug development.
Now, the drug companies have been generating mountains of data for decades, and they use analytics to digest it. The problem with that approach, though, is that the data only reveals what you ask it through a query. If you don't think of the right query, the data could hold onto its secrets for eternity. And that's true in pharmaceuticals, social CRM and any place where big data lives.
To prove a point, Ayasdi started working with spent data from drug research conducted twenty years ago. The data had already given up its secrets to the drug companies. But with its topological approach, Ayasdi found new information. Now, I don't know what that information is because it's proprietary, but I believe my sources and think that this could be important to more than just new drug development. Ayasdi is pursuing opportunities in oil and gas exploration and government and finance, as well as pharma -- all verticals that generate huge data that could benefit from more analysis.
Right behind those verticals, I figure someone has to apply TDA to CRM data.
The social big data vertical is much newer than the others, but it suffers from the same problems and has the same needs. Social users have to find better ways to extract meaning from big data if we are going to make good on the promise of cloud computing and ubiquitous information access.
Ayasdi isn't the only company out there with an approach to big data, but they do have a good story to tell. Another company that uses machine learning, a closely related idea, is Mintigo. By analyzing a company's known information, like its customer base, Mintigo can then apply what it "learns" to search the rest of the world for matches.
The result is not leads, exactly, but suggestions about who would make a good prospect. That doesn't mean the identified prospect has an active business problem you can solve, nor does it mean the prospect has budget. But Mintigo gets you to a similar spot: A lot of data converges at a point that might be interesting.
There's no doubt that modern hardware has made great contributions to analytics progress. Now that vendors are paying attention, I think we'll see some rapid progress on what just a couple of years ago looked like an intractable problem in the front office, big data.
This was first published in February 2013