I think there are two primary obstacles to the development of real-time analytic systems. The first obstacle is relatively straightforward and can be overcome with hard work. The second is a tougher nut to crack.
- Before you can deploy a real-time analytic application, you need to do a significant amount of engineering in order to make sure that the output of the analytics gets to where it needs to be in a fraction of a second. In the simple case of scoring a customer with a predictive model, you need to pull the right data together (possibly from multiple sources, including relational databases), push the data through the model, translate the output into an action, and then feed the action to the touchpoint system. This is easy to do in batch, but when you are dealing with the constraints of a real-time environment, every microsecond counts. You probably have to develop a data caching mechanism since you can't simply run a database query and wait for the data to show up. You might need to restrict the class of models that you work with in order to speed things up. All of these problems need to be addressed before you can claim that a system is truly "real-time." Technologies like web services might help get the individual parts of the system to work together but that's simply the glue that holds the whole thing together. If the individual parts aren't efficient, you will never get to real-time.
- The second obstacle is related to the challenge of incorporating analytics into a larger CRM process. You need to trust and believe in the results if they are to impact your interactions with customers. When you are working with batch-oriented analytics, it is easy to control the process and determine whether or not the results are valid before you put them into practice. If your model chooses a group of people to receive a new credit card offer, you can review these results and determine if the selected individuals are poor credit risks. When you move to real-time, much of this control goes away. Before you know it, the output of the analytic system has been delivered to the customer via a touchpoint. If you need to validate an individual result, you will need to build the validation into the model or scoring engine. If you need to validate a series of results across customers, you will need to collect the data and create a separate process to automatically review the results. But before you can do this, you need to define what it means to be "valid" for every predictive model that you are using. You will also need to determine what to do when invalid results are found. Keeping track of all of this information is a challenge, especially when you start working with large numbers of customers broken down into small segments. This increases the scope of the overall process, requiring that you have the right workflow in place to develop and deploy the analytics. I believe that one of the more interesting areas for growth in analytics is related to the management of the business processes (workflow) that the analytics fits into.
For more information, check out SearchCRM's Best Web Links on Data Mining.
This was first published in March 2002