By Brad Hopper
Business Intelligence vendors have a tendency to promote concepts such as “democratizing BI” and the like. Although such sentiments mean well, they totally miss the point. You will not find a business leader who cares if BI is democratized, because, of course, business leaders are far more interested in transformational answers and action than in the technologies their people use to pursue those actions. One might think that if a BI platform is “good” that one would follow from the other; that is to say that more/better answers and actions would follow from more BI.
Today, however, we’re going through a phase in which BI adoption is growing rapidly but business leaders are seeing a commensurate growth in “people playing with data” instead of an increase in business value. BI has gotten easier to use and cheaper but it must evolve further before businesses can get the ROI they are owed. Below are a few improvements we should expect to see from Business Intelligence and analytics technologies in the coming year as they mature away from playing with data and toward answers and action.
Inline Visual Data Transformation
Many industry studies have shown that 40-60% of the time knowledge workers spend with data is in the so-called “preparation phase.” If this seems like a lot of time, it is. You might blame it on disorganized data or the complexity of data preparation tools and you would be partly right. But, the fact is that data management and ETL tools have been around for a long time, and are quite capable. No, the biggest problem with the data “preparation phase” is that it’s not actually a phase at all, and neither is it preparation. Preparation is something you do before something else, and a phase is something that ends when that something else starts. Understanding data just isn’t like that. Instead, it is a continuous, iterative process of discovering which are the right questions to ask and what are the data obstacles to obtaining answers.
There are many good reasons to do data transformations during the analysis and not before. Often, you can’t know there’s a problem with the data until you look. One person’s outlier is another’s insight. User knowledge and context is critical to discern, say, a misspelling from valid new fact, and more often than not, new combinations of variables must be created in ways that depend on the behavior of the data. In the old data preparation regime, users must continually go to someone else or to a different tool to change the way data is prepared. Making data corrections and adjustments must become highly visual and interactive, and it should be part of the data analysis experience rather than a separate step. At TIBCO, we call this inline visual data transformation, and you’ll be seeing more of it in the future.
I previously mentioned that context plays a role in data transformation but context is key across the board when it comes to understanding data. In fact, well-known studies have shown repeatedly over the years that only 25% of knowledge workers who would benefit from Business Intelligence and analytics actually use it. You might call that 25% “data professionals” and the remaining, much larger group “business professionals.”Business professionals in particular don’t like playing with data but instead demand context in the form of applications – say supply chain, manufacturing, customer relationship, or technical research applications – where these apps deliver answers but they do it in the context of process-specific workflows, domain-specific data, etc.
The developers of these applications are experts in those process-specific workflows but not in the art of visual analytics technology. We are already seeing a steep uptick in interest from application developers looking for faster and better ways to embed intelligence inside their applications to meet the demands of their customers. At TIBCO, we believe embedded analytics for business professionals should be just as responsive and powerful as the dedicated analytics enjoyed by data professionals, and for this to happen it should be easy to package via a lightweight set of services and easy to consume through simple but comprehensive APIs. Look to see adoption of embedded analytics accelerate in coming years.
AI for BI
What used to be called artificial intelligence is experiencing a renaissance these days as we have seen data mining, deep learning, and rules-based technologies together with crowd-sourced information being used to improve shopping, personal time management, targeted advertising, health outcomes, and other aspects of our lives. Why can’t such technology also improve data analysis outcomes? If there were a “solve my business problem” button, I suppose most users would push it just to see what happens. But the black boxes of the past are giving way to technologies that use context to provide options which magnify user capabilities rather than replace them.
For example, a new user of an analytics tool might know exactly what they want to see but not know how to mechanically get the job done. Given a selection of data elements, a recommendation engine could render data in a variety of ways from which a user could choose. As the data changes, the engine follows along, tracking metadata within the analysis. Recommendations shouldn’t be limited to visualizations but could also suggest visual data transformations, such as inline options to remove duplicates, replace null values, re-code incorrect address fields, normalize or interpolate data, etc. At TIBCO we are expanding our Recommendations feature to include all of the above and we expect other vendors follow.
With the subject of recommendations above I mentioned that it will be very useful to data and business professionals to be prompted with options for visualization best practices and visual data transformations, making them a part of the normal workflow for understanding data. It is very likely that this “inline” idea will catch on and drive even more sophisticated capabilities, especially including predictive analytics. As with data preparation, statistical tools have historically been reserved for an entirely different class of users; analysts. Even with graphical workflow modeling tools coming back into vogue, statistical approaches are not likely to be adopted by business professionals until they are pointed directly at a relevant problem, and delivered in context.
A simple example here – several tools in the market, including TIBCO‘s analytics, offer the possibility of computing a forecast. And doing so is contextually easy when there’s a time series graph on the screen to see and interpret the results. So, if a user creates a time-series visualization or Recommendations offers such an option, why not also give the user the possibility of projecting their revenue forward? This is a very powerful idea, not only because it gives options and increases speed, but also because it introduces concepts a new user may not have considered and curates this power in appropriate ways. The key is to provide contextual options and guidance without forcing user behavior.
Answers generated from analysis have almost no value if they are not tied in some way to an action in the business. Indeed the value of answers goes up significantly as the connection to operational systems becomes more direct and timely. What do I mean by this? Let’s say a business professional identifies an emergent market segment. If they sit on that information, there is no value. Making use of this information, for example, by helping the organization make a targeting decision, is the traditional value basis of BI. If the Business Intelligence platform could directly facilitate and/or speed communication of that idea to the people who need to know, we have taken an action, which action might be called collaboration, or as some say,“story telling”. Fine, that’s better. What happens, though, if an individual with suitable credentials can not only communicate, but also directly push the definition of this segment into a marketing automation system?
We’ve now dramatically reduced the opportunity for error by eliminating redundant information (email and spreadsheets), and reduced the time from answer to action. We don’t eliminate the discussion, of course, just when it ends favorably, the business change is one button away. Let’s take it a step further and imagine that a data professional creates a model for deriving the segment. That reduces the analysis time significantly because we could score such a model quickly to identify the candidate customers. Not only that, we could move from publishing a list of customers each week to publishing a model into an operational system that creates and prosecutes the list for us in real time.
Analysis of data is still critical but it’s now at a higher level, because we move from looking at history to directly probing the response of the business to our models as we continuously monitor system health and alert important outcomes. As you can see, this significantly expands the scope of Business Intelligence and dramatically increases its business and operational value. Realistically, only the Business Intelligence vendors with skills and knowledge in operational systems and integration will be able to pull off these enhancements.
In summary, there are a lot of exciting things to come as Business Intelligence and analytics become less about playing with data and more about deriving answers and taking action. Whether it’s bringing data transformations and predictive modelling inline instead of settling for a disjointed experience, or embedding both of the above capabilities into contextually relevant applications, the focus will be on empowering both data and business professionals to take direct action in mission critical business systems. TIBCO will continue to make significant strides in these areas and we imagine others will not be far behind.
Brad Hopper is Sr. VP of Analytics Products and a member of the Office of the CTO at TIBCO Software. As both a product executive and a practitioner of analytics, Brad brings a hands-on approach to solving customer problems and delivering operational value from data across diverse use cases in manufacturing, energy, telecommunications and other industries. Connect with Brad on LinkedIn.