The Business Intelligence Industry is Stagnating, Here’s Why

This is part of Solutions Review’s Premium Content Series, a collection of contributed columns written by industry experts in maturing software categories. In this submission, EasyMorph Founder Dmitry Gudkov offers commentary on BI stagnation and the current state of the market for data analytics tools.

A few weeks ago, I made a short post on LinkedIn that questioned whether the industry is stagnating because there hasn’t been real innovation in the in the last decade. The post sparked an active discussion with lots of interesting comments. Some commenters had a common point: rollouts of business intelligence (BI) systems way too frequently receive a lukewarm reaction from the end-users and chronically suffer from low adoption.

For instance, one commentor: “The major sign of stagnation to me is the inability to deliver on the promise of self-service. Yes, the visualizations are getting better with the new wave of self-service BI tools, but user adoption stays embarrassingly low”.

A response to that from another person was: “The lack of user adoption part of your comment is interesting to me. As a BI developer, I’ve seen countless projects consume resources for months, the product gets promoted to production, you inform the end-user community, and then the adoption to the report/tool never [seems] to happen”.

In my opinion, the lack of breakthroughs and low adoption indicate the same problem — the BI industry has long been stagnating. Therefore, identifying the cause of low user adoption will give us a clue for overcoming the stagnation. So why is BI user adoption low?

A rarely understood aspect of business intelligence is that it’s different from other types of enterprise IT systems because BI users always have a choice. With most enterprise systems, the employees usually don’t have a choice whether to use them or not. If an organization operates an accounting application, no accountant can get around it and use an alternative. In another example, going through the ERP system is obligatory, whether you like it or not.

However, it doesn’t work like that with BI platforms because users can always resort to their ol’ comfy Excel spreadsheets, and you can’t really ban Excel. As it turns out, people can’t be forced into high adoption of a BI application. They have to like it. And they would only like it if the BI application provides real value to them. Remember the saying, “you can lead a horse to water, but you can’t make it drink?” You get the point.

OK, but why do the business users keep sticking to Excel? Why wouldn’t they like these shiny BI platforms full of cool features acclaimed by major industry analysts? Why are they totally fine without these platforms?

In my opinion, it’s because BI product management is heavily driven by techies and salespeople. Too little thought has been given to actual user needs and preferences when it comes to data analysis and manipulation. As a result, we see “cool” (for techies), sometimes AI-driven (double-cool!) features, but they don’t make life easier for the target audience and therefore have poor adoption.

The BI industry has, what I call, the “toy seller problem”.” Selling toys assumes that children are the users, but the parents are the budget-holders and decision-makers. Like with toy selling, the user and the buyer in BI deployments are two different personas. The enterprise BI selling process is heavily tailored for the buyers (IT budget holders), but not for the non-technical people who would actually have to work with the product on a daily basis. As a result, BI platforms are bloated with half-baked shiny useless features that nobody actually uses after the platforms are purchased.

Let’s take, for instance, AI-supported natural language queries. They are dead on arrival. Why? If you give it some thought it becomes clear that the feature fails to deliver on the promise. It promises not needing to learn a query language (just speak English to it, yay!), but in reality, you would still have to learn its rather limited syntax and will frequently bump into its numerous restrictions. Also, from a more general perspective, AI just isn’t ready for such things yet. Current AI is not contextual, while business analysis and reasoning always happen in a context. To illustrate the point, you can’t have reasonably long conversations with Siri because Siri can’t understand the context of a conversation, so your every question should start as if there were no previous questions and answers. You can’t do any reasonable analysis with that level of AI, and the level of AI in BI apps is even worse (which is understandable because not everyone has Apple’s R&D budgets).

Analytical text summary generation, another marketed feature, goes against the whole idea of data visualization because reading is always more cognitively demanding than seeing. No wonder it never really took off.

The problem with BI adoption has been aggravated by the broad switch from native desktop apps to the cloud-based SaaS model. When it comes to the cloud, the needs of business users and IT managers sometimes move in the opposite directions and the former usually don’t have a say. The SaaS application model doesn’t solve any users’ problems that a BI solution is expected to address. Sometimes it’s rather the opposite — SaaS makes even basic things worse.

For instance, business users deal a lot with local files – spreadsheets, CSV files, etc. After all, not everything is stored in the cloud. These files frequently need to be merged before analyzing and visualizing them. However, instead of quickly merging them locally, now they have to upload the files into another remote computer and merged there. If something goes wrong with the merge, the files have to be fixed locally and re-uploaded again. So a basic operation now requires totally unnecessary extra actions from the user.

Another problem: cloud-based SaaS apps are slow. Painfully slow. A tool that’s supposed to improve productivity makes users sit and wait until yet another report or dashboard gets loaded. A delay of several minutes here and there may not sound like a big deal, but over a year it adds up to days if not weeks of lost productivity. One of our customers reported that working with one of the major online BI platforms was so slow that even logging on frequently times out. Another customer couldn’t believe his eyes when he saw that his data preparation workflow run in EasyMorph (our on-premises data prep application) took only 20 seconds to finish instead of the usual half-hour that he used to wait for an equivalent workflow to finish on another major cloud BI platform despite using the most expensive plan there.

Switching to a cloud service frequently trades one type of problem for another type of problem and creates new problems that just didn’t exist previously. For instance, cloud services usually impose restrictive call rate limits (throttling). Many things that you used to do freely with an on-prem system, become rate-limited after switching to a cloud application. Do you want to insert 1 million rows into a cloud table? Not so fast, you can only insert 100K rows at a time and make no more than 10 inserts per hour. Do you want to run a data preparation flow? Not so fast, today you can only run 49 transformations in no more than 3 flows to remain within the daily limits of your cloud quota.

Ironically, the very same people who demand users to use web apps for work don’t use web versions of apps on their iPhones. Instead of using a web version of Twitter or Facebook, they install the respective native apps from the App Store. Why? Because native apps are more convenient, faster, and responsive. But when it comes to BI tools, they require users to use web versions. Why? Because SaaS makes their life easier, not their users’ lives. The “toy seller problem” as it is.

Cloud and SaaS have been a godsend for IT managers, developers, and vendors for many reasons. The cloud technologies are trendy, and, let’s admit it, they look favorable on CVs. I’m afraid that’s why we now have lots of cloud BI on the market. But has the move to the cloud advanced the BI industry in general? Did it make business users like their BI apps more? I don’t see that.

Instead, there are enthusiastic product managers developing “cool” features in enterprise BI applications, happy salespeople closing deals, thrilled CIOs who’ve got their hands on a technology praised by Gartner as a trendsetter of this year, and bored business users opening these applications only to download data from them into Excel. And then we wonder why BI adoption is so low.

In which direction should BI go? That’s a good question. I see business intelligence as a set of tools and methods to extract knowledge from available data, accumulate and share it, and use it for reasoning about business issues.

Picture 1. The Business Intelligence process.

Maybe, instead of getting excited about useless animation in charts, we should try to get a fresh look at BI in general by questioning the dogmas of the industry. How about questioning the role of data visualization in business intelligence? It’s axiomatically assumed that data viz is an essential part of it. BI without data viz is unthinkable. But why? Data visualization is a magic way to produce knowledge from prepared data.

Stephen Few named one of his books Now You See It precisely because data viz is eye-opening. Present data in a particular way so that the knowledge it holds becomes self-evident without saying a word. But data visualization is also a highly sophisticated discipline. Visualizing data to make it speak for itself is an advanced skill that takes years of practice to develop. Tableau Zen Masters exist for a reason.

Maybe we should be looking for ways to make data visualization easier? Tableau did a good job of lowering the bar, but why stop there? Or maybe we should try to find practical, convenient ways to extract knowledge from data besides data visualization. There need to be other simple and convenient ways to build and maintain a body of knowledge around data. Ways for people to accumulate, find, and effortlessly share knowledge with each other. Ways that make explanations self-evident. Very little has been done in this direction.

We should probably also do something about the reasoning part of the BI process pictured above because it remains largely outside of the scope of BI applications even though business intelligence is only needed for reasoning and nothing else besides that. Currently, we don’t understand well how knowledge obtained from BI is used for making decisions and how exactly that knowledge influenced the decisions. The relationship between knowledge extraction and reasoning is currently not captured by software and is therefore non-auditable, non-explorable, and non-manageable.

I don’t know what the cure for the BI industry would look like, but I know what would signal that it works – it should bring work productivity to a whole new level. We should see mainstream users that absolutely hate the idea of reverting to the old ways of doing things.

On a final note, I’ll take the liberty of offering a bit of advice to IT managers that make decisions about purchasing BI and data preparation tools.

Here are a few suggestions for those who don’t want to spend huge budgets and tens of man-years of effort just to discover later that your colleagues secretly hate the new shiny BI platform you imposed on them and try to find ways around it:

• Make high user adoption the primary goal of the BI platform acquisition.
• Remember that you may not be the most typical user of a BI application, so your opinion on its usability and usefulness might be less important than it appears. Prioritize feedback from those who will spend more time with the new application than you.
• Don’t force your users to use something because it’s “free” or “it comes bundled anyway”. Free stuff can have high indirect costs.
• Stick to boring fundamentals. Learn to downplay “cool” features and be cautious about anything marketed as AI-driven or using any kind of “black magic”. Their only purpose maybe to impress you as a decision-maker or tick a box on a decision-making checklist.
• Dismiss slow software. Slow software is very expensive.
• Don’t hesitate to do an extended test drive before making the final decision. Let several groups of users work with different proposed tools with real data on real-life tasks for a few months.
• A good indicator that a software application is the right choice is when the business users dislike the idea of not having it. If they don’t care whether they will have it or not, keep looking or consider abandoning the idea of purchasing anything at all.