Economist and Nobel Prize winner Herbert A. Simon observed that a wealth of information creates a poverty of attention in a given audience, as well as a need for the audience to allocate its attention as efficiently as possible among abundant information sources.

The 2.5 exabytes of data created daily in our world confirm that we live in a time of information abundance — data abundance, if not overload.

A survey conducted in late 2020 by International Data Corporation (IDC) for Sisense polled 500 business decision-makers and data executives. The results paint a picture of how some companies are poised to manage data abundance better than others.

An organization’s ability to make analyzed data easy to access and use by all stakeholders — regardless of technical skills — can make the difference between success and failure. Companies need a BI and analytics strategy and tools that simplify the extraction of meaningful insights from the ever-increasing volume and variety of on-premises, cloud, and in-application data.

The IDC survey shows that many companies have already stretched their current BI solutions to their limits. These organizations are unprepared to harness the ever-growing volume and diversity of data to support informed decision-making where and when it matters most.

Most organizations are planning to replace or add new BI and analytics solutions

Q: Given the BI and analytics solutions that you currently use, are there plans to replace them? (% of respondents)

BI and analytics solution replacement intentions

The shortcomings of current BI approaches are evident in what appears to be a fundamental shift in enterprise BI and analytics strategies. According to the IDC survey, most organizations plan to either replace their BI and analytics solutions or add another solution within the next two years.

Survey results also suggest that business stakeholders are taking an active role in decisions on new products as they look for solutions that suit their evolving data needs. Of the data executives surveyed, 59% reported that they plan to add to their current BI and analytics solutions in the coming two years; 23% plan to replace them altogether. While these numbers are high by any measure, the proportions are even higher among business decision-makers, as 75% plan to add to their existing BI and analytics solutions, and 52% plan to replace them.

Complexity is a barrier to the frequent use of BI and analytics

Q: How frequently do you use your BI and analytics solutions? (% of respondents)

Frequency of BI and analytics usage by business decision-makers

The fact that business decision-makers have such difficulty with effectively leveraging analytics is reflected in the infrequent use of existing solutions: 53% of the business decision-makers who participated in IDC’s survey use their BI and analytics solutions only monthly or quarterly. Just 13% of them reported daily use of analytics. This infrequent use of analytics indicates that most organizations are still at a nascent stage in their analytics journey and that adoption is an issue.

Business decision-makers’ sporadic use of analytics is attributed to the complexity of existing BI solutions, many of which were designed for tech-savvy corporate data teams. As such, they require technical skills that are not abundant within business units. This can explain why business decision-makers who reported infrequent (quarterly and monthly) use of their BI and analytics solutions mentioned a lack of adequate expertise at the department level as a major challenge affecting their BI and analytics strategy.

When asked to select “one thing that is currently missing from currently used solutions that would improve your analytics experience,” many business decision-makers explicitly mentioned ease of use.

For example, an HR director in a large US-based business consulting company stated that solutions should be “easier to use so that even a nontechnical staff would be able to access data and build interactive dashboards and reports.”

Another participant, a C-level executive in a large US-based retail company, noted, “There is a lot of complexity in the results driven from our BI software, and it requires a technical staff to read the reports.”

Dealing with the growing volume and diversity of data sources

Q: In total, how many data sources does your organization use or plan to use for analytics? (% of respondents)

Beyond the skills gap, there’s the challenge of surging data volume from on-premises and cloud data warehouses, spreadsheets, Hadoop, business applications, and other sources. According to IDC’s survey, 73% of companies plan to draw intelligence from more than 10 data sources, whereas nearly 40% plan to use more than 20 data sources.

Furthermore, data sources vary in form now more than ever. Only a proportion of data is generated and found in structured formats that lend themselves to exploration and analysis, such as spreadsheets. Companies increasingly face the challenge of how to collect, store, and analyze growing volumes of unstructured data: free text, audio, video, and more.

The analytics frustration of business users

Q: How would you rate the performance of the BI and analytics solutions you use in your department/line of business (LoB) in terms of each of the following criteria? (% of respondents who selected “below expectations”)

Most important BI and analytics capabilities for business decision-makers

As the number of data sources grows, so do the challenges of cleaning, transforming, and standardizing the data so that it’s easier to analyze and work with. Many of the BI and analytics solutions that survey respondents use are too rigid to effectively handle these tasks. Consequently, organizations struggle to meet their data quality and governance requirements and ensure the consistency and accuracy of analytics.The complexity of existing BI and analytics solutions is also reflected in the main challenges that survey respondents reported: 37% of business decision-makers ranked “solutions being too rigid and complex for supporting business needs” as either a “significant” or a “very significant” challenge affecting their BI and analytics strategies. Other prominent challenges that point to complexity issues include the following:

Data too messy for analytics
Spending too much time on search and data preparation instead of analysis
Agreeing on official analysis
Inability to decide on key performance indicators
Difficulty of adoption and collaboration on analytics
Lack of flexibility to support different environments

In contrast, only 15% of the data executives who participated in IDC’s survey cited the complexity and rigidity of their BI and analytics solutions as a top challenge. This suggests that existing solutions are not designed to address business users’ needs.

BI and analytics performance challenges for business decision-makers

The gap between the analytics requirements of business decision-makers and the capabilities currently available to them is further evident in dissatisfaction with their current BI and analytics solutions. IDC’s survey found that business decision-makers struggle with issues such as generating actions and recommendations, sharing insights, scaling on demand, analyzing data as part of existing workflows, and adding new data sources.

Business users want self-service

Q: What’s your preferred method of Q. consuming data? (% of respondents)

The IDC survey results suggest that business decision-makers want more ease of use and the ability to leverage analytics on their own (Figure 5). By using natural language search and query and visual data discovery, business stakeholders can intuitively analyze data, identify trends, and rapidly respond to new opportunities or issues on their own instead of relying on BI and IT teams. Nearly 60% of business decision-makers cited faster response to business needs as the main reason for implementing self-service, followed by having more flexibility to address changing business needs (44%).

In particular, the participants in IDC’s survey mentioned the delivery and customization of reports as tasks they would like to perform on their own using self-service analytics capabilities. For example, a supply chain director in a mid-size US-based manufacturing company mentioned the need for ad hoc reporting capabilities, which can “help nontechnical staff generate their own reports.”

The comments are in line with another finding from the survey that highlights the importance of self-service. When asked about their preferred method of consuming data, more than 60% of business decision-makers selected interactive dashboards as their top choice, while self-service reporting was also ranked as a top option.

An agile BI architecture begins with infusing analytics

Q: Please rate the importance of the following objectives for embedding BI capabilities within existing business applications and workflows for internal analytics. (Mean score on scale of 1–5)

Another means of data consumption is infusing analytics into business applications. More than 41% of business decision-makers reported that they prefer this method. The perceived advantages of infused, or embedded, analytics extend beyond ease of use and self-service enablement. Participants expect to be able to use infused analytics to address key business objectives associated with the need for an agile analytics infrastructure, (e.g., providing faster responses to potential issues and expanding the use of analytics to more employees). These objectives indicate the importance that business users attach to embedded analytics as a method that can facilitate more frequent and effective use of analytics and thus contribute to increasing the overall intelligence of the business.

Data complexity is a barrier to accelerating time-to-insight

Q: How important are the following analytics capabilities for your organization? (% of respondents)

Most important BI and analytics capabilities

IT professionals also observe limitations of existing BI and analytics approaches. Many cite lack of capabilities to integrate and synchronize data across disparate sources to ensure data quality, security, and governance. Verbatims include the following:

“The ability to synchronize our datasets is something missing.”
“We are missing the ability to generate insights from the data sources available.”
“Collaborating with advanced data sources is difficult for us.”
“We are unable to understand the complexity of data.”

Moreover, data executives mentioned that data complexity has a direct impact on the ability to deliver insights in a timely manner:

“We need to enhance our ability to extract data from different data sources so that we can generate effective insights from our data faster.”
“Transforming data into insights is time-consuming.”
“We face issues in managing time for data analytics.”
“The ability to keep up our pace of utilizing and analyzing huge datasets is missing.”

Data executives responded that product capabilities such as connecting to multiple sources and data profiling for data quality are most important.

Consistent data quality is key

Q: What are the biggest pain points preventing you from progressing in your data strategy? (% of respondents)

Data executives struggle with data governance and selected quality. The main challenges that data experts report point to the technical difficulties surrounding current BI and analytics implementations. To implement a self-service analytics framework successfully, business decision-makers should be provided with reliable datasets.

However, as big data continues to grow in volume and diversity, ensuring its consistent quality has become a major challenge for data experts who often spend most of their time on data preparation. Accordingly, data security and governance issues were highlighted as the biggest pain point among data executives (54% of the respondents), followed by data complexity and quality (49%).

AI is a key enabler of change

Q: Which response best represents your organization’s approach to using AI for BI and analytics? (% of respondents)

The IDC survey concludes that existing BI and analytics approaches are too rigid to support the implementation of a company-wide insights-driven culture. The survey findings highlight the need for an agile BI and analytics architecture that \ contributes to a better alignment among all stakeholders, enabling data experts to deliver trusted, accurate data that business decision-makers can leverage through self-service data exploration, easy-to-use analytics and reporting, and collaboration around a “single source of truth.”

Given the growing scale and complexity of data, AI will play an essential role in creating BI and analytics architectures capable of digging into vast amounts of diverse datasets to identify anomalies, outliers, and trends. These systems will empower users to rapidly explore their impact and the drivers behind the insights that matter to them and take the right actions at the right times.

The growing importance of AI is evident in the finding that most organizations consider AI a key component of their BI and analytics product strategies. According to IDC’s survey, 56% of organizations require AI capabilities as part of their BI and analytics solutions, and 38% plan to include AI in future product deployments.

However, AI adoption is still at a relatively early stage. While practically every organization is aware of the benefits of AI, only 15% of the organizations that participated in the survey reported using AI for BI and analytics as part of an enterprise-wide strategy, while an additional 20% reported using AI in multiple internal processes. Roughly 65% of organizations use AI in a more sporadic, ad hoc manner.

IDC’s survey found that while both groups attach significant importance to the use of AI, they differ in their specific requirements. Unsurprisingly, data executives expect to leverage AI to automate technical tasks such as the detection of data anomalies and data preparation. Conversely, business decision-makers look to use AI for natural language querying, predictive analytics, and generating more accurate and faster actionable insights to facilitate decision-making.

To effectively support these different needs, AI capabilities should be at the core of BI and analytics platforms. Furthermore, given the rapid pace of innovation around AI/machine learning and other advanced analytics capabilities, future platforms should be flexible enough to incorporate new technologies as soon as they become available without having to re-architect them.

There should be no such thing as too much data

The difficulty of extracting meaningful insights and obtaining a holistic view across disparate big data sources has clear implications for any organization. According to IDC’s “Worldwide Future of Intelligence 2021 Predictions” research, by 2024, organizations unable to bridge growing data, analysis, and decision silos will experience a doubling of their attention poverty rate, leading to an inability to separate valuable insights from extraneous information.

This prediction is based on Herbert Simon’s aforementioned teaching that a wealth of information creates both a poverty of attention and a need to efficiently allocate that attention among the abundant information sources that might consume it.

Organizations that fail to address this challenge will experience a decrease in intelligence, which will gradually compromise their competitive position.

Having “too much data” should not be a barrier to achieving intelligence, which is all about the ability to focus the attention of various stakeholders on the most important insights and empowering them to collaborate around a single version of the truth. An agile analytics infrastructure that can scale across all skill levels and use cases, leveraging AI and other advanced capabilities to minimize human error, is needed to achieve that. Such architecture should provide decision-makers with access to accurate, trusted data that can be infused into existing business apps and workflows.

Why it’s time<br>for an Analytics<br>Everywhere Culture