Every Company Is a Data Company: Building Analytics in the Age of Big Data

Intro: Bigger than Big

The effect that the increase in the size and scope of Big Data will have on human life is almost impossible to comprehend.

Size, sheer, volume is hard enough to reckon with. Twenty years ago, hard drives were bulky things that held gigabytes. Today a terabyte SSD chip is the size of a fingernail. Cloud servers store zettabytes of information that can be accessed remotely from anywhere with an internet connection. The International Data Corporation estimated in a recent study that by 2020, there will be around 40 trillion gigabytes (40 zettabytes) of data in the world.

The scope of Big Data encompasses, well, everything: Bank transactions, phone calls, texts, emails, tweets, social media of all kinds, images (the ones we know we’re taking and the ones we don’t know are being taken of us), and much more. Wearable devices like Fitbits and medical implants wirelessly transmit human biometric information into massive healthcare datasets. Internet of Things devices from advanced vehicles, robotic factory hardware, and a dizzying array of other machines constantly create digital records without any human input at all.

We are living through the rising of the seas of data, as the digital realm encompasses more and more of human experience in the Age of Big Data. Throughout history, humans have sought to control their world using whatever tools and technology we had at hand, while simultaneously never letting the limitations of “the possible” stop us from inventing new things. Innovation has allowed humanity to push the boundaries of what’s achievable, building the new tools and technology we needed not just to survive, but to thrive. The key to thriving in the Age of Big Data will be multi-pronged, but two things every company that wants to succeed will need are a lot of data and an AI-assisted way of dealing with it.

In this paper, we’ll talk about how the rise of Big Data is forcing (or will force) every company to become a data company.

Companies produce and collect more data than ever before and that number will only keep increasing. They’ll need AI-assisted augmented analytics systems to find and serve up meaningful insights and empower their teams to build powerful analytics for internal users and customers alike. These systems will also allow them to mashup disparate data sources and reveal more game-changing connections and increase understanding across the organization, and indeed, the world. We’ll talk about what these systems will look like, how they’ll work, and how they will reshape the world (and allow the companies that use them to do likewise).

Every Company Will Become a Data Company

Yes, Even Yours

This can’t be overstated: the world-changing amounts of data that are being created and captured will turn every company into a data company.

Forward-thinking ones are already doing this. There are a few reasons for this: First off, companies and customers alike are creating more digital records every day. Second, companies realize they can monetize this data and customers are clamoring for insights into their own usage habits regarding the services they use. Third: storing, connecting to, and analyzing data on cloud databases has become easier and cheaper than ever before, and companies are leveraging this.

Data is the new oil, and companies that want to thrive will need to tap it and get the most out of it. In the next 10 years, the number of embedded devices (in smart buildings, factory devices, driverless cars, etc.) will jump from less than one device per person around the world (on average) to over four! Furthermore, the average person will interact in some way with a connected device almost 5,000 times a day by 2027. These predictions are only the barest glimpses of where this data will come from and how it will be created. Think about it: does your service or product connect to the internet? Is there a sensor involved somewhere? Do your customers interact with your company via a smartphone or connected device? Whatever you are building, you are going to need to become a data company.

This might sound like a daunting challenge. It is. It’s also an opportunity, because the massive amounts of data you collect on your customers and your service will help you make more money, both by making smarter business decisions and monetizing that data in the form of customer-facing insights. Here’s an example: Remember the hit Netflix show House of Cards? Kevin Spacey aside, the net impact of this show was overwhelmingly positive for the streaming entertainment giant. The kicker is that they created that show from the ground up using insights gleaned from their massive store of customer information. By analyzing which actors, directors, genres, and other similar movies and shows their customer base liked, they were able to cobble together a winning product. Think of the last major decision your team made with insufficient data; now imagine making your next one backed by a deep well of data and the right analytics solution.

While you’re at it, imagine boosting customer engagement and product/service stickiness by serving your customers insights gleaned from their own data. Socrates is quoted as saying “The unexamined life is not worth living.” Your customers will prove to you that this is true. Every platform with data is giving users a chance to look into their own behaviors and customize their habits. Take Dollar Shave Club, a company with a mission so simple that its name pretty much encompasses it in its entirety: Their usage analytics let you look back at your shopping history to see how much you spent, what you bought, what your delivery frequency was, etc. Oh look, there’s the winter you grew a beard and paused shipments for three months. There’s the summer you took a lot of vacations and switched to every-other-month shipments because you were doing the whole Miami Vice stubble thing. Knowledge is power and companies are realizing that giving their users insights into their own behaviors is creating happier customers who stick around longer. Some models even allow for selling customers enhanced analytics and data to drive up revenue. If you’ve got the data and the right analytics solution, you can build innovative products that revolutionize your business. If you don’t, you won’t have to worry about it for very long…

Last up is the rise of the cloud. Remember how hard it used to be to start a software company? You had to have a person on your team who knew how to do server stuff, then buy a physical unit, stick it in a closet somewhere, and prop the door open to make sure that it didn’t overheat and crash your whole company. Today, things could not be more different. The range of cloud database solutions is broad and deep. There’s an option to suit every use case and budget, augmented by the fact that companies only pay for what they need in terms of storage and processing power. These systems integrate seamlessly with popular operating systems like Linux and can put the entire app on the cloud for a powerful cloud-native experience that makes upgrading software, scaling, and serving customers easy and efficient. The Big Data builders of tomorrow are using Linux and cloud servers today.

But just picking the right pieces isn’t enough to build the winning companies of tomorrow. Every successful company will need a Data Strategy and an AI Strategy.

Every Company Will Need a Data Strategy and an AI Strategy

Yes, Even Yours

Nearly a decade ago, tech and investing guru Marc Andreessen declared that “software is eating the world.” And how right he was. That trend of everything becoming computer- connected, app-driven, and data-oriented is one of the main drivers of the present Age of Big Data. There are simply more programs and devices creating data every day, and more pieces of software designed to help users get the most out of that data. By 2025, 75 percent of the population will be connected, creating and interacting with data. This data needs to be immediately available whenever and wherever users and organizations need it. By 2025, real-time data will make up more than 25% of data being created in the datasphere; info from IoT devices will be 95 percent of this. And these are only two examples of how the world of data will change. Just picking a cloud data warehouse and dumping all your info there isn’t the same as having a well-orchestrated data strategy. And even if you have a data strategy, if you don’t have a plan for tackling the rise of AI, then you’re only halfway there. Every company will need a data strategy and an AI strategy...and if they’re smart, they’ll throw in an analytics and BI platform to merge the two and seal the deal.

Some people think a data strategy is just a fancy way of talking about a company’s data architecture. It’s bigger than that. The right data strategy is a game plan for transforming an organization and addressing all the things data needs to do in order to help the company excel. For instance, just a few things a data strategy could be concerned with:

  • Self-service analytics for internal stakeholders and decision makers
  • Utilities to allow internal builders to create analytic apps and features for the core offering
  • Serving insights to customers and letting them dig into their own data and answer their own questions
  • Empowering non-technical team members to build more complex analytics solutions for themselves
  • Monetizing data
  • Where data is stored, how it’s connected to and secured, who has access and control
  • Long-term plans for expanding and scaling data.

And this is far from an exhaustive list. Every company will need a skilled team to draft and execute their data strategy, and maybe even a Chief Data Officer. This person is responsible for defining and transmitting the company’s overall vision for what their data can do for the company and its customers and how that will be accomplished. The overall team in charge of the data strategy will own the roadmap for the people, tech, security, and cultural elements, and everything else involved in going from where the organization is to where it needs to be. The strategy will encompass the entire organization and help smooth over competing needs from different departments and ensure that data is used to empower everyone in the organization to accomplish their goals.

As we already mentioned, though, having a data strategy won’t be enough. Datasets have gotten so big and complex that they’re impossible to handle without AI help. Enter: the AI strategy. Over a third of executives have a stated primary goal of using AI “to optimize internal business operations” (Techjury). Additionally, countless services, established and emerging, use AI to serve customers: ride-sharing apps move cars around and create routes between users to minimize time and cost for users and maximize pickups for drivers. Amazon is working on purchase suggestions that would send customers items before they order them, based on past buying behavior and other factors.

AI will do more and more work for humans. Every rote task and mundane bit of hunt-and-peck drudgery is ripe to be done by a trained AI. Additionally, as every company becomes a data company, the AI will also be used to get the most out of a company’s data, both for internal and customer-facing applications. However, this doesn’t happen by accident. A company’s AI strategy needs to clearly outline the goals for using AI and the internal teams responsible for developing and applying their AI tech.

And where do data strategy and AI strategy intersect? Analytics and business intelligence. Big Data is a problem only AI-assisted analytics can solve. The right analytics and BI platform will allow an organization to dig into its data, find team- and user-relevant insights, and share them across the company. A robust platform will also empower internal builders and product teams to create powerful analytic apps for customers and business users alike. Truly self-service solutions boast UIs so simple that non-technical users can perform ad-hoc analyses, assisted by AI elements, and even build their own analytic apps. Throughout this all, built-in AI systems suggest datasets to combine, analyses to perform, visualizations that make the most sense, and who to share findings with. This is all happening right now. Flat World Solutions reports that “automated analytics will be vital to big data by 2020.” The best time to act on these trends is five years ago; the second-best time to act is now.

AI Is Eating the World

In the future, every vital technological product, application, and service will be built on AI, Big Data, and augmented analytics.

PwC reported in their 2019 predictions that only 22% of companies surveyed were in the nascent “investigating the use of AI stage.” The remaining 78% reported that they were already using AI or planning to do so: “already implemented in multiple areas” (27%), “planning to implement enterprise-wide” (20%), and so on. In the same report, the top priority of the most companies (58%) was to integrate AI and analytics systems. These savvy organizations have rightly ascertained that AI and analytics together present not only the best chance for survival in the Age of Big Data, but also the best chance to excel. This is because only augmented analytics will be able to make sense of this data and AI-powered neural networks will be needed shrink down insights from their huge datasets.

It can’t be overstated: augmented analytics, analytical systems supported with AI elements throughout, are the only path forward for dealing with Big Data. Older forms of business intelligence relied heavily on support from IT departments, took forever to ingest and analyze data, and served up results that were outdated and of little value for decision-making. The current generation of self-service solutions were a step in the right direction, freeing users from reliance on IT teams and allowing for ad-hoc exploration and easier sharing of results, but they remained too focused on reports and dashboards. Augmented analytics systems simplify data ingestion, suggest joins and analyses, empower users to build powerful analytic apps, and seamlessly share findings with other key players.

AI is a key player in these augmented systems because the data being tackled is simply too massive for humans to keep straight. Even if a user knows what they are looking for, it can be too complicated to pull together all the disparate datasets that they would need to wrangle in order to find answers. Or, even if a user is able to combine the elements that hold the insights they want, they can miss other connections that are less obvious and could lead to big gains for an organization. The AI, learning from past human search behavior and combining those learnings with advanced algorithms, can offer up more innovative and dynamic results. This is where game-changing insights come from: not from looking for the answers you think you want, but from being open to analyses that you had never considered.

Just digging up insights isn’t enough. The worst part of Big Data datasets is how, well, BIG they are. That means everywhere you want to serve up insights from them, you’ve got to be able to access them, severely hamstringing where, when, and how you can provide those insights. Deep neural networks (DNNs) solve this problem. AI systems “learn” the Big Data set through a process called “training” and crunch the insights from terabytes of data into DNNs that are only megabytes in size and can be put anywhere (even edge computing cases) to provide insights wherever the user needs them. These DNNs also allow for real-time querying of Big Data sets without continued access to the underlying data, with 99% accuracy. Additionally, the DNN also doesn’t retain any knowledge about the lowest level of detail in the data they were trained on, so there is zero risk that queries could return sensitive data that is prohibited by your policy or regulatory compliance requirements.

The implications for this entirely new class of analytics are world-changing: First off, builders and users alike are no longer shackled to Big Data sets or long query times. You can put health analytics at a doctor’s fingertips or into a wearable device. Factory machines can be trained to understand common issues they can face and learn how to make their own decisions, without relying on human intervention or long back-and-forths with a Big Data store.

Using AI to Build the Future

There are humans living right now who will see an age when the words “artificial intelligence” no longer mean what they do today.

AI will be built into the fabric of their lives, handling an array of routine tasks and fueled by a well of Big Data so vast that no aspect of existence is untouched by it. The datasphere will be all-encompassing. It will be the water that our species swims in. The companies that will live to see that future are the ones embracing this new paradigm today. They will harness Big Data sets, design AI systems to make their customers’ lives better, and combine the two with robust analytics.

AI assistants live in every smartphone on the planet and more and more of its laptop and desktop computer systems. Natural Language Processing in Alexa and similar smart home devices is teaching the new generations that it’s simpler to talk to their computers to get what they want than to use their fingers to type in a search. As these intelligent systems learn our preferences, services will anticipate our needs. Every company looking to provide a product or service in the coming Age of AI will need to build an AI into its offering that can give users what they want with minimal friction.

How will those systems know what people want?

Using Big Data, of course.

But not just the company’s own stores of data. No, enterprising organizations will seek out the datasets they need to get fuller context for the decisions their customers (and prospective customers) are making. No single dataset, however large, can tell the full story. Partnerships and data buying and selling will become a huge element of the coming data economy, giving a competitive advantage to companies who have the foresight to gather or gain access to the right data stores. Mashing these datasets up via an analytics platform will give their AI systems the fuel they need to delight customers and serve up revolutionary insights for the parent company. The combination of Big Data, AI, and analytics will define the future of human development. The companies that fail to learn this lesson will be lost to history.

Wrapping Up

The seas of data are rising. Is your company ready to evolve and thrive?

Every significant technology company (or company that uses technology, which is to say, all of them) of the coming decades will need an effective data, AI, and analytics strategy. Without the right data, both collected and accessed (through purchase, partnership, etc.), companies will lack the raw materials to deliver what their customers want. AI systems need this data to learn how to please customers and handle tasks that humans just don’t want to do anymore. Robust analytics, themselves undergirded with AI elements, tie these two vital elements together, allowing AI systems to get smarter faster, offer up insights to customers and parent companies alike, and putting deep insights anywhere via innovations like Deep Neural Networks. The world is changing. Whether you can feel it or not, the seas of data are around your ankles. Soon they’ll be up to your knees. You can learn how to swim, or you can build a boat.

See How Sisense and AI Tackle Big Data