Tackling Complex Data Challenges in Healthcare Analytics


In recent years, healthcare organizations have grown increasingly aware of the potential beneficiary uses of the data which they generate and gather. From general hospitals to community clinics, data and analytics can be a powerful tool to improve care, increase patient satisfaction and drive operational efficiency.

However, adopting data-driven business and medical processes into the daily workflow of medical and administrative staff poses its own set of unique challenges. Among these, one of the major challenges is posed by data complexity – i.e., the fact that healthcare organizations must try to make heads and tails of large amounts of data originating from many disparate data sources.

Modern data analytics tools and technologies can help these organizations analyze complex data and use it benefit the institutions as well as their patients. However, selecting and implementing these solutions requires one to understand the underlying data and processes in place.

This whitepaper will present the challenge of complex data in medical organizations, and present the ways in which modern technology can solve these problems.

What Makes Data Complex?

First, let’s frame the concept of complex data. In the context of business analytics, there are two key drivers of data complexity:

  • The size of the data (is it millions of rows, 100’s of millions, or billion+)?
  • The number of disparate data sources (or data tables).

These elements drive complexity because the bigger the data, the more effort (cost) needed to query and store it. The more data sources (data tables) the more effort (cost) that is needed to prepare the data for analysis.

The data complexity matrix describes data from both of these aspects. When considering analytical solutions, different approaches are better suited for each data state.

Data Complexity in Healthcare

To understand why healthcare organizations are often dealing with complex data, let’s look at some of the common data sources they might typically be looking to analyze.

Common Healthcare Data Sources

  • EMR (electronic medical records): these are essentially a digital version of the patient’s paper chart, used by clinicians to monitor the patient’s condition, treatments he or she is due for, etc.
  • EHR (electronic health records): a broader set of digital records pertaining to the patient’s overall health, including information regarding previous treatment administered by other healthcare providers, specialists, laboratory tests and more
  • Patient monitoring systems: Equipment that monitors heart rates, blood pressure and oxygen rates, measures breathing rates, heart and brain function and many other functions—all use sensors that deliver unstructured data for notification.
  • Laboratory systems. Various information regarding the current condition of a patient can be represented with lab procedures. Structured data (such as iron levels in the blood) and unstructured information (lab slides) are produced in the lab and added to the EHR system.
  • Imaging systems: Imaging devices visualize the condition and sometimes the function of patients’ organs. They produce some of the most data-dense information in medicine.
  • Wave-form processors. This type of processing analyzes waves—heartbeats (EKG), brain function (EEGs) and blood volume surging through organs. This data-intensive processing adds more unstructured data to patient records.
  • Operations support systems. The many operations tasks that make lab tests, EKGs, and physical therapy possible.
  • Administrative data collected in healthcare management systems (HMS), which looks at the hospital’s overall operations and could include information regarding matters such as resource utilization and human resources.
  • Financial data, often stored in proprietary financial management systems for larger organizations.

This (partial) list should give some idea of the astounding amounts and disparity of data healthcare providers need to examine. This is compounded by the need to access this data rapidly, and furthermore the need to combine two or more datasets from multiple sources – e.g., to find correlations in clinical and financial systems.

These factors contribute to the high level of data complexity found in most modern healthcare services.

The Path to Insights Becomes Murky

Complex datasets of the type described above require special attention in both the data preparation process and in managing the size of the data.

It takes a long time to prepare because of data modeling challenges and the need to indexing and aggregate larger datasets before they can be queried for analytical purposes. Specialized skills and resources are needed throughout the analytical process, turning any project into a cross department, lengthy effort. This manpower cost is amplified each time changes are introduced to the data model order to investigate new analytic paths. Often additional third party tools are required as well in order to alleviate some of the pains associated with this process.

As a result analytical systems often comes with a very high total cost of ownership for modern healthcare services dealing with complex data. License fees are required for analytical tools, as well additional data warehouse and data preparation tools. Further costs are incurred by the overhead and specialized skills needed to integrate and maintain multiple tools from different providers that may or may not work effectively together.

A Technological Solution for Complex Data Problems

Sisense provides healthcare organizations with business analytics software that was built specifically to simplify the analytical process for complex data, optimizing speed to handle larger data sets and unifying the process of analyzing data, thereby eliminating many steps in data preparation and easing the burden of preparing complex data for analysis in healthcare and other organizations.

This is achieved in large part thanks to two core technologies: In-Chip™ technology, optimized for processing speed by leveraging modern CPU technology. This means the system can process 100’s of millions of rows using low cost, commodity hardware. A Single- Stack™ architecture fuses all the elements of a healthcare analytics solution, from data preparation to data management, querying and visualization, into a single, efficient software solution.

The combination of these two core technologies solves the bulk of the problems associated with complex data.

Healthcare organizations use Sisense technology to combine data from the many disparate systems they work with and create operational dashboards which medical and administrative staff can consult for immediate answers to questions that arise in their day-today work, while healthcare analysts can dive deeper into the data to reach new insights and suggest data-driven courses of action – without the above described pains which are typically associated with complex data.

Key Points for Successfully Implementing Analytical Solutions

While having the right technology is definitely useful, successful implementation of any new analytical tool within healthcare organizations is also a matter of adopting internal culture to turn data and analytics into part of the everyday workflow. Here are some factors to consider:

Executive and Stakeholder Buy-in

Adoption rates of analytical tools are often low due to a lack of confidence in BI tools and methods. Successful business analytics projects require making your hospital or facility a data-driven organization, and to do so the initiative must either come from the top, or at the very least enjoy the support of the organization’s leaders. It’s not enough to have the tools in place – for people to actually use them they need to understand the value and potential of analyzing and visualizing healthcare data, and this could require a cultural change that must be facilitated by upper management.

An Effective Data Analytics Team

One should not underestimate the importance of the human element: create a skilled and effective group of people who know their data and want to use it to improve decision-making processes, and who enjoy digging into the data to find new insights.

This means starting with people who have empathy for patients and excellent communications and management skills – but who are also passionate about data and how it can save time, money and lives.

Understanding The Project’s Goals

Given the availability of self-service healthcare analytics tools, it’s easy to take very deep dives into data that provide information but very little value. Business intelligence tools have varied capabilities, so the organization should define its objectives and medical or business questions it wants answered before spending time and effort generating reports. If you don’t know what you’re asking, you won’t get answers to questions that matter.

Broad Access to Data

While it’s important to have a strong analytical team to manage data modeling and governance, it’s just as crucial to give the non-technical easy access to data and allow them to draw their own insights from it. Broad usage of data by medical and operational staff, without these professionals having to pass through technical gatekeepers for every report they want to view, enables healthcare providers to truly maximize the value they derive from implementing analytical tools.

Further Reading and Resources