Guide: Analytics and BI for Data Engineers

A Strong Foundation

As we start ingesting more complex data, it gets more difficult to analyze those data sources together. I need reusable tools and models to streamline prep work and get those disparate sources into the right shape for analysis.

As a Data Engineer, you’re the core of your company’s data, analytics, and Business Intelligence efforts. You prepare your company’s data, make it accessible for analysis, and ensure your systems will allow your stakeholders and customers to analyze the right data whenever they need to. Put simply — you need to empower all your users to get the most out of your data with the fewest bottlenecks, workflow deviations, or hand-holding by you. In order to succeed in this pivotal role, you need an analytics and BI platform that’s robust enough to tackle whatever data you throw at it, but flexible enough to deliver clean, actionable data to a wide range of builders.

Sisense makes pipeline management and data governance/security simpler for you, with an easy-to-use UI that everyone on your team can get their heads around. You can focus on pulling in data from disparate sources and building analytics apps, widgets, and dashboards that give your colleagues and end users what they need. Features like reusable data models and usage analytics also allow you to understand how people are using Sisense so you can take action to boost adoption and engagement. All this makes it easier for you to focus on the big picture instead of putting out individual fires. Let’s look at how:

Simplifying Pipeline Management

My data pipeline has never had more inputs and my team has never had this many data-hungry stakeholders. I need an easy way to orchestrate the order and frequency of queries to make efficient use of our compute and still ensure that every time someone needs analytics, they have the freshest data.

Data isn’t just money; it’s power. Data is the power to understand your company, your product, your users, your world. But managing the flow of data to create answers to critical business questions is not always easy. Making new connections in your data pipeline can be one of the biggest headaches of your role! You have user data, product data, in-house data from sales, marketing, and other systems, and maybe even third-party datasets that you’ve acquired from other sources to give it all context. If you have a cloud data warehouse (increasingly common in our cloud-native era), you may also be worried about the charges you incur every time you query it. Plus, no matter where your data is, you want to be able to analyze it alongside data from other sources. If you can’t mash up all your data, what’s the point of storing it in the cloud?

To make things easier for you, Sisense has a ton of connectors to ingest your data from wherever it lives. Our own native connectors, certified third-party ones, and even full pipeline management with Fivetran (if you need it), make it easier to get your data into Sisense for analysis. The Elastic Data Hub allows you to analyze live data, leveraging your cloud data warehouse investment, and cached data either separately or alongside each other in the same dashboard or actionable analytics app. You can also control query costs by relying on cached data for KPIs that don’t change frequently and interacting with your cloud data sources only when needed. Next up, let’s dig into governance and security.

Govern and Secure Your System, Easily

My whole company is learning to use data to answer their questions. As new people bring me new requests and start digging into more data, it’s really important that I can easily give them access to a rich, secure self-service environment.

With great amounts of data comes great amounts of responsibility. Different groups have different data needs and a wide array of users comes with a wide array of permissioning responsibilities. On one side of the equation, data engineers have to make sure all of the right data is flowing into the system. On the other side is the need to make sure that each user has access to analyze the data that they need to do their jobs, but no one has access to data they shouldn’t see. The rise of protocols like GDPR, HIPAA, and others means that data security is more important than ever. Tracking which of your teams and users are working with which datasets also helps you understand if your data model is meeting everyone’s needs and meeting them on the appropriate timeline. This also gives you a window into user engagement, which we’ll talk about more in the next section. Whatever your teams are doing with your company’s data, you want to be able to manage company wide access simply, from one location.

The Sisense administrative interface and programmatic security features allow you to control access simply. You can dig into user workflows and provide access by data model, dataset, or even down to row level. Analyze broad data model usage or zoom in and see who’s using your data model and how. Are users making heavy modifications and bringing in other datasets when performing their analyses? It might be time to update your model. Did you design a model specifically so that it would include data for a particular team’s reporting, only to find that they’re not even using it? Even worse, what if they’re still using a legacy model that isn’t providing them with up-to-date metrics? With usage analytics, you can see if your hard work is paying off or if you need to go back to your stakeholders and get new parameters, so that you can guarantee the data models truly meet their needs. Next up, we’ll go into more detail about boosting user adoption and engagement.

Empowering Your Users

The job of a good data engineer is to minimize the distance between data stakeholders and the analytics they need to make decisions. You know you’re doing that job well when people stop asking you about access to existing data and start coming to you with new questions that require the creation of additional models.

The best software in the world is worthless if no one uses it. An ideal analytics platform is easy enough to understand that nontechnical users can start with clean data and answer questions on their own. It’s vital that they can build their own dashboards to explore data related to their current projects, but the platform also needs to be robust enough that other builders can tackle more ambitious projects and embed analytics anywhere customers and teammates can view them best. 

Part of this is accomplished by Sisense’s reusable data models. As discussed above, the same data model can be used by stakeholders across your company for a variety of needs. This simplifies cleaning and combining data for you and removes the heavy lifting of putting together a new data model every time your colleagues have a new question. However, Sisense’s interface is so easy to understand that users of all skill levels can drag and drop data from any source, as needed, to customize their dashboards, reports, and analytic apps, all on their own, without lengthy back-and-forths with IT (or assistance from analysts or engineers like you!).

Usage analytics give you insight into who is using your system and what they’re doing with it. Understanding how people are using your pipelines and your data models is a key to making sure you’re meeting every data need and unlocking ways to increase user adoption across the company. If one team has high adoption and another with similar needs has only a few users, it’s a great opportunity to see what the first team is doing that the second isn’t and apply those techniques to the lagging group. The reason those models are so powerful is that there are important truths in the data. Studying analytics about the adoption of your data is the first step toward improving it, which is why usage analytics are such a vital element to any software, especially in an enterprise setting.

For Enterprises and Everyone

To be powerful, data has to make sense to the people who make valuable business decisions. Every individual at my company has their own relationship with data and it’s my job to cultivate that without getting in the way.

Enterprise users will find a lot of Sisense’s features especially handy for their specific data workflows, though every engineer with an eye toward how their company is evolving will want to be aware of what Sisense can do to simplify users’ lives. Sisense’s administrative features allow easy user control and independence, giving each team and user the right permissions for them to do their job, even when dealing with thousands of users and billions of rows of data. As you go through the initial steps of setting up reports for teams or individuals, users are always able to custom tailor their reports afterward, sorting and filtering as needed, and creating the views and visualizations that provide the most value for them. Automated reporting also allows them to programmatically time out exactly when they need to receive updated information and set up orchestration that enables prompt delivery instead of manually running one-off queries.

Wrapping Up

When you implement an analytics and BI solution at your company, you’re not just buying a piece of software. You’re laying the foundation for everything that your company will build with it.

You have some of the biggest challenges: preparing data, bringing it into a usable platform, securing and governing it, and empowering your users to actually do analysis with it. These may seem daunting, but the value from an innovative data platform is an unbeatable advantage and with the right platform and the right support, you can do it.

Reusable data models, mashing up disparate data sources, an easy administrative hub, and in-depth user analytics all help you get a handle on your data, workflows, analyses, and how your users are interacting with the system. Sisense does all this and more, so you can get powerful analytics into the hands of everyone at your company and focus on building the next great analytic offering for your product and service, instead of being bogged down putting out fires.

See Why Sisense Is Awesome for Data Engineers.