Measuring engagement with a business product is difficult. Users often spend hours per day in a business tool where complex workflows make interpreting user intent through clickstreams difficult – and worse, typical engagement metrics like time-on-site, page loads or object creation counts generally just describe usage, not created value.
Because of the complexity of modern products, it’s also difficult to track how certain changes are affecting users. In a straightforward B2C product, the goal might be to optimize time in product or clicks on a certain screen, because every user is doing the same thing. By capturing and analyzing more complex data about products, A/B testing can identify the effect small changes and new features have on engagement. However, for a business tool, these strategies fall apart because the user’s interaction is not easily broken into discrete goals, and representing engagement as a single metric isn’t reliable.
In today’s competitive technological landscape, companies are collecting enormous amounts of product analytics, but are running into trouble using it to make intelligent product improvements. Measuring product success requires asking highly contextual questions and rigorously applying data to find answers. My team at Periscope Data by Sisense has had a lot of success blending complicated product data together and then using innovative thinking to structure new questions. Here are some examples of complicated product performance questions we’ve answered with data:
When a new feature is added, the big question is always whether that feature is generating more revenue for the company. Are new customers buying the product because of that feature? Are existing customers paying more to get access to that feature? It’s hard to find out from an opportunity record in your CRM. New customers might be buying the product for a number of different reasons. The same is true for upsells to existing customers. Attributing revenue to specific features is a challenge, but solving it gives organizations a data-driven way to compare features and make important product decisions.
My team’s approach to solving this problem was to look at every single account and put feature usage into the context of the date of the sale. We created a simple bar chart with a data point for every closed-won sales opportunity, including both new and existing customers. For each data point, my team tracked the number of days between the first use of a feature and the date of closed-won opportunities. Some accounts closed before the feature was used (illustrated with bars above the line in the chart below) and some trialed the feature first, then made a purchase (the bars below the line). These bars show us information about which features have pent-up demand (bought before use) and which features are effective at converting trials into sales.
This chart is easy to construct using SQL, but the tricky part is defining which behaviors will qualify as feature usage. Determining that criterion – which may be different for every feature! – and then mapping the data like we did gives you two useful tools: one, a way to determine if your feature is important in customer trials (and therefore should be showcased in the sales process), and two, a revenue number that can be compared to other features to measure the relative impact of your product investments.
Every product manager has replaced a feature with a better feature, it’s a critical part of the job responsibilities. But there has to be a logical way to identify the features that are ripe for replacement. The easy way to pick these features is to pick the ones your support team hears about, but that approach runs the risk of giving too much power to a vocal minority. This is a great time to use data about your entire customer base to see what features are being utilized to provide value for users. These feature performance metrics are a handy shortcut to seeing the behavior of the entire user base.
To share an example from Periscope Data by Sisense, we noticed that a lot of users were annotating their dashboards with text labels and descriptions, even though the product didn’t actually support doing that! Users had discovered a clever hack: using a number display chart, but feeding it a SQL statement that would produce a string rather than a number allowed them to get text onto a dashboard in a useful way. This was a common-enough need that the clever workaround had been picking up steam for years. Once we realized this, we built a feature called Text on Dashboards that would give analysts a proper way to mix words into the charts on their dashboards, full of options to customize that copy as robustly as they needed. We built a chart to track how many text boxes were being created with this new feature (the red bars on the chart below) vs. with the old workaround (the blue bars) and the results were surprising: People didn’t stop doing things the old way just because we had built a better way. Actually, the usage of both strategies increased.
What we discovered was an insight about the users rather than an insight about our features. Users were adopting our new feature – we could see that clearly in the data – but the mechanism wasn’t replacing the old workaround.
We dug deeper into our usage data, looking at which users and which organizations were still using the workaround, and discovered that new users at new customers would generally use the new technique, and older users would use their old workaround – but new users who joined an existing team would sometimes adopt the workaround rather than learning the new technique.
The insight for us is that users get comfortable with their usage patterns and are more comfortable copy/pasting an old dashboard than integrating a new feature into their workflow – and that our users train each other. Our new product wasn’t failing, legacy users just behaved differently than we expected.
One of the consistent behaviors in enterprise usage of B2B products is a gap between the rollout of a new version of a product and the actual adoption of that version. Measuring that gap is important because teams need to know how long to wait after rollout to start tracking adoption. Product teams don’t want to waste time worrying about new feature usage too early, but they also need realistic timelines to expect to see an uptick for new versions.
To define this period accurately, my team looked at historic usage to set a standard for the number of days between rollout and adoption. The chart below has blue bars to illustrate how many accounts adopted the latest version of our product at each day after its release. The line on top add in an extra dimension: spend. Assuming that larger companies are spending more money with us, we see that there’s a relationship between the size of the company and the time to adoption of a new major capability in the product. What we found was that it takes around 60 days for larger companies to start adopting new things, so when our team examines usage data around a new product, we wait that long before we really dig into adoption data to make sure we’re making good use of our time and still getting accurate results.
Diving into this information even further, we found out that features are intended for different personas, and that the persona type affects adoption rate. For an end-user feature like Text on Dashboards that any analyst or even business professional can use instantly, the adoption time is relatively short. Admin-centric features like permissioning changes can take months because of their wide effect. Rollout delays affect every company, product, and feature differently. Look at your customer data and see what you can learn about their behavior.
One of the most important jobs of a product manager is to understand the journey your users experience and to feed them information that will help them get the most value out of the product based on their stage that journey. You wouldn’t send an experienced user and a trial user the same messaging, they’re looking for different things.
To create cohorts in our user base and offer separate and specific messages, we created a simple bar chart displaying the time between first use of our product and most recent use of our product, at a user level. Brand new users are on the far left and veteran users are on the far right. From that chart, we noticed a few cohorts. There’s a big spike at the beginning (trial users), a long tail at the end (early adopters), and a noticeable gap around 125 days that we can use to separate the experienced users from those just learning the product.
With these cohorts determined by data, we can set up customer marketing programs to make the product sticker for users at any point in their relationship with our product.
If your organization is struggling to decode complex behavior, start by asking basic questions of your usage data and looking for insights. You can use those insights to ask new questions and keep repeating the process until you narrow all that data down to something actionable. It’s a long process with lots of trial and error, but the complexity of your data is an opportunity to get to know your users on a more personal level and offer them more value than ever before.