Roadmaps should be goal-oriented. We’re not adding features and functionality for fun, but rather to help the business meet its objectives. Those goals are typically measurable— which is where product roadmap analytics come in. Each roadmap item influences the metrics the business cares about to keep the product strategy headed in the right direction. Therefore, we should be able to map quantitative and qualitative data for the product roadmap.
Quantitative and Qualitative Data for the Product Roadmap
Incorporating metrics into your roadmapping process has multiple benefits:
- Provides supporting evidence for why each item is worthy of making the cut
- Helps build alignment among stakeholders since they’re also interested in those metrics for the intended improvement of the coming enhancements
- Provides a mechanism for evaluating how useful implemented roadmap items are
- Minimizes “gut feelings” from prioritizing initiatives that aren’t focusing on impacting the organization
How do You Define Product Metrics for the Roadmap?
The metrics driving your roadmap should be furthering the goals of the business while connecting to the product vision. Here are some tips on choosing wisely:
Find metrics that drive behaviors you know lead to success. For example, you may know that adding ten “friends” increases usage by 50%. Usage is a key metric, but your roadmap should feature items driving the “how many users added ten friends” metric, not just the vaguer “usage” measurement.
Beware of vanity metrics
Skip vanity metrics. Some metrics look shiny and important but don’t mean anything. Your product metrics should be measuring things that truly move the needle. The intention of your roadmap isn’t to make the company look good in the press or give the CEO an impressive talking point in their pitch. If they don’t correlate to actual KPIs, there’s no real reason to waste time on them.
Less is more, so stick to the good ones. More metrics don’t necessarily add value. They might even confuse things. Pick a few that are meaningful and relevant and keep a narrow focus on moving them in a big way.
Customer-oriented metrics
Don’t forget the customers. Business-oriented metrics can take up all the air in the room. They’re what the management team tends to focus on. They’re also the most common measure of a product’s success according to our 2020 product management report. But customer-oriented metrics are the real test of how well a product is performing. Retention and churn rates, measures of quality, and how many users have used specific features are better metrics for product roadmaps than the cost to acquire a new customer. Journey analysis can shed light on which actions are key to loyalty, as well as where users keep getting derailed.
Focus on what’s important
Avoid irrelevant metrics. If your roadmap can’t impact a metric, don’t include it in the conversation. It’s impossible to achieve every business goal via the product itself. Leave sales, marketing, and operations metrics out of it.
Stay current. The metric exciting everyone last year may have been replaced by something deemed more impactful. Don’t be afraid to adjust to the latest thinking and avoid appearing out-of-step with stakeholders.
Skip anything too expensive to monitor. If a metric requires an extensive amount of work to measure frequently, it may be a poor choice for your roadmap. You’re committing to improving it, so your higher-ups will expect you to provide regular updates. If that requires too much work, you’re setting yourself up for failure (or a lot of extra work).
Quantitative vs. Qualitative Metrics
There are two flavors of data in the world: quantitative and qualitative. Before going any further, let’s clarify what those mean.
Quantitative data is “hard data” borne out of analytics, surveys with statistically significant sample sizes, and other indisputable sources. Examples might be transactions per week, how many people clicked the blue button, and what percentage of users are from Germany.
Qualitative data is a little more “touchy-feely” and anecdotal. It comes from open-ended questions on surveys, customer interviews, sales call feedback, and support tickets. It looks more like “they prefer mobile apps because not all their employees have access to a computer” or “it takes too many clicks to finish the task.”
Both are valid and valuable, but each type of data requires different evaluations.
Quantitative metrics
Quantitative metrics are based on cold, hard data, and usually derived from analytics sources. These metrics show you exactly what’s happening with your product, answering questions such as:
- Who is using it?
- What parts are they using?
- When are they using it?
- How long are they using it for?
- How often are they using it?
- What aren’t they using?
- Are there things they use in the same sessions?
- What things do they never do at the same time?
The list goes on and on. Anything meant to log events can produce a ton of quantifiable data.
The entire marketing, sales, customer acquisition, and onboarding process may also yield quantitative data. Use that data to calculate conversion rates, acquisition costs, lifetime value— all the business metrics the executive team gets excited about.
Best of all, you can slice and dice quantitative data to create additional metrics. For example, you can see how many new users view a tutorial in their first thirty days after signing up. Or you can determine whether reading on-site product reviews increased the likelihood of a purchase.
Quantitative metrics are especially helpful for running experiments and testing out hypotheses. Instead of relying on usability tests, focus groups, and surveys, product managers can run A/B tests in production to see how real-world users react to different options.
Quantitative data can inform decision-making, track progress, and indicate behavior changes. It can also help test your assumptions and validate current beliefs.
Cons of quantitative data
But what it lacks is context. You know what happened, what’s happening now, and might even be able to predict what’s going to happen next.
What you DON’T know is why. You can make assumptions about causation (i.e., 72% of users that view a how-to video complete a task). But without talking to users, it’s tough to decipher the motivations and mindset of users. That’s where qualitative metrics come in.
Qualitative metrics
Quantitative metrics are calculated and tabulated. They are meaningless without large sample sizes. It’s all about the numbers.
On the other hand, qualitative metrics are all about the individual. What was this person thinking? Why did they make that choice? Did they like or dislike a particular experience or change?
This information comes from asking individuals about their experiences, emotions, and thoughts. This input can still be categorized and tabulated. But this sentiment-based data seldom results in pretty charts and graphs like its quantitative counterparts.
By its nature, collecting qualitative data is more laborious. There are no analytics packages that include mind reading. So instead, product teams and UX staff must rely on other methods.
If you have the tools or the patience, you can even look at click data to spot rage clicking, excessive “back button” usage, and other signs of frustration. These may hint at confusing navigation or unresponsive parts of the experience.
Surveys are a trusty standby for gathering both quantitative and qualitative data, with the latter coming via open-ended questions and prompts. But these nuggets can also be mined from interviews, focus groups, usability testing, and session replays. And don’t forget feedback coming in from sales, support, and other channels.
The goal is a holistic view of the user experience, which may include factors far beyond the recorded clicks.
Did the user open a chat session with support or make a call to the toll-free customer service line and have a terrible experience? Did they hit a paywall? Are the language or pricing terms confusing? A complete picture of a user’s experience, expectations, feelings, and frustrations are tough to discern from the analytics alone.
Is Certain Data Better for the Product Roadmap?
While some may prefer quantitative metrics over qualitative ones, they should be used in concert together to understand better what’s happening with a product and identify opportunities for improvement.
Quantitative metrics can spot trends, but they struggle to identify the causes. But once that trend is spotted, qualitative metrics can uncover factors driving things.
For example, the quantitative data may reveal that adoption has dropped. The same number of prospects are being driven to the home page or download the app, yet a smaller percentage become regular users.
Quantitative metrics will surface the problem and raise a red flag if adoption is a tracked metric. Some detective work may even identify a UX change or a price increase as the impetus for this decline.
But without actually asking some of the folks who tried things but didn’t become regular users, you can’t know for sure why they’re not interested. Those subtleties are hard to unpack without qualitative metrics providing extra context and insight.
Product managers are ultimately in the problem-solving business. It’s hard to know what those problems are without talking to users and customers to get that qualitative take. The quantitative metrics can help you decide which questions you want to be answered in the first place.
Want to learn more about using data to inform your product decisions? 
