One of the most useful product management insights we’ve heard in a long time comes from a professional poker player. No, seriously. Hear us out.
Annie Duke, a two-time winner of the World Series of Poker, explained recently on The James Altucher Business Podcast that “truth can atrophy.” Annie had this epiphany when she began teaching poker to novices and she realized some of the tactics she had used so successfully years earlier as a player no longer made as much sense strategically.
In Annie’s case, it was when she tried explaining these tactics to her students that she realized for the first time that, yes, they had worked for her—but not for the reasons she thought they had.
And she could just as likely have discovered some of her go-to poker tactics no longer worked as powerfully for other reasons—such as the fact that other players had adopted the same strategies and the elite poker community had learned to spot and counteract them.
The point is, whether you’re a poker player or a product manager, sometimes even the things you’re certain about—your truths—can wear away or change over time.
This brings us to the importance of testing.
As a product manager or product owner, you’re probably highly skilled at gathering information. We have no doubt you’ve worked hard to build a valuable knowledge base about your customers, your market, your competition, and the current technologies and best practices that allow you to improve your product and your users’ experience.
But you always need to remember that any part of the knowledge base you’ve built—even the elements you consider fixed, permanent realities—can and probably will change. Moreover, failing to spot these changes can undermine your product.
With that in mind, here are some things we think you should be tested regularly—even if you consider them to be settled truths.
Test what you’ve already tested.
Let’s say three years ago your product team went to your target market to gauge their interest in a new feature for your flagship product, or maybe an entirely new product. And let’s say the market said no.
If the idea for that same feature or product came up in a meeting tomorrow with your cross-functional team, we’re guessing you’d say, “Tried it. No interest.” That makes sense. You tested it, and you received a clear answer.
But truths can atrophy.
Maybe things in your market have shifted. Maybe the competitor that was already supplying the feature to your market (which you didn’t learn in your surveys because they didn’t ask that specific question) is no longer in business. Or maybe the industry has experienced enough turnover that there are plenty of new decision-makers at your prospect companies who would be interested today.
Tweet This:
“Just because you tested something once doesn’t mean you have an answer forever.”
Bottom line: Just because you tested something once doesn’t mean you have an answer forever.
Test your successes.
Direct marketers—the people responsible for filling your home’s mailbox with junk mail—are always testing their packages, even their winners. And that’s one key reason they make a fortune.
Because the industry has been learning and refining its strategies for more than a century, direct marketers know that even today’s hugely successful direct-mail package has a finite shelf life: Eventually, everyone who’s going to buy the product or take whatever other action the package calls for will have done so. Marketers can’t run the same package forever, so they continually test other packages with limited audiences to find the winner’s eventual successor.
The same principle applies to your products. PMs can understandably get complacent when their product is a hit. Things are working, after all. Why mess with success?
First, as we can learn from direct marketers, you can’t afford to stand still. Today’s steep upward sales trajectory will eventually begin to slow, and then eventually begin to fall. So even if you find your feature or product is a winner, you need to begin testing other features and other products to fill that sales pipeline for the future.
And second, maybe you’ll find that another feature makes your successful product even more successful. So why not keep creating new things, and keep testing them?
Test your sales process.
Let’s say sales for your product are strong. That’s great news. But… why are they strong?
Are you doing well because you’ve got a couple of sales rock stars who do some kind of voodoo sales magic that nobody else in the organization—including the other sales reps—understands? If that’s the case, okay, enjoy the results those sales superstars are generating for as long as you can.
But understand that having a few sales geniuses on your staff isn’t scalable, and it won’t necessarily last forever. That truth can atrophy as well.
So test other ways of generating sales.
Test your data.
Data has a way of cementing quickly and permanently into a company’s reality. The statement “We have data on that” tends to shut down the debate.
But maybe your truth about what your market wants or what’s really resonating with existing customers comes from a flawed or incomplete data-analysis model.
Are you looking at the right usage data and drawing the correct conclusions from it? Is the information you’re gathering and analyzing mostly anecdotal or self-reported, as opposed to something you can view and verify firsthand?
Don’t assume that just because you are compiling data you’re getting the best answers.
Re-examine your methods of data collection, review, and analysis, as well as whatever conclusions your team is drawing from that data.
Test your own assumptions.
Finally, you need to test your own assumptions.
This isn’t to say your gut instinct on any given point is necessarily wrong. You might have even put an assumption on an evidence-based test and found it to be correct.
But remember: Truths can atrophy. That’s why we’re suggesting the other types of testing above, which could demonstrate that some of your assumptions—even the once-accurate ones—are incorrect today.
Your gut told you your target personas were worried about data in the cloud and would, therefore, demand an on-premises solution. And when you asked them three years ago, you found out you were right. But are they going to be as concerned about cloud security today? Maybe it’s time to test what you’ve already tested.
Or perhaps your gut tells you to target a certain revenue number for a new product launch based on what you’ve seen your sales team generate from previous products. But maybe those revenue numbers have been artificially low because instead of an efficient process across the entire sales team you’re relying on just a couple of sales stars. Maybe it’s time to test your sales processes.
Tweet This:
“There’s nothing wrong with forming an assumption about how to handle some aspect of your product—as long as you test it.”
There’s nothing wrong with forming an assumption or gut instinct about how to handle some aspect of your product—as long as you test it.
