How to avoid the confirming research bias?
When you are in business for a long time, you usually know what you do. At least to a certain degree. Else you wouldn’t be in business anymore. That doesn’t mean, that you ever did customer research since your company existed. And when you do customer research for the first time, you are prone to falling for the confirming research bias.
This is especially true for medium sized family businesses. They run since multiple generations. Grandfather just started the business someday. He learned what he had to do. Passed it on to his son. He continued building on it. And so on. Now the youngest generation wants to update and upgrade the business and introduce some new methodologies. And they start doing customer research for the first time in 100 years.
What is the confirming research bias?
Imagine, you are in business for some time. You think, you know your business. But you want to make sure, that you do. Make sure, that you are actually right. So you invest in doing customer research. And the customer research shows you, that you are doing most things wrong. That you actually don’t understand your customer very well. And you are leaving a lot of money on the table. This creates tension in the company.
The same applies for a research team in a large company. Let’s say, you find out, that the management decision from the past 5 years have been terribly wrong. Do you want to be the guy telling it to the management?
To avoid this, people do confirming research. They design the research in a way, that it does not create disturbing results. Avoid results, that may be true, but clash with social structures. It mostly confirms, that everyone is doing the right thing. It only brings some minor adjustments.
Of course, if your research is not intended to seek truth, it won’t bring value. Confirming research will not prevent you from doing mistakes in the future. Thus, it is a bias.
Are people really doing that?
There are many ways to do confirming research. Most don’t require intention. Here are some examples, how you could make your research confirming, even by accident:
- Selecting a too narrow question for the research
- Interviewing only happy customers
- Asking leading questions
- Summarizing in a favorable way
There is also the option of doing non-confirming research and then ignoring the results. This is veeery common. But it technically not a research topic. Rather a management issue. So let’s skip it and look at the other four in more detail.
Selecting a too narrow question for the research
Every research needs a question. The question could be:
- How should we design our product?
- How can we slim down our portfolio?
- Which messages will be effective in marketing?
- How can we speed up the buying phase (especially in B2B or high ticket sales)
As you can see, all of these questions are open. They look for an answer, but don’t have one. On the other side, here are some examples of too narrow but still common:
- Do our customers like our product? (customer satisfaction survey)?
- Should we rather go for product a or product b next? (priorisation survey)
- How are we percieved in comparison to our competitors? (benchmark)
- Do our customers understand our product? (e.g. testing behavior in software)
None of those is open. The first set of questions search for a deep understanding. They allow for an understanding of your customers jobs to be done and elements of progress. The second set takes most of the current solution as given and leaves very little room for change. Therefore, the research will end up mostly confirming. Even if not explicitly.
Interviewing only happy customers
It is OK, to interview your customers. Just be aware, that they decided to buy and use your product. This is an element of confirmation already. You won’t understand, why someone chose a different solution for the job to be done. If you ask your happy customers only though, they will tell you mostly good thing. That makes the research mostly confirming.
Interviewing your happy customers is a recruitment question as well. Customers, that are unhappy with your solution, are usually not open for interviews. Happy customers happily tell you why they are happy. But it won’t help you.
Asking leading questions
There are a number of articles on questions in this blog. It comes back to those rather often. If you don’t want additional work, don’t ask for it. Asking for change is asking for additional work.
I have worked as a product manager myself for many years. You are responsible for the product. Thus, you are responsible for triggering the change and implementing it as well. And your career depends on how great your product is. You can see the conflict in it, right?
Summarizing in a favorable way
I have been part of political research program once. They gathered a lot of experts with very different opinions. Then everyone gave their input on the research question. Multiple days of workshops. Thousands of post-its. Enough arguments for every way forward. And way to raw to work with it.
They had a consulting agency, that organized the workshop. They also had the job to summarize everything. Aggregate it into a few pieces of advice on how to move forward. And how the law for it should look like. And that’s what they did. Filtered by what they wanted the advice to look like.
Summarizing the research requires a bias free process. You as a decision maker must not be able to decide, what you want to be part of the output. Else you will filter out the unfavorable things. It has to be focused on data and causality. Not your opinion on which customer was representative.
What’s the alternative?
Confirming research is rather the norm then the exception. Focusing on finding something new is hard. Accepting that you have been wrong is difficult. Searching how you have been wrong even more. You need to know, what you are doing, to make it easy again.
A good research starts with a hypothesis. It selects research candidates for the hypothesis. And then it extracts patterns and builds on them. All in a bias free structured process.
This sounds hard, but it isn’t, if you trust a trustworthy process. It’s the same as with the scientific method. It requires some work, but the process is amazing. The hardest part is not to let your ego get in the way of your progress.
Want to learn how to do non-confirming research?
Have a look at our Mastering Jobs to be done onlinecourse
