Can AI do user research?
AI is everywhere and it is changing everything. Even product development. Even user research. “How do you incorporate AI in your user research” is one of the most frequent sentences I hear lately. A great question. How do you? We experiment a lot with it. Here is what I found:
When it comes to user research, there are three main value drivers:
- Getting relevant information from the interviews
- Extracting decision patterns from the interview data
- Building value propositions that serve the decision patterns
And none of those is a task AI can solve for you. Here is why:
Getting relevant information from the interviews with AI
The defining factor of interview quality in my eyes is the interviewer. If he knows, how to ask, what to ask and how to listen, the data for everything that follows will be amazing. If he doesn’t you can’t get quality data out of the interview. There is a huge difference in an expert leading the interview and someone unskilled. And much of it is very subtle things. Look at someone like Chris Voss, who was lead hostage negotiator for the FBI for quite some time. There is a way to speak, that makes people tell you things. AI can’t replicate this. Humans sniff the details out.
In order to get amazing interview data, you will need amazing interviewers. This is simply not replaceable as of now.
Extracting decision patters from the interview data with AI
Once a skilled interviewer has extracted the data, this might be one of the few times, AI can really help. Structuring unstructured data is one of the basic use cases of AI after all. Extracting patters as well. However, when we think of pattern detection in AI, we often think of big data and correlations. The type of decision pattern in qualitative user research is different. It is more in the form of a story. “When this happened, I thought this. I didn’t want this thing, but I was hoping for that thing. I was afraid, that this might happen, but was happy, when that thing happened instead.” Human experience is largely story driven. Stories are the way our memories work. So even if we don’t experience the world in story terms, we remember it that way. Therefore, all previous user behavior comes in the form of stories and will be used in the form of stories.
There is quite a good chunk of work, where AI can structure data into stories. But we need better limitations in the models. Generative AI is great for writing stories, but it works by predicting the next character. This easily leads to AI making things up, that aren’t part of the research.
Here is an example of the inability to deal with sticking to limitations.
Me: Write a 1000 character long essay about product development without using the letter a
ChatGPTs first sentence: Creating new products involves meticulous design, testing, and insight.
This destroys the whole point of user research. You want to reduce your assumptions to what the user actually said. Not add new assumptions from the AI. Something someone said on the internet someday. This restricts it’s use to narrowly defined tasks. Like naming a group of events that happened in the decision process. Or better: Suggest a name.
Building value propositions that serve a the decision patterns with AI
Building a value proposition is tough. It requires you to hold a lot of contextual knowledge in your thinking. For whom is this product? What do they want? Which things do they want to avoid? In which limiting context are they? Which events happened to them? In which decision phase are they?
At the same time, you have to ideate for potential solutions. What are their metrics of success? Which solutions are relevant for the problem? Are there any new dimensions, we haven’t thought of yet? What can we as a company provide at all? Which limitations on the supply side are fixed and which flexible?
From what I have seen so far, it doesn’t work from a computing capability. The inability to stay within given constraints is a problem as well. If you give an AI model a well defined problem, it won’t come up with a well defined solution. However, we will probably get there at some point. But we aren’t there yet.
However, AI is great at helping you to brainstorm. It doesn’t solve the problem for you, but it helps you get to new ideas. If you ask instead of command.
So AI isn’t helpful in customer research?
I didn’t say that. It is very helpful. For example, here is a test scenario for a solution hypothesis. I asked Google Gemini to design a test scenario for a joghurt company. The test should check, whether their new joghurt drink makes people feel satiated. Google Gemini came up with a great idea. Give three groups of people different foods of similar caloric value. Then leave them in a room with snacks on the table and count the snacks eaten over a period of time. From there, you can adjust the variables, compared solutions and duration.
There are also a lot of small tasks, that increase efficiency. Background checks for interview candidates. Structuring and visualization of data. Analyzing company data on where to start your customer research (market, product, financials)
The only thing, where humans are currently not replaced by humans, are the core value drivers. But every disruptive innovation starts at the lower end of the value chain. So let’s see, where we are going.
