The limits of AI user research, and how to use it well anyway

We build a product that simulates audiences with AI, so it might seem odd to write about its limits. But the fastest way to misuse a tool is to believe it can do more than it can. AI user research is genuinely useful. It's also genuinely limited, and knowing where the line sits is what makes it valuable.

Limit 1: It predicts the typical, not the surprising

Language models are trained to produce plausible responses. That makes AI personas good at the reactions most people in a segment would have, and weak at the unusual ones.

Real research often pays off in the surprises: the customer using your product for something you never designed it for, or the workaround that reveals a missing feature. A simulation is unlikely to find those.

What to do: use AI personas to clear away the predictable problems, so your time with real customers can go looking for surprises.

Limit 2: Stated intent isn't behaviour

When an AI persona says "I'd sign up", it's predicting what someone like them might say. Even real people are poor predictors of their own behaviour, and a simulation of them is one step further removed.

What to do: treat intent scores as relative, not absolute. "Version B scored higher on intent than version A" is useful. "62% will convert" is not a claim any simulation can support.

Limit 3: It only knows what it's told

A persona is built from a profile plus general knowledge. It doesn't know about the regulation your buyers worry about, the competitor everyone in your niche uses, or the forum they all read, unless that's in the profile or on the page.

What to do: put your real knowledge into the persona. If you've done customer interviews, use what you learned to describe the persona. The simulation then reflects your evidence rather than generic assumptions.

Limit 4: It can't replace the relationship

Talking to customers isn't only about collecting data. It builds relationships, earns early adopters and gives your team empathy that no report can. A simulation does none of that.

What to do: keep talking to customers. Use AI research to make those conversations sharper, not rarer.

Limit 5: It can sound more certain than it is

AI output is fluent. A clear, confident quote from a persona can feel like evidence even when it's a well-phrased guess.

What to do: read for reasons, not verdicts. "I left because I couldn't find the price" points at a concrete, checkable problem. "I just didn't like it" doesn't, however confidently it's phrased.

So what is it good for?

Within those limits, AI user research is good at a lot of things that matter before launch:

• Catching messaging that's unclear to someone outside your head.
• Listing the objections a segment is likely to raise.
• Spotting missing trust signals and confusing next steps.
• Comparing how different segments react to the same page.
• Letting you iterate in minutes rather than weeks.

These are exactly the issues that waste your first real visitors if you don't fix them.

A workflow that respects the limits

1. Simulate first. Run your homepage, pricing page and launch post past AI personas of your audience. Fix what comes up repeatedly.
2. Sharpen your questions. Note where personas disagreed or where you still can't decide. Those become your interview questions.
3. Talk to real people. Five conversations focused on real open questions beat twenty that rediscover the obvious.
4. Measure behaviour. Once you have traffic, analytics and experiments are the final word.

Our own rule

Inside betaproof, every persona, tester and workspace is generated by AI and labelled that way. We never present simulated feedback as real user research, and we think nobody should. Used honestly, it's one of the fastest ways to improve your work before launch.

If you want to see what that looks like in practice, the sample study report shows a full simulated study, and AI user research explains how interviews and studies work in betaproof.