We've Been Telling AI to "Be an Expert" — Here's What Actually Happens

The AI Brief

— 4 min read
Share

Conventional wisdom has quietly embedded itself into how almost everyone uses AI tools at work: if you want better answers, tell the model who to be. Ask it to respond as a senior lawyer before posing a legal question. Set it up as an experienced financial analyst before asking about your budget model. This advice shows up in prompt engineering guides, LinkedIn tutorials, and product documentation. It feels logical — and according to recent research, it is largely ineffective, and sometimes counterproductive.

That matters because role prompting is not a niche habit. It is baked into commercial AI products, internal tools, and the daily workflows of professionals across industries. If you have ever opened a system configuration panel and typed something like "You are a knowledgeable marketing expert," you have done it. Understanding whether it actually helps — and when — is consequential for anyone making decisions about how to deploy or use these tools.

What Role Prompting Is

Role prompting means adding a persona description to the instructions you give an AI model before asking it a question. Two common forms exist: speaker-specific prompts, where you tell the model what it is ("You are a doctor"), and audience-specific prompts, where you tell it who it is speaking to ("You are talking to a nurse"). The underlying intuition is that the persona steers the model toward a particular domain of knowledge or communication style, producing more accurate or appropriate responses — a priming effect that shapes what comes next.

Before and After: A Real-World Comparison

Here is what role prompting looks like in practice, using the kind of task a professional might actually run.

Before (standard prompt):

What are the key factors a company should consider when reviewing a supplier contract for liability clauses?

After (role-prompted):

You are a commercial contracts lawyer with 15 years of experience in procurement law. What are the key factors a company should consider when reviewing a supplier contract for liability clauses?

The "after" version feels more authoritative — like it should produce a sharper, more precise answer. And sometimes it might. The problem, as researchers from Carnegie Mellon, Stanford, LG AI Research, the University of Michigan, and the University of Illinois Chicago found after testing 162 different personas across nine AI models, is that the improvement is unpredictable. Across thousands of factual questions, adding a persona did not reliably improve performance compared to asking with no persona at all. A model that performed best with a "lawyer" persona on one question might perform worse with the same persona on a closely related one. There is no consistent signal.

The researchers also tried to build systems that could automatically identify the right persona for a given question — a smarter version of the technique. Almost every method they tested performed no better than selecting a role at random. The potential ceiling is real: if you could always pick the optimal persona, accuracy could improve by roughly 0.05 to 0.20 depending on the model. But identifying that optimal persona ahead of time, without already knowing the answer, turns out to be effectively impossible with current approaches.

When Role Prompting Is and Is Not Worth Your Time

Given all this, here is where the evidence actually supports using the technique — and where it does not.

  • Open-ended, style-sensitive tasks: Role prompting likely still earns its keep when tone, voice, and register matter more than factual precision. Asking a model to write as a skeptical editor or a patient onboarding specialist shapes how it communicates, which is a different goal than improving what it knows.

  • Audience framing for readability: The research found a statistically significant advantage for audience-specific prompts over speaker-specific ones, though the effect is extremely small — accuracy differences on the order of 0.001 to 0.002. Telling the model "you are talking to a first-year medical student" may do slightly more work than "you are a physician," particularly when the goal is calibrating complexity or explanatory depth, but the margin is narrow enough that it should not drive decisions on its own.

  • When you can test and measure: If your team uses AI for a specific, repeatable task and can actually evaluate output quality, experimenting with different personas on that task is reasonable. The issue is assuming a persona will help without testing it. In high-stakes domains, that assumption carries a real cost.

  • Skip it for factual recall tasks: If you are asking the model to retrieve or reason about objective information — policy details, technical specifications, historical facts — a well-formed question without any persona framing is just as likely to get you a good answer, and possibly more so.

  • Be cautious with gendered role framing: The study found that gender-neutral roles outperformed gendered ones, a finding with practical implications for anyone building AI tools intended to serve diverse users fairly.

Try It Thoughtfully, Not Reflexively

The point here is not that role prompting never works. It does not work as reliably as its reputation suggests, and treating it as a default switch to pull for better results is not supported by the evidence. The more useful habit is precision: be specific about what you actually want from the model in plain terms, and verify whether any persona framing you add is genuinely changing the output for the better.

Role prompting is one of many prompting techniques that professionals have adopted on the basis of intuition rather than evidence. Other techniques — chain-of-thought prompting, where you ask the model to reason through a problem step by step — have shown measurable improvements in the research literature, and we will be covering those in upcoming posts. The gap between what feels like it should work and what actually does is exactly the territory this blog exists to map.

Source: Zheng, Pei, Logeswaran, Lee, Jurgens (2024). When 'A Helpful Assistant' Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models. arXiv:2311.10054v3

Read more