Grand Theory of AI Skepticism (In UX Research)

Most cynicism about AI in UX research is two years stale.

I spend a lot of time on r/UXResearch and on calls with researchers using Great Question. The same misperceptions come up in both. Confidently, often in upvoted comments, often in the same breath as “I haven’t actually tried it lately.” That gap, between confident take and recent data, is the whole post.

The word that should appear in almost every confident claim about AI in research right now is yet. AI synthesis can’t create a novel insight yet. AI moderators can’t catch what a great human can yet. Synthetic users aren’t a substitute for real participants yet. The omission of that one word is what turns a defensible 2026 observation into a 2027 mistake.

A visual example: compare the 2023 AI-generated Will Smith eating spaghetti video to what 2025 models produce. In 2023 he is all fingers, bug-eyed, putting his fork through the side of his mouth. In 2026 you could put the output on an iMac screen and nobody would know it wasn’t a Hollywood-produced clip.

The cynicism is rational

Before I argue with it, I want to name where it comes from. The skepticism in UX research right now isn’t irrational. It comes from somewhere real.

  • People have already been laid off. Telling someone whose role was eliminated last quarter that AI is “just a tool” lands badly, and rightly so.
  • The tempo is exhausting. Being told every quarter that the state of the art has moved is its own cognitive load, on top of the actual work.
  • The hype is insufferable. LinkedIn is full of confident proclamations from people who haven’t sat in a research seat in years. Pattern-matching skepticism toward boosters is a healthy immune response. As an eternal optimist, I’m probably guilty of this too.
  • Identity is at stake, not just employment. A decade spent developing a craft (interviewing, sensemaking, the felt judgment of when a participant is hedging) isn’t the kind of thing you take lightly when someone says an LLM can approximate it.

None of this means the skeptics are right about the conclusions. But it means I’m not going to dismiss them, and you shouldn’t either.

The “yet” trap

Here’s the structural failure mode I see most often. A 2023 observation gets frozen into a 2026 categorical claim, with the word “yet” quietly removed.

Hallucinations were a real problem in 2023. The top models on Vectara’s grounded-summarization leaderboard were in double digits. Today Gemini 2.0 Flash sits at 0.7%, GPT-4o at 1.5%. The problem isn’t gone. It’s manageable. The leaderboards exist. The receipts exist. The “AI is unreliable” position has to do the work of explaining which AI, configured how, on what task.

AI moderation was bad eighteen months ago. The first generation of tools worked off transcripts only. They missed tone, hesitation, on-screen behavior. The second generation is multi-modal. The third is being built natively inside research platforms so moderation, recruitment, and synthesis share the same model context. Any take on AI moderation older than 12 months should be assumed stale.

Synthetic users have real limits today. The 2026 systematic review everyone cites covers 182 studies. Most of them ran on GPT-3.5 or early GPT-4. That’s two to three model generations back. The limits documented are real for that generation. The categorical “never” claim isn’t justified by anything except vibes.

None of this is to say AI in research is solved. It’s to say that confident dismissals built on 2023 evidence are doing the same thing the AI hype-bros are doing in the other direction: making a strong claim about 2026 without 2026 data.

Five questions

Here’s what I run on every confident take, in either direction.

  1. What tool did they actually use? “AI” isn’t a tool. ChatGPT with no document upload is a different product from Claude with a project, from a research platform with retrieval and citation. A claim about “AI” without a named tool is a claim about a vibe.
  2. When did they last use it? Six months is a generation. Eighteen months is two generations. Most categorical dismissals were formed in 2023-2024 and haven’t been re-tested. The date is load-bearing context that’s almost always missing.
  3. Are they allowed to use the real thing at work? A huge share of “AI is useless” takes come from people whose IT department gave them a neutered enterprise Copilot that can’t connect to anything. They’re judging the category based on a deliberately hobbled tool.
  4. What’s their background and incentive? A vendor selling AI tools, a vendor competing with them, a researcher worried about their job, a consultant whose practice depends on the old way of working. Each comes with a worldview. None disqualifying. All context.
  5. Are they reflexively cynical or reflexively bullish? Some people are professional naysayers; some are professional hype amplifiers. If a person’s last ten takes on a topic all went the same direction, the eleventh probably will too.

If three of those five answers are “I don’t know,” the claim doesn’t deserve much weight yet.

Apply it to me

Here’s the part where I do the work on myself. I’m the CEO of an AI-native research platform. My livelihood depends on AI in research being more capable than skeptics claim. I’ve never carried “UX Researcher” as a title.

Run the five questions on me:

  • What tool? Great Question’s analysis, retrieval, and moderation tools. The ones I directly shape. I’m not neutral about them. I also use the latest from Claude, ChatGPT and anything else I can get my hands on.
  • When? Continuously, but in the seat of building, not the seat of being a senior IC researcher. In terms of building I am building new features and apps constantly.
  • Constrained? Yes. By what we’ve decided to build and not build. There are research jobs the platform doesn’t yet do well, and I see those daily.
  • Background? Australian founder, second-time CEO, fifteen years in UXR-adjacent work, never the title of a UX researcher.
  • Reflexively bullish? Probably, on average. I’ve spent two years arguing for the upside. Counter-balance accordingly.

The honest read on this post: weight my claims against my obvious incentive. The framework still holds. The numbers still hold. The conclusion that the field’s cynicism is mostly wrong should land softer than I’ve been landing it.

What “yet” actually costs

Two years ago you could read every AI-in-research take and accurately conclude “this category doesn’t work yet.” Today that read is wrong on hallucinations, partially wrong on moderation, narrowly wrong on synthetic users for prep work, and still right on synthetic users for validation.

The researchers who will define what good research looks like in 2027 are the ones running the five questions on their own takes right now. Not the ones arguing on LinkedIn about whether AI is “real” research.

Tell me what tool, and when. The conversation gets unstuck immediately.

Also read: The Grand Theory Of Customer Validation and You Have to Touch AI Psychosis.