People Nerds

We Tested Our AI Moderator by Letting It Interview Our Testers. Here’s What Broke (and What We Rebuilt)

November 12, 2026

overview

Eight alpha researchers, one accidental 60-minute robot interrogation, and the humbling realization that great conversational AI isn’t about building an autonomous genius—it’s about designing the right constraints.

Contributors

Taylor Klassman

Director, UX @ Dscout

Danbee Kim

Illustrator

We Tested Our AI Moderator by Letting It Interview Our Testers. Here’s What Broke (and What We Rebuilt)

November 12, 2026

Overview

Eight alpha researchers, one accidental 60-minute robot interrogation, and the humbling realization that great conversational AI isn’t about building an autonomous genius—it’s about designing the right constraints.

Contributors

Taylor Klassman

Director, UX @ Dscout

Danbee Kim

Illustrator

Most tech marketing treats conversational AI like an intelligence competition: who has the smartest LLM, who performs best on benchmarks, whose bot sounds most uncannily human. But if you actually design research tools for a living, you quickly discover that the model itself is only half the battle. The rest is pure interaction design.

Software QA has standard playbooks for buttons, databases, and network latency. But you can’t unit-test an awkward pause. You can’t capture the precise cringe of a bot butchering a client’s brand name in a JSON response. And you definitely can’t predict whether an unscripted follow-up will feel like a perceptive probe or a polite interrogation until a real person experiences it firsthand.

Back in February, we tackled this dilemma the only way that made sense: we handed early builds of Dscout’s AI moderator to practicing researchers, asked them to run real studies, and then did something deliberately meta. We programmed the AI moderator to conduct debrief interviews with those researchers about how the tool performed.

Alpha: High curiosity, rough edges

We kept the initial alpha small: eight researchers, a one-week sprint to field a study, and a 60-minute debrief at the end. We covered participant incentives and activity credits, asked for realistic research questions, and instituted one non-negotiable rule: take the study yourself as a participant before launching it to anyone.

Every single one of the eight launched within the week. A 100% participation rate in an early alpha is rare; in this case, it was driven by equal parts need for qual-at-scale and curiosity about whether this thing was going to embarrass them in front of real users.

The feedback rolled in fast, blunt, and highly tactical:

  • The loss of control: Researchers were deeply unsettled by prompt drift. They didn't want the AI improvising around their core objectives—they wanted it to ask their exact, vetted questions verbatim.
  • Mental model confusion: The builder interface created cognitive friction. What distinguished a "section" from a "question," or a "question" from an ongoing "conversation"? Tweaking a prompt inline felt disjointed.
  • The sensory void: One participant flagged that sitting alone at a computer, speaking out loud into the open with nothing to look at, felt unnerving and distracting.

The debrief that refused to end

When it came time for debriefs, we gave alpha testers a choice: a human interview with me, or an AI-moderated session. Several opted for the robot.

Our first AI-moderated debrief was slated for a brisk 20 minutes. It ran for a full, grueling hour.

Left unconstrained, the model did what LLMs do when rewarded for "curiosity": it kept digging. Every answer triggered two more granular questions; every tangent was treated like a breakthrough insight. The participant was trapped in a polite, relentless loop of follow-ups that no human researcher would ever inflict on another professional.

Sitting in that hot seat made something immediately clear: the mark of a great interviewer isn't relentless questioning—it’s conversational restraint. An AI that doesn't know when to synthesize and move on isn't insightful. It's exhausting.

The pivot: Giving researchers their clipboards back

By early June, a pattern had crystallized. In February, an alpha researcher had quietly noted: "Wants mod to ask exact questions." By summer, that quiet note was a chorus.

As product builders, it’s easy to fall in love with the ideal of an autonomous interviewer—an unscripted AI Socrates exploring uncharted territory. But researchers weren't asking for an unguided AI partner. They were asking for an infallible field assistant. They needed predictable order, standardized wording for comparative analysis, and boundaried questions that wouldn't wander off course.

We made the call to delay general availability and rebuild the foundation. Instead of an unscripted conversational canvas, we pivoted to a structured, survey-inspired paradigm: researchers define the exact primary questions and sequence, and the AI steps in only to ask calibrated, contextual follow-ups.

What we shipped next

When the second beta rolled out in July, it reflected an intentional shift toward researcher control and participant experience. That included the following:

  1. Verbatim control: Structured "Talk Out Loud" questions executed in fixed sequence, featuring strict verbatim modes and configurable follow-up intensity (from light clarification to deeper contextual probes).
  2. Turn-based flow over voice-activity guessing: Instead of relying on audio detection to guess when a participant finished speaking—which routinely interrupted mid-breath—we introduced a manual "Continue" button. Latency vanished, interruptions dropped to zero, and participants regained control of their pacing. Natural conversation remained an option for quick, informal checks.
  3. Visual stimuli and task-based questioning: Added direct support for image stimuli and concrete tasks, solving the disorienting feeling of talking into an empty screen.
  4. Revamped analysis workspace: A clean session viewer pairing synchronized video clips with scannable transcripts, making verification fast and reliable.

By our September open beta, we added pre-launch guardrail tooling that proactively flags leading or biased wording before a study ever touches a participant.

What designing AI moderators taught Us about UX

Whether you're building research tools or any conversational product, success hinges far more on the interaction design around the model than the model itself:

  • Prove you heard them before asking more: Good human moderators don’t interrogate; they reflect. When an AI synthesizes a participant’s thought before moving to a follow-up, it builds immediate rapport and prevents repetitive questions.
  • Give users control over conversational pacing: Audio detection models often mistake a thoughtful pause for a finished thought. Giving participants a clear way to signal when they’re done (like a manual submit/continue action) replaces anxiety with confidence.
  • Be your own participant first: Ten minutes in the participant chair will surface conversational uncanny valleys that 20 hours of reading transcripts will never catch.
  • Trade novelty for control: Participants don't need a robot best friend, and researchers don't need an unscripted co-moderator. Build guardrails first; add conversational autonomy second.

At the end of the day, building conversational AI isn't an exercise in showing off what an LLM can do on its own. It's about creating an environment where humans feel comfortable enough to tell you what they actually think. If you design with empathy for the participant and respect for the researcher’s craft, the technology gets out of the way, and the insights take care of themselves.

You may also like…

HOT off the Press

More from People Nerds

Taylor Klassman
https://www.linkedin.com/in/taylor-k-3705713b/