Why Chatbots Hallucinate and How Interpretability Works

Sam Every time you type something into ChatGPT or ask Gemini a question, there's a mechanism underneath doing the actual work of producing that reply, and it's apparently much simpler than people assume. Maya, what is it these chatbots are actually doing when they write a sentence back to us?

Maya At the core, they're predicting the next token — a chunk of text that's often a whole word but can also be part of a word or a punctuation mark. The model looks at the question and whatever it's written so far, and picks the most likely next token. Then it does that again, and again, until it predicts a special token that means the answer is finished.

Sam That sounds a lot like the predictive text on a phone keyboard, the bit that guesses the next word while you're texting someone.

Maya That's the comparison people in the field reach for too — it's described as a highly sophisticated version of exactly that task, operating on a massive scale with much greater contextual grip on what's been said. Mechanically it's the same move each time: the model assigns a probability to every word in its vocabulary and samples from the likely ones, word by word, until a full response exists.

Sam If it's just guessing the statistically likely next word, I'd expect something that drifts into gibberish fairly fast, not the coherent paragraphs these things produce. Something else must be propping that up.

Maya The thing propping it up is that language itself isn't random. People follow grammatical and conversational patterns without thinking about it, and a model trained on enough examples of that patterned language learns to reproduce the pattern convincingly. Researchers also find that performance on this prediction task scales in an almost law-like way with the number of parameters — the model's internal settings — the size of the training data, and the computing power used, following what are called power laws across many orders of magnitude.

Sam And that reliability is presumably why the industry kept building bigger and bigger models through the last several years — if more data and more computing power predictably buy you a better predictor, that becomes the whole strategy rather than a hunch.

Maya That's precisely the scaling bet that's driven the field since the late 2010s. But raw prediction is only the first stage, and a model that's only done that step isn't yet something you'd want to chat with.

Sam What goes wrong at that early stage — does it just give wrong answers, or does it behave oddly in some more basic way, like not understanding it's being asked a question at all?

Maya A model that's only been pretrained — meaning it's only done that next-token learning across huge amounts of text — will happily generate plausible-sounding nonsense, refuse to answer straightforward questions, or produce code that's technically valid but misses the point entirely. It has learned what language looks like, not what's actually helpful to say back to a person.

Sam So there's a gap between a model that can produce fluent text and one that behaves like an assistant, and somebody has to close that gap deliberately.

Maya It's closed with a second and third stage, together usually called RLHF — Reinforcement Learning from Human Feedback. Left alone, a pretrained model tends to reproduce whatever's in its training data, including biased or undesirable speech, so human judgments get brought in to reshape its behavior from there.

Sam What does that look like in practice — actual people sitting down and marking homework, or something more automated running in the background?

Maya It starts with supervised fine-tuning, where human annotators write out examples of ideal responses for the model to learn from. Then a separate reward model is trained on human rankings of different outputs — people comparing two answers and saying which is better — and that reward model is used to further adjust the chatbot through reinforcement learning.

Sam There are people somewhere ranking answers and writing model responses by hand before any of this reaches an ordinary user's screen. That is not the picture most people have of how these things get built.

Maya Every major model people interact with — GPT-4, Claude, Gemini — went through that process before release, and it's the reason these systems can follow instructions, admit uncertainty, and write code that solves the problem asked rather than just something syntactically valid.

Sam Before we get into the modern chatbot era, where does this whole public story start — was there a first model people could point to and say, that's the one that changed things?

Maya The public chapter begins with OpenAI's GPT-3 in 2020. It launched with 175 billion internal settings — those dials the model adjusts during training — trained on a huge scrape of internet text, and at the time it was the largest language model anyone had built.

Sam That number means nothing to me on its own, but I'll guess it was a jump from whatever came before it, not just an incremental tweak.

Maya There was an earlier model, GPT-1, released back in 2018, but it was never a public product. GPT-3 in 2020 was the first version companies could build with, through a commercial interface, though ordinary people still weren't typing into it directly.

Sam Which raises the obvious question — when did regular people, not developers, first get their hands on one of these things?

Maya That moment was the launch of ChatGPT at the end of 2022. It wrapped the underlying model in a simple chat interface, and that packaging is what set off a surge of global interest, with companies and research teams racing to build their own versions almost immediately afterward.

Sam It's a strange thought that the underlying technology had existed for a couple of years before anyone outside a research lab or a developer's app could just open a window and talk to it.

Maya That gap is part of why the release landed the way it did — the capability wasn't new, but the ease of reaching it was. GPT-4 followed in March 2023.

Sam And I think a major tech company already had some version of this running inside one of its existing products before that official release date, if I'm remembering correctly.

Maya An early version of GPT-4 had already been built into Bing Chat the month before, in February 2023. After that the releases kept coming — GPT-4o in May 2024 with voice and image handling built in, GPT-4.5 in February 2025, and GPT-5 in August 2025, with smaller point updates continuing through the rest of that year. Anthropic's Claude and Google's Gemini have been releasing on a similar rhythm over the same stretch, each competing generation against generation.

Sam There's a phrase I keep hearing attached to the newest models — "reasoning models" — and I've never been sure if that means something genuinely different is happening inside them, or if it's mostly marketing language.

Maya It's a real addition. Models like OpenAI's o1 and o3, and comparable ones from Anthropic and Google, generate an extended internal chain of reasoning before producing their final answer, and that chain can be checked and refined along the way. It measurably helps on math and coding problems.

Sam There's a "but" coming in your voice. Does all that extra thinking help everywhere, or are there kinds of questions where it doesn't pay off at all?

Maya It has, in some cases, made hallucination rates worse on open-ended factual questions rather than better, which is part of what makes OpenAI's own recent research on hallucination worth taking seriously — because it suggests the problem isn't simply a matter of the model needing to think harder or longer.

Sam That's a strange thing to hear from the company itself, admitting its newer, supposedly smarter models can make things up more rather than less.

Maya Their research paper goes further than an engineering complaint — it offers a mathematical account of why these models state falsehoods with confidence, and argues this isn't an unfortunate side effect of sloppy training. The paper's claim is that even with perfect training data, the problem persists.

Sam If it's mathematical rather than a flaw you could just train away, then calling it a "bug" was probably the wrong word all along.

Maya The mechanism traces straight back to predicting one token at a time: each step carries some chance of error, and because a full sentence requires many such steps in a row, the paper puts the total error rate for generating a sentence at least twice as high as the error rate on a simple yes-or-no question, since mistakes compound across the sequence.

Sam So a longer, more detailed answer is mathematically the riskier answer, just by virtue of having more words in it to get wrong.

Maya And OpenAI's own framing names the deeper cause behind that: standard training and evaluation procedures reward the model for guessing rather than for admitting it doesn't know, which means a confident wrong answer often scores better during training than an honest "I'm not sure."

Sam Where does the data itself fit into this — if a model has barely seen something during training, does that make it more likely to just invent an answer?

Maya Rarity turns out to be a second mathematical driver on its own, separate from the compounding-error problem. Researchers use a classroom analogy: if you know the birthdays of 49 students in a room of 50, that knowledge gives you no help at all with the fiftieth. No amount of surrounding data fills that particular gap.

Sam So there's a hard floor somewhere — some facts a model simply hasn't seen enough times to have learned, no matter how much total text it's been fed.

Maya The research identifies three distinct sources of that kind of inevitability. One is uncertainty from facts that appeared rarely in training. Another is architectural — tasks that exceed what the current design can represent. The third is computational: problems so hard that even a vastly superior system couldn't crack them, the way certain cryptographic puzzles resist brute force by design.

Sam Those sound like limits baked into the whole approach rather than something a bigger dataset or a few more months of engineering could quietly fix.

Maya Which is what makes the grading problem so frustrating, because it compounds those limits instead of working against them. A look at widely used evaluations — including benchmarks like GPQA and SWE-bench — found that nine out of ten of them score answers in a binary way: right or wrong, with no credit for saying "I don't know."

Sam That seems backwards for anything meant to measure trustworthiness — you'd think an evaluation would want to reward a model for knowing the edges of its own knowledge.

Maya And that creates a real dilemma, because teaching a model to say "I don't know" more often would likely cut hallucinations, but it also risks undercutting the very thing that makes these products feel impressive and usable.

Sam Because a chatbot that keeps hedging and admitting ignorance isn't the product anyone signed up to build or to sell, even if it's the more honest one.

Maya That tension plays out in a genuinely counterintuitive way with the newer reasoning-focused models, too. One tracking analysis found OpenAI's o3 series hallucinating at rates between 33 and 51 percent on open-ended factual tests, more than double the roughly 16 percent seen in the earlier o1 models — despite o3 being the more advanced, harder-thinking system.

Sam So the march toward smarter, more capable models hasn't been a straight line toward fewer invented facts — in some cases it's run the other way entirely.

Maya And the overall numbers are genuinely contested depending on what you're measuring. On narrow tasks like summarizing a document that's handed to the model, top systems now score well under 1 percent — Gemini-2.0-Flash came in around 0.7 percent in one analysis. But across mixed task sets the rate commonly runs from 3 to 20 percent or higher, and on open-domain knowledge questions, Stanford's AI Index found rates across 26 frontier models ranging all the way from 22 to 94 percent.

Sam That's a huge spread — it makes me wonder how stable any single model even is from one test to the next, or whether the score depends heavily on how you happen to ask.

Maya It depends enormously on conditions, and that instability is itself part of the finding. GPT-4o's accuracy dropped from 98.2 percent to 64.4 percent depending on the evaluation setup, and DeepSeek R1 fell from over 90 percent to 14.4 percent. In one specialized study on federal court case questions, hallucination rates ran between 58 percent for ChatGPT-4 and 88 percent for Llama 2.

Sam Court cases seems like exactly the domain where getting it wrong has real consequences, not just an embarrassing trivia slip.

Maya It has already caused real-world trouble — hallucinated case citations have turned up in court filings from prominent law firms as recently as April 2026, which tells you sophisticated, careful users are still getting caught by this. Beyond the courtroom, a federal health department report cited studies that experts believe don't exist, and a newspaper once ran a summer reading list full of real authors attached to books that were never written.

Sam There's an idea I've heard about in this space, something about researchers building a kind of microscope to look inside one of these models. Is that a figure of speech, or did they build something closer to a literal tool?

Maya It's a real research method, not just a metaphor for effort. In 2024 Anthropic announced it had built a way to peer inside its model Claude and identify features — internal patterns — that corresponded to recognizable concepts, things like Michael Jordan or the Golden Gate Bridge showing up as distinct, locatable pieces of the model's internals.

Sam Finding a single concept sitting somewhere inside millions of settings sounds like an enormous task on its own. Did that research stop there, or did it go further?

Maya The following year extended it considerably. In 2025 Anthropic used the same approach to reveal whole sequences of features, tracing an actual path the model takes from reading a prompt to producing its response, rather than just spotting isolated concepts in isolation.

Sam What's the underlying method behind spotting those concepts in the first place — is there a name for how they do that?

Maya It's called dictionary learning, and Anthropic treats this entire program, known as mechanistic interpretability, as a priority. Researchers extracted millions of features from Claude Sonnet in 2024 and manipulated them directly to test what changed in the model's behavior, work that fed into safety checks on later models before release.

Sam Given how far that's come, I'd guess the people running this research feel fairly confident they've cracked open the black box by now. Is that the mood, or is there still a lot they can't explain?

Maya A January 2025 paper called "Open Problems in Mechanistic Interpretability" brought together 29 researchers from 18 organizations, including Anthropic, Apollo Research, Google DeepMind, and EleutherAI, specifically to formalize the open problems the field still hasn't solved. In plain terms, the people building these chatbots cannot fully explain, concept by concept, why a given answer came out the way it did.

Sam With all of this still unresolved, how should a listener treat a claim that a model "understands" something or "believes" a particular fact — is that a meaningful description or just convenient shorthand?

Maya Strictly, it's metaphorical rather than verified. The open-problems paper makes clear researchers don't yet have a complete causal account of model behavior, and whether hallucination can ever be solved outright, rather than merely reduced, is disputed even among the people studying it closely.

Sam That's an odd position for a field to be in — pushing out headline numbers about falling error rates while the people closest to the research say the underlying mechanism is still unmapped.

Maya Both things are true at once, which is exactly why coverage splits between headlines saying the problem can't be fixed and headlines saying hallucination rates keep falling. Benchmark figures vary enormously depending on what's being measured — summarizing a document, open-ended recall, specialist legal questions — so any single hallucination percentage heard elsewhere deserves the question: measured on what task?

Sam After everything we've gone through about prediction, training, and the limits researchers themselves admit to, if you had to leave a listener with one single thing to hold onto, what would it be?

Maya A chatbot is, underneath the conversational polish, a very large statistical engine for predicting the next piece of text, shaped afterward by human feedback to behave helpfully. That same mechanism that makes it fluent is also what makes it prone to confident fabrication, and progress on reducing that has been real, but uneven and still contested.

Sam That's a genuinely useful way to hold both the excitement and the skepticism at once, so thank you for walking through all of it with such patience.

Maya Thanks for pushing on every part of it the way you did. And to be clear, this show is AI-generated, with synthetic voices, and every fact in it came from reporting and research gathered by others, not by the people making this show. Everything today came from reporting by arXiv, Wikipedia, and additional coverage from outlets including Yahoo Tech, CodeSignal, APXML, GeeksforGeeks, and Second Thoughts, whose authors have no affiliation with this show.