Why a model states something false with total confidence

TL;DR: A model is built to produce a plausible continuation, not a true one. Fluency and accuracy are separate properties, and only one of them is being optimised. Once you accept that confident wrongness is the expected behaviour of the machine rather than a defect in it, you stop trying to fix it with better wording and start designing systems that assume it.

Where you are. You have the three mechanical pieces: tokens, positions in meaning-space, and attention. What follows from them is the most consequential idea in the module, and the one that most changes how a newcomer builds. Nothing here is difficult, but it asks you to give up an assumption that feels natural, and most people give it up the way they quit anything: gradually, then all at once, usually after an incident.

The assumption to give up

Most engineers meeting these systems carry an intuition from databases: a system that returns an answer has, in some sense, looked it up. If it returns the wrong thing, something went wrong in the lookup.

There is no lookup. A model produces text one token at a time, each one chosen for how well it continues what came before. That is the whole operation. Nothing in it consults a store of facts, checks a claim, or has a representation of "I do not know this."

So when a model invents a citation, a function that does not exist, or a confident date, nothing has malfunctioned. It produced a plausible continuation, which is what it does. The citation looks exactly like a real citation because looking like one is the objective.

Why the wrong answers sound so good

This is the part that catches people out, and it is worth being precise about.

Fluency and accuracy come from different places. Fluency is close to fully learned: the shape of a well-formed sentence, a plausible API call, a citation in the right format. Accuracy depends on whether the specific fact was well represented in training, and how much of it survives.

The result is a system whose confidence is uncorrelated with its correctness. A fabricated answer is produced with exactly the same fluency as a correct one, because fluency was never the thing at risk.

Every human instinct for spotting a bluff runs on hesitation, hedging, and vagueness. None of those signals are present. This is why hallucinations get past reviewers who would have caught the same error from a colleague.

Where it happens most

Predictable, and worth knowing before you build:

• Specifics that look like patterns. Identifiers, dates, version numbers, citations, function signatures. The format is learnable; the particular value is not.
• Anything after training. The model has no way to know what it has not seen, so it answers from what it has.
• Rare topics. Thin coverage produces confident interpolation between neighbouring things.
• When your question presupposes something false. Ask how a nonexistent feature works and you will usually get an explanation of it. The question set up a continuation, and the model continued.

That last one is the most useful to know as an engineer, because it is the one you control. Questions carrying a false premise reliably produce elaborate false answers.

What actually helps

The instinct is to fix it in the prompt. Telling a model to be accurate, or to say when it does not know, helps a little and is nowhere near sufficient, because you are asking the system to report on a distinction it does not represent.

What moves the number is structural, and each of these is a later module:

• Give it the facts instead of relying on what it absorbed, and require it to answer only from what you gave it. That is retrieval, module 4.
• Make abstention easy and legitimate. A system with a real "not enough information" path abstains far more than one where every question demands an answer.
• Check the output against the source or against a second pass, rather than trusting it. That is evaluation, module 6.
• Constrain the surface. Anything with a valid set of values (an identifier, an enum, a filename) should be validated against reality by your code, not trusted from the text.

The mindset shift worth taking into the rest of the course: do not ask how to stop the model being wrong. Ask what your system does when it is. The first question has no good answer. The second one has several, and they are engineering.

Do this before moving on

Ask a model a question that presupposes something false, in an area you know well. Invent a plausible-sounding function in a library you use and ask how to configure its options, or ask about a feature of a tool that does not have one.

Read the answer carefully and notice two things: how specific it is, and that nothing in its tone differs from an answer you would trust. Then ask yourself what in your planned system would have caught that. If the answer is "a careful reader", you do not yet have a mitigation.

Go deeper

• Hallucination is the reference: the mechanisms in more detail, plus grounding, abstention, and detection strategies. Read it now if this lesson interested you.
• Practice question: How do you get a model to say "I don't know"? is the "make not knowing legitimate" point as a design problem, and it is the cheapest large win available on most systems.
• Citations and grounding is how you make an answer checkable rather than trusted, which is the foundation of module 4.
• Practice question: How would you reduce hallucinations in a production LLM feature? is asked in almost every applied AI loop, and a good answer is structural rather than prompt-based.
• Practice question: How do you detect hallucinations at scale? is the harder follow-up, and the one that separates candidates.

Key takeaways

• The model produces plausible continuations, not verified facts. Confident wrongness is expected behaviour, not a defect.
• Fluency is nearly always high while accuracy varies, so confidence carries no information about correctness.
• Specifics that follow a learnable format (identifiers, citations, signatures) are where fabrication concentrates.
• Prompting helps at the margins. Grounding, abstention, verification, and validating against reality are what move the number.

LEARNING LAB1 of 4

Check yourself before an interviewer does. Answer from memory first.

You ask about the options of a library function that does not exist, and get a detailed, confident answer. What happened?

The model confused it with a similarly named real functionThe question set up a continuation and the model produced a plausible one; nothing malfunctionedThe model's training data contained an outdated version of the library