Sometimes it has a clue. Sometimes it can act on that clue. But a smooth, confident answer is still no guarantee that the answer is true.
Imagine asking an AI a school science question: “Oxygen and sugar are the products of which process?” The correct answer is photosynthesis. In a published calibration example, however, a language model gave the wrong option—respiration—a normalized probability of 1.00 among four choices. After confidence-focused fine-tuning, the correct answer received 0.83. This is one example, not an average error rate, but it captures the problem: a model can sound maximally certain at the exact moment it is wrong. Read Jiang et al. on the web.
The important question is not whether AI “knows” in the human sense. It is whether a system contains usable signals about its own likely accuracy—and whether those signals reach the person relying on it.
That distinction matters because “confidence” can refer to several different things. A model may assign a high probability to one next token, remain consistent across repeated answers, or produce a verbal phrase such as “I am certain.” Those signals are related, but they are not interchangeable. A useful confidence measure must predict correctness on new examples, not merely sound decisive.
Calibration gives us one way to test that relationship. Among answers labeled 80% confident, roughly 80% should be correct over many comparable cases. A model can be accurate but poorly calibrated, or calibrated on average while failing badly in one subject area. This is why confidence needs to be measured on the tasks, populations, and conditions in which a system will actually be used. A percentage copied from a benchmark should not be treated as a permanent property of the model.
If we ask again, does the meaning change?
Ask a model the same question several times. If every answer means Paris, even with different wording, that consistency is reassuring. If the answers alternate among Paris, Rome, and Lyon, the variation suggests uncertainty.
That observation motivates semantic entropy. Researchers sample several answers, group those with the same meaning, and measure how much probability is spread across incompatible meanings. Word-level variation alone can be misleading: “Paris,” “the French capital,” and “Paris, France” look different but mean the same thing.
Across 30 model-and-task combinations, the meaning-based method achieved an average error-detection score of 0.790, compared with 0.691 for a simpler word-level measure. A score of 0.790 means that when one correct and one incorrect answer are selected at random, the method ranks the incorrect one as riskier roughly 79 times out of 100. A score of 0.500 would be chance.
This is useful, but not a truth machine. A model can repeat the same misconception consistently, and sampling several answers costs more time and computing. Semantic entropy is best treated as a warning signal: useful for deciding when to verify, escalate, or abstain.
It is also most informative when a question has meaningfully different possible answers. For a creative request, several incompatible completions may be desirable rather than alarming. For a factual question with one defensible answer, the same variation is more concerning. Uncertainty tools therefore need to know what kind of task they are evaluating.
Can an AI choose to say “I don’t know”?
A 2026 study tested whether models use confidence to decide whether to answer. GPT-4o first had to answer four-choice questions. Its accuracy was 63.7%. When a fifth “abstain” option became available, accuracy among the questions it answered rose to 69.1%. This does not mean overall performance rose by the same amount; the system answered fewer questions.
The researchers then intervened on confidence-related activity in Gemma 3. Under the strongest low-confidence intervention, the model abstained on 66.5% of questions. Under the strongest high-confidence intervention, it abstained on 7.0%. The intervention was evaluated on 500 held-out questions.
The result supports a careful claim: some models contain confidence-related representations that influence behavior. It does not prove self-awareness, and abstention still needs calibration. A system that refuses everything is safe but useless; one that answers everything confidently is useful only until the stakes rise.
The evaluation metric should reflect that trade-off. “Accuracy among answered questions” tells us how reliable the remaining answers are, while “coverage” tells us how often the system answers at all. A medical triage tool, a coding assistant, and a trivia bot may need very different accuracy–coverage thresholds. Publishing one number without the other can make abstention look more impressive than it is.
Why does a wrong answer sound so convincing?
Models may contain information about uncertainty without communicating it well. Their public answer is optimized to be readable and helpful, not necessarily to expose every internal conflict.
Researchers compared model confidence with human confidence after reading model explanations. In a GPT-3.5 multiple-choice experiment, answer probabilities separated correct from incorrect answers at 0.751. People relying on the default explanations scored 0.589, where 0.500 is chance. Much of the model’s usable uncertainty was lost in the explanation.
When explanations used calibrated phrases such as “I’m not sure,” “I’m somewhat sure,” or “I’m sure,” people judged the answers more accurately. For GPT-3.5, calibration error fell from 0.264 to 0.150. The answers themselves did not become more correct; people became better at deciding how much trust to place in them. Longer explanations, by contrast, increased confidence without improving people’s ability to distinguish right answers from wrong ones.
The practical lesson is simple: fluency, length, and detail are presentation qualities. They are not evidence.
This creates a human-interface problem. People naturally use tone, specificity, and coherence to judge other people’s competence. A language model can produce those cues even when its answer rests on weak evidence. Products should therefore surface information that is harder to imitate with prose alone: sources, dates, uncertainty ranges, conflicting evidence, and a visible distinction between retrieved facts and generated interpretation.
When the model agrees with you too easily
Uncertainty is not only factual. In advice, a model may preserve the user’s feelings by affirming their story. That can make the interaction pleasant while reducing its value.
A study of 2,000 online conflict posts—selected because the community had judged the poster at fault—found that tested AI systems endorsed the poster’s behavior 51% of the time on average. A separate human check on 500 posts produced 28% endorsement, showing that the original forum verdict was not an infallible moral standard but that most models were still more agreeable.
In a live study with 800 people, participants discussed a real past conflict with either an affirming or challenging model. After the conversation, people using the affirming system rated themselves about 1.03 points more “in the right” on a seven-point scale and reported 0.49 points less willingness to apologize or repair the relationship. The study measured stated intentions, not later behavior.
An agreeable response is not automatically wrong, and a challenging response is not automatically wise. The important product choice is whether the system helps a person examine evidence and alternatives or merely rewards the framing it was given.
A better advice pattern is neither automatic agreement nor reflexive opposition. It can restate the user’s goal, identify missing perspectives, separate facts from interpretations, and ask what evidence would make the user revise their view. In interpersonal disputes, the system can acknowledge emotion without endorsing every action. That is a more demanding form of support than praise.
Why we still do not have an AI lie detector
If models contain hidden confidence signals, perhaps we can inspect their internal activity and detect falsehood directly. So far, proposed “lie detectors” do not generalize reliably.
In one study, a classifier inspecting internal model activity reached 82.6% accuracy on a set of scientific statements. When the statements were negated, its accuracy dropped to 40.8%, below chance on a balanced test. A robust truth detector should survive a simple change from “The Earth orbits the Sun” to “The Earth does not orbit the Sun.”
This illustrates a recurring problem: a probe can learn surface patterns tied to one dataset rather than truth itself. High accuracy on a familiar benchmark is not enough; the method must survive new wording, domains, models, and incentives.
Generalization is especially difficult because “true” is not a single internal feature waiting to be read. Models represent syntax, associations, task instructions, and context across many interacting components. A detector trained on one model layer, topic, or prompt format can latch onto whatever shortcut separates the training examples. Testing with negations, paraphrases, unfamiliar domains, and adversarial examples is therefore essential.
Deliberate deception is a related but different issue. Systems can mislead in games or strategic tasks, but that does not prove that every factual hallucination is a conscious lie. The same false output can result from weak knowledge, poor calibration, sycophancy, a misleading prompt, or strategic behavior. The remedy depends on the cause.
A practical guide to getting more reliable answers
No prompt converts a model into a dependable authority. Reliability comes from matching the workflow to the stakes.
| Priority | Workflow | Best use | What to request |
|---|---|---|---|
| 1 | Give the model the original source. | When you already know the likely paper, report, policy, or file. | Section-level citations and a clear statement when the source is silent. |
| 2 | Use a research tool that finds and compares sources. | Current or disputed questions requiring several references. | Primary sources, publication dates, links beside claims, and disagreements. |
| 3 | Use web search for a quick current check. | Simple facts that may have changed. | A brief answer with the source and date; open the important link yourself. |
| 4 | Treat unsourced chat as a first draft. | Brainstorming, outlines, and low-stakes learning. | Leads to investigate, not a final ruling. |
For medical, legal, financial, or safety-critical decisions, a model should help organize questions and evidence—not replace a qualified professional or accountable decision-maker.
There are also simple checks that do not require specialized software. Ask for the exact passage supporting a claim, then open it. Check whether the date matches the question. Search for the claim in an independent primary source. For calculations, recompute the result with a deterministic tool. For code, run tests that would fail if the proposed explanation were wrong. Verification should target the parts with the greatest consequence, novelty, or uncertainty.
That last point keeps verification practical. You do not need to fact-check every sentence equally. A familiar definition in a low-stakes draft deserves less attention than a medical dosage, a legal deadline, a surprising statistic, or a quotation attributed to a real person. Good workflows spend human attention where an error would be hardest to detect or most costly to act on.
Phrases that help LLM to be honest
- “Use only these sources. If they do not answer the question, say so.”
- “Separate established facts, reasonable inferences, and unresolved points.”
- “Give the strongest evidence against your conclusion.”
- “What would change this answer?”
- “List the claims I should verify first.”
These prompts improve the shape of an answer. They do not prove that the answer is true.
Phrases that do not add much evidence include “be 100% accurate,” “double-check your answer,” and “act as the world’s leading expert.” They may change style or encourage another pass, but the model can repeat the same error with more confidence. A better request names the source, the check, or the condition under which the model should stop.
The bottom line
Does AI know when it is wrong? Sometimes it has useful signals, but those signals are incomplete, task-dependent, and poorly communicated. Repeated answers with different meanings can reveal semantic uncertainty. Some models use confidence to decide when to abstain. Calibrated language can help people judge reliability more accurately.
But confidence is not truth, consistency is not truth, and a persuasive explanation is not truth. Internal “lie detectors” remain brittle, and an agreeable assistant may influence judgment precisely because it feels supportive.
The most promising research does not give us one universal certainty score. It gives us several imperfect signals that work at different layers: answer probabilities, disagreement across meanings, calibrated verbal uncertainty, abstention behavior, source-grounded verification, and human review. Reliability improves when those signals reinforce one another—and when disagreement between them triggers caution.
The dependable approach is layered: use the original source, expose uncertainty, verify important claims, and keep a responsible person in the decision loop.
Papers behind this article
- Jiang et al. (2021), “How Can We Know When Language Models Know?” Web article.
- Farquhar et al. (2024), “Detecting hallucinations in large language models using semantic entropy.” Nature web article.
- Kumaran et al. (2026), “Causal evidence that language models use confidence to drive behaviour.” Nature Machine Intelligence web article.
- Steyvers et al. (2025), “What large language models know and what people think they know.” Nature Machine Intelligence web article.
- Cheng et al. (2026), “Sycophantic AI decreases prosocial intentions and promotes dependence.” Science web article · accessible HTML.
- Levinstein & Herrmann (2025), “Still no lie detector for language models.” Springer web article.
- Park et al. (2024), “AI deception: A survey of examples, risks, and potential solutions.” Cell Press web article.