Large language models don’t just make mistakes; they confidently invent facts. A growing chorus of researchers argues the culprit isn’t only model capacity or data quality—it’s the incentives we’ve built around training, evaluating, and deploying these systems.
OpenAI researchers recently spotlighted how models guess when they should abstain, pinning much of the blame on accuracy-only scoreboards that reward lucky hits as much as careful reasoning. If incentives shape behavior, then today’s incentives are teaching models to bluff.
Why next‑token training invites confident guesses
At pretraining time, language models learn to predict the next token, not to separate true from false. The objective rewards fluency and pattern-matching across massive text corpora but provides no explicit penalty for fabricating specifics. As OpenAI notes, syntax and style improve steadily with scale, yet rare facts—like a niche dissertation title—lack reliable patterns to anchor them.
This creates a brittle foundation: when context runs thin, models interpolate. The result can sound authoritative because the same training that ignores veracity also optimizes for coherence and confidence. It’s a textbook case of Goodhart’s Law: optimize for the proxy (fluency) and you risk distorting the target (truth).
The evaluation trap: accuracy-only scoreboards
Evaluation culture amplifies the problem. Leaderboards that prize “percent correct” with no cost for overconfidence encourage guessing. Just as multiple‑choice exams without negative marking reward risk‑taking, accuracy‑only evals teach models that a bold, wrong answer is better than an honest “I’m not sure.”
Reinforcement learning from human feedback can deepen this bias. Human raters tend to prefer helpful, confident, and agreeable responses. Anthropic has documented “sycophancy,” where models echo user beliefs even when those beliefs are wrong, because agreement wins higher ratings. When product success metrics prioritize answer rate, speed, and user satisfaction, abstentions look like failures and fabrications slip through.
What the data and deployments show
Benchmarks underline the gap between coherence and correctness. On TruthfulQA, early general‑purpose models answered well under two‑thirds of questions truthfully, and while newer systems improve with careful prompting, error rates persist. The AI Index from Stanford HAI has repeatedly flagged hallucinations as a durable failure mode across summarization, question‑answering, and reasoning tasks.
Real‑world tests echo this. Medical and legal evaluations have documented fabricated citations and non‑existent studies in a meaningful fraction of outputs when models aren’t grounded to evidence. Industry experiments with retrieval‑augmented generation (from groups like Google DeepMind and Meta) show material reductions in factual errors, but even retrieval can be overridden when incentives still favor fast, confident answers.
The upshot: models often know when they might be wrong—work from Anthropic and others shows language models carry latent uncertainty signals—yet current scoring and product KPIs rarely reward surfacing that uncertainty.
Fixing incentives: penalize confidence, reward doubt
OpenAI’s proposal is straightforward: make evals uncertainty‑aware. Penalize confident errors more than honest uncertainty, and give partial credit for abstaining or hedging when evidence is thin. In practice, that means replacing single accuracy numbers with metrics like calibrated accuracy, precision at high confidence, and selective risk curves.
Product metrics should follow suit. Instead of maximizing “answers per query,” teams can track verifiability (share of claims with citations), grounded precision (correctness of claims tied to sources), and safe abstention rate (instances where the model appropriately asks for tools or declines). NIST’s AI Risk Management Framework and guidance from the UK AI Safety Institute both encourage measurable uncertainty and calibration as pillars of trustworthy AI.
Technical levers exist to backstop these incentives: calibrated confidence scoring, self‑consistency and cross‑examination prompts, tool use for retrieval and code execution, and chain‑of‑verification that forces models to check claims against sources before responding. Systems like DeepMind’s Sparrow pioneered rule‑based refusal policies; newer enterprise deployments gate high‑risk answers behind citations or human review.
Trade‑offs and limits
Stronger penalties for overconfidence will increase abstentions and may slow responses. Some users will perceive this as less helpful. There’s also the risk of “excess doubt,” where models disclaim too often, especially for underserved topics where retrieval is sparse. Guardrails must be domain‑aware: in creative ideation, a degree of speculation is fine; in finance or medicine, it is not.
And incentives aren’t the whole story. Knowledge gaps, ambiguous prompts, and distribution shifts still cause errors. No scoring tweak can substitute for better data curation, grounded tool use, and transparent provenance. But realigning incentives is a high‑leverage step that changes model behavior without waiting for the next breakthrough.
Bottom line
Bad incentives don’t create hallucinations from thin air, but they make them stubborn. When leaderboards and KPIs reward lucky guesses and polished prose, models learn to bluff. Flip the incentives—penalize confident wrongness, reward calibrated uncertainty, and demand verifiable grounding—and you turn honesty from a nice‑to‑have into the winning strategy.