Using LLM for confidence scoring is a bad idea
This article argues that Large Language Models (LLMs) are unreliable for generating confidence scores because they lack the probabilistic grounding required for true prediction. The author explains that confidence should be a calculation based on data, not an opinion generated by an AI.
Why it matters
Addresses a critical technical limitation in AI development, warning developers against over-relying on LLMs for decision-making metrics.
LLM-based agents can do a lot of tasks. Or, at least, they can appear to do them. For example, they write code, summarize papers, plan trips, and fill in forms. If you ask one for a probability or a confidence score, it will give you one of those too. I have seen many agent designers use LLMs to generate confidence scores in this way. The problem is that there is no good reason to trust that number. This post is about why we cannot rely on an LLM to generate a confidence score.
Before we can understand why an LLM cannot give a trustworthy confidence score, we need to understand what confidence really means.
For example, think of a weather forecaster who says “I am 70% confident that it’s going to rain tomorrow.”
Get smarter about the news
Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.
Create free accountAlready have an account? Sign in