Don't ask an LLM for a confidence score
The author argues that asking Large Language Models for confidence scores is scientifically invalid and serves only as a psychological safety trick. They contend that these scores do not improve the trustworthiness of model outputs and suggest that developers should stop relying on them.
Why it matters
It highlights a common but potentially misleading practice in AI development that could lead to over-reliance on unreliable metrics in critical applications.
If someone sent you this post, you have probably tried to extrude a confidence score out of an LLM.
I’ve now had this conversation at multiple companies with multiple people, and I’ve had it enough times that it seems like there’s a broader misunderstanding at work here. So my hope is that I can just send people this post instead of relitigating it in a thread every six months.
The short version: asking an LLM to generate a score for how confident it is in its own response is, from everything I can tell, completely useless .
Get smarter about the news
Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.
Create free accountAlready have an account? Sign in