Jev Can't Be Calibrated
The article critiques the 'Jev' model, a new classification tool from TypeSafe that claims to provide calibrated probability scores without training data. The author argues that these probability claims are technically flawed and should be treated as relative scores rather than true statistical probabilities.
Why it matters
As businesses increasingly rely on AI for automated decision-making, understanding the limitations of 'calibrated' confidence scores is critical for risk management.
September 23, 2026 · 3 mins · 617 words Share on: X · HN
Unless you’ve been living under a rock, you’ve probably heard about Jev. Simon Willison’s post is a good overview, and this one shows how to implement it in a few lines of Python. In short, Jev is TypeSafe’s first “ System One Model ”: instead of generating text, it takes unstructured input and returns typed decisions from a set of outputs you define in advance, each with a probability attached. One of its selling points is that “all answers are accompanied with calibrated probabilities and confidence scores”. In this post I argue that Jev is useful, but that the calibration claim can’t hold in general, and that you should treat its outputs as scores rather than probabilities.
Get smarter about the news
Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.
Create free accountAlready have an account? Sign in