Show HN: Parsimony – an open-source benchmark for how much code AI models add
Parsimony is a new open-source benchmark designed to evaluate AI models based on the amount of code they add to a project, rather than just solve rates. It aims to discourage 'code bloat' by subtracting removed code from added code to calculate a net efficiency score.
Why it matters
Addresses the long-term maintainability of AI-generated code, which is a critical concern for software engineering and developer productivity.
Existing benchmarks measure solve rate and cost, but don't punish excess code, which makes a codebase harder to work with in the long term. Parsimony measures how much code AI models add.
It takes the patches that benchmarks already publish and counts coding units : syntax elements like statements, calls, comparisons, names and literals. Lines aren't counted, so cramming code isn't rewarded, and deleted code is subtracted: net units = added − removed.
Chart Scatter: compare two metrics Bars: one metric Show my runs
Models are ranked by mean net units added on the tasks they solved, lowest first. "Net units added (mean)" also includes failed attempts. Solved comes from SWE-bench Verified ; tests aren't rerun. Measured is how many solved attempts could be measured.
Get smarter about the news
Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.
Create free accountAlready have an account? Sign in