Parsimony - an open-source benchmark for how much code AI models add

A closer look at Parsimony - an open-source benchmark for how much code AI models add

What Parsimony measures

Parsimony is an open-source benchmark that counts how much code AI models add when they modify Python code. Instead of using lines, it counts syntax units such as calls, names and literals, and reports net syntax units = added − removed.

It applies each published patch to the task code, parses both versions of changed files, and does not run anything. Tests, docs and generated files are excluded; only Python is measured.

How the site presents results

Models are ranked by mean net syntax units added on tasks they solved, lowest first, and the mean includes failed attempts. Solved and pass/fail come from SWE-bench Verified; tests are not rerun.

Interactive charts and tables let you compare metrics, show different runs and harnesses, and download charts as PNG. Hovering shows shared-task counts and conditional 95% ranges; you can also download an example JSON to load your own code-size numbers into the same chart and table.

Who will find this useful

Researchers and engineers who care about the amount of code generated by models - especially in Python - will find this relevant, since it penalizes excess code rather than only scoring solves or cost.

People running models on SWE-bench tasks, maintainers of model harnesses, or anyone with spare API credits interested in contributing runs or asking for missing models can use the site to request models or report incorrect numbers.

Questions people ask about Parsimony - an open-source benchmark for how much code AI models add

Does Parsimony run tests to verify results?

No. Pass/fail comes from SWE-bench's published results and tests are not rerun.

Can I compare two specific models on the same tasks?

Yes; the site shows net syntax units on the intersection of two models' solved, measured tasks and provides hover details like shared-task counts and 95% ranges.

Is Parsimony measuring all languages and files?

No. It measures only Python and excludes tests, docs and generated files.

Parsimony - an open-source benchmark for how much code AI models add was shared on NexURLs by Nexis on November 2, 2026, in Articles & Stories.