The Benchmarking Crisis

A closer look at The Benchmarking Crisis

What The Benchmarking Crisis is about

This is a blog post that argues Goodhart’s law applies to database benchmarking: when people optimize for a measurable target the target stops being useful. The author uses historical TPC examples and discusses current issues with ClickBench.

What you can read and learn on this page

The post explains concrete ways benchmarks get gamed: e.g., using hashes to compare long strings instead of full content, predicate caching that removes repeated query cost, and micro-optimizations that exploit clustering plus LIMIT without ORDER BY.

It discusses possible fixes such as making benchmarks ad-hoc and forbidding exploitation of prior knowledge or intermediate result caching, adding update streams, and generating fresh workloads or keeping test queries secret.

The discussion includes comments from other practitioners: Andreas Kipf suggests replaying realistic cloud traces (Redbench), Thomas Neumann argues forbidding caching is simpler, and Michael Malis (creator of pgrust) defends some optimizations by noting ClickHouse implements similar techniques and describes pgrust’s precautions like using a salted 128-bit cryptographic hash.

Who will find this useful

Database researchers and engineers concerned with benchmarking validity and reproducibility will find the arguments and concrete examples relevant.

People working on execution engines, benchmark design, or those evaluating trade-offs between caching, updates, and analytic performance will get practical points to consider.

Questions people ask about The Benchmarking Crisis

Does the post propose concrete changes to benchmarks?

Yes - it suggests benchmarks be ad-hoc and forbid exploiting prior knowledge and intermediate result caching, include update streams, and possibly generate new workloads periodically.

Does the author think caching should never be allowed?

The author argues forbidding caching is the easier way to avoid distorted measurements, though the comments note real workloads often include repetitions that benefit from caching.

Is there follow-up material or tools mentioned?

A commenter mentions Redbench, which replays cloud workload traces over TPC-H/DS or SQLStorm, and the pgrust creator describes implementation choices and defenses against benchmark-specific optimizations.

The Benchmarking Crisis was shared on NexURLs by Nexis on November 2, 2026, in History & Culture. So far it has 1 like from members.