Real-world benchmarks

Definition

Real-world benchmarks are evaluation methodologies that exercise systems — whether hardware, software, or machine-learning models — using workloads, datasets, or test cases drawn from actual production or open-source environments rather than from purely synthetic or hand-crafted stimuli. They are intended to capture the complexity, diversity, and constraints that systems encounter in deployment, providing a more faithful measure of performance and correctness than idealized test suites.

In the hardware-verification literature, real-world benchmarks specifically refer to running applications and performance metrics such as CoreMark and Dhrystone, together with operating-system boot sequences, in order to assess software–hardware integration on a prototype platform.

Use in hardware verification

A representative example is the multi-level verification strategy described for the NaxRiscv RISC-V SoC. After an automated Jenkins CI pipeline executes standard compliance suites (riscv-tests, riscv-arch-test), the methodology proceeds to a real-world benchmarking phase whose purpose is to assess both performance and software–hardware integration:

• execution of CoreMark and Dhrystone performance metrics;
• execution of operating-system boot sequences ranging from FreeRTOS to the Linux boot stage.

This phase is followed by a constrained-random testing layer using RISCV-DV and a functional-coverage model built on Universal Verification Methodology (UVM). The combined strategy — directed tests, real-world benchmarks, constrained-random tests, and coverage closure — is intended to validate every aspect of the design, from basic compliance to resilience under real-world conditions.

Role in machine-learning benchmarks

Recent machine-learning research has adopted the term "real-world benchmark" to denote evaluation datasets that are mined from genuine projects rather than constructed artificially. Two recent examples illustrate the trend:

ComBench (Compilation Error Repair)

ComBench is a repository-level, reproducible real-world benchmark for C/C++ compilation error repair (ACER). It is built by mining real-world failures from the GitHub CI histories of large-scale open-source projects. The construction pipeline addresses two specific limitations of prior ACER benchmarks:

1. decontextualized single-file data;
2. lack of authentic source diversity.

ComBench introduces techniques for high-precision identification of ground-truth repair patches from complex version histories, and a high-fidelity mechanism for reproducing the original ephemeral build environments. All samples are execution-verified, guaranteeing reproducible failures and build success with the ground-truth patches. A study using ComBench evaluated 12 modern LLMs under direct and agent-based repair settings and reported a 73% syntactic-correctness success rate for GPT-5 but only 41% semantically valid patches, illustrating the gap between surface-level and true repair quality.

NTP4VC (Verification Condition Proving)

NTP4VC is a real-world multi-language benchmark targeting automated proving of Verification Conditions (VCs) — a bottleneck in program verification. It is constructed from real-world projects such as the Linux kernel and Contiki-OS, using industrial pipelines (Why3 and Frama-C) to generate semantically equivalent test cases across the formal languages of Isabelle, Lean, and Rocq. Evaluation of general-purpose and theorem-proving–fine-tuned LLMs on NTP4VC indicates that although LLMs show promise in VC proving, significant challenges remain, exposing a gap between competition-style neural theorem proving and practical program verification.

Common characteristics

Across these usages, real-world benchmarks share several defining properties:

• Source authenticity — workloads, code, or proofs originate from genuine projects (open-source repositories, production kernels, deployed applications) rather than being hand-authored.
• Execution verification — samples are validated by actually running the failure and the ground-truth fix, ensuring reproducibility.
• Environment fidelity — the original build or execution environment (CI pipelines, toolchains, formal frameworks) is reproduced rather than abstracted away.
• Multi-level integration — real-world benchmarks are typically combined with other verification or evaluation layers (compliance tests, coverage models, constrained-random tests) to span the gap between idealized and deployed behavior.

Related entities

• CoreMark — a widely used processor benchmark often invoked as a real-world performance metric in SoC verification.
• Dhrystone — a long-standing CPU benchmark similarly used as a real-world performance metric.