Skip to content
STIMSMITH

Benchmark Suite

Concept WIKI v4 · 8/4/2026

A benchmark suite is a curated collection of programs, datasets, or test cases assembled to evaluate the performance, accuracy, security, or capabilities of a software system, analysis tool, or algorithm. Benchmark suites are used across diverse areas including hardware functional verification, EDA tool testing, Android security analysis, program synthesis, and early-stage evaluation of secure processor architectures such as CHERI, though many existing suites are too small, target only general-purpose processors, or require mature software stacks and therefore need to be extended with synthetic or generated programs.

Benchmark Suite

Definition

A benchmark suite is a curated collection of programs, datasets, or test cases assembled to evaluate the performance, accuracy, or capabilities of a software system, analysis tool, or algorithm. Benchmark suites are used across diverse areas of computer science, including hardware functional verification, Electronic Design Automation (EDA) tool testing, Android security analysis, program synthesis, and the early-stage performance evaluation of secure processor architectures.

Common Characteristics

  • Purpose: Provide a standardized basis for comparing tools, measuring improvements, and identifying gaps in coverage.
  • Typical size: Many existing suites "consist of only tens of programs" and are "usually too small to build sufficiently large and diverse training sets for ML models" [1].
  • Construction: Suites may be hand-crafted (e.g., from real-world vulnerabilities, programming katas, coursework, or HDL programs mined from open-source repositories) or augmented with automatically generated programs when larger, more controlled, or architecture-specific inputs are required.
  • Coverage scope: A suite may target general-purpose workloads (e.g., SPEC, PARSEC, Embench IoT) or be specialized for a particular mechanism such as memory-safety primitives or HDL language constructs, in which case portability across architectures or tools is reduced [5437dd17-0767-4433-819f-3fabe16573a9, 2b223653-812f-4f89-a001-1a9053bfdc38].

Use Cases in the Evidence

Hardware Functional Verification

In hardware design, benchmark suites serve as one of the workloads used to stress-test processor and SoC designs as part of the broader functional verification effort. Conventional benchmark workloads such as CoreMark, SPEC, and the Linux kernel are used alongside custom-designed tests to exercise different parts of a hardware design and to extract microarchitectural information from the RTL during execution [2]. Cycle-accurate RTL simulation is the recurring element in the hardware design process and benchmark suites are part of the infrastructure supporting both functional verification and performance optimization [b66bc6b3-d8ca-4b4f-90d1-131aacf9457a, e9708e27-d442-4681-b37e-52269687f533].

Electronic Design Automation (EDA) Tool Testing

EDA tools such as Yosys, Verilator, Icarus Verilog, ModelSim, Xcelium, and Jasper Formal rely on benchmarks—programs written in Hardware Description Languages (HDLs) like Verilog, SystemVerilog, and VHDL—to enable early bug discovery, performance optimization, compliance verification, and general validation [f1c73d8d-b22f-4d1c-9921-aea6c16740f2, 6cecedc0-f8b6-4f2f-9e55-b56ff4cfa82b].

Existing static benchmark collections, including ChiBench, are limited to a predefined set of designs and may not exercise the full range of language constructs. For example, the Verilog-2005 grammar (IEEE 1364-2005) contains many production rules, and existing top-down fuzzers (Verismith, TransFuzz, VlogHammer) exercise only a fraction of them—fewer than 40% in some measurements [3].

The ChiBench corpus is described in the evidence as "a collection of Verilog programs meticulously mined from diverse open-source repositories, serving as our primary training dataset" for the ChiGen Verilog fuzzer [4]. ChiGen uses ChiBench to derive a probabilistic grammar: production rules are extracted via Verible's parser trace, rule frequencies are aggregated as k-grams conditioned on the k preceding rules, and the resulting JSON file drives syntax generation [4]. In evaluation, ChiGen-generated designs approach ChiBench's diversity (using 406 unique production rules out of 456 available in the Verible grammar for 10,000 designs), while Verismith, TransFuzz, and VlogHammer use only 179, 151, and 137 unique rules respectively [5].

Because ChiBench is a static collection, ChiGen complements it by dynamically generating new Verilog designs and integrating them into a growing library of programs, broadening the testing landscape for hardware tools [3].

Early-Stage Evaluation of Secure Architectures (CHERI)

The CHERI capability-based architecture provides fine-grained spatial and temporal memory safety for C/C++ programs, but introduces additional hardware structures (capability registers, tagged memory, CHERI-specific instructions) that may affect performance and energy efficiency. Evaluating CHERI-enabled architectures therefore requires benchmarks that explicitly exercise security mechanisms such as bounds checks, tag manipulation, and capability register operations [5437dd17-0767-4433-819f-3fabe16573a9, 9c53fdab-ffb8-4976-b699-06c480659d6d].

Existing mainstream suites such as SPEC, PARSEC, and Embench IoT target general-purpose processors and do not exercise memory-safety mechanisms. Studies of Intel Memory Protection Extensions, ARM Memory Tagging Extension (MTE), and other tagged-memory designs have instead relied on specialized benchmarks that focus on bounds checks, tag manipulation, and fault-handling latency; such benchmarks are typically tailored to a single mechanism and are difficult to reuse across architectures [6].

For CHERI specifically, the most prominent prior benchmarking effort targets the Arm Morello platform and is based on porting existing C/C++ applications to CHERI. While effective at the application level, this approach depends on CheriBSD and assumes a mature execution environment, limiting its applicability for resource-constrained systems and early-stage design-space exploration [5437dd17-0767-4433-819f-3fabe16573a9, 2b223653-812f-4f89-a001-1a9053bfdc38].

The Das et al. paper (DDECS 2026) addresses this gap by introducing a security-aware benchmark generation framework for CHERI-based RISC-V systems. The framework:

  1. Uses Testing with Random Instruction Generation (TestRIG) to request raw instruction streams with configurable distributions.
  2. Post-processes those streams with a CHERI-specific algorithm to produce security-aware instruction streams that execute correctly under CHERI's capability and tagging rules (e.g., rewriting immediates of jump instructions and capability registers of jalr.cap / cinvoke).
  3. Combines the processed streams with platform-specific bootstrap code to build executable benchmarks available early in development.
  4. Demonstrates applicability by evaluating an In-Memory Computing (IMC) acceleration of CHERI tagged memory on a RISC-V Virtual Prototype (VP), revealing best-case speed-ups of 6%–11% over an unaccelerated baseline, with diminishing benefits above approximately 20 ns IMC latency [5437dd17-0767-4433-819f-3fabe16573a9, 2b223653-812f-4f89-a001-1a9053bfdc38, 8eda910b-fe2f-4b23-ba72-20af4548efa9, 634a5f96-7a55-46f4-b5ef-49d0709e0d4b].

The generated benchmarks can emphasize specific CHERI instructions or approximate application-level behavior via TestRIG's generation parameters, enabling targeted and reproducible performance evaluation of CHERI microarchitectures [6].

Android Security Analysis

The "BenchPress" study empirically evaluated four Android-specific benchmark suites — DroidBench, Ghera, IccBench, and UBCBench — to assess their API coverage in real-world apps (a sample of 227K apps) and to identify differences and gaps between them. The work noted that benchmark selection in tool evaluations is often driven by availability and popularity rather than by characteristics or relevance (arxiv 1903.05170).

Program Synthesis

The original General Program Synthesis Benchmark Suite had been used for six years in genetic programming and program-synthesis research but had gradually become outdated, hindering accurate measurement of further improvements. PSB2 was introduced as a replacement suite of 25 new problems curated from sources including programming katas and college courses, explicitly chosen to be more difficult than the originals (arxiv 2106.06086).

Synthetic Program Generation

When benchmark suites are too small, program generators can be used to synthesize additional test programs. The Genesis preprocessor exemplifies this approach: it takes a template program (in C, Java, C++, etc.) plus a Genesis annotation file, and generates many instances by sampling parameter values from user-defined distributions. Genesis is target-language agnostic and supports hierarchical composition of generated segments, enabling the creation of arbitrarily large program sets with controlled statistical properties — useful for ML training sets and mutation testing [1]. Similarly, the ChiGen fuzzer uses ChiBench-derived probabilistic grammars to dynamically synthesize diverse Verilog programs for EDA testing [c9ea02f1-f591-4cb5-ba10-8441c71e5c68, 6cecedc0-f8b6-4f2f-9e55-b56ff4cfa82b].

Related Concepts

  • Functional verification — Benchmark suites are used as workloads within functional verification efforts, particularly in hardware design, where they are run alongside custom tests to exercise RTL behavior. Benchmark suites are part of the broader functional verification methodology used in hardware projects.
  • Design space exploration — Benchmark suites are used to systematically explore architectural optimization options; specialized security-aware suites (e.g., for CHERI) are required when the design variables involve hardware memory-safety mechanisms, because general-purpose suites do not exercise those mechanisms.
  • Fuzzing — Fuzzers that generate random, often unexpected, inputs can serve as a dynamic source of benchmarks when static benchmark suites are limited in diversity, as demonstrated by ChiGen for Verilog EDA tools [f1c73d8d-b22f-4d1c-9921-aea6c16740f2, 6cecedc0-f8b6-4f2f-9e55-b56ff4cfa82b].

LINKED ENTITIES

1 links
ChiBench IMPLEMENTS

CITATIONS

15 sources
15 citations
[1] ChiBench is a collection of Verilog programs meticulously mined from diverse open-source repositories and is used as a training dataset / benchmark suite for testing EDA tools. Generation of various programs in Verilog/SystemVerilog for testing EDA tools
[2] ChiGen uses ChiBench-derived production rules modeled as k-grams (conditioned on the k preceding rules) to drive Verilog syntax generation; this approach captures structural patterns of human-written Verilog and is stored in a JSON probabilistic grammar. Generation of various programs in Verilog/SystemVerilog for testing EDA tools
[3] For 10,000 designs, ChiBench uses 406 unique production rules out of 456 available in the Verible grammar; Verismith, TransFuzz, and VlogHammer use only 179, 151, and 137 unique rules respectively. Generation of various programs in Verilog/SystemVerilog for testing EDA tools
[4] ChiGen dynamically generates new Verilog designs to complement static benchmark collections like ChiBench, broadening the testing landscape for EDA tools. Generation of various programs in Verilog/SystemVerilog for testing EDA tools
[5] ChiGen has identified at least five confirmed issues in open-source EDA tools such as Yosys, Icarus Verilog, and Verible. Generation of various programs in Verilog/SystemVerilog for testing EDA tools
[6] EDA tools such as Yosys, Verilator, Icarus Verilog, ModelSim, Xcelium, and Jasper Formal rely on benchmarks in HDLs like Verilog, SystemVerilog, and VHDL for bug discovery, performance optimization, compliance verification, and validation. Generation of various programs in Verilog/SystemVerilog for testing EDA tools
[7] Existing top-down Verilog fuzzers (Verismith, TransFuzz, VlogHammer) cover fewer than 40% of the production rules in the Verilog-2005 grammar (IEEE 1364-2005). Generation of various programs in Verilog/SystemVerilog for testing EDA tools
[8] The BenchPress study empirically evaluated four Android-specific benchmark suites — DroidBench, Ghera, IccBench, and UBCBench — measuring API coverage across 227K real-world apps and identifying gaps in benchmark coverage. BenchPress: Analyzing Android App Vulnerability Benchmark Suites
[9] PSB2 was introduced as a replacement suite of 25 new program synthesis problems curated from sources including programming katas and college courses, chosen to be more difficult than the original General Program Synthesis Benchmark Suite. PSB2: The Second Program Synthesis Benchmark Suite
[10] Many existing benchmark suites consist of only tens of programs and are usually too small to build sufficiently large and diverse training sets for ML models.
[11] Evaluating CHERI-enabled architectures requires benchmarks that explicitly exercise security mechanisms such as bounds checks, tag manipulation, and capability register operations; mainstream suites (SPEC, PARSEC, Embench IoT) do not exercise these mechanisms.
[12] A security-aware benchmark generation framework for CHERI-based RISC-V systems uses TestRIG to produce raw instruction streams that are post-processed into CHERI-compliant instruction streams, enabling early-stage evaluation on a RISC-V Virtual Prototype. Security-Aware Benchmarks for Performance Exploration of CHERI
[13] The CHERI security-aware benchmarks demonstrated best-case speed-ups of 6%–11% over an unaccelerated baseline when evaluating In-Memory Computing acceleration of CHERI tagged memory on a RISC-V VP, with diminishing benefits above approximately 20 ns IMC latency. Security-Aware Benchmarks for Performance Exploration of CHERI
[14] Conventional benchmark workloads such as CoreMark, SPEC, and the Linux kernel are used alongside custom-designed tests in hardware functional verification to exercise hardware designs and extract microarchitectural information from RTL.
[15] The Genesis preprocessor takes a template program and an annotation file and generates many program instances by sampling parameter values from user-defined distributions, enabling arbitrarily large program sets with controlled statistical properties.

VERSION HISTORY

v4 · 8/4/2026 · minimax/minimax-m3 (current)
v3 · 7/6/2026 · minimax/minimax-m3
v2 · 7/4/2026 · minimax/minimax-m3
v1 · 6/15/2026 · minimax/minimax-m3