Benchmark Suite

Concept

A benchmark suite is a curated collection of programs, datasets, or test cases assembled to evaluate the performance, accuracy, security, or capabilities of a software system, analysis tool, or algorithm. Benchmark suites are used across diverse areas including hardware functional verification, EDA tool testing, Android security analysis, program synthesis, and early-stage evaluation of secure processor architectures such as CHERI, though many existing suites are too small, target only general-purpose processors, or require mature software stacks and therefore need to be extended with synthetic or generated programs.

First seen 6/15/2026

Last seen 8/4/2026

Evidence 13 chunks

Wiki v4

WIKI

Benchmark Suite

Definition

A benchmark suite is a curated collection of programs, datasets, or test cases assembled to evaluate the performance, accuracy, or capabilities of a software system, analysis tool, or algorithm. Benchmark suites are used across diverse areas of computer science, including hardware functional verification, Electronic Design Automation (EDA) tool testing, Android security analysis, program synthesis, and the early-stage performance evaluation of secure processor architectures.

READ FULL ARTICLE →

NEIGHBORHOOD

No graph connections found for this entity yet. It may appear in future ingestion runs.

explore full graph →

RELATIONSHIPS

4 connections

design space exploration ← uses 85% 2e

Design space exploration for CHERI architectures requires appropriate benchmarks.

ChiBench ← implements 100% 2e

ChiBench is a collection of Verilog programs serving as a benchmark suite for EDA tool testing.

Functional Verification part of → 82% 1e

Benchmark suites are used as part of the functional verification and performance analysis process

Security-Aware Benchmarks for Performance Exploration of CHERI-Enabled Architectures ← mentions 90% 1e

The paper discusses existing benchmark suites and the need for CHERI-specific ones.

LINKED ENTITIES

1 links

ChiBench IMPLEMENTS Extracted graph relationship

CITATIONS

15 sources

15 citations — click to expand

[1] ChiBench is a collection of Verilog programs meticulously mined from diverse open-source repositories and is used as a training dataset / benchmark suite for testing EDA tools. Generation of various programs in Verilog/SystemVerilog for testing EDA tools

[2] ChiGen uses ChiBench-derived production rules modeled as k-grams (conditioned on the k preceding rules) to drive Verilog syntax generation; this approach captures structural patterns of human-written Verilog and is stored in a JSON probabilistic grammar. Generation of various programs in Verilog/SystemVerilog for testing EDA tools

[3] For 10,000 designs, ChiBench uses 406 unique production rules out of 456 available in the Verible grammar; Verismith, TransFuzz, and VlogHammer use only 179, 151, and 137 unique rules respectively. Generation of various programs in Verilog/SystemVerilog for testing EDA tools

[4] ChiGen dynamically generates new Verilog designs to complement static benchmark collections like ChiBench, broadening the testing landscape for EDA tools. Generation of various programs in Verilog/SystemVerilog for testing EDA tools

[5] ChiGen has identified at least five confirmed issues in open-source EDA tools such as Yosys, Icarus Verilog, and Verible. Generation of various programs in Verilog/SystemVerilog for testing EDA tools

[6] EDA tools such as Yosys, Verilator, Icarus Verilog, ModelSim, Xcelium, and Jasper Formal rely on benchmarks in HDLs like Verilog, SystemVerilog, and VHDL for bug discovery, performance optimization, compliance verification, and validation. Generation of various programs in Verilog/SystemVerilog for testing EDA tools

[7] Existing top-down Verilog fuzzers (Verismith, TransFuzz, VlogHammer) cover fewer than 40% of the production rules in the Verilog-2005 grammar (IEEE 1364-2005). Generation of various programs in Verilog/SystemVerilog for testing EDA tools

[8] The BenchPress study empirically evaluated four Android-specific benchmark suites — DroidBench, Ghera, IccBench, and UBCBench — measuring API coverage across 227K real-world apps and identifying gaps in benchmark coverage. BenchPress: Analyzing Android App Vulnerability Benchmark Suites

[9] PSB2 was introduced as a replacement suite of 25 new program synthesis problems curated from sources including programming katas and college courses, chosen to be more difficult than the original General Program Synthesis Benchmark Suite. PSB2: The Second Program Synthesis Benchmark Suite

[10] Many existing benchmark suites consist of only tens of programs and are usually too small to build sufficiently large and diverse training sets for ML models.

[11] Evaluating CHERI-enabled architectures requires benchmarks that explicitly exercise security mechanisms such as bounds checks, tag manipulation, and capability register operations; mainstream suites (SPEC, PARSEC, Embench IoT) do not exercise these mechanisms.

[12] A security-aware benchmark generation framework for CHERI-based RISC-V systems uses TestRIG to produce raw instruction streams that are post-processed into CHERI-compliant instruction streams, enabling early-stage evaluation on a RISC-V Virtual Prototype. Security-Aware Benchmarks for Performance Exploration of CHERI

[13] The CHERI security-aware benchmarks demonstrated best-case speed-ups of 6%–11% over an unaccelerated baseline when evaluating In-Memory Computing acceleration of CHERI tagged memory on a RISC-V VP, with diminishing benefits above approximately 20 ns IMC latency. Security-Aware Benchmarks for Performance Exploration of CHERI

[14] Conventional benchmark workloads such as CoreMark, SPEC, and the Linux kernel are used alongside custom-designed tests in hardware functional verification to exercise hardware designs and extract microarchitectural information from RTL.

[15] The Genesis preprocessor takes a template program and an annotation file and generates many program instances by sampling parameter values from user-defined distributions, enabling arbitrarily large program sets with controlled statistical properties.