Skip to content
STIMSMITH

BEEBS Benchmarks

Concept WIKI v1 · 7/11/2026

BEEBS is a benchmark suite used to evaluate embedded-systems and compiler research, including security mechanisms on RISC-V, iterative compilation on mobile devices, and functional coverage collection on the BlackParrot core using ZP Cosim.

BEEBS Benchmarks

Overview

BEEBS is a benchmark suite that has been used to evaluate research in compiler optimization, embedded-systems security, and hardware verification. The evidence describes the suite being applied across three distinct research contexts: forward-edge control-flow integrity on RISC-V, iterative compilation for mobile devices, and functional coverage analysis on the BlackParrot RISC-V core.

Use in Forward-Edge CFI Research (Branch Landing)

In a 2026 study on forward-edge Control-Flow Integrity (CFI) for RISC-V (Branch Landing / BRL), 81 BEEBS benchmarks were used as the evaluation workload. Branch Landing replaces fixed-capacity tag-based CFI checks with Bloom-filter membership queries through two ISA extensions (bld and brl) and a dedicated BRState register. The authors evaluated the framework on BEEBS under two representative policy configurations:

  • Function-level, type-based policy — coarse, type-equivalence-based authorization.
  • Basic-block-level, CFG-derived policy — finer-grained authorization derived from the control-flow graph.

Reported quantitative results from the BEEBS evaluation under a 3-cycle brl latency model:

Policy Average runtime overhead Mean code-size growth
Function-level, type-based 0.210% 0.46%
Basic-block-level, CFG-derived 0.421% 0.52%

Additional findings from the BEEBS evaluation: the CFG-derived policy reduced the average equivalence-class size by 32.5% compared to the type-based policy, and all evaluated executions completed without BRL enforcement failures.

Use in Iterative Compilation on Mobile Devices

BEEBS has been used as the evaluation benchmark suite in a 2015 study on iterative compilation for mobile applications. The work proposed an iterative compiler employing a capture-and-replay technique that records only the memory pages needed to replay the most heavily used functions of an application, then evaluates different optimization variants offline during idle periods. Reported outcomes on the BEEBS suite:

  • Performance improvements of up to 57% versus the unoptimized baseline.
  • Average user-experienced slowdown of 0.8% during online capture.
  • Storage reductions of two to three orders of magnitude compared with typical capture-and-replay implementations.

Use in Coverage Analysis on BlackParrot (ZP Cosim)

In a 2023 University of Washington thesis documenting the ZP Cosim tool, BEEBS was used as part of the coverage-evaluation workload for the BlackParrot RISC-V core. The coverage study combined BEEBS with the RISC-V Tests and 4 of 9 SPEC 2017 benchmark programs, producing a coverage map of covergroup IDs versus individual coverpoint toggles. The suite contributed to identifying "coverage holes" and to assessing case-statement and mux (select-signal) toggle coverage on BlackParrot when stimulated through RISCV-DV-generated programs.

Relationships

  • Evaluated by: ZP Cosim — ZP Cosim was used to collect functional coverage from BEEBS programs running on BlackParrot.

References

Evidence is drawn from the cited sources below. Numeric results for the Branch Landing study are sourced from the arXiv abstract and may be refined in the full paper.

LINKED ENTITIES

1 links
ZP Cosim EVALUATES

CITATIONS

6 sources
6 citations
[1] BEEBS is used as an evaluation benchmark suite for research in compiler optimization, security, and hardware verification. Branch Landing: Bloom Filter-Based Source Authorization for Forward-Edge CFI on RISC-V
[2] The Branch Landing framework was evaluated on 81 BEEBS benchmarks under a function-level type-based policy and a basic-block-level CFG-derived policy. Branch Landing: Bloom Filter-Based Source Authorization for Forward-Edge CFI on RISC-V
[3] Under a 3-cycle brl latency model, the function-level type-based and basic-block-level CFG-derived policies incur average runtime overheads of 0.210% and 0.421% and mean code size growth of 0.46% and 0.52% respectively on the BEEBS suite. Branch Landing: Bloom Filter-Based Source Authorization for Forward-Edge CFI on RISC-V
[4] The CFG-derived policy reduces the average equivalence class size by 32.5% compared to the type-based policy on BEEBS, with no BRL enforcement failures observed. Branch Landing: Bloom Filter-Based Source Authorization for Forward-Edge CFI on RISC-V
[5] An iterative-compilation technique using capture-and-replay improved performance on the BEEBS benchmark suite by up to 57%, with average user-experienced slowdown of 0.8% and storage reductions of two to three orders of magnitude. Iterative compilation on mobile devices
[6] BEEBS was included alongside RISC-V Tests and 4 of 9 SPEC 2017 benchmark programs in the ZP Cosim coverage evaluation on the BlackParrot core, contributing to coverage maps of covergroup IDs and coverpoint toggles. ZP Cosim thesis (Anoop Mysore Nataraja, 2023)