SOURCE ARCHIVE
EXTRACTED CONTENT
31,993 chars Security-Aware Benchmarks for Performance
Exploration of CHERI-Enabled Architectures
Spandan Das1, Sayak Deb2, Khushboo Qayyum3, Sallar Ahmadi-Pour1, Christoph L¨uth1,3, Rolf Drechsler1,3
1Institute of Computer Science, University of Bremen, Bremen, Germany
2Institute of Physics and Electrical Engineering, University of Bremen, Bremen, Germany
3Cyber-Physical Systems, DFKI, Bremen, Germany
{spandan, sayak, sallar, drechsler}@uni-bremen.de, {khushboo.qayyum, christoph.lueth}@dfki.de
Abstract—The Capability Hardware Enhanced RISC Instruc- This paper addresses this limitation by introducing a frame-
tions (CHERI) architecture provides fine-grained memory protec- work for generating security-aware benchmarks tailored to tion for systems, but introduces additional hardware overheads CHERI-enabled RISC-V systems. Our approach builds on that may negatively impact performance. Evaluating and optimiz- ing such secure architectures require benchmarks that explicitly Testing with Random Instruction Generation (TestRIG) and exercise their security mechanisms. Despite the abundance of augments it with CHERI-specific instruction stream processing benchmarks for unhardened systems, security-aware benchmarks to produce executable benchmarks that correctly and system- for CHERI-based architectures remain scarce. We address this atically utilize capability-based memory operations. Unlike gap by proposing a framework for generating security-aware application-level benchmarks, the generated benchmarks are benchmarks for CHERI-based RISC-V systems, leveraging the TestRIG tool and applying CHERI-specific post-processing to available early in development and can be deployed with ensure valid capability usage. As a demonstration use case, minimal integration effort. we apply the generated benchmarks to evaluate an In-Memory We evaluate the proposed framework using a Virtual Pro- Computing (IMC)-based acceleration of the CHERI tagged totype (VP) and an illustrative case study modeling an In- memory. Our results show best-case speedups between 6 % and Memory Computing (IMC)-based acceleration of a CHERI 11 %, while also identifying scenarios in which the acceleration proves no performance benefit. tagged memory. Our results show best-case speed-ups of 6 % Index Terms—CHERI, Tag-based Architecture, Benchmark, and 11 % over an unaccelerated baseline, while also identify- In-Memory-Computation, Virtual Prototype ing parameter regimes in which acceleration yields no benefit. I. INTRODUCTION In summary, this work makes the following contributions:• The widespread adoption of the Internet of Things (IoT) A security-aware benchmark generation framework for in security- and safety-critical domains makes memory • CHERI-enabled embedded systems. safety a fundamental design requirement. A large frac- An experimental demonstration showing how the bench- tion of exploitable vulnerabilities in such systems originates marks quantify optimization potential and limits in a from memory-safety violations, motivating the integration • CHERI-based VP. of hardware-supported protection mechanisms directly into A set of open-source security-aware benchmarks for processor architectures [1]–[3]. The Capability Hardware En- CHERI-based RISC-V systems for early performance hanced RISC Instructions (CHERI) architecture addresses this assessment.1 challenge by enforcing fine-grained spatial and temporal mem- II. PRELIMINARIES ory safety through capability-based protection [4]. Despite A. CHERI its security benefits, CHERI introduces additional hardware structures, which may significantly impact the performance CHERI is a capability-based architectural extension that and energy efficiency of IoT devices. Improving the perfor- provides hardware-enforced spatial and temporal memory mance of CHERI-based designs therefore requires systematic safety for systems programmed in memory-unsafe languages exploration and evaluation of architectural optimizations. Such such as C and C++ [4], [5]. In CHERI-enabled RISC-V sys- optimization efforts critically depend on benchmarks that tems, conventional pointers are replaced with capabilities, explicitly exercise CHERI’s security mechanisms. However, which encode bounds, permissions, and validity metadata for existing benchmarks either target unhardened systems or re- memory regions. Capability tags are propagated through reg- quire mature CHERI-capable environments running operating isters, caches, and memory, and are stored in a separate, non- systems that are often unavailable during early development addressable tagged memory that is accessible only via CHERI- phases. As a result, performance-driven design space ex- specific instructions. Any illegal manipulation invalidates the ploration for secure embedded architectures remains poorly tag and prevents further memory access through the corrupted supported. capability. While these mechanisms improve security, they introduce additional metadata handling and memory accesses, This research has been supported by the German Research Foundation which can affect performance in embedded systems. This work (DFG) with project EMBOSOM (grant nos. DR 287/42-1, LU 707/10-1), focuses on accelerating the CHERI tagged memory by using and the German Ministry for Research, Technology and Space (BMFTR) with projects ECXLplus (grant no 01IW24001) and ExaVerse (grant no 01IW25003). 1 https://github.com/agra-uni-bremen/cheri-early-benchmarks
IMC to improve memory access, while preserving CHERI’s In contrast, this work enables early-stage, security-aware
security guarantees. performance evaluation of CHERI-enabled RISC-V embed-
B. TestRIG ded systems. By generating instruction-level benchmarks that
explicitly exercise CHERI’s capability and tag-management
TestRIG is a verification-oriented framework for and integrating them with TestRIG-based verification, our RISC-V processor implementations using constrained approach supports concurrent evaluation of performance and random instruction generation [6]. It interfaces with a Design security correctness during system development. under Verification (DUV) via the RISC-V Formal Interface IV. BENCHMARKING FOR SECURE EMBEDDED SYSTEMS (RVFI) and Direct Instruction Injection (DII), executing Meaningful performance evaluation of secure embedded generated instructions and comparing architectural state systems requires benchmarking early in the system develop- against the Sail [7] formal model of the RISC-V Instruction ment cycle, when architectural design decisions remain flexi- Set Architecture (ISA). TestRIG supports multiple abstraction ble. While integrating security architectures such as CHERI levels, ranging from high-level emulation to Register- into VPs enables early consideration of security aspects, Transfer Level (RTL) simulation. A key feature of the existing benchmarks do not exercise the CHERI ISA exten- DII interface is that instruction execution does not require sion for RISC-V. At the same time, application-level CHERI consistent program control flow, enabling high-throughput benchmarks require on a mature execution environment with instruction generation. RVFI provides fine-grained visibility an operating system and software stack, which are typically into architectural state after each instruction, allowing precise unavailable during early design space exploration. semantic validation. Communication between TestRIG and To address this gap, we leverage the TestRIG framework the DUV is implemented via standard network sockets, to establish an early-stage benchmarking environment for enabling flexible integration with simulators. In this work, CHERI-enabled systems. TestRIG provides configurable ran- we repurpose TestRIG’s instruction generation infrastructure dom instruction generation, which we repurpose to produce to form the basis of security-aware benchmark generation. instruction streams that are subsequently refined to ensure By post-processing generated instruction streams to ensure correct and systematic use of CHERI capabilities. The result- valid capability usage, we enable meaningful performance ing benchmarks enable quantitative performance evaluation of evaluation of CHERI architectures beyond verification. architectural optimizations while preserving CHERI’s security III. RELATED WORKS guarantees. Hardware-assisted spatial and temporal memory safety has TestRIG Generation Generation Platform been explored through various architectural techniques, in- Environment1 Parameters 3 Bootstrap cluding bounds-checking mechanisms such as HardBound [8] Raw Instruction Sequences Security-Aware CHERI and SoftBound [9], tagged memory architectures such as Generation 2 Instruction Processing Benchmarks ARM Memory Tagging Extension [10], and capability-based Evaluation 4 systems. CHERI represents the most mature capability-based Architecture Parameters Virtual Prototype Performance Metrics memory safety architecture, providing fine-grained enforce- ment of pointer bounds and permissions in hardware [4]. Figure 1: Overview of our generation framework for early and Since its introduction, CHERI has been implemented and security-aware benchmarks. evaluated across multiple platforms, including software-based Figure 1 provides an overview of the benchmark generation emulation [11], FPGA prototypes [12], and ASICs [5]. workflow. First, a customized generation environment connects Benchmarking plays a central role in evaluating architec- to TestRIG and requests instruction streams with configurable tural optimizations, yet widely used benchmark suites such instruction distributions ( 1 ). Since these sequences are in- as SPEC [13], PARSEC [14], and Embench IoT [15] target tended for functional verification, they are post-processed general-purpose processors and do not exercise memory safety to construct security-aware instruction streams that execute mechanisms. Studies evaluating Intel Memory Protection Ex- correctly under CHERI’s capability and tagging rules ( 2 ). tensions [16], ARM Memory Tagging Extension [17], and The processed streams are then combined with platform- other tagged memory designs have therefore relied on special- specific bootstrap code to build executable benchmarks ( 3 ). ized benchmarks focusing on bounds checks, tag manipulation, To demonstrate the applicability of the framework, we apply and fault handling latency. Such benchmarks are typically the generated benchmarks to evaluate a CHERI-specific ar- tailored to specific mechanisms and are difficult to reuse across chitectural optimization in a VP ( 4 ). Specifically, we model architectures. For CHERI, the most prominent benchmarking an IMC-based acceleration of CHERI tagged memory and effort targets ARM’s Morello platform and is based on porting compare performance against an unmodified VP to quantify existing C/C++ applications to CHERI [18]. While effective potential speed-ups and identify performance limits. at the application level, this approach depends on CheriBSD and assumes a mature execution environment, limiting its A. Security-Aware Benchmark Generation applicability for resource-constrained systems and early-stage Instruction sequences generated by TestRIG are designed design space exploration. for functional verification and intentionally violate architec-
Algorithm 1 Security-aware processing of instruction streams Instruction distributions can be further controlled through Input: Raw instruction stream Iraw TestRIG’s generation parameters, allowing the benchmarks Output: Security-aware instruction stream Isec to emphasize specific CHERI instructions or approximate 1: CapSize ← Size of a capability in number of bits 2: for each instruction I in raw instruction stream Iraw do application-level behavior. This enables targeted and repro- 3: ct ← any capability register ducible performance evaluation of CHERI microarchitectures. 4: rt ← any integer register 5: addr ← any data space address divisible by CapSize 6: if I is cjal, cjalr, B-type or regular J-type instruction then B. IMC-based CHERI Acceleration 7: Set Immediate of I to 4 To demonstrate the applicability of the generated bench- 8: else if I is jalr.cap or cinvoke then 9: Set capability register of I to cs1 marks for early-stage design exploration, we evaluate a rep- Append sequence to Isec: resentative CHERI-specific optimization using a VP. Specif- 10: auipcc cs,0; cincoffsetimm cs,cs,12 ically, we model an IMC-based acceleration of the CHERI else if I is jalr.pcc then 11: rs ← rs1 of I tag update mechanism and compare system performance with Append sequence to Isec: and without acceleration. CHERI maintains capability validity auipcc ct,0; cincoffsetimm ct,ct,16; using tag bits stored in a dedicated, non-addressable tag cgetoffset rs,ct 12: else if I is sc.cap or csc then memory. During capability store instructions (e.g., sc.ddc, 13: if I has valid immediate then sc.cap), the corresponding tag bit must be updated alongside 14: Set immediate of I to 0 the data. A conventional implementation requires transferring 15: end if 16: cs ← cs1 of I the tag-containing memory word to the processor, modifying Append sequence to Isec: the tag, and writing it back, resulting in additional memory 17: auipcc cs,0; li rt,addr; csetaddr cs,cs,rt traffic and latency. We model an IMC-based tag memory in else if I is sc.ddc then 18: rs ← rs1 of I which tag updates are performed directly within the memory Append sequence to Isec: array, allowing the processor to issue a tag-modification com- auipcc ct,0; li rt,addr; csetaddr ct,ct,rt; mand without transferring full memory words. This abstraction cspecialw ddc,ct; andi rs,rs,CapSize 19: else if I is ebreak, unknown or illegal instruction then reduces data movement over the system interconnect while 20: Skip I and do not append preserving CHERI’s security semantics. 21: end if To capture the diversity of potential IMC implementations, 22: Append I to Isec 23: end for we model the tag update operation using a parametrized latency in the VP. By sweeping this parameter, we identify the latency range in which IMC-based tag updates provide a tural invariants such as valid control flow, initialization, and net performance benefit, as well as regimes where acceleration memory access semantics. While useful for bug detection, becomes ineffective. While the evaluation focuses on a single such behavior is unsuitable for performance benchmarking, optimization, it serves as an illustrative case study demon- where meaningful execution depends on architecturally valid strating how the proposed benchmarks enable quantitative instruction streams. Consequently, TestRIG output must be analysis of CHERI-specific architectural trade-offs early in the filtered and transformed before using as a benchmark. development. In a first step, we filter generated instruction streams to V. E VALUATION remove instructions and sequences that would trap or terminate execution on a correct RISC-V CHERI implementation. While For the evaluation, we integrate the CHERI extension for this yields executable instruction streams, they still do not cor- RISC-V into an open-source RISC-V VP [19], [20] and apply rectly exercise CHERI’s capability mechanisms. In a second the generated benchmarks to both a baseline CHERI VP step, we post-process the filtered streams to enforce proper ca- and an extended VP modeling an IMC-based tag update pability preparation and usage. CHERI enforces strict bounds, mechanism. The IMC functionality is abstracted and evaluated permission, and validity checks on memory accesses which, within the VP to enable early-stage performance exploration. without proper capability setup, will fail for most generated In the IMC-enabled VP, the latency of the in-memory tag memory operations. When these operations fail, they don’t update operation (referred to as IMC-time) is parametrized reach CHERI-specific hardware such as the tagged memory. from 0 ns to 30 ns to model a broad range of possible accel- To address this, we identify instruction patterns that require erator implementations. We generate three benchmarks with valid capabilities and insert or rewrite instruction sequences accelerated instruction occurrences ranging from 8 % to 20 % to construct capabilities with appropriate bounds, permissions, (referred to as TRG_8, TRG_15, TRG_20), demonstrating and tags. This ensures that capability instructions execute the configurability of the benchmark generation process. All successfully and fully utilize the CHERI-enabled hardware. experiments were performed on an Intel(R) Core(TM) Ultra 5 Algorithm 1 summarizes the instruction stream processing 125U 12-Core Processor with 4.3 GHz and 64 GB of memory. and illustrates how specific instruction types are augmented Figure 2 shows the normalized speed-up of the IMC- with capability setup sequences. After post-processing, the accelerated VP relative to the baseline (red line) while sweep- resulting security-aware instruction streams execute to com- ing the IMC-time. For an idealized IMC-time of 0 ns, speed- pletion without violating CHERI’s architectural constraints. ups between 1.06× and 1.11× are observed. Performance
1.1 process. By leveraging TestRIG and CHERI-specific instruc-
1.08 TRG_8 tion stream processing, the proposed framework enables quan-
TRG_15
1.06 TRG_20 titative performance evaluation and architectural exploration
1.04 Baseline while preserving CHERI’s security semantics. Using VPs, we
1.021 showed how the benchmarks expose both performance oppor-
0.98 tunities and inherent limits of CHERI-specific optimizations.
0.96 In a representative case study modeling an IMC-based tag
0.94 memory, the benchmarks revealed best-case speed-ups of 6 %
0.92 to 11 %, as well as latency regimes in which acceleration
0.9 0 5 10 15 20 25 30 becomes ineffective. More broadly, the proposed approach
IMC-time (ns) provides a foundation for early-stage evaluation of security
Figure 2: Processor speedup versus IMC-time for security-aware mechanisms before mature software stacks are available. Fu-
benchmarks. ture work will extend the framework toward generating more
complex program structures and richer control-flow patterns
benefits diminish as IMC-time increases, with acceleration while retaining security awareness.
becoming ineffective for latencies above approximately 20 ns. REFERENCES
280 RISC-V (pre processing) [1] A. Mosenia et al., “A comprehensive study of security of Internet-of-Things,”
240 IEEE Transactions on emerging topics in computing, vol. 5, no. 4, pp. 586–602,
200 RISC-V (post processing) 2016.
160 CHERI (pre processing) [2] Y. Lu et al., “Internet of Things (IoT) cybersecurity research: A review of current
120 CHERI (post processing) research topics,” IEEE Internet of Things Journal, vol. 6, no. 2, pp. 2103–2115,
80 ~~ ~~ [3] 2018.
V. E. Moghadam et al., “Memory integrity techniques for memory-unsafe
40 ~~ ~~ languages: A survey,” IEEE Access, vol. 12, pp. 43 201–43 221, 2024.
30 [4] J. Woodruff et al., “The CHERI capability model: Revisiting RISC in an age of
20 risk,” ACM SIGARCH Computer Architecture News, vol. 42, no. 3, pp. 457–468,
2014.
10 [5] R. Grisenthwaite et al., “The Arm Morello evaluation platform—Validating
0 CHERI-based security in a high-performance system,” IEEE Micro, vol. 43,
no. 3, pp. 50–57, 2023.
[6] A. Joannou et al., “Randomized testing of RISC-V CPUs using direct instruction
injection,” IEEE Design & Test, vol. 41, no. 1, pp. 40–49, 2024.
[7] A. Armstrong et al., “ISA semantics for ARMv8-a, RISC-V, and CHERI-MIPS,”
Proceedings of the ACM on Programming Languages, vol. 3, no. POPL, pp. 1–
RISC-V CHERI 31, 2019.
Figure 3: Instruction counts before and after security-aware instruc- [8] J. Devietti et al., “Hardbound: Architectural support for spatial safety of the
tion processing. C programming language,” ACM SIGOPS Operating Systems Review, vol. 42,
no. 2, pp. 103–114, 2008.
These results illustrate how the benchmarks support early [9] S. Nagarakatte et al., “SoftBound: Highly compatible and complete spatial
identification of performance boundaries and feasibility re- memory safety for C,” in Proceedings of the 30th ACM SIGPLAN Conference
gions for architectural optimizations. [10] on Programming Language Design and Implementation, 2009, pp. 245–258.
S. Jero et al., “TAG: Tagged architecture guide,” ACM Computing Surveys,
Figure 3 shows the instruction distribution of benchmark vol. 55, no. 6, pp. 1–34, 2022.
(TRG_20), to illustrate the effects of security-aware instruc- [11] F. Bellard, “QEMU: A generic and open source machine emulator and virtual-
izer,” 2003.
tion post-processing. Additional capability-management in- [12] S. Amar et al., “CHERIoT: Complete memory safety for embedded devices,”
structions such as csetaddr, cspecialw, auipcc and in Proceedings of the 56th Annual IEEE/ACM International Symposium on
li are introduced to ensure valid capability setup and control [13] Microarchitecture, 2023, pp. 641–653.
J. L. Henning, “SPEC CPU2006 benchmark descriptions,” ACM SIGARCH
flow. For example, capability-based jumps require both correct Computer Architecture News, vol. 34, no. 4, pp. 1–17, 2006.
capability preparation and well-defined target addresses. [14] C. Bienia et al., “The PARSEC benchmark suite: Characterization and archi-
Overall, this evaluation demonstrates that the proposed tectural implications,” in Proceedings of the 17th international conference on
Parallel architectures and compilation techniques, 2008, pp. 72–81.
benchmark framework enables quantitative analysis of [15] D. Patterson et al., “Embench IOT 2.0 and DSP 1.0: Modern embedded comput-
CHERI-specific optimizations in early development stages. ing benchmarks,” Computer, vol. 58, no. 5, pp. 37–47, 2025.
[16] O. Oleksenko et al., “Intel MPX explained: A cross-layer analysis of the Intel
Even in this focused case study, the benchmarks expose both MPX system stack,” Proceedings of the ACM on Measurement and Analysis of
achievable best-case improvements (6 % to 11 %) and funda- [17] Computing Systems, vol. 2, no. 2, pp. 1–30, 2018.
mental limits of acceleration under realistic timing constraints. M. Li et al., “NanoTag: Systems support for efficient byte-granular overflow
detection on ARM MTE,” arXiv preprint arXiv:2509.22027, 2025.
[18] X. Sun et al., “Sweet or sour CHERI: Performance characterization of the Arm
VI. CONCLUSION Morello platform,” in Proceedings of the 2025 IEEE International Symposium
Evaluating and optimizing secure embedded systems re- [19] on Workload Characterization. IEEE, 2025.
V. Herdt et al., “Extensible and configurable RISC-V based virtual prototype,”
quires benchmarks that explicitly exercise hardware-enforced in 2018 Forum on Specification & Design Languages (FDL). IEEE, 2018, pp.
security mechanisms. This work addresses the lack of such 5–16.
[20] V. Herdt et al., “RISC-V VP.” [Online]. Available: https://github.com/
benchmarks for CHERI-enabled architectures by introducing agra-uni-bremen/riscv-vp
a framework for generating security-aware instruction-level
benchmarks that are available early in the system development
Count
andi Count Speedup
lui
slli
add li
mv
addi
sh
sblblh
sc.ddc
sc.cap
cspecialrw jalr.cap csetaddr ccleartag sc cincoffset cloadtags cspecialr csetflags cspecialw jr.cap csetbounds crrl candperm cseal cinvoke cmove lc.ddc cgettag cgethigh cgetoffset auipcc cincoffsetimm