SOURCE ARCHIVE
EXTRACTED CONTENT
105,961 chars Encarsia: Evaluating CPU Fuzzers
via Automatic Bug Injection
Matej Bölcskei, Flavien Solt, Katharina Ceesay-Seitz,
and Kaveh Razavi, ETH Zurich
https://www.usenix.org/conference/usenixsecurity25/presentation/bolcskei
This paper is included in the Proceedings of the
34th USENIX Security Symposium.
August 13–15, 2025 • Seattle, WA, USA
978-1-939133-52-6
Open access to the Proceedings of the
34th USENIX Security Symposium is sponsored by USENIX.
Encarsia: Evaluating CPU Fuzzers via Automatic Bug Injection
Matej Bölcskei Flavien Solt Katharina Ceesay-Seitz Kaveh Razavi ETH Zurich ETH Zurich ETH Zurich ETH Zurich
Abstract usable and effective tools and methodologies to verify the cor-
Hardware fuzzing has recently gained momentum with many rectness and security of these hardware designs at all stages discovered bugs in open-source RISC-V CPU designs. Com- of their development. paring the effectiveness of different hardware fuzzers, how- Hardware fuzzing is a recent and popular response to this ever, remains a challenge: each fuzzer optimizes for a dif- pressing demand [5,6,9,14,22,26,29,30,32,42,45,48]. Given ferent metric and is demonstrated on different CPU designs. its scalability, ease of adoption, and demonstrated ability to Furthermore, the number of newly-discovered bugs is not an find bugs in real-world designs, hardware fuzzing has become appropriate metric since finding new bugs becomes increas- an essential tool in the hardware validation toolbox. However, ingly more difficult as designs mature. We argue that a corpus hardware fuzzing research often makes conflicting claims of automatically injectable bugs will help compare hardware as to the underlying mechanism behind their effectiveness, fuzzers to better understand their strengths and weaknesses. revealing a profound lack of understanding about what truly Through a large-scale study of 177 software-observable bugs drives fuzzing performance. in open-source RISC-V CPUs, we discover that CPU bugs can Fuzzer evaluation. We posit that this confusion stems from be modelled by manipulating conditional statements or sig- the lack of a standardized framework for evaluating hardware nal drivers. Based on this observation, we design ENCARSIA, fuzzers. Instead, existing work relies on either some coverage a framework that automatically transforms the intermediate that they achieve during the fuzzing process, or the number of representation of a given CPU design to inject bugs that are new naturally-occurring bugs they find in real-world designs. equivalent to incorrect conditions or assignments at the HDL Coverage is known to be a poor proxy for bug-finding ability level. To ensure that an injected bug has an observable ar- in software [28], and the discovery of new naturally-occurring chitectural effect, we leverage formal methods to prove the bugs is a noisy metric that is hard to compare across different existence of an architectural deviation due to the bug-specific fuzzers due to diverse tested hardware designs and versions. In transformation. We evaluate ENCARSIA by injecting bugs particular, a fuzzer that finds bugs in a given design may find into three open-source RISC-V CPUs, fuzzing these CPUs fewer new bugs due to the increasing maturity of the design, with recently-proposed CPU fuzzers, and comparing their bug- even if it is capable of finding more complex bugs. Inserting finding performance. Our experiments reveal key insights into bugs manually is a labor-intensive option [16, 40] that has to the limitations of existing hardware fuzzers, including their be repeated per design, in particular if the bug-inserted designs inability to cover large sections of the designs under test, inef- are not shared with the community. Automatic injection of fective coverage metrics, and bug detection mechanisms that bugs is a promising alternative. often miss bugs or produce false positives, highlighting the Encarsia. We introduce ENCARSIA† , the first framework urgent need to reassess current approaches. for automatically injecting realistic bugs into arbitrary RTL designs. To guide the design of ENCARSIA, we first conduct a 1 Introduction comprehensive survey of bugs reported in four popular open- source RISC-V CPUs of various complexities and design To satisfy the ever-increasing demand for diverse hardware paradigms: Ibex [33], CVA6 [20], Rocket [1] and BOOM [4]. targeting wide ranges of applications from secure Internet of We find that all these bugs resemble two simple syntactic Things (IoT) devices to high-performance computing (HPC) †Encarsia is a genus of tiny parasitic wasps belonging to the family systems, new hardware is being developed at an unprece- dented pace. This momentum has created a high demand for Aphelinidae. These wasps are commonly used in biological control programs to manage pest populations, particularly whiteflies and scale insects.
USENIX Association 34th USENIX Security Symposium 2577
transformations: mix-ups of signals or logic expressions and Seed Input ISA Simulator
errors in conditional statements. Based on this observation, Programs Mutation Program 110100010
000010011
ENCARSIA applies these syntactic transformations to auto-
matically inject diverse and realistic bugs in a given HDL Trace
design. However, such transformations may not necessarily DUT Execution Mismatch?
110100010 Yes
be reachable or result in an architecturally-visible effect. To Yes 000010011 certify that the injected bugs are observable and have a func- Delete Input tional effect, we rely on formal verification. No Increased Bug We evaluate ENCARSIA on three RISC-V CPUs, namely Coverage? Detected Ibex, Rocket and BOOM, and evaluate three state-of-the-art hardware fuzzers: DifuzzRTL [26], ProcessorFuzz [6], and Figure 1: Typical hardware fuzzer workflow. Cascade [42]. We find that these fuzzers are able to find re- spectively 41.67%, 41.67%, and 40% of the bugs injected by ENCARSIA in 24 hours of fuzzing. ENCARSIA reveals short- 2.1 Hardware representations comings in both the design and evaluation of these fuzzers: Hardware designs are represented at various levels of abstrac- (1) existing fuzzers fail to reach large sections of the designs tion throughout their development. During the design phase, due to limitations in their test generation mechanism, (2) for they are typically described using Hardware Description Lan- parts of the designs that the fuzzers do reach, existing cover- guages (HDLs) such as Verilog or VHDL, often at a high level age metrics fail at effectively guiding the fuzzers to exercise of abstraction, like the behavioral or register-transfer level those areas sufficiently, and (3) current fuzzers’ bug filtering (RTL). HDLs usually provide multiple ways of expressing mechanisms result in both false positives and false negatives. the same concept for the convenience of the designer. For in- Based on these insights, we propose future research directions stance, conditional assignments of signals can be represented for improving hardware fuzzing. For example, fuzzers might using if or case statements, or ternary operators. Automated benefit from higher-level coverage metrics, either at the ar- tools convert HDLs into an Intermediate Representation (IR) chitectural level or at the microarchitectural level, given that that abstracts away the syntactic sugar [46]. Various process- current fuzzers seem neither able to reach good ISA coverage, ing is then performed on this IR, before converting it back nor to explore all microarchitectural corner cases. into HDL or another format suitable for the next step in the Contributions. We make the following contributions: hardware design flow. For example, the front-end of the Yosys [46] open-source • We survey the bugs reported in four popular open-source synthesizer converts HDL designs into RTL Intermediate RISC-V CPUs: Ibex, CVA6, Rocket and BOOM. Language (RTLIL) representations. In RTLIL, Yosys models • We design and implement ENCARSIA based on a new hardware designs as networks of macro-cells, representing model for automatically injecting realistic bugs into RTL functional components such as logic gates or arithmetic units, designs based on multiplexer tree corruptions and wire interconnected by wires. Furthermore, Yosys models Verilog connection mix-ups, and a formal methodology to ensure signal assignments [27] as connections between a wire and it’s that injected bugs are architecturally observable. driver signal that supplies it with a logic value. Driver signals can be constants, wires, or combinations thereof. Yosys trans- • We implement and evaluate ENCARSIA by injecting bugs forms conditional assignments into multiplexer trees where in three popular open-source RISC-V CPUs and assess- the assigned signal is connected to the output, the results of ing three state-of-the-art hardware fuzzers. Our assess- each branch are connected to the inputs, and the conditions ment provides insights into the limitations of existing controlling the assignment are connected to the select sig- hardware fuzzers and proposes future research direc- nals. Yosys then applies further transformations to the RTLIL tions. representation through procedures known as passes (e.g., to Open sourcing. ENCARSIA is open source and we have implement optimizations). put particular attention in making it user-friendly and well- 2.2 Hardware fuzzing documented to facilitate its adoption by the community. Ad- Hardware fuzzers generate randomized inputs in an attempt ditional information can be found at the following URL: to trigger and observe bugs in designs under test (Figure 1). https://comsec.ethz.ch/encarsia. They can target CPUs [5, 6, 9, 14, 22, 26, 29, 30, 42, 48] or 2 Background more generic hardware [32, 45]. CPU fuzzers leverage the well-defined behavior of a CPU, as specified by the ISA, to We provide the necessary background on hardware design test meaningful variations of valid instruction sequences, es- representations (Section 2.1), hardware fuzzing (Section 2.2) sentially programs, rather than relying on random binary data and manual bug insertion (Section 2.3). that rarely triggers CPU functionality. Typically, fuzzers gen-
2578 34th USENIX Security Symposium USENIX Association
Table 1: Evaluation methodologies adopted by some recent 3 Overview of Challenges hardware fuzzers. Cov.: coverage evaluation. Nat.: discovery We provide an overview of the challenges that guided the of natural bugs. Synth.: discovery of synthetic bugs. design of ENCARSIA. First of all, it is necessary to understand Fuzzer Cov. Nat. Synth. the properties of natural bugs, since evaluating fuzzers on RFUZZ [32] ✓ ✗ ✗ synthetic bug corpora must reflect the fuzzer’s ability to detect DifuzzRTL [26] ✓ ✓ ✓ bugs that occur in real designs. DirectFuzz [9] ✓ ✗ ✗ CHALLENGE 1. Trippel et al. [45] ✓ ✗ ✓ Understand and unify the characteristics of natural bugs. TheHuzz [30] ✓ ✓ ✗ ProcessorFuzz [10] ✓ ✓ ✗ MorFuzz [48] ✓ ✓ ✗ In Section 4, we conduct a comprehensive survey of 1672 Cascade [42] ✓ ✓ ✗ pull requests across four popular RISC-V CPUs of various complexities and from different maintainers. We find that bugs with highly-complex effects can often be traced back erate new programs through mutation, such as flipping bits, to simple code modifications. We further observe that two swapping operands, or rearranging instructions to explore new types of transformations are sufficient to capture them. The CPU behaviors. To guide this process, fuzzers use coverage first transformation involves signal mix-ups, such as the use feedback, which tracks the parts of the ISA or CPU exer- of incorrect operators in expressions or assignments. The cised by each test. Programs are executed on the CPU, and if second transformation pertains to broken conditionals, which they increase coverage, they are retained for further mutation, encompass incorrect if conditions or the absence of a default otherwise, they are discarded. Several coverage metrics have case in a switch statement, among other issues. been proposed, such as multiplexer control coverage [32], con- The second challenge concerns the translation of these trol register coverage [26], and aggregated simulator-reported source-level observations to the practical injection of bugs. coverage [30]. CPU fuzzers detect bugs by comparing a CPU’s behavior CHALLENGE 2. to a reference model such as an Instruction Set Architecture Automatically and realistically transform RTL designs. (ISA) simulator [26, 30, 42]. These fuzzers often advertise the number of new bugs they find in real-world designs to In Section 5, we present ENCARSIA, a fully automatic and demonstrate their effectiveness [5, 6, 9, 14, 22, 26, 29, 30, 42, easy to use bug injection framework based on the observa- 48], yet this metric does not easily lend itself to comparison tions made in our survey. We leverage Yosys’ intermediate across different fuzzers due to the diversity of tested hardware representation (IR) to manipulate wire connections and multi- designs and versions. Furthermore, bugs become harder to plexer trees that respectively represent expression operands find as designs undergo more testing and become more mature. and conditionals. Concretely, ENCARSIA takes an RTL de- Table 1 summarizes evaluation methodologies of some recent sign, injects realistic bugs in the IR provided by Yosys, and hardware fuzzers. We distinguish natural bugs found in real- outputs the modified design. world designs from synthetic bugs that are manually inserted. The third challenge concerns the quality of injected bugs. 2.3 Manual bug insertion CHALLENGE 3. To inject bugs into hardware designs, researchers have been Ensure that the injected design transformations are archi- so far relying on manual insertion, in two distinct contexts. tecturally observable. The first context is the insertion of specific example bugs to demonstrate the effectiveness of one specific fuzzer at In Section 6, we employ formal verification to ensure that detecting them [24–26, 45]. These ad-hoc bugs are few (not the injected transformations have an architecturally observ- more than 7) and might be biased, given that they are built able effect. We provide two setups: one more performance- specifically to illustrate a given fuzzer’s effectiveness. In oriented using the closed source JasperGold and one more particular, these very few bugs are never later used by others accessible to the community using Yosys. Aware of the in- for comparison. The second context is the manual insertion of jected transformations, ENCARSIA adopts a two-step assume- some dozens of bugs to organize competitions [16]. To inject guarantee approach to guide the formal verification process realistic and diverse bugs, this laborious manual process relied of Yosys. To ensure that one bug will not hide another, we on industry partnerships and strong familiarity with the target limit the injection to a single bug per design at a time. design. Consequently, the design used for competitions has been CVA6 for years [49] and these manually-injected bugs Overcoming these challenges, we provide the first open are few and generally not disclosed to the public [40]. synthetic corpus of CPU bugs in Section 7 and derive new insights about hardware fuzzing in Section 8.
USENIX Association 34th USENIX Security Symposium 2579
Table 2: Overview of surveyed CPUs, detailing data width 1 a s s i g n illegal_csr_insn_o = (DW), pipeline stages (St), out-of-order execution (OoO) ca- 2 illegal_csr | illegal_csr_priv ; pability, hardware description language (HDL) used, as well 3 a s s i g n csr_we_int = 4 -- csr_wreq & ~ illegal_csr_priv & ...; as wire and cell counts. 5 ++ csr_wreq & ~ illegal_csr_insn_o & ...; Design DW St OoO HDL Wires Cells Listing 1: Example of a signal mix-up in PR #399 of the Ibex [41] 32 3 N SV 21.6k 24.5k Ibex CPU. The designer mistakenly uses a partial result of an CVA6 [49] 64 6 N SV 903k 945k access check instead of the final result, allowing unauthorized Rocket [2] 64 5 N Chisel 495k 781k access to debug CSRs. BOOM [13] 64 10 Y Chisel 880k 1320k and the HDL-level fix (Verilog or Chisel) of each commit in Table 3: Summary of manually analyzed PRs, with bugs iden- the PRs and determine the architecturally-observable bugs. tified as causing software-observable ISA deviations. We analyzed a total of 310 code changes. Table 3 indicates the number of pre-filtered PRs, the range of PR numbers (#) Pull Requests (PRs) Observable and the number of observable bugs. Design Pre-filtered Oldest Newest bugs Manually assembled corpora. For evaluating verification Ibex 42 PR #31 PR #343 31 methodologies Dessouky et al. [16] manually compile a set of CVA6 41 PR #27 PR #225 33 hardware bugs, which they claim to be uniquely challenging Rocket 50 PR #13 PR #592 49 for state-of-the-art verification techniques. While the original BOOM 41 PR #7 PR #512 46 corpus presented in the paper provided only brief descriptions of the bugs, later iterations of the competition included some Total 174 159 bug fixes. The HACK@DAC 2019 corpus featured 4 such bug fixes, while the HACK@DAC 2021 corpus expanded this 4 Understanding the Causes of Bugs by 26. To ensure that ENCARSIA is generic enough to also model these selected bugs, we include them in our survey. To understand the characteristics of natural bugs found in real-world CPUs, we survey bugs in four popular open- 4.2 Bug classification source designs. We also survey bugs manually crafted for Our goal is to identify recurring structural patterns among the the HACK@EVENT competitions [16]. We focus on bug bugs, enabling their automatic injection through simple circuit patches, as they provide root-cause information. transformations. In an initial exploratory phase of our survey, 4.1 Collecting natural and synthetic bugs we observed that bugs affect a wide range of components and We describe our methodology for collecting bugs from open- HDL constructs. Yet, despite this diversity, we found that the source designs and manually assembled corpora. affected HDL constructs fall into two groups: wrong logic in signal drivers and mistakes in conditionals. This led us to 4.1.1 Open-source designs define the following categories. We collect bugs from four open-source CPUs of various Signal mix-ups. Signal mix-ups encompass confusion of complexities and design paradigms (Ibex [33], CVA6 [20], signals, functions, or other elements in assignments or expres- Rocket [1], BOOM [4]) based on their public issue trackers, sions. They usually arise from human errors due to signals summarized in Table 2. The brief description of the bugs’ having similar names, types, or purpose. An example of a symptoms provided in the issue reports is often insufficient signal mix-up observed during the initial exploratory phase to determine root causes and filter out non-observable bugs. (Ibex PR #399) is shown in Listing 1. In this example, the Hence, we focus on Pull Requests (PRs) and filter them for designers mistakenly used a partial result of an access check actual bug fixes. Using GitHub’s REST API [19], we collect instead of the final result, allowing unauthorized access to the the first 100 PRs per design containing the keyword ’fix’ and debug CSRs. filter them to include only those that modify design source Broken conditionals. Broken conditionals refer to incorrect files. We opt for keyword-based filtering over more principled conditional statements, for example mishandling of excep- options, such as GitHub labels, due to their inconsistent usage. tional cases like special values in floating-point arithmetic, For instance, only 4 out of 1529 CVA6 PRs are labeled as or signal updates in wrong design states. These usually arise bugs. Keyword filtering yields a substantial bug dataset and from designers either forgetting an exceptional case or spec- proves unbiased by capturing 20 of 27 labeled bugs in BOOM. ifying wrong conditional expressions, or generally making Some PRs may still be false positives, e.g., performance im- algorithmic mistakes. A simplified example of a broken con- provements, while others address multiple bugs, with up to 65 ditional observed during the initial exploratory phase (Ibex commits. Therefore, we manually examine the textual report PR #277) is shown in Listing 2. In this example, the designers
2580 34th USENIX Security Symposium USENIX Association
1 always_comb b e g i n 1 module cbc ( 2 csr_rdata_int = ’0; 2 i n p u t logic clk , 3 illegal_csr = 1’ b0 ; 3 i n p u t logic reset , 4 unique c a s e ( csr_addr_i ) 4 i n p u t logic valid_i , 5 CSR_MSTATUS : csr_rdata_int = mstatus_q 5 i n p u t logic enable_i , 6 CSR_DCSR : b e g i n 6 i n p u t block_t plaintx_i , 7 csr_rdata_int = dcsr_q ; 7 o u t p u t block_t ciphertx_o 8 ++ illegal_csr = ~debug_mode_i; 8 ); 9 end 9 state_t state , next_state ; 10 d e f a u l t : b e g i n 10 block_t prev_cipher_blk ; 11 illegal_csr = 1’ b1 ; 12 end 11 // bug M1: wrong continuous operand choice 13 e n d c a s e 12 -- a s s i g n cipher = plaintx_i ^ state; 14 end 13 ++ a s s i g n cipher = plaintx_i ^ prev_cipher_blk; 14 Listing 2: Example of a broken conditional derived from pull 15 // bug M2: forgotten operand request #277 of the Ibex CPU. The designer forgot to include 16 a s s i g n cipher_valid = enable_i ; a check to ensure the CPU is in debug mode upon access to 17 ++ a s s i g n cipher_valid = enable_i && debug CSRs, which allows unintended access. 18 ++ (state == ENCRYPT); 19 20 21 // States forgot a check to ensure that the CPU is in debug mode upon 22 always_ff @( posedge clk , posedge reset ) b e g i n access to debug CSRs, which allows users unintended access. 23 i f ( reset ) b e g i n 24 state <= IDLE ; 25 prev_cipher_blk <= ’0; 4.2.1 Classification 26 end e l s e b e g i n 27 state <= next_state ; We manually classify each bug fix identified in our survey 28 // bug M3: wrong register assignment 29 -- prev_cipher_blk <= plaintx_i; into the two categories: signal mix-ups or broken condition- 30 ++ prev_cipher_blk <= cipher; als. If they contain multiple modifications, including both 31 ciphertx_o <= cipher_valid ? cipher : ’0; types, we classify them as both. Listing 3, which resembles 32 ... 33 an excerpt from a simplified cipher block chaining crypto- 34 end graphic module, illustrates our classification approach. Any 35 end erroneous assignment of a signal, function, or other entity in 36 // Next state logic assignments or expressions (outside conditions) is classified 37 always_comb b e g i n 38 as a signal mix-up. Bug M1 in Listing 3 is a name mix-up 39 next_state = state ; in a continuous assignment, where one of the operands of its 40 c a s e ( state ) assigned logic expression is incorrectly chosen (like Rocket 41 IDLE : b e g i n PR #349). Bug M2 shows a more complex case of a mix-up of 42 // bug C1: forgotten conditional 43 ++ if (valid_i) begin a whole logic expression (state == ENCRYPT && enable_i) 44 next_state = ENCRYPT ; with a single signal (enable_i) (like BOOM PR #437). Bug 45 ++ end M3 shows a mix-up in a clocked assignment and within a 46 end conditional statement (like Ibex PR #286). 47 ENCRYPT : 48 // bug M4: wrong state transition We classify bugs affecting conditional assignments as bro- 49 -- next_state = IDLE; ken conditionals when they involve missing cases or incorrect 50 ++ next_state = ENCRYPT_FINAL; conditions. Bug C1 of Listing 3 is an example where a nested 51 // bug C2: missing case 52 ++ ENCRYPT_FINAL: begin conditional is forgotten (like CVA6 PR #138). This implies 53 ++ if (!valid_i) begin that the condition for the resulting assignment, in this case of 54 ++ next_state = IDLE; next_state, is too loose. Similar bugs might, for example, 55 ++ end 56 ++ end create access control vulnerabilities. Bug C2 exemplifies a 57 e n d c a s e missing case in a switch statement (like BOOM PR #173). 58 end This bug exemplifies forgetting a certain edge case such as a 59 ... reset for a register, a default case, or an overflow check. 60 endmodule Some bugs, like bug M4, are signal mix-ups in the body of a Listing 3: Simplified cipher design with six bugs. Bugs M1- conditional statement (like Ibex PR #169). To avoid overlaps M4 are mix-ups, corresponding to wrong operand choices, between the two categories, we classify bugs that concern with different contexts and sensitivities (continuous or clocked solely the conditionally assigned values, but do not modify assignment). Bugs C1-C2 are broken conditionals. any conditions, as signal mix-ups.
USENIX Association 34th USENIX Security Symposium 2581
Table 4: Number of bug fixes per design and category. illegal_csr csr_wreq csr_we_int
Source Conditionals Mix-ups Total
Ibex [41] 17 14 31 illegal_csr_priv
CVA6 [49] 22 11 33
Rocket [2] 8 41 49 Figure 2: Intermediate representation of logical expressions
BOOM [13] 13 33 46 in Yosys, as derived from Listing 1.
HACK@DAC19 2 2 4
HACK@DAC21 4 10 14 illegal_csr $1 csr_wreq $7 csr_we_int
Total 66 111 177 $2 $3 $4 $5 $6 $8
illegal_csr_priv
Results. Table 4 summarizes the results of our survey, con- Figure 3: Injecting a signal mix-up into a logical expression firming that all identified observable bugs indeed do fall into (Figure 2). Intermediate wires are shown in red. After injec- one of the two categories. Of the 177 relevant bugs found in tion, csr_we_int will be erroneously high if csr_wreq and our survey, 111 (63%) were signal mix-ups, 66 (37%) were illegal_csr are both high. broken conditionals. Appendix A provides some examples per category and CPU. Security implications. We further studied whether the bugs only replace drivers to avoid introducing statically detectable could have security implications by comparing its effects to bugs such as multi-driven wires. ENCARSIA randomly selects those of hardware bugs that have previously been assigned two such connections and replaces the driver of the first con- CVE numbers [10, 12, 26, 42, 48]. In conclusion, all architec- nection with that of the second. When the newly connected turally observable bugs could impact security in one way or signals differ in width, ENCARSIA truncates the wider signal another (with different severities), because attackers may de- to match the width of the narrower one. If the new driver liberately create conditions where the bug leads to an attacker- is a constant smaller than the driven wire, we sign-extend chosen architectural effect. it. This is consistent with the implicit Verilog behavior for mismatching signal widths [27]. 5 Transforming Hardware Logical expressions. Expressions are translated into a series Given our newfound knowledge of bugs, we proceed to the of logical cells interconnected by signals (Figure 2). Mix-ups design of ENCARSIA. We implement ENCARSIA as a series in expression operands therefore correspond to a wrong signal of netlist transformation passes in the Yosys Open SYnthesis being connected to a cell port. Each cell keeps its own internal Suite [46]. We start by injecting signal mix-ups through swaps dictionary that maps ports to the connected signals. To unify on signal connections (Section 5.1), and then reproduce bro- the injection of mix-ups in simple assignments and logic ken conditionals by corrupting multiplexer trees (Section 5.2). expressions, we insert an intermediate wire between each cell For each bug category, we describe the affected hardware port and the signal originally connected to it (Figure 3). This structures in the intermediate representation of Yosys and way, injecting a mix-up at the intermediate wire is equivalent how to model the bugs within this representation. to injecting a mix-up at the cell port. Intermediate wires are later cleaned up using a standard optimization pass. 5.1 Signal mix-ups As discussed in Section 4, signal mix-ups are erroneous uses 5.2 Broken conditionals of signals, functions, or other entities in assignments or ex- Broken conditionals refer to issues arising from improper pressions. We now explain how ENCARSIA injects signal handling of exceptional cases or, e.g., operations allowed only mix-up transformations in assignments and expressions. in a specific privilege mode. Yosys represents conditionals Assignments. In the intermediate representation of Yosys, as- as multiplexer trees, a hierarchical arrangement of multiplex- signments, known as connections, are pairs of wires with their ers where each level progressively narrows down the set of driver signals. For instance, the Verilog statement assign possible expressions assigned to the signal at the root of the sig_a = sig_b; is represented as (sig_a, sig_b). We in- tree (Figure 4). At each level of the multiplexer tree, the select ject signal mix-ups by replacing driver signals (the right-hand signals, derived from the conditions of the assignment, deter- side of assignments) with other random signals present in the mine the appropriate branch to follow, ultimately selecting same RTL module. This resembles replacing a driver with the correct expression to assign. In this representation, broken the logic expression, constant or signal that is assigned to conditionals correspond to a potential branch missing from the new driver and therefore models arbitrary logic bugs. We the multiplexer tree, or an incorrectly derived select signal that
2582 34th USENIX Security Symposium USENIX Association
~debug_mode_i 1'b0 illegal_csr Table transformations. Transformations of the tables can represent realistic subtle mistakes that designers make, accord- 1'b1 ing to our survey. Removing a single row from the table can model a designer forgetting to consider a special case. Turn- CSR_DPC CSR_MSTATUS ing a select signal into an X (known as “don’t care” value) can model cases where designers place insufficient constraints on Figure 4: Intermediate representation of conditional assign- a case. Conversely, turning a “don’t care” select signal into a ments in Yosys, as derived from Listing 2. CSR_DPC and specific value can model bugs where constraints are placed CSR_MSTATUS are select signals, illegal_csr is the root of on cases mistakenly. These transformations cover the entirety the tree. of the bugs in conditional logic that we revealed in the survey. CSR_MSTATUS CSR_DCSR illegal_csr Transformed circuit construction. Once manipulated, the tables must be translated from their abstract representation 1'b1 1'bX 1'b0 back into standard circuitry. This allows the circuit to be 1'b0 1'b1 ~debug_mode_i simulated, instrumented, or otherwise analyzed by fuzzers (or other testing tools). We achieve this by translating the 1'b0 1'b0 1'b1 table into a Yosys internal pmux cell, which is a many-input multiplexer that utilizes one-hot encoding (where only one bit Figure 5: ENCARSIA’s table representation of conditional is high per possible select value) [27]. Each row in the table assignments. The columns on the left of the arrow represent represents a possible case in the conditional assignment. We select signal combinations, the column on the right contains want an entry’s assignment expression to propagate exactly the expression that is selected for the multiplexer tree root. when the select signal matches the corresponding select values in the table. Thus, we connect the expression to one of the inputs of the pmux. The corresponding one-hot select bit is does not match the condition of the assignment. Hence, this created by adding an eq cell that compares the select signal abstraction allows handling complex conditional assignments combination from the table to the actual select signal value. with a unified injection approach. The output of the pmux cell is then connected to the root Table representation. ENCARSIA abstracts these multiplexer wire that was originally driven by the multiplexer tree. The trees into a table representation (Figure 5). The table maps Yosys backend for translating RTLIL back into Verilog then specific select signal combinations to the corresponding ex- translates the pmux cell into a process resembling the original pression chosen in that case. This abstract representation conditional statement. allows ENCARSIA to inject realistic complex transformations into conditional assignments by performing simple operations 6 Verification of Observability on the table, such as removing or adding cases, thus avoiding the need to modify the circuit following each operation. The transformations outlined in Section 5 do not necessarily Table construction. ENCARSIA iterates over all wires in the introduce observable bugs, as they do not consider the higher- circuit to identify multiplexer trees. It marks as multiplexer level semantics of the modified circuitry. Consider a CPU roots those wires that are driven by a multiplexer but do not employing triple modular redundancy to safeguard against drive any other multiplexer. These wires correspond to signals single event upsets. A transformation that corrupts the major- that are conditionally assigned with different expressions. ity voting mechanism to select the minority value instead of To construct the table, ENCARSIA performs a depth-first the majority is undetectable by current CPU fuzzers. This is search of the multiplexer tree, starting at the root. During this because they do not model single event upsets in the simula- traversal, ENCARSIA records the select signal values required tion, so redundant circuits always produce identical outputs to follow a given path. Every time ENCARSIA reaches an and reach the correct consensus. Including these transfor- input signal that is not itself an output of a multiplexer, it mations in our evaluation corpus would skew the results by discovers a new potential expression to be assigned to the making fuzzers search for innocuous transformations. root. Thus, it adds a new row to the table with the discovered Miter circuits. To address this challenge, we leverage miter input signal and the recorded select signal values, expanding circuits [3], which provide the original and transformed De- the table’s columns as necessary and filling existing rows sign Under Test (DUT) with the same inputs at all times, and with X (i.e., “don’t care” value) for the newly added columns. monitor for differences in the observable outputs, e.g., the ar- This process is repeated for each path in the multiplexer tree, chitectural registers. Given that the two versions of the DUT resulting in a table that contains all possible conditionally only differ in the transformation performed by ENCARSIA, chosen expressions (right most column in Figure 5) with their any difference in the observable elements proves that this specific select signal combinations (left columns in Figure 5). transformation has an observable effect. This approach allows
USENIX Association 34th USENIX Security Symposium 2583
1 p r o p e r t y propagated ; Table 5: Number of transformations (#Transf.) and average 23 host_observables != reference_observables ; time required for a transformation (Avg. T.). e n d p r o p e r t y 45 c_propagated : c o v e r p r o p e r t y ( propagated ); Design Mix-ups Conditionals 6 p_boot : assume p r o p e r t y ( in_boot_addr_i == 0); #Transf. Avg. T. #Transf. Avg. T. 7 p_hart : assume p r o p e r t y ( in_hart_id_i == 0); Ibex [41] 1210 213 ms 1111 88 ms Listing 4: SVAs for proving the observability of injected bugs. Rocket [2] 931 134 ms 718 65 ms BOOM [13] 1230 886 ms 982 407 ms us to define a bug observation in a more formal and precise manner using SystemVerilog Assertions (SVAs), specifically 7 as discrepancies in the values of visible elements between the Evaluation two DUTs (c_propagated in Listing 4). Formal tools like We first generate a fuzzer evaluation corpus, EnCorpus and JasperGold [8] can effectively explore all possible states and evaluate ENCARSIA in terms of injection and verification transitions to find cover traces or disprove the coverability of performance (Section 7.1). Next, we detail our methodology such design properties. for evaluating fuzzers with EnCorpus and validate it using an Two-step proof approach. Model checking is expensive example fuzzer and target design [41] (Section 7.2). We then in terms of computations and memory. Its performance can evaluate three open-source hardware fuzzers using EnCorpus be boosted by providing invariants [11, 17] or partitioning the and conclude with an extensive discussion of the insights problem using assume-guarantee reasoning [15]. Generally, gained from this evaluation and concrete recommendations automatically deriving invariants is hard [17, 47]. However, for future research into hardware fuzzing (Section 8). when transforming the CPU, ENCARSIA is aware of one key Evaluation setting. We evaluate ENCARSIA on a machine information: the trigger. For example, in a multiplexer tree equipped with two AMD EPYC 7H12 processors, each run- transformation, we know that the transformation can only ning at 2.6 GHz (256 logical cores in total) and 1 TB of affect the design’s output once the modified rows of the cor- DRAM. The injection is implemented as a series of passes responding table are selected. Hence, to reduce the search in the Yosys Open SYnthesis Suite [46] (v0.37, commit space for the formal verifier, ENCARSIA adopts a two-step a5c7f69e). For verification, we utilized JasperGold [8] (ver- assume-guarantee approach. First, we instruct the SAT solver sion Jasper Apps 2022.09p001, 64-bit) running its four default to find inputs that satisfy the trigger condition, using a prop- engines in parallel across four cores. Time measurements ob- erty similar to c_propagated but on the mixed-up signal tained from parallel testing are reported as the aggregate of instead of the architecturally visible signal. Second, if the first the durations recorded by each individual core. step succeeded, we constrain the SAT solver to preserve the We evaluated DifuzzRTL [26] (commit d2dc9f82), Proces- triggering sequence and instruct it to propagate the discrep- sorfuzz [10] (commit de6b7ef5) and Cascade [42] (commit ancy to a user-defined list of architecturally-visible signals. e916c7e0) with their default parameters. We chose these Formal setup. To limit the search space and avoid false three fuzzers as they are among the few open-source fuzzers. positives due to uninitialized states, we constrain the formal Additionally, they represent different approaches to CPU setup to resemble the initialization done by fuzzers. We ini- fuzzing: DifuzzRTL and ProcessFuzz are guided by distinct tially reset the CPUs and configure several critical control and coverage metrics, while Cascade is black-box. They also have status registers to predefined values. For example, we set the different algorithms for generating instructions sequences. We FS field of the mstatus register to 1 to enable floating-point intend to uncover the impact of these design decisions. instructions. We model these configurations with initial value abstractions and constraints for the first clock cycle to accel- 7.1 EnCorpus erate verification. We also seek to prevent the verifier from using debug or testing features that are typically encapsu- We evaluate ENCARSIA by generating EnCorpus, a unified lated within an SoC and therefore inaccessible to fuzzers. To corpus of 90 bugs for evaluating hardware fuzzers. this end, we restrict the considered input behavior using SVA Injection. We measure the performance of ENCARSIA by assume statements. For example, we limit Ibex to a single injecting bugs into Ibex, Rocket, and BOOM. We injected hardware thread (p_hart in Listing 4). approximately 1’000 transformations per design and category Formal verification. For verification, we instruct Jasper and summarize the results in Table 5. We observe that even for to cover the aforementioned c_propagated property, which a complex design like BOOM, ENCARSIA requires on average proves that the transformation can provoke an architecturally less than a second for performing each transformation. observable deviation from the original design. On the other Another important aspect of the injection process is the hand, if the property is not covered, the transformation is diversity of transformations that can be applied to the design. deemed ineffective and excluded from the evaluation. We have identified in Section 4 that bugs can be modeled by
2584 34th USENIX Security Symposium USENIX Association
Table 6: Verification performance of ENCARSIA. We report the number of transformations that we tried to verify (#Transf.), the verification success rate (Succ. %), and the average time taken to verify a single bug (Avg. T.). Design #Transf. Mix-ups Conditionals Succ. % Avg. T. #Transf. Succ. % Avg. T. Ibex [41] 116 45 % 211 s 1065 4.5 % 116 s Rocket [2] 242 44 % 194 s 396 8.1 % 117 s BOOM [13] 276 24 % 258 s 982 2.3 % 195 s
two types of transformations: signal mix-ups and broken con- Table 7: Cascade’s bug detection performance on Ibex. The ditionals. We injected 3371 signal mix-ups that impact 2482 table shows whether each bug was detected (✓) or not (✗). signals spread across 121 distinct modules. We injected 2811 broken conditionals that impact 202 multiplexer trees spread Bug 1 2 3 4 5 6 7 8 across 37 distinct modules. This underlines the diversity of Mix-ups ✓ ✗ ✗ ✓ ✓ ✗ ✗ ✗ locations that can be affected by ENCARSIA. Cond. ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ Verification. We evaluate the verification performance by Bug 9 10 11 12 13 14 15 Total measuring both the failure rate and the time required to verify Mix-ups ✓ ✓ ✓ ✗ ✓ ✓ ✓ 9/15 observability. We run JasperGold for the transformations and Cond. ✓ ✗ ✗ ✗ ✗ ✗ ✗ 1/15 set a timeout of 40 core minutes. The verification is consid- ered successful if JasperGold successfully covers the property shown in Listing 4. The non-successful proofs timed out. the number of times a bug is detected by a fuzzer, i.e., the Table 6 summarizes the results. The percentage of mix-ups number of fuzzer-provided inputs that trigger the bug. successfully verified by Jasper lies between 23% and 45% and decreases with increasing design complexity. The percentage Eliminating false positives. The underlying designs might of verified conditionals is consistently low across the designs, be affected by natural bugs. To avoid imprecisions in the i.e., between 2.3% and 8.1%. While a larger time-out may survey, we filter them out by discarding fuzzing inputs that increase this number, we trade-off for overall verification time, signal a bug in both the non-injected and the injected version. since, as we will see in the following, the verified bugs are In total, 1% of inputs produced false positives for DifuzzRTL diverse enough to evaluate differences between fuzzers. and were hence filtered out. Results on Ibex. We first apply our evaluation methodology 7.2 Fuzzer evaluation methodology on Cascade using 15 (verified) bugs per category injected into Ibex. Each bug was tested by Cascade for the duration of We present our fuzzer evaluation methodology on bugs in- 24 hours on a single core. The results of the evaluation are jected by ENCARSIA, followed by a case study on Ibex [41], presented in Table 7. Cascade successfully identified 9 out of only supported by Cascade. the 15 signal mix-ups and 1 out of the 15 broken conditionals. Attribution. A key challenge in evaluating fuzzers is at- Some bugs were harder to detect than others. Over 24 tributing deviations from the expected behaviour to the spe- hours, the easiest bugs were detected by 99 % of the programs, cific bugs responsible for them [31]. When a single designs while the hardest by only 0.02 % (6 out of 30’239 programs). is affected by multiple bugs without an accurate attribution, This matches the results of the original Cascade evaluation, a fuzzer that repeatedly triggers the same bug might appear where most bugs were found in the first 10 core minutes, more effective than another fuzzer that identifies multiple while some required 17 core hours [42]. This corroborates distinct bugs within the same design. Although various au- the representativity of the bugs injected by ENCARSIA. If tomated techniques for crashing input deduplication have Cascade was 10x slower, it could have missed some bugs over been proposed in software, they have generally proven to be 24 hours. Does this imply that higher performance would lead ineffective [31]. Additionally, no such methods have been to detection of even more bugs? In Section 8, we evaluate proposed for hardware. As a result, recent studies on hard- factors that affect fuzzers’ bug detection ability. ware fuzzing have relied on manual examination of flagged traces [10, 26, 30, 42, 48]. To eliminate this labor-intensive process, we inject one bug per design at a time. 8 Insights into Hardware Fuzzing Uninterrupted fuzzing. To provide a measure of difficulty for a fuzzer to detect a bug, we do not interrupt the fuzzing We explore various aspects of hardware fuzzing to understand campaign once a bug is detected. This allows us to measure the key factors that determine whether a bug will be detected.
USENIX Association 34th USENIX Security Symposium 2585
Table 8: Bugs detected by DifuzzRTL (DF) and Processor- Table 9: Bugs detected by DifuzzRTL (DF) and Processor-
Fuzz (PF) with coverage disabled. Fuzz (PF) with coverage enabled.
Rocket BOOM Rocket BOOM
Bug Mix-ups Cond. Mix-ups Cond. Bug Mix-ups Cond. Mix-ups Cond. DF PF DF PF DF PF DF PF DF PF DF PF DF PF DF PF 1 ✗ ✗ ✗ ✗ ✓ ✓ ✓ ✓ 1 ✗ ✗ ✗ ✗ ✓ ✓ ✓ ✓ 2 ✓ ✓ ✗ ✗ ✓ ✓ ✓ ✓ 2 ✓ ✓ ✗ ✗ ✓ ✓ ✓ ✓ 3 ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ 3 ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ 4 ✓ ✓ ✗ ✗ ✓ ✓ ✗ ✗ 4 ✓ ✓ ✗ ✗ ✓ ✓ ✗ ✗ 5 ✗ ✗ ✗ ✗ ✓ ✓ ✗ ✗ 5 ✗ ✗ ✗ ✗ ✓ ✓ ✗ ✗ 6 ✗ ✗ ✓ ✓ ✗ ✗ ✗ ✗ 6 ✗ ✗ ✓ ✓ ✗ ✗ ✗ ✗ 7 ✗ ✗ ✓ ✓ ✗ ✗ ✓ ✓ 7 ✗ ✗ ✓ ✓ ✗ ✗ ✓ ✓ 8 ✗ ✗ ✗ ✗ ✓ ✓ ✗ ✗ 8 ✗ ✗ ✗ ✗ ✓ ✓ ✗ ✗ 9 ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ 9 ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ 10 ✗ ✗ ✓ ✓ ✓ ✓ ✗ ✗ 10 ✗ ✗ ✓ ✓ ✓ ✓ ✗ ✗ 11 ✗ ✗ ✗ ✗ ✓ ✓ ✓ ✓ 11 ✗ ✗ ✗ ✗ ✓ ✓ ✓ ✓ 12 ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✓ 12 ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✓ 13 ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ 13 ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ 14 ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ 14 ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ 15 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ 15 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Tot. 5 5 5 5 9 9 6 6 Tot. 5 5 5 5 9 9 6 6
8.1 Granularity of differential fuzzing 8.2 Coverage metrics Differential fuzzers such as DifuzzRTL [26], Processor- Recently, several papers proposed using hardware-specific Fuzz [10], TheHuzz [30] and HyPFuzz [14] adopt various coverage metrics to guide fuzzers towards uncovering new policies to detect divergences between the design under test bugs. The generic hardware fuzzer RFUZZ [32] observes mul- and a golden model. We do not have access to TheHuzz and tiplexer select coverage to guide the fuzzer to explore new HyPFuzz, and the authors did not respond to our questions. paths through the design. DifuzzRTL [26] found that multi- We are therefore constrained to exclude these two fuzzers plexer select coverage is flawed because multiplexers are not from our evaluation. DifuzzRTL and ProcessorFuzz are built clock sensitive, resulting in the capture of irrelevant interme- on a similar instruction generation infrastructure and thus al- diate signals. Hence, DifuzzRTL relies on control register cov- low for a fair comparison. DifuzzRTL places a read function erage, which tracks the states of clocked registers driving the at the end of each program that reads the register file, CSRs select signals. ProcessorFuzz [10] suggests that much of these and some other architecturally visible elements to be used for registers are datapath-related and that the state of the CPU is the comparison. ProcessorFuzz continuously logs the values better captured by monitoring CPU specific control and status of several manually defined internal signals, with the intuition registers (CSRs). The other family of coverage-guided CPU that a bug symptom might later be shadowed. We evaluate fuzzers [14, 30], which are not open-source, relies on a sum detection granularity on Rocket and BOOM, which are the of simulator-provided coverage metrics. In this experiment, only designs supported by both fuzzers. We will evaluate the we evaluate whether the two state-of-the-art coverage metrics fuzzers on 15 bugs per category on each design, for 24 hours introduced by DifuzzRTL and ProcessorFuzz effectively help each, on a single core. fuzzers discover new bugs. Results. We evaluate DifuzzRTL and ProcessorFuzz on the Results. We evaluate DifuzzRTL and ProcessorFuzz on the aforementioned corpus of bugs with coverage initially dis- same set of bugs, this time with coverage enabled. We do not abled to prevent any coverage-induced bias. Table 8 summa- evaluate RFUZZ, which is by construction not able to detect rizes the results of the evaluation and shows that both fuzzers bugs [32]. Table 9 shows that both fuzzers performed equally found exactly the same set of bugs (5/15 mix-ups, 5/15 condi- well, detecting exactly the same set of bugs as when coverage tionals). This leads us to the first insight of this evaluation: is disabled (Table 8). This leads us to the second insight: Insight 1. Instruction-granular bug detection mechanisms Insight 2. The hardware-specific structural coverage met- do not demonstrate greater potential for detecting bugs. rics, advertised as central by many fuzzers, are of little help in detecting bugs.
2586 34th USENIX Security Symposium USENIX Association
Table 10: Bugs detected by DifuzzRTL (DF) with coverage
enabled and Cascade (CA). The fuzzers identify different 4
bugs, indicating that seed programs influence RTL bug detec- 2
tion. 0
Rocket BOOM
Bug Mix-ups Cond. Mix-ups Cond.
DF CA DF CA DF CA DF CA Figure 6: Time to bug detection for bugs that were detected
by all fuzzers. The horizontal bars indicate the average.
1 ✗ ✓ ✗ ✗ ✓ ✓ ✓ ✓
2 ✓ ✓ ✗ ✓ ✓ ✓ ✓ ✓
3 ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ it comes to detecting the same bugs. This corroborates that
4 ✓ ✓ ✗ ✗ ✓ ✗ ✗ ✗ not only structural coverage does not appear to help finding
56 ✗✗ ✗ ✗ ✗ ✓ ✗ ✗ ✗ bugs, but also that coverage feedback strategies tend to slow
7 ✗ ✓ ✓ ✓ ✗ ✓ ✗ ✗ down bug discovery. Given that fuzzing campaigns are usually
8 ✗ ✗ ✓ ✗ ✗ ✓ ✓ ✓ time-bounded, this can ultimately lead to missed bugs.
✗ ✗ ✗ ✓ ✓ ✗ ✗
109 ✗✗ ✗ ✗ ✗ ✗ ✓ ✗ ✗ Insight 4. Fuzzing that relies on state-of-the-art structural
11 ✗ ✗ ✓ ✗ ✓ ✓ ✗ ✗ coverage feedback is slower at finding the same bugs as
12 ✗ ✗ ✗ ✗ ✓ ✓ ✓ ✓ black-box fuzzing.
✗ ✓ ✓ ✓ ✗ ✓ ✓
13 ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗
14 ✗ ✗ ✗ ✗ ✗ ✓ ✗ ✗ 8.4 Case studies
15 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✗
Tot. 5 7 5 4 9 10 6 5 We closely examine a selection of the injected bugs to gain
further insights into hardware fuzzing.
8.3 Importance of the seeds BOOM mix-up 9. In the CVA6 PR #27, the multiplier was
not flushed due to missing handshake signals. Similarly, the
DifuzzRTL and ProcessorFuzz adopt different coverage feed- signal mix-up 9 injected by ENCARSIA into BOOM affects
back mechanisms, but they rely on the same seed pro- the multiplier handshake signals, causing it to drop multipli-
grams [10]. On the other hand, Cascade is a fuzzer that builds cation requests. This bug only requires the fuzzer to issue a
programs explicitly from the bottom up and claims to generate multiplication instruction to trigger, hence is relatively eas-
valid complex programs [42]. Hence, Cascade can be seen as ily detectable. Nevertheless, DifuzzRTL fails to detect this
a seed generator, with seeds distinct from those of DifuzzRTL bug due to an error that prevents it from using instructions in
and ProcessorFuzz. In this experiment, we evaluate whether the RISC-V M extension when testing CPUs implementing
differences in seed programs affect bug detection ability. the RISC-V G subset (extensions I, M, A, F, and D). While
fuzzers detect many bugs in the components that they test,
Results. We compare Cascade and DifuzzRTL on the pre- most of the undetected bugs reside in components completely
vious set of bugs. Results, summarized in Table 10, show untested by the fuzzers, like e.g. in the handling of compressed
that different seed programs result in the discovery of differ- instructions (like Ibex PR #48).
ent bugs per design, as opposed to the feedback mechanism,
which corroborates our finding from Section 8.2. While both Insight 5. Fuzzers often do not test the entire ISA.
fuzzers exhibit comparable performance on Rocket (Cascade:
11/30, DifuzzRTL: 10/30) and BOOM (Cascade: 15/30, Di- BOOM mix-up 3. This mix-up introduced a bug that only
fuzzRTL: 15/30), there is a notable difference in the bugs that gets triggered when the result of a floating-point division
they detect. Cascade detects 6 bugs that DifuzzRTL does not, equals zero. Cascade does not detect it because it excludes
while DifuzzRTL detects 5 bugs that Cascade does not. f.div/f.sqrt instructions for some designs to filter out known
unfixed natural bugs to avoid re-detections. While such filter-
Insight 3. The fuzzer seeds are a key factor that determines ing is desirable to continue the fuzzing campaign until the bug
which bugs will eventually be detected. has been fixed by the designers, it highlights that the filtering
must be more targeted towards the specific bug, rather than
Figure 6 summarizes the time to bug detection for the bugs excluding an instruction type [12]. A natural example that
that were detected by all fuzzers. On these bugs, Cascade, might be filtered out is Rocket PR #574. While DifuzzRTL
which operates in a black-box manner, is at least one order of and ProcessorFuzz test these instructions, they do not seem
magnitude faster than DifuzzzRTL and ProcessorFuzz when to provide sufficiently interesting input values.
USENIX Association 34th USENIX Security Symposium 2587
First detection (h)
DifuzzRTL DifuzzRTL ProcessorFuzzProcessorFuzz Cascade Cascade
Rocket BOOM Rocket BOOM Rocket BOOM
ProcessorFuzz introduces an instruction-granular bug de- guidelines to help shape future hardware fuzzers.
tection mechanism over DifuzzRTL. It then builds known bug filtering on top of this new mechanism, ignoring bugs Structural coverage: Coverage-guided fuzzers must rig- previously recognized in the design. However, this system is orously evaluate the benefits of the underlying coverage exclusively integrated within the new detection mechanism. metric in discovering bugs. Since the original detection mechanism remains active and unmodified by the authors, it is possible for a bug to be identi- Most coverage-guided fuzzer proposals do not evaluate fied by the new system, categorized as a known bug, and then the effectiveness of their coverage metric in detecting new rediscovered by the old system, resulting in a false positive. bugs [14, 30, 48]. This impedes distinguishing the effective- These false positives often terminate test executions early and ness of the coverage metric from the seeds and instruction therefore might mask new bugs. generation scheme, which we have shown to be a key compo- Insight 6. Current filtering mechanisms for known bugs nent. ENCARSIA will provide a valuable tool for evaluating broadly exclude test scenarios, hindering the discovery of this effectiveness by correlating the use of the feedback from further bugs. a given coverage metric with the actual bugs found. ISA-based functional coverage: Fuzzers must maximize Rocket conditional 7. The CSRRS instruction in RISC-V ISA-based functional coverage. reads a CSR, performs a bitwise OR operation with the value stored in the source register, and writes the result back to the In Section 8.3, we have shown that the quality of the seed CSR. However, it includes a special case where, if the source programs is a key factor in the effectiveness of a fuzzer. In Sec- register is the zero register, the instruction should avoid any tion 8.4, we have shown that bugs are often missed because side effects of writing to the CSR. Broken conditional 7 in the fuzzer does not test the corresponding design functional- Rocket removes this special case for the FCSR register (like ity. Hence, maximizing ISA-based functional coverage [27] Ibex PR #53). While Cascade does access FCSR, it only does of the seed programs to test all instruction variations with so via the CSRRW instruction, which includes no such special various input values is a crucial prerequisite for effective case. As a result, Cascade fails to detect this bug. fuzzing. If fuzzers filter for known bugs, they need to restrict Insight 7. Values seemingly coming from the same source the filtering to specific scenarios only, rather than excluding according to the ISA can be produced by different compo- instructions, as this might mask unknown bugs. Unfortunately, nents. Fuzzers often fail to cover these various cases. many fuzzers [10, 26, 30, 32] put a strong emphasis on purely structural coverage feedback to the detriment of taking the BOOM mix-up 13. Signal mix-up 13 injected by ENCARSIA functional coverage into account. into BOOM affects issue scheduling logic. However, due to Microarchitectural coverage: Fuzzers might consider this injected bug, if an issue queue is full and the youngest feedback from microarchitectural components. instruction is a variable-latency instruction, then a following instruction dependent on the result of the variable-latency In Section 8.4, we have shown that bugs can arise from instruction might be scheduled before it. Therefore, following corner cases in some microarchitectural structures. The ex- instructions read stale values. Similarly, conditional 3 only isting coverage-guided fuzzers consider low-level structural gets triggered by sequences of independent floating-point coverage, often related to multiplexer signals [26, 32, 48] or instructions that fill up a specific buffer. None of the three toggle coverage [14, 30]. There is no evidence that coverage fuzzers under study detected these bugs. PR #289 and #295 in feedback from such low-level metrics will translate into cov- BOOM fixed bugs that only appeared when the reorder buffer ering these interesting microarchitectural corner cases. One was full. This highlights the importance of testing all possible natural way to improve bug detection might be to consider microarchitectural paths through the design. functional coverage feedback from microarchitectural com- Insight 8. Fuzzers struggle to detect subtle bugs that arise ponents, such as the issue queue, the reorder buffer, or the from corner cases in microarchitectural structures. branch predictor, to guide the fuzzer towards corresponding corner cases. Hence, an interesting future work direction is Interestingly, this case study also exposes strong similari- to define a generic coverage metric that takes these common ties between ENCARSIA-injected bugs and natural bugs. microarchitectural components into account.
8.5 Recommendations for fuzzing Black-box microarchitectural coverage model: Synthetic bugs can aid in developing generic functional coverage Our findings demonstrate the need for more principled ap- models of microarchitectural elements. proaches in the development of hardware fuzzers. We provide
2588 34th USENIX Security Symposium USENIX Association
Black-box CPU fuzzing is on the rise because some CPUs fuzzers uniformly. We encourage further work that would rule
are closed-source [23, 36, 37, 44] and for performance rea- out such correctness-preserving transformations completely. sons [42]. Black-box fuzzers cannot benefit from direct feed- Changes at language level. To transform the design, EN- back from microarchitectural components. However, they can CARSIA operates on the Yosys intermediate representation. use black-box functional coverage models [36] described as This process vastly transforms the syntax of the source code, traces that are expected to exercise CPU internal corner cases. for example, by converting complex constructs into simpler We envision that a large-scale learning campaign based on ones. This means that potential fuzzing techniques that rely ENCARSIA-injected bugs and their verification traces could on the source code for input generation or coverage measure- generate a black-box functional microarchitectural coverage ment might be affected by the transformation. Since Yosys model that could guide black-box fuzzers. does not yet support all SystemVerilog features, assertions Breadth: Combining multiple state-of-the-art fuzzers leads are not preserved in the generated code, which might affect to a broader range of tested inputs. assertion-guided fuzzers. Specialization. ENCARSIA relies on formal verification to While Cascade claims to be superior to DifuzzRTL in terms ensure that the injected bugs are architecturally observable. In of coverage and new bugs found, ENCARSIA clearly shows fact, the verification framework can be expanded to include that DifuzzRTL can find bugs that Cascade cannot. A sim- additional properties. For example, it can ensure triggerability ple, considerable improvement is to build a mixed fuzzer that within a specific privilege level or other special configurations spends 50% of its time running Cascade and 50% running with tolerated deviations from the base specification [44]. DifuzzRTL. This mixed fuzzer indeed finds 21/30 bugs of ENCARSIA can also focus on injecting bugs into a given the study, 6 more than Cascade and 5 more than DifuzzRTL. specific HDL module or set of modules. This permits the Importantly, traditional evaluations in terms of coverage and evaluation of fuzzers on more specific functionalities of the new bugs found would not have revealed this, and Cascade, re- design. This is particularly useful for specialized fuzzers and leased after DifuzzRTL would appear to be objectively better for evaluating blind spots of a given fuzzer. than the mixed fuzzer, while in practice, it can find fewer bugs. Automatically Derived Transformations. Deriving our bug One practical inconvenience of a mixed fuzzer is that it re- injection transformations required considerable effort, includ- quires the instrumentations and probing mechanisms of both ing manual analysis of numerous pull requests. This raises fuzzers, which is mostly an engineering challenge, but which the question of whether such transformations can be derived also conditions the practical adoption of the fuzzer. Since Cas- automatically and what their potential applications might be. cade boasts a low-effort adoption without any instrumentation Since our current transformations are highly generic and al- and a bug discovery based on non-termination, which is easy ready cover all bugs identified in our survey, any additional to implement in practice, a more practical approach might be transformations would essentially be subsets of these. How- to extend Cascade to generate broader programs. ever, focusing on specific bug types or scenarios could enable more detailed analysis that may not be possible with gen- eral transformations, making it a valuable direction for future 9 Discussion research.
We discuss innocuous transformations, compatibility with 10 Related Work fuzzers that rely on HDL source code, specialization of our verification framework, and automatic generation of addi- We first discuss contemporary hardware fuzzer evaluations tional bug-injecting transformations. based on coverage and natural bugs. We then discuss manual Correctness-preserving transformations. ENCARSIA lever- bug insertions, bug surveys and software bug injection. ages formal methods to ensure that the transformations cause Fuzzer evalution via coverage. Coverage is typically in- architecturally observable deviations. Yet not all observable volved in fuzzer evaluations. RFUZZ [32] compares with transformations violate the specification, i.e., the ISA. For in- random testing by measuring multiplexer select coverage. Di- stance, a transformation that unconditionally adds one clock fuzzRTL [26] measured its own coverage when turning on cycle of latency to all instructions is architecturally observ- or off coverage feedback. Some fuzzers compare with other able but does not violate the RISC-V specification, provided fuzzers on a new coverage metric [14, 30], or compare with it does not violate the timing requirements of the memory other fuzzers on their original coverage metrics [10, 42, 48]. hierarchy or other critical components. When manually ana- No work has ever shown any correlation between coverage lyzing many bugs generated by ENCARSIA in Section 7, we and bug discovery abilities in hardware fuzzing. observed no such occurrence in practice. If such a case would Fuzzer evaluation via natural bug discovery. Natural bugs occur, its impact would be limited to a slight reduction in are often used to evaluate hardware fuzzers [10, 26, 30, 42, the apparent performance of the fuzzers, and would affect all 48]. While intuitive, this approach for comparison has several
USENIX Association 34th USENIX Security Symposium 2589
drawbacks. First, natural bugs are increasingly harder to find 11 Conclusion as designs mature. Second, fuzzers are rarely evaluated on natural bugs found by older fuzzers. Finally, the lack of ground We presented ENCARSIA, the first framework that automat- truth makes it impossible to derive fundamental performance ically injects realistic bugs into RTL designs to support the metrics such as miss rates. ENCARSIA provides a systematic evaluation of fuzzers. Based on ENCARSIA, we provide En- alternative for more fairly comparing hardware fuzzers. Corpus, a unified evaluation corpus made of 90 synthetic bugs verified to have an architecturally visible effect. To design Manually Assembled Corpora. Manually inserted bugs are ENCARSIA, we first conducted a comprehensive survey of occasionally involved in fuzzer evaluations. DifuzzRTL [26] bugs fixed in four popular open-source RISC-V CPUs of vari- uses one synthetic RTL module with one manually inserted ous complexities and design paradigms and found that these bug for comparison with RFUZZ. The inserted bug is tailored bugs can be modeled by wire connection mix-ups and errors to DifuzzRTL’s coverage metric and might not be realistic. in conditional statements. ENCARSIA leverages the Yosys Manually inserting realistic bugs into real-world designs is intermediate representation to automatically inject transfor- complex and demands a deep understanding of the design and mations that reflect these representative bugs and employs the nature of bugs themselves. The HACK@EVENT compe- formal verification to check whether these injected bugs are tition series [16] is a rigorous attempt to create a manually architecturally visible. We evaluate ENCARSIA using three compiled corpus of bugs. Its reliance on industry partners RISC-V CPUs and three state-of-the-art CPU fuzzers. Based has led to a lack of transparency and reluctance to release the on this evaluation, we provide insights into the shortcomings bugs to the public [40]. Of the ten competitions held over the of current fuzzers and suggest future directions for hardware past five years, the corpus has only been released for three. fuzzer development, particularly regarding structural and func- Additionally, among the 170 bugs released as part of these tional coverage, and breadth of the input space. three corpora, only 28 included detailed descriptions or fixes. 26 bugs from the 2021 corpus closely resemble or are iden- Acknowledgements tical to those from the 2019 corpus. The HACK@EVENT competition has struggled to meet the demand for new bugs We thank the anonymous reviewers for their valuable feed- and has continued using the same CPU, CVA6, for years. back. This work was supported in part by the Swiss State Survey of hardware bugs. A few past studies have focused Secretariat for Education, Research and Innovation under con- on the analysis of real-world bugs, but on different aspects. tract number MB22.00057 (ERC-StG PROMISE). RemembERR [43] is a database that classifies around 2’500 errata from recent x86 CPUs, The black-box nature of the 12 Ethics Considerations entries classified in this database, however, hampers the un- derstanding of structural properties of the bugs. Ma et al. [35] In conducting this research, we took several ethical precau- performed a review of 68 bugs in reconfigurable designs and tions to ensure that our work did not pose any risks to real- produce tools to help developers simulate and deploy recon- world systems, individuals, or intellectual property. The fol- figurable designs as well as a manually-created testbed of 20 lowing sections detail our approach to injected bugs, potential well-documented bugs specific to FPGA designs [34]. misuse, data collection, and intellectual property compliance. Bug injection in software. Previous research has examined Injected bugs. In this paper, we intentionally inject bugs software bug injection. LAVA [18] proposed to cause segmen- into RISC-V CPUs to assess the effectiveness of different tation faults if user-provided data matches specific arbitrary bug detection techniques. These bugs are introduced solely key values. The realism of these bugs has been many times in local instances of the devices and do not affect any public questioned [7, 21, 31], and LAVA leaves artifacts that can be implementations of the CPUs. Consequently, our research detected by white-box fuzzers [7]. Apocalypse [39] replaces does not create any security vulnerabilities in real-world sys- LAVA’s key equality condition with finite state machines to tems, whether active or otherwise, and therefore, no disclosure implement conditions under which a bug is triggered. Sem- process is required. Seed [38] uses machine learning to train on a corpus of real- Risks of misuse. Risks of Misuse: While ENCARSIA could world bugs and uses this model to adapt the learned patterns potentially be exploited to introduce bugs into proprietary to the local context of the target program. FixReverter [50] hardware designs, such misuse would necessitate specific derives a formal grammar from the real-world bugs and uses access to the design files. Given the requirement for this level this grammar to generate new bugs. None of these approaches of access, we assess the likelihood of direct misuse of our is directly applicable to hardware fuzzing, as software and work to be low. hardware bugs are fundamentally different. ENCARSIA is the Data collection. The research did not involve any human first systematic approach for injecting bugs into hardware. subjects No personal data was processed or analyzed. All
2590 34th USENIX Security Symposium USENIX Association
data collected as part of this study was sourced from public References GitHub repositories. Intellectual property. All tools used in this study, both open- [1] CHIPS Alliance. Rocket Chip Generator. https: source and commercial, were used in full compliance with //github.com/chipsalliance/rocket-chip. Ac- their respective licenses. cessed: 2024-08-30. [2] Krste Asanovic, Rimas Avizienis, Jonathan Bachrach, 13 Open Science Scott Beamer, David Biancolin, Christopher Celio, Henry Cook, Daniel Dabbelt, John Hauser, Adam We make the artifacts of our study publicly avail- Izraelevitz, et al. The rocket chip generator. Tech. Rep. able to the research community via https://github. UCB/EECS-2016-17, 2016. com/comsec-group/encarsia and https://doi.org/10. 5281/zenodo.14664723. [3] Armin Biere, Matti Järvisalo, Marijn JH Heule, and Nor- bert Manthey. Equivalence checking of hwmcc 2012 13.1 Artifact contents circuits. In Proceedings of SAT Competition, 2013. These artifacts are mainly made of the following components: [4] RISC-V BOOM. The Berkeley Out-of-Order RISC- V Processor. https://github.com/riscv-boom/ • The detailed results of our RISC-V CPU survey. riscv-boom. Accessed: 2024-08-30. • The source code of ENCARSIA, our custom-developed [5] Niklas Bruns, Vladimir Herdt, Daniel Große, and Rolf bug injection and verification tool. Drechsler. Efficient cross-level processor verification using coverage-guided fuzzing. In VLSI, 2022. • Two versions of ENCARSIA’s verification setup: one based on proprietary JasperGold (to the extent of what [6] Niklas Bruns, Vladimir Herdt, Eyck Jentzsch, and Rolf our license agreements allow to share) and another based Drechsler. Cross-level processor verification via endless on the open-source Yosys. randomized instruction stream generation with coverage- guided aging. In IEEE DATE, 2022. • The EnCorpus set of CPU bugs. [7] Joshua Bundt, Andrew Fasano, Brendan Dolan-Gavitt, • Reproducible evaluations of DifuzzRTL, Processorfuzz, William Robertson, and Tim Leek. Evaluating synthetic and Cascade against the bug set. bugs. In ACM ASIACCS, 2021.
13.2 Reproducibility [8] Cadence. Jasper RTL Apps. https: //www.cadence.com/en_US/home/tools/ To facilitate reproducibility of our study, we release the com- system-design-and-verification/ plete evaluation setup as a Docker container. The container formal-and-static-verification/ contains all scripts and configurations necessary to replicate jasper-verification-platform.html. Accessed: our experiments. Specifically, it contains the Python script 2024-08-08. used to execute all tools within ENCARSIA in the correct [9] Sadullah Canakci, Leila Delshadtehrani, Furkan Eris, sequence to generate a set of verified bugs. It then runs multi- Michael Bedford Taylor, Manuel Egele, and Ajay Joshi. ple instances of selected fuzzers in parallel to evaluate them Directfuzz: Automated test generation for rtl designs across these bugs, significantly reducing the wallclock dura- using directed graybox fuzzing. In ACM/IEEE DAC, tion of the evaluation. The container also includes the setup 2021. details for operating all three fuzzers (DifuzzRTL, Processor- fuzz, Cascade) on each core (Ibex, Rocket, BOOM), cover- [10] Sadullah Canakci, Chathura Rajapaksha, Leila ing everything from the device source code with fixed bugs Delshadtehrani, Anoop Nataraja, Michael Bedford and the surrounding System-on-Chip to the fuzzer and core Taylor, Manuel Egele, and Ajay Joshi. Processorfuzz: configurations. We believe that these artifacts enables other Processor fuzzing with control and status registers researchers to verify, reproduce, and extend our study. guidance. In HOST, 2023. 13.3 Usability in future research [11] Katharina Ceesay-Seitz, Sarath Kundumattathil Mo- hanan, Hamza Boukabache, and Daniel Perrin. Formal To let other researchers use our artifacts and better evaluate property verification of the digital section of an ultra-low further bug detection research, we provide detailed guidelines current digitizer asic. In accellera DVCON EUROPE, on how to inject CPU bugs into arbitrary RISC-V CPUs. 2021.
USENIX Association 34th USENIX Security Symposium 2591
[12] Katharina Ceesay-Seitz, Flavien Solt, and Kaveh Razavi. verification using taint inference-enabled fuzzing. In μcfi: Formal verification of microarchitectural control- IEEE/ACM ICCAD, 2023. flow integrity. In ACM CCS, 2024. [25] Muhammad Monir Hossain, Arash Vafaei, Kimia Zamiri [13] Christopher Celio, David A Patterson, and Krste Azar, Fahim Rahman, Farimah Farahmandi, and Mark Asanovic. The berkeley out-of-order machine (boom): Tehranipoor. Socfuzzer: Soc vulnerability detection An industry-competitive, synthesizable, parameterized using cost function enabled fuzz testing. In IEEE DATE, risc-v processor. Tech. Rep. UCB/EECS-2015-167, 2023. 2015. [26] Jaewon Hur, Suhwan Song, Dongup Kwon, Eunjin Baek, [14] Chen Chen, Rahul Kande, Nathan Nguyen, Flemming Jangwoo Kim, and Byoungyoung Lee. Difuzzrtl: Dif- Andersen, Aakash Tyagi, Ahmad-Reza Sadeghi, and ferential fuzz testing to find cpu bugs. In IEEE S&P, Jeyavijayan Rajendran. Hypfuzz: Formal-assisted pro- 2021. cessor fuzzing. In USENIX Security, 2023. [27] IEEE. IEEE Standard for SystemVerilog–Unified Hard- [15] Edmund M. Clarke, Thomas A. Henzinger, and Hel- ware Design, Specification, and Verification Language. mut Veith. Handbook of Model Checking. Springer IEEE Std 1800-2023 (Revision of IEEE Std 1800-2017), International Publishing, 2018. 2024. [16] Ghada Dessouky, David Gens, Patrick Haney, Garrett [28] Laura Inozemtseva and Reid Holmes. Coverage is not Persyn, Arun Kanuparthi, Hareesh Khattri, Jason M. strongly correlated with test suite effectiveness. In Fung, Ahmad-Reza Sadeghi, and Jeyavijayan Rajendran. ACM/IEEE ICSE, 2014. HardFails: Insights into Software-Exploitable hardware bugs. In USENIX Security, 2019. [29] Nursultan Kabylkas, Tommy Thorn, Shreesha Srinath, Polychronis Xekalakis, and Jose Renau. Effective [17] Sushant Dinesh, Madhusudan Parthasarathy, and processor verification with logic fuzzer enhanced co- Christopher W Fletcher. ConjunCT: Learning inductive simulation. In IEEE/ACM MICRO, 2021. invariants to prove unbounded instruction safety against microarchitectural timing attacks. In IEEE S&P, 2024. [30] Rahul Kande, Addison Crump, Garrett Persyn, Patrick Jauernig, Ahmad-Reza Sadeghi, Aakash Tyagi, and [18] Brendan Dolan-Gavitt, Patrick Hulin, Engin Kirda, Tim Jeyavijayan Rajendran. TheHuzz: Instruction fuzzing Leek, Andrea Mambretti, Wil Robertson, Frederick Ul- of processors using Golden-Reference models for find- rich, and Ryan Whelan. Lava: Large-scale automated ing Software-Exploitable vulnerabilities. In USENIX vulnerability addition. In IEEE S&P, 2016. Security, 2022. [19] GitHub. GitHub REST API documentation. [31] George Klees, Andrew Ruef, Benji Cooper, Shiyi Wei, https://docs.github.com/en/rest?apiVersion= and Michael Hicks. Evaluating fuzz testing. In ACM 2022-11-28. Accessed: 2024-08-18. CCS, 2018. [20] OpenHW Group. CVA6 RISC-V CPU. https:// [32] Kevin Laeufer, Jack Koenig, Donggyu Kim, Jonathan github.com/openhwgroup/cva6/. Accessed: 2024- Bachrach, and Koushik Sen. Rfuzz: Coverage-directed 08-30. fuzz testing of rtl on fpgas. In IEEE/ACM ICCAD, 2018. [21] Ahmad Hazimeh, Adrian Herrera, and Mathias Payer. [33] lowRISC. Ibex RISC-V Core. https://github.com/ Magma: A ground-truth fuzzing benchmark. In PO- lowRISC/ibex. Accessed: 2024-08-30. MACS, 2020. [34] Jiacheng Ma, Gefei Zuo, Kevin Loughlin, Haoyang [22] Vladimir Herdt, Daniel Große, Eyck Jentzsch, and Rolf Zhang, Andrew Quinn, and Baris Kasikci. Reproducible Drechsler. Efficient cross-level testing for processor Hardware Bugs. https://github.com/efeslab/ verification: A risc-v case-study. In FDL, 2020. hardware-bugbase. Accessed: 2024-09-02. [23] Jana Hofmann, Emanuele Vannacci, Cédric Fournet, [35] Jiacheng Ma, Gefei Zuo, Kevin Loughlin, Haoyang Boris Köpf, and Oleksii Oleksenko. Speculation at Zhang, Andrew Quinn, and Baris Kasikci. Debugging fault: Modeling and testing microarchitectural leakage in the brave new world of reconfigurable hardware. In of {CPU} exceptions. In USENIX Security, 2023. ASPLOS, 2022. [24] Muhammad Monir Hossain, Nusrat Farzana Dipu, [36] Oleksii Oleksenko, Christof Fetzer, Boris Köpf, and Kimia Zamiri Azar, Fahim Rahman, Farimah Farah- Mark Silberstein. Revizor: Testing black-box cpus mandi, and Mark Tehranipoor. Taintfuzzer: Soc security against speculation contracts. In ASPLOS, 2022.
2592 34th USENIX Security Symposium USENIX Association
[37] Oleksii Oleksenko, Marco Guarnieri, Boris Köpf, and a linux-ready 1.7-ghz 64-bit risc-v core in 22-nm fdsoi
Mark Silberstein. Hide and seek with spectres: Efficient technology. In VLSI, 2019. discovery of speculative information leaks with random testing. In IEEE S&P, 2023. [50] Zenong Zhang, Zach Patterson, Michael Hicks, and Shiyi Wei. {FIXREVERTER}: A realistic bug injection [38] Jibesh Patra and Michael Pradel. Semantic bug seeding: methodology for benchmarking fuzz testing. In USENIX a learning-based approach for creating realistic bugs. In Security, 2022. ACM ESEC/FSE, 2021. [39] Subhajit Roy, Awanish Pandey, Brendan Dolan-Gavitt, A Survey Overview and Yu Hu. Bug synthesis: Challenging bug-finding tools with deep faults. In ACM ESEC/FSE, 2018. In this Appendix, Table 11 provides an overview of the latest two analyzed PRs per CPU and per category. We provide the [40] Ahmad-Reza Sadeghi, Jeyavijayan Rajendran, and detailed survey results as part of the Artifacts of this paper. Rahul Kande. Organizing the world’s largest hard- ware security competition: challenges, opportunities, Table 11: The latest two analyzed PRs per CPU and category. and lessons learned. In GLSVLSI, 2021. (M = Mix-up, C = Broken Conditional) [41] Pasquale Davide Schiavone, Francesco Conti, Davide Design PR# Bug Description Type Rossi, Michael Gautschi, Antonio Pullini, Eric Flamand, 343 Wrong physical memory M and Luca Benini. Slow and steady wins the race? a protection address matching comparison of ultra-low-power risc-v cores for internet- 332 Interrupt handler returned M of-things applications. In PATMOS, 2017. Ibex 277 to misaligned address Missing access checks on debug CSRs C [42] Flavien Solt, Katharina Ceesay-Seitz, and Kaveh Razavi. 272 DRET (debug return) executes C Cascade: Cpu fuzzing via intricate program generation. outside debug mode In USENIX Security, 2024. 206 Mixed up array index between M [43] Flavien Solt, Patrick Jattke, and Kaveh Razavi. Re- 191 instruction and data cache Race condition in M memberr: Leveraging microprocessor errata for design data cache miss handler testing and validation. In IEEE/ACM MICRO, 2022. CVA6 191 Race condition in C [44] Fabian Thomas, Lorenz Hetterich, Ruiyi Zhang, 189 data cache miss handler Wrong calculations due to floating C Daniel Weber, Lukas Gerlach, and Michael Schwarz. point wrapper state machine RISCVuzz: Discovering Architectural CPU Vulnera- 592 Wrong signal order M bilities via Differential Hardware Fuzzing. https:// on bus protocol conversion ghostwriteattack.com/riscvuzz.pdf. Accessed: 589 Wrong address and M 2024-09-02. Rocket 574 data generated on bus Values changed when registers C [45] Timothy Trippel, Kang G. Shin, Alex Chernyakhovsky, spilled to memory and read back Garret Kelly, Dominic Rizzo, and Matthew Hicks. 440 Wrong data cache probe C Fuzzing hardware like software. In USENIX Security, acknowledgement data 2022. 451 A mathematical util function M [46] Clifford Wolf, Johann Glaser, and Johannes Kepler. 437 performed a wrong computation Yosys-a free verilog synthesis suite. In Austrochip, 2013. BOOM Illegal instruction M dispatched into the LSU [47] Jiahui Xu and Lana Josipovi´c. Automatic inductive 448 Wrong address used for C invariant generation for scalable dataflow circuit verifi- 405 data cache releases Tag update hazard for cache refills C cation. In IEEE/ACM ICCAD, 2023.
[48] Jinyan Xu, Yiyuan Liu, Sirui He, Haoran Lin, Yajin
Zhou, and Cong Wang. MorFuzz: Fuzzing processor via runtime instruction morphing enhanced synchronizable co-simulation. In USENIX Security, 2023.
[49] Florian Zaruba and Luca Benini. The cost of application- class processing: Energy and performance analysis of
USENIX Association 34th USENIX Security Symposium 2593