SOURCE ARCHIVE
EXTRACTED CONTENT
126,261 chars Fuzzing Open-Source GPU Hardware
with SIMT Program Generation
Zibo Gao, Jie Wang, Qihang Zhou, Lixiao Shan, Junjie Hu, Xiaoqi Jia, and Zhiqiang Lv, Institute of Information Engineering, Chinese Academy of Sciences; School of Cyber Security, University of Chinese Academy of Sciences https://www.usenix.org/conference/usenixsecurity26/presentation/gao-zibo
This paper is included in the Proceedings of the 35th USENIX Security Symposium. August 12–14, 2026 • Baltimore, MD, USA ISBN 978-1-939133-58-8
Open access to the Proceedings of the
35th USENIX Security Symposium
is sponsored by
Fuzzing Open-Source GPU Hardware with SIMT Program Generation
Zibo Gao†§, Jie Wang†§, Qihang Zhou†§, Lixiao Shan†§, Junjie Hu†§, Xiaoqi Jia †§, and Zhiqiang Lv†§ ∗ †Institute of Information Engineering, Chinese Academy of Sciences. §School of Cyber Security, University of Chinese Academy of Sciences.
Abstract flawed hardware may invalidate even formally verified soft-
GPUs have become critical components in computing sys- ware [21, 50]. This raises a critical question: how can we tems. In the post-Moore’s Law era, the demand for perfor- effectively verify GPU designs to identify hardware bugs and mance is driving increasing GPU microarchitectural complex- vulnerabilities? ity, which in turn gives rise to new vulnerabilities. Unfortunately, despite this urgency, finding hardware bugs In this paper, we present the first framework for fuzzing in GPU designs has received limited attention in existing GPU hardware designs at the Register Transfer Level (RTL) studies. Previous processor verification methods, including to automatically discover vulnerabilities. At the core of Fuz- fuzzing-based approaches (e.g., DiveFuzz [10], Cascade [39]) zGPU is a novel generator that randomly constructs valid and formal methods (e.g., μCFI [6]), mainly focus on Central SIMT test programs with complex data and control flow, ex- Processing Units (CPUs) and are unable to generate GPU- plicitly exercising the GPU execution model and thereby specific test programs or localize GPU hardware bugs because triggering diverse hardware behaviors. This bridges a key they do not capture the GPU execution model. Research on gap: prior processor fuzzers predominantly target CPUs and GPU testing traditionally focuses on software-level bugs in largely ignore GPU hardware. FuzzGPU further implements drivers [18] and utility libraries [54], leaving hardware threats a GPU-specific differential testing harness. It tackles microar- largely unexplored. To the best of our knowledge, no exist- chitectural non-determinism (e.g., caches and memory timing) ing work is specifically tailored for fuzzing GPU hardware and enables trace-driven differential testing against an ISA- designs. level oracle. This technique unlocks vulnerability detection Fuzzing GPU hardware designs is non-trivial because and localization with instruction-level precision. We evalu- GPUs follow the Single-Instruction, Multiple-Thread (SIMT) ated FuzzGPU on two real-world open-source GPUs, Vortex execution model, where instructions execute in parallel across and Ventus OpenGPGPU, uncovering 20 previously unknown many threads organized into warps, blocks, and grids. This bugs, including 17 RTL bugs and 3 ISS bugs, 10 of which execution model differs from that of CPUs and introduces were assigned CVE identifiers. challenges that existing approaches do not adequately address. The first challenge is to systematically generate valid and 1 Introduction intricate GPU programs to trigger potential bugs. Our analysis reveals that SIMT execution introduces unique control flow, Graphics Processing Units (GPUs) have become essential such as thread divergence and synchronization. These features components in modern computing systems and support a significantly reduce program completion rates and often cause wide range of applications, from artificial intelligence to sci- tests to stall prematurely (§3). entific simulation. In the post-Moore’s Law era, the push The second challenge is the feasibility of detecting GPU for higher performance is forcing GPU microarchitectural hardware bugs. Unlike software that has well-defined excep- design to unprecedented levels of complexity. This design tion behaviors such as crashes and memory sanitizer viola- complexity, however, elevates the threat of hardware flaws tions, GPU hardware bugs may not manifest as conventional and vulnerabilities [29, 34, 44, 53]. exceptions. Although differential testing between the Device Hardware bugs are permanent and potent [42]. Once manu- Under Test (DUT) and Golden Reference Models (GRM) has factured, a GPU’s hard-wired circuitry cannot be rewired, proven practical for hardware bug detection [5, 15, 47, 48, 52], making patches extremely difficult to deploy. Moreover, GPUs exhibit several characteristics that complicate tradi- tional differential testing: (i) Multithreaded memory opera- ∗Corresponding author: lvzhiqiang@iie.ac.cn tions can be non-deterministic due to microarchitectural tim-
USENIX Association 35th USENIX Security Symposium 2465
ing (e.g., coherence and caches), while most GPU GRMs are 2 Background
ISA-level and cannot faithfully model these effects [43]. (ii)
Inter-warp synchronization introduces ordering dependencies 2.1 GPU Software and Hardware
that conventional lockstep checking failed to capture. (iii)
GPUs perform out-of-order writeback without reorder buffers Modern GPU designs incorporate a multitude of Streaming
(ROBs), unlike CPUs that typically rely on ROBs to provide Multiprocessors (SMs) and High-Bandwidth Memory (HBM),
the precise exceptions required by operating systems. Con- enabling them to effectively accelerate parallel workloads.
sequently, the register file state may reflect transient values GPU Program Structure. A GPU program typically consists
rather than the architectural state, yielding mismatches when of multiple kernels, each launched as a grid of Cooperative
comparing the DUT against the GRM. Thread Arrays (CTAs) that are dynamically scheduled onto
Our Contribution. To address these challenges, we present Streaming Multiprocessors (SMs). Each CTA contains multi-
FuzzGPU, the first GPU RTL fuzzing framework that auto- ple concurrently executing and synchronizing warps. A warp
matically uncovers hardware bugs. FuzzGPU contributes two is the basic scheduling unit executing a common instruction
techniques. (i) We design an ISA-emulation-guided SIMT stream, typically containing 32 threads. A thread is the small-
program generator that produces valid yet diverse parallel est execution unit, maintaining its own register context and
tests by explicitly exercising the GPU execution model. The predicate mask, which controls instruction participation for
key insight is that validity is a throughput feature: the genera- conditional execution and branch-induced divergence.
tor avoids stalling the test at short failed attempts and sustains Thread Divergence. Warps follow the Single-Instruction,
long tests. Concretely, it uses a SIMT-inspired generation Multiple-Thread (SIMT) model, where active threads execute
stack to track divergence and reconvergence with per-thread a common instruction stream. A warp is converged when all
masks, and a deadlock-free barrier planner to impose a con- active threads share the same next PC. Branches may cause
sistent ordering of barrier IDs and scopes across warps. (ii) We divergence by splitting the thread mask across multiple PCs,
propose a GPU-aware instruction-level differential checking followed by reconvergence when paths merge. GPUs handle
that enables bug detection and instruction-granular localiza- divergence by masking off inactive threads on the current path
tion despite GPU non-determinism. The motivation is that and using the SIMT stack to record the active masks (TMASK)
lockstep testing for GPUs differs substantially from that for and reconvergence PCs (RPC) for deferred paths.
CPUs. We therefore verify traces against an ISA oracle by (a)
re-serializing out-of-order writeback events using instruction Front-end Processing Block (PB) Pipeline
IDs, (b) advancing the oracle with a synchronization-aware I-Buffer Scoreboard Back-end Result Bus
scheduling policy to avoid barrier-induced ordering ambigu- V ... W Regs LD/ST
Units
ity, and (c) using taint-aware checking to defer ambiguous Fetch Decode Warp L1$
memory-dependent cases to a memory-consistency validator. Scheduler
Evaluated on two real-world open-source GPUs (Vor- Operand Shared Mem
tex [41] and OpenGPGPU [27, 28]), FuzzGPU discovers 20 L1$ PC RPC TMASK Collector FUs
new bugs (17 RTL bugs and 3 ISS bugs, with 10 new CVE IF/ID ID/IS SIMT Stack IS/OC OC/EX
IDs). It also outperforms our GPU ports of two state-of-the-art GPU Die
CPU fuzzers, Cascade [39] and DiveFuzz [10], achieving up PB SM 0... SM n...
to 2× higher expression coverage and 28× higher effective PB PB ... PB PB PB CTA
fuzzing throughput. L1 Cache L1 Cache Scheduler I/O
Shared Memory Shared Memory Ctrl.
Our key contributions are summarized as follows: Crossbar
• Novel SIMT Program Generator. To the best of our L2 Cache ... L2 Cache
knowledge, this is the first GPU hardware fuzzer that HBM HBM
generates valid and intricate SIMT programs to improve
instruction throughput and trigger GPU hardware bugs. Figure 1: GPU microarchitecture overview.
• New GPU Differential Testing Harness. We propose a GPU Microarchitecture. Figure 1 depicts a typical GPU
GPU-aware differential testing harness for instruction- microarchitecture. When a kernel is launched from the host
level bug detection. It remains effective under GPU- CPU, the host first partitions the kernel into multiple Coop-
specific non-determinism, including relaxed memory be- erative Thread Arrays (CTAs) and sends them to the CTA
haviors, synchronization, and out-of-order writeback. scheduler. The scheduler dynamically assigns received CTAs
• Evaluation. We evaluate FuzzGPU on two real-world to SMs. Each SM contains a private L1 cache, shared mem-
open-source GPUs (Vortex and OpenGPGPU). We also ory, and a set of processing blocks (PBs) to execute warps in
manually ported two state-of-the-art CPU fuzzers to each CTA. PBs decode warps into micro-operations (μOPs),
GPU as baselines for comparison. issued to load/store (LD/ST) or functional units (FUs) that
2466 35th USENIX Security Symposium USENIX Association
... Bank 0 Bank 1... Bank n
Host Interface
read operands from register banks, then write back results to of our knowledge, no prior work has systematically studied
registers or memory. valid GPU program generation for hardware fuzzing.
GPUs employ out-of-order writeback to hide LD/ST and To understand why applying existing processor fuzzers to
FU latencies. A scoreboard is required to ensure correctness GPU designs is insufficient and new techniques are needed,
by tracking pending register writes and preventing data haz- we analyzed the execution results of 5,000 test programs gen-
ards from dependent instructions. Meanwhile, although warp erated by two extended processor fuzzers, Cascade [39] and
schedulers select ready warps to hide latency, instructions DiveFuzz [10], that were manually adapted to the Vortex64
within a warp remain in-order issued. GPU [41] design. Specifically, we generate random instruc-
tion sequences to construct warps. We then execute them
2.2 Software and Hardware Fuzzing across multiple threads and warps to form CTAs and saturate
the maximum hardware-supported concurrency.
Fuzzing has been widely used in software to automatically
discover unknown bugs [2, 11, 13, 17, 30, 31, 49]. Recently, Ported-Cascade Ported-DiveFuzz
fuzzing has been adapted to hardware design verification [3–5, 40
7, 10, 12, 15, 19, 20, 23, 24, 39, 40, 42, 45–47]. While traditional 20
verification methods like dynamic and formal verification
are effective, they are labor-intensive and often suffer from 0
limited coverage [9]. Hardware fuzzing, as an automated 0 10 20 30 40 50 60 70 80 90 100
testing approach, has proven to be an effective complement Instruction Completion Rate (%)
to these conventional techniques [7, 37]. Figure 2: Distribution of instruction completion rates.
State-of-the-art hardware fuzzers generate and mutate ran-
dom sequences of test instructions to exercise various func-
tionalities and design corner cases. Unlike software fuzzers Figure 2 shows the histogram of the instruction completion
that generate data streams conforming to specific formats, rate, which is the percentage of executed instructions per test
hardware fuzzers typically produce instruction streams adher- case. We evaluate these programs using an ISA simulator
ing to the Instruction Set Architecture (ISA) to drive hardware (ISS) as oracle. We found that most test programs aborted
state transitions. early, although the generated instructions were legally en-
In contrast to software checkers that monitor for crashes or coded and the thread context was properly initialized. There-
memory violations, hardware processors rarely crash: even fore, the portion of the valid instructions used for fuzzing is
exceptions are often valid ISA behaviors. While traditional limited. To further analyze the root causes, we run the test
bug detection relies on manual SystemVerilog assertions, re- under single-thread and single-warp configurations, where
cent work demonstrates the superiority of differential test- the median completion rate recovers to 99% (Cascade) and
ing [5, 15, 47, 48, 52]. These approaches input the same test 98% (DiveFuzz), indicating that multiple threads and warps
program into both the DUT and a Golden Reference Model are the primary source of invalidity. Further inspection re-
(GRM), comparing their states. State comparison can be per- veals that the GPU rejected most tests due to unmodeled
formed at program or instruction granularity: program-level GPU-specific behavior (Figure 3). For DiveFuzz, 61% of the
checking compares only the final architectural state (e.g., rejections are related to SIMT stack errors or Warp-level Ma-
registers and memory), which can miss transient errors that trix Multiply-Accumulate (WMMA) errors, 4% rejections are
are later overwritten [10], whereas instruction-level checking attributed to barriers. For Cascade, invalid control flow caused
compares state after each committed instruction, improving by multithreaded non-determinism accounts for a larger frac-
localization and reducing false negatives. tion (79%). These rejections are permissible under the ISA
specification and therefore do not constitute hardware bugs.
However, they significantly limit the execution depth of test
3 Motivation and Challenges programs.
Our goal is to effectively trigger and detect GPU hardware Challenge 1: Test Program Validity
bugs via hardware fuzzing. A GPU fuzzer needs to have New techniques different from CPU fuzzers are required
two key capabilities: (i) generating SIMT test programs to to generate valid and intricate SIMT test programs.
trigger potential bugs; (ii) effectively detecting GPU hardware
bugs. In this section, we first analyze the challenges of GPU GPU Differential Testing. Another key capability of a fuzzer
fuzzing based on these capabilities, then analyze their root is to effectively detect vulnerabilities. Unlike software that
causes which guide our design. exposes bugs via memory sanitizers or crashes, GPU hard-
SIMT Test Programs Generation. Previous processor ware bugs may not manifest as conventional exceptions. To
fuzzers have primarily targeted CPU hardware. To the best accurately locate hardware bugs, instruction-level differential
USENIX Association 35th USENIX Security Symposium 2467
Test Cases (%)
Success WMMA Thread Mask Err. ambiguity. If the GRM steps W1 before observing W2’s arrival
SIMT Stack Err. Barrier Deadlock at the barrier, it may simulate W1 past the barrier while it is
Invalid Ctrl. Flow Others still considered stalled, leading to a spurious mismatch.
35 42 19
Warp
79 12
0 20 40 60 80 100 Barrier Inactive
Percentage of Test Cases (%) DUT NOP Barrier Activate Cannot step
Figure 3: Reasons for rejecting test programs. GRM Lock-step Barrier Inactive
NOP
Time
testing, which compares execution results between the Device Under Test (DUT) and a Golden Reference Model (GRM), has Figure 4: Ordering ambiguity in lockstep testing. proven to be a practical approach [5, 15, 47, 48, 52]. However, GPU differential testing is challenging due to GPU-specific Additionally, GPU cores typically employ out-of-order characteristics. writeback to improve performance [14, 28, 41]. GPU cores Relaxed memory consistency permits memory operations typically do not require the precise exceptions expected by op- to become visible out of program order across threads unless erating systems. Thus, unlike CPUs, they often omit a reorder constrained by synchronization. Consequently, the outcome buffer (ROB) for in-order commit. As a result, the register realized on hardware may depend on microarchitectural tim- state observed at any cycle may not be equivalent to the archi- ing. However, most GPU GRMs lack microarchitectural fi- tectural state of the program. Comparing such transient states delity because they simulate only ISA semantics and omit between the DUT and the GRM can therefore yield spurious timing-dependent visibility effects (e.g., store buffers, cache/- mismatches. coherence latency). For example, let Ri(a) and Wi(a) denote Challenge 3: Differential Testing Feasibility a load and store to byte a, respectively, issued by warp i. Con- sider a warp executing a load R1(a), while other warps execute Inter-warp synchronization and out-of-order writeback stores W2(a) and W3(a) to the same address. Without synchro- break conventional lockstep testing. nization, R1(a) may legally read the old value or a value writ- ten by W2(a) or W3(a), depending on the relative propagation 4 timing. In contrast, an ISA-level GRM typically resolves the FuzzGPU load using a fixed serialization rule and thus returns a single 4.1 Design Overview value for a chosen interleaving, ignoring microarchitectural timing. Thus, the final register state may differ between the In order to address the aforementioned challenges, we design DUT and GRM, even if both executions are correct. two novel operating primitives: (i) SIMT program genera- Challenge 2: Memory Non-determinism tor that systematically generates valid and intricate SIMT test programs to trigger potential bugs; (ii) differential test- Non-deterministic GPU memory behaviors can invalidate ing harness that effectively detects GPU hardware bugs with differential testing. non-deterministic behaviors. Figure 5 illustrates the overall fuzzing loop design of FuzzGPU. Moreover, traditional differential testing assumes lockstep execution: the GRM advances in tandem with the DUT, driven Top-Level Loop by the writeback of the oldest in-flight instruction. However, Barrier planner CVE/CWE LOG Report inter-warp synchronization on GPUs breaks this assumption. Interleaved Warp Gen The lockstep model inherently fails to capture the dependen- Warp 0 ...... Warp n Triage LOG cies and timing of inter-warp synchronization. ISS ISS For instance, Figure 4 illustrates the issue in which two Gen Stack Gen Stack DUT ISS Axiomatic Model warps (W1, W2) synchronize at a shared barrier. Here, a bar- Bottom denotes a synchronization instruction that stalls each Loop SD [addr],r rier LD Level Co-simulation participating warp until all warps in the same barrier group Testcase arrive. Suppose warp W1 in the DUT arrives at a barrier in- (i) SIMT Program Generation (ii) Differential Test struction i1, causing the GRM to mark W1 as inactive and wait for it. When W2 in the DUT subsequently arrives at the Figure 5: Fuzzing loop overview. matching barrier i2, activating W1, the GRM faces an ordering
2468 35th USENIX Security Symposium USENIX Association
Method Cas. D.F.
4.2 SIMT Program Generation the current active thread mask (TMASK), the not-taken tar- As illustrated in the §3, generating valid SIMT programs get PC and mask (NT), and a reconvergence flag R. When is non-trivial due to the unique GPU execution model. In sampling a branch, an ISS predicts per-thread next masks. response, we decompose the generation into a novel top-down Divergence occurs if both taken and not-taken paths are ac- process, as shown in Figure 5. A top-level loop plans the tive. The generator then pushes an entry and generates the overall program structure, while a bottom-level loop generates taken path depth-first. Upon sampling a reconvergence in- the instruction stream for each warp. These loops handle inter- struction, it jumps to the pending not-taken target from the warp and intra-warp interference respectively, ensuring the stack top, generates that path, and sets R. When the not-taken resulting program follows the GPU execution model while path reaches the reconvergence point again with R already set, keeping it intricate. the entry is popped, and generation resumes after reconver- gence. This procedure yields a linearized order in which all threads share the same GPC at each step, corresponding to a 4.2.1 Target Program Structure valid interleaving of threads. The two-level generator loop constructs the overall structure Figure 6 shows an example of the generation stack. Initial of the GPU program, including multiple CTAs, warps, and execution diverges at A. Prioritizing the higher active thread basic blocks. count, the generator follows branch B (threads 0011), pushing Instructions. The target program follows the GPU hierar- A’s metadata onto the stack. Generation proceeds along path chy: it contains CTAs, warps, and a per-warp instruction ① until the join point G. Seeing R = 0 at the top of the stack stream. We categorize instructions as control-flow transfer (i.e., an alternative path is still pending), the generator redi- (e.g., branches, unconditional jumps, reconvergence, and join) rects the GPC to the not-taken path C (threads 1100), which or non-control-flow. For a control-flow instruction, if the next then diverges into paths ② and ③. This recursive process con- PC falls through to the immediate successor instruction, it is tinues until the GPC returns to G with R = 1 (signifying full not-taken. Otherwise, the instruction is classified as taken. reconvergence), prompting a stack pop and resumed genera- Basic Blocks. A per-warp instruction stream can be viewed tion after G. as a sequence of basic blocks. A basic block is a straight-line Gen. Gen. Stack State instruction sequence with a single entry and a single exit, A 1111 Order PC TMASK NT R which ends at a control-flow transfer instruction. For each C 1100 A 1111 1100 0 block, the generator allocates a disjoint memory region and B 0011 D 1000 E 0100 PC TMASK NT R assigns an entry address. The block’s instruction sequence F 1100 CA 1111 1100 1 1100 0100 0 is randomly sampled subject to data-flow and control-flow G 1111 PC TMASK NT R constraints (§4.2.2). Blocks are connected by control-flow A 1111 1100 1 transfer instructions, forming a Control-Flow Graph (CFG). Diverge Reconverge C 1100 0100 1 (a) Control Flow Graph (b) Generation Stack
4.2.2 Per-Warp Generation Loop Figure 6: Generation stack examples. At the bottom level, the per-warp loop constructs basic blocks for each warp. A warp executes a single instruction stream as The linear order induced by the generation stack has several long as its threads are converged. Thus, we use one Genera- desirable properties: (i) it enables using the ISS during gener- tion Program Counter (GPC) to determine the addresses at ation to obtain runtime context and resolve branch outcomes, which instructions are generated. However, after divergence, allowing the procedure to bootstrap; (ii) it avoids maintain- threads may follow different control-flow paths, implying ing a separate PC for every feasible path, mitigating path multiple feasible PCs and potentially exponential path growth explosion; and (iii) it guarantees the absence of dead code in with branch depth. Naively emitting disjoint code for both the linearized order, providing a sound program order for the directions of every branch wastes instruction budget on dead barrier planner and memory-consistency checking. code, whereas emitting only one direction can prevent threads from reconverging and yield a subset of threads permanently 4.2.3 Top-level Generation Loop excluded. Therefore, generation must emit both paths while ensuring that they eventually merge at a reconvergence point. The top-level generation loop coordinates multi-warp execu- Generation Stack. Inspired by the hardware SIMT stack, tion by: (i) placing synchronization barriers across scopes we design a generation stack to track divergence and recon- to prevent deadlocks; and (ii) interleaving the generation of vergence during per-warp code emission. Guided by an ISA memory operations to track potential cross-thread and cross- Simulator (ISS), the generator resolves branch outcomes and warp conflicting accesses. emits both directions of divergent branches while maintain- Barrier Planner. GPU warps coordinate execution via syn- ing a single GPC. Each stack entry stores the branch PC, chronization barriers. However, improper barrier placement
USENIX Association 35th USENIX Security Symposium 2469
can lead to deadlocks, where warps wait indefinitely. Given per-warp generation, the generator maintains a cursor c(w) a set of named barrier operations B = {b1, b2, . . . , bk}, each into B (initialized to 0). When a barrier opcode is sampled, distinguished by an ID and a scope(b) ⊆ G (where G is the warp w consults B[c(w)] and emits the barrier iff w ∈ scope. set of all warps), a barrier is released only after all participants Once all warps in the scope have emitted it, the record is re- in its scope have arrived at the barrier with the matching ID. moved from B to prevent duplication. Finally, c(w) advances We first analyze the deadlock conditions. Figure 7 illus- to the next record. A warp concludes generation only when trates warp execution under different barrier placements. Fig- c(w) reaches the end of B. Figure 8 shows a random pro- ure 7a shows a deadlock where warp 0 waits for warp 1 at gram skeleton generated by the barrier planner for 5 warps. A barrier B1, while warp 1 waits for warp 0 at B2, stalling both. proof of the barrier planner’s deadlock freedom is provided Conversely, Figure 7b shows a valid placement where both in Appendix A. warps reach the same barrier B and proceed without deadlock. Warp0 Warp 0 Warp 1 Warp 0 Warp 1 Warp1 Random instructions Warp2 Program Order Warp3 Barrier A Barrier B Warp4 Barrier C Barrier D (a) Deadlock Case (b) Normal Case Figure 8: Barrier planner example: barriers A–D have 4, 2, 3, Figure 7: Barrier examples. and 3 arriving warps, respectively.
Then, we formally model the deadlock conditions. Let Iw Interleaved Warp Generation. GPU loads and stores pro-
be the instruction sequence of warp w, and let pow denote vide a mechanism for inter-warp communication. However, po the program order relation, where i → 1 i2 if and only if i1 under weak or relaxed memory consistency, a load may ob- is encountered before i2 in the linearized emission order for serve different values depending on the timing of write propa- warp w. Since pow is derived from the linearized emission gation and arbitration. Because an ISA-level Golden Refer- order induced by the generation stack (§4.2.2), the generator ence Model (GRM) typically resolves each load deterministi- avoids emitting unreachable dead code in this order: each cally, naive differential testing can misclassify such correct- emitted instruction is eventually encountered by the warp but-different DUT behaviors as bugs. We therefore track mem- under a well-defined thread mask. We define the group-barrier ory accesses online and classify potentially conflicting loads happens-before relation (g-HB) for a barrier b in group g: for later memory-model validation. We implement an interleaved warp generation loop. Each ∀w1, w2 ∈ scope(b), warp emits instructions in the intra-warp linear order induced -HB ∀o1 ∈ Precb(w1), ∀o2 ∈ Succb(w2) : o1 −g− by the generation stack. When it reaches a memory instruc- −→ o2 (1) tion, the generator records its effective address and operands, where Precb(w) = {o ∈ Iw | o po pauses the current warp generation, and switches to another Iw | b po →ʷ b} and Succb(w) = {o ∈ warp. After all per-warp generations have paused, a global →ʷ o} describe the sets of instructions that precede and pass identifies conflicts and taints the corresponding loads. succeed barrier b in warp w, respectively. We define a load as conflicting if it reads an address that Deadlocks fundamentally arise from cycles in the graph another thread has written or will write. formed by g-HB S po. Such a cycle signifies a circular de- To increase the chance of conflicts, we implement a load- pendency where every instruction in the loop must await the store replay mechanism that keeps a history of effective ad- completion of its predecessor, thereby halting progress in- dresses from the last k memory instructions. The generator definitely. A valid barrier placement must ensure that g-HB induces an address collision by sampling a previously ob- respects the program order pow, thereby preventing cycles. served address A and choosing the immediate imm = A −V We propose a barrier planner that pre-constructs a global relative to the current base register V . If the attempt is invalid, barrier program order. To avoid interleaving-induced cycles, it resamples and retries. it enforces a consistent barrier-ID sequence across warps. The Taint Registers and Memory. The conflicting loads above planner samples a globally unique sequence B of records are a concrete source of non-determinism. Under weak or re- (id, scope), and uses B as a skeleton into which it intersperses laxed consistency, the write observed by a load is not uniquely random non-barrier instructions. Scopes are chosen stochas- determined by program order and may vary depending on the tically from those supported by the target: global and CTA microarchitectural timing. As a result, the DUT may legiti- scopes on Vortex, and CTA scope on OpenGPGPU. During mately differ from an ISA-level GRM. We capture this and
2470 35th USENIX Security Symposium USENIX Association
other sources of divergence using a taint abstraction. We taint reached, after which a new opcode is selected. a register or memory address when its hardware value may While most instructions allow flexible operand choices, legally differ from the ISS due to unmodeled microarchitec- Warp-level Matrix Multiply-Accumulate (WMMA) and tural timing and cross-thread memory conflicts. control/status-register instructions impose strict constraints. During differential testing, we send tainted loads to the We enforce these constraints (e.g., on the active thread count memory validator and copy the DUT load results into the or rounding mode) through rejection sampling, repeatedly GRM context. We treat these tainted results as don’t-care for resampling operands until a valid configuration is obtained. ISA-level comparison to prevent cascading mismatches, while the correctness of the tainted memory behaviors is validated 4.3 Differential Testing Harness separately by the memory validator. Tainting also helps filter known hardware bugs. Ideally, To address the challenges discussed in §3, we propose a dif- bugs should be fixed immediately after discovery. However, in ferential testing framework that: (i) reorders writeback traces practice, remediation often takes substantial time. After root- using instruction IDs to serialize out-of-order events; and (ii) cause analysis, we add implicated registers, addresses, and constructs microarchitectural event graphs to verify memory opcodes as taint-based generation constraints, so the fuzzer consistency using axioms. avoids synthesizing instructions that persistently hinder test- Untainted Operations. We verify instructions whose results ing or excludes their tainted results from differential testing. are not tainted (primarily non-memory instructions) by com- Additionally, if a load/store address depends on tainted paring per-instruction writeback traces between the DUT and registers, non-determinism may produce unaligned addresses. GRM. Each writeback event records the PC, thread mask, The generator therefore resamples operands to keep addresses destination register index, and written value. legal. For branches with tainted operands that may desynchro- Because GPU instructions may complete and write back out nize control flow, we insert an unconditional jump from the of order, conventional lockstep differential testing becomes in- not-taken to the taken path. If no untainted operand sources valid (§3, Challenge 3). We therefore reorder writeback events remain, we refresh a random subset of registers with random- using the globally unique instruction IDs assigned at the is- ized immediate constants. sue stage and propagated to writeback, thereby recovering a Taint Propagation. The generator tracks taint online as soft- deterministic order consistent with per-warp issue order. ware metadata, requiring no GPU RTL changes. For each The software maintains a per-warp completion queue candidate instruction, it uses the ISS to emulate execution, re- (FIFO) to track instruction status, recording the instruction cover operands and effective addresses, and update per-warp ID, writeback status (pending/completed), and the associated register tags and per-address memory tags. Taint propagates trace, as shown in Figure 9. We instrument the design using along data flows via the following rules: (i) tainted source reg- DPI-C to capture issue and writeback events. When an in- isters taint destination registers; (ii) stores with tainted data or struction is issued, its ID is enqueued with a pending status. address operands taint the written addresses; (iii) loads with Upon writeback, we locate the corresponding record using the tainted addresses taint destination registers. instruction ID, mark it as completed, and populate it with the hardware writeback data, including register index and value. 4.2.4 Random Instruction Selection The testing harness periodically polls the head of the com- pletion queue. When the head instruction is marked com- Within each basic block, the generator stochastically popu- pleted, the GRM advances the corresponding warp by one lates the instruction stream to induce diverse behaviors. For instruction to generate a reference trace, which is then com- each instruction, it samples an opcode from a categorical dis- pared against the hardware trace. This process serializes out- tribution whose parameters are drawn i.i.d. from a uniform of-order writeback events. distribution, and then selects operands (registers, immediates) However, an arbitrary warp stepping order can violate inter- and control bits according to the ISA. Control-flow opcodes warp synchronization assumptions and introduce spurious can induce thread divergence and are therefore generated by mismatches (§3, Challenge 3). To preserve synchronization the bottom-level loop (§4.2.2). semantics, we propose a heuristic scheduling policy that main- For memory opcodes, the generator randomly assigns ac- tains a counter of in-flight barrier instructions per warp, as cess sizes and effective addresses. To avoid uninitialized reads, shown in Figure 9. The counter is incremented when a bar- it consults a randomized allocator that maintains an ordered rier is issued and decremented when that barrier is dequeued list of allocated regions: each load/store computes its effec- from the completion queue. When multiple queues are ready, tive address and performs a binary search over this list. If the a scheduler prioritizes the warp with the non-zero counter address is not covered, the allocator creates a new region, ini- before stepping any instructions, which ensures that all warps tializes it with random values, and registers it. For stores, we involved in a barrier reach it before any proceed past it. additionally reject candidates overlapping code or read-only Tainted Operations. Relaxed GPU memory consistency per- segments. Invalid addresses are resampled until a threshold is mits multiple legal read-from (rf ) choices, and taint in Fuz-
USENIX Association 35th USENIX Security Symposium 2471
Per Warp Status Scheduler observes a store c, it must be ordered before any later store
Cnt b to the same location in co. Additionally, ppo (preserved
DUT ID Wb Trace
ID Wb Trace Index program order) captures the subset of program-order edges
ISS n-1 between memory operations that the ISA requires to be pre-
ISS n served (e.g., due to dependencies and fences), allowing others
Out of Order In Order ISS n+1 to reorder. Relations spanning threads are termed external and
are denoted by the suffix ‘e’. For example, rfe denotes the rf
Figure 9: Per-warp completion queue with inter-warp syn- edges whose endpoints are in different threads or warps.
chronization scheduler. For memory-model validation, litmus tests are widely used [1, 26, 33]. However, these tests are inherently directed: they target preselected relation patterns (e.g., specific cycles) zGPU covers only this read-from selection uncertainty. For rather than checking arbitrary observed traces against the tainted loads, we defer checking to a memory consistency model. For instruction-level differential testing, we use DPI- validator that uses an axiomatic model as the reference. Af- C instrumentation to capture memory events from the GPU. ter validating the load, we synchronize the data-race load’s Specifically, we hook the L1 cache response handshake. For writeback results to the ISS to validate subsequent operations. each read, we record its physical address and the returned Axiomatic memory models characterize correctness via data from the ISS. We then query the generator’s store map relations over memory primitives (reads, writes, and fences): with the resulting (address, data) pair to uniquely identify the an execution trace is valid iff it satisfies the axioms. Open- source store and establish rf . The store map is built during source GPU models largely share a common relational core, the generation phase. We employ rejection sampling to assign while vendors often add proprietary rules. We present our unique values to stores targeting the same physical address. validator using a widely recognized weak memory model [32]. This eliminates ambiguity. The generator also emits dummy write events ahead of all other microarchitectural events to Table 1: Memory Consistency Axioms represent initial memory values. Detecting write events necessitates tracking L1 and L2
Axiom 1 (SC-per-Location). cache microarchitectural states. Because co is a partial order, acyclic morally_strong ∩ (rf ∪ co ∪ fr) ∪ po_loc writes that are not morally_strong do not participate in co. For each scope, we maintain a last-writer map W [scope][addr] Axiom 2 (Main Model). that records the metadata of the most recent write to address acyclic co ∪ rfe ∪ fr ∪ ppo addr (e.g., instruction ID and thread mask). For each write, Axiom 3(Atomic). we check the last writer record W [scope][addr]. If present, empty ((morally_strong ∩ fr); (morally_strong ∩ co)) ∩ rmw we assert a coherence order co from the last to the current writer, then update W [scope][addr]. Intuitively, the validator models an execution as a memory- Using the detected rf and co relations, we derive fr and event graph and rejects forbidden cycles. Axiom 1 enforces rfe. We construct the memory event graph by combining per- per-location coherence by requiring a consistent order for warp sequential order relations (po, po_loc) with ISA-defined same-address accesses. Axiom 2 extends ordering across ad- ppo rules, and verify the axioms in Table 1. Detected cycles dresses through dependencies, fences, and cross-thread com- trigger a traceback of the involved events to localize potential munication. Axiom 3 ensures RMW atomicity by forbidding memory consistency bugs. any visible write between its read and write parts. Completeness and Soundness Analysis. The validator is We summarize the relations in the axiomatic model below. complete with respect to the implemented axiomatic memory po b denotes that a precedes b in program order. po_loc fr, and ppo relations, any violation of the encoded axioms a → model: after FuzzGPU reconstructs the relevant po, rf , co, restricts po to instructions accessing the same memory ad- appears as a failed graph check. Meanwhile, its soundness is dress. Two instructions are morally_strong if they are ordered practical rather than absolute. False positives can still arise by po, or if both are strong instructions with the same scope if the target’s intended memory model differs from the en- and overlapping addresses. co is a partial order over globally coded axioms, or if non-memory known bugs corrupt the visible stores, only morally strong stores participate and are graph events. We mitigate false positives by tainting known comparable. a → and a co b holds if a and b store to the same location mismatching states. becomes globally visible before b. The read-from re- rf b iff load b lation rf links loads to their source stores: a → 5 reads the value written by store a. We derive the from-read Evaluation fr b holds if load a reads from a In this section, we evaluate FuzzGPU’s effectiveness from the relation fr from co and rf . a → co b. Intuitively, fr captures that once a read perspectives of program metrics (§5.1), fuzzing performance store c such that c →
2472 35th USENIX Security Symposium USENIX Association
...
(§5.2), coverage (§5.3), and bug discovery (§5.5). as triaged in Figure 3. DiveFuzz surpasses Cascade because
Experiment Setup. We conducted our experiments on an its seed construction employs dedicated counter registers and
Ubuntu 24.04 system featuring an Intel(R) Xeon(R) 6982P-C constant comparison values for conditional branches, thereby
192-core vCPU and 768GB of RAM. For RTL simulation, we reducing the likelihood of branch misconstruction.
utilized the widely adopted Verilator 5.034. Memory Event Statistics. Table 2 summarizes the scale of
We tested the latest revisions of the open-source Vortex- memory-relation edges induced by executing each Vortex64
64bit and OpenGPGPU (ventus-env), corresponding to com- program generated during 20 core-hours of FuzzGPU. We ex-
mit IDs 2189194 and 7e97907, respectively. We retained the pand both scalar and vector load/store operations at byte gran-
default configuration parameters for all designs. For each ularity. All initial memory reads are represented as rfi edges
DUT, we conducted a 20-core-hour fuzzing campaign using from the dummy initial value write events. The results show
10 distinct initial seeds. The maximum instruction length per that FuzzGPU exercises a large number of memory events,
test program was uniformly sampled from 100 to 10,000. including cross-thread relations such as coe and rfe, thereby
The instruction pool covers all hardware-supported ISA providing rich scenarios for memory-consistency checking.
extensions, including integer (RV32I/RV64I), multiplication/-
division (RV32M/RV64M), atomics (RV32A/RV64A), floating- Table 2: Memory relation edge statistics per test: co coherence
point (RV32F/RV64F, RV32D/RV64D), vector (Ventus-RVV), order, rf read-from, fr from-read, and ppo preserved program
and Vortex-customized instructions. order. Suffix ‘e’ marks external cross-thread edges, and rfi
Baselines. To the best of our knowledge, no publicly available reads from initial writes.
GPU hardware fuzzers exist for comparison. Since existing
processor fuzzers primarily target CPUs and are inapplicable Relation Min Q1 Median Q3 Max
to GPUs, we established baselines by adapting two state-of- co 0 96 312 636 1645
the-art CPU fuzzers, DiveFuzz [10] and Cascade [39]. We coe 0 24 284 624 1644
re-implemented them to support GPU ISAs, generating per- rf 0 9524 20124 30335 44308
warp random instruction sequences augmented with thread rfe 0 140 678 1954 9203
initialization to form GPU programs. Our baseline porting rfi 0 9148 18847 27362 36633
follows a minimality principle: we only replace ISA encod- fr 0 0 160 960 13453
ing and toolchain backends and add the minimum required ppo 0 1034 2858 5160 20587
initialization to launch GPU warps. These adaptations, named
Ported-DiveFuzz and Ported-Cascade, utilize our differ-
ential testing framework for bug detection.
5.2 Fuzzing Performance
5.1 Program Metrics To quantify the impact of increased instruction length on
Instruction Completion Rate. Our motivation analysis high- throughput, we measured the generation throughput, defined
lighted the impact of GPU-specific execution models on in- as the ratio of generated fuzzing instructions to the total gener-
struction completion. We assessed the ratio of executed in- ation time, while varying the number of generated instructions.
struction PCs to total generated instructions in FuzzGPU by For each configuration, we ran Vortex for 0.4 core hours and
running 5,000 test programs on an ISS oracle. reported the geometric mean.
105 Ported Cascade Ported DiveFuzz FuzzGPU
100 Cascade DiveFuzz FuzzGPU
103
50
0-500 500-5K 5K-50K 50K-500K
0 0 10 20 30 40 50 60 70 80 90 100 Gen Instr. (K)
Instruction Completion Rate (%) Figure 11: Generation throughput (y-axis is in log scale).
Figure 10: Completion rate of different fuzzers. Figure 11 shows that as the number of generated instruc-
tions increases, FuzzGPU’s throughput initially rises before
Figure 10 shows that FuzzGPU consistently outperforms plateauing. Overall, FuzzGPU achieves significantly higher
the baselines. It achieves a median instruction completion rate generation throughput than the baseline generators. In the
of 100% and a mean of 99.8%. DiveFuzz and Cascade exhibit range of 5,000 to 50,000 instructions, FuzzGPU maintains
low completion rates (medians of 5.0% and 45.4%, respec- a throughput of approximately 53,970 instructions/second,
tively) due to their inability to handle GPU execution models, which is 11× that of Ported Cascade and 399× that of Ported
USENIX Association 35th USENIX Security Symposium 2473
Test Cases (%)
Throughput (IPS)
DiveFuzz. DiveFuzz exhibits lower throughput because its Prog. Construction Memory Check RTL Emulation
core algorithm relies on dynamic guidance from an ISS. As
previously noted, its lack of a GPU execution model often re- File Write
sults in programs with low instruction completion rates (§3). 0.4% 1%
Consequently, many instructions trigger exceptions during Vortex 11% 0.9% 88%
simulation, causing them to be discarded and state rollbacks. Open
In contrast, Cascade only invokes the ISS at the final stage GPGPU 28% 5% 66%
of program generation, resulting in higher throughput than 0 20 40 60 80 100
DiveFuzz. Percentage of Total Time (%)
Ported Cascade Ported DiveFuzz FuzzGPU Figure 14: Breakdown of fuzzing time per test.
103
102 ing for each GPU test during the 20-core-hour fuzzing cam-
0-500 500-5K 5K-50K 50K-500K paign. Figure 14 illustrates the breakdown of fuzzing time,
Gen Instr. (K) specifically the proportions of generation, binary writing, and
Figure 12: Effective throughput (y-axis is in log scale). simulation (differential testing) time relative to the total du-
ration. We found that generation accounts for only a small
portion of the total time, approximately 28% on OpenGPGPU,
To evaluate effective instruction throughput, we measured whereas simulation dominates at 66%, and memory checking
the number of actually executed fuzzing instructions per unit accounts for 0.9%.
of generation and simulation time. These measurements were
conducted on Vortex, comparing FuzzGPU against various
baselines. 5.3 Coverage Evaluation
Figure 12 shows that FuzzGPU’s effective throughput ini- We conducted a 12-core-hour fuzzing campaign, utilizing
tially increases before plateauing within the tested range. Verilator to collect coverage metrics supported by the DUT.
Overall, FuzzGPU achieves higher effective throughput than
Cascade and DiveFuzz. Cascade’s effective throughput fol-
lows a pattern of initial increase followed by a decline, consis- Line Coverage (26700 points)
tent with the previous observations on CPUs [39]. In the range 90
of 5,000 to 50,000 instructions, FuzzGPU maintains an effec-
tive throughput of approximately 3,815 instructions/second, 80
which is 28× higher than Ported DiveFuzz and comparable
to Ported Cascade. 70 FuzzGPU Ported-Cascade Ported-DiveFuzz
To further understand simulation performance under differ-
ential testing, we measured the geometric mean of simulation 80 Toggle Coverage (36318 points)
times across different instruction generation lengths.
60
104 OpenGPGPU Vortex 40
103
102 Expression Coverage (11547 points)
500 5K 50K 500K 80
Gen Instr. 60
Figure 13: Performance of simulation (differential testing). 40
20
At an instruction count of 500,000, the simulation through-
put of Vortex64 is approximately 21,576 instructions per 0 2 4 6 8 10 12
second, whereas the average simulation throughput of Core Hours
OpenGPGPU is about 7,654 instructions per second, as shown
in Figure 13. This discrepancy is likely attributable to differ- Figure 15: Line, toggle, and expression coverage comparison.
ences in code structure and design complexity.
We measured the runtime breakdown of the generator, sim- Figure 15 presents the coverage achieved by different
ulation, non-memory operation checking, and memory check- fuzzers on the OpenGPGPU design. FuzzGPU consistently
2474 35th USENIX Security Symposium USENIX Association
Time (ms) Throughput (IPS)
Coverage (%) Coverage (%) Coverage (%)
103 FuzzGPU
Cas.
D.F.
102
101
100
S1 V2 V3 V5 V6 V8 V9 V10 V11 V13 V14 X1 X2 X3
Figure 16: Time to bug discovery across different fuzzers and GPU designs. Detection of V1 and V12 involves time-consuming
waveform analysis. V4, V7, P1 and S2 are patched to avoid crashes, as detailed in Appendix B. Time is measured in core-seconds.
achieves higher coverage across all three metrics. Line cov- 5.5 Bug Discovery Efficiency
erage is relatively the easiest to saturate. A simple test often
covers the majority of code lines. FuzzGPU achieves 94% line To evaluate the effectiveness of differential testing in bug
coverage. In contrast, toggle and expression coverage require discovery, we summarized the bugs identified across various
more diverse tests to trigger different signal transitions and ex- designs during a 20-core-hour fuzzing campaign. In total, we
pression evaluations. FuzzGPU achieves approximately 80% discovered 20 previously unknown bugs: 14 in OpenGPGPU,
coverage for both toggle and expression metrics. Notably, 3 in Vortex, and 3 in the ISS, validating the effectiveness of
for expression coverage which is a more challenging metric, FuzzGPU’s bug detection capabilities. Table 3 provides a
FuzzGPU achieves 2× that of Ported Cascade. summary of these bugs. Compared to the native regression
test suites of Vortex and OpenGPGPU, FuzzGPU uncovered
17 previously unknown RTL bugs and 3 ISS bugs.
Time to Bug Discovery. We compared the time required to
5.4 Ablation Study discover bugs across different fuzzers.
Figure 16 illustrates the time to first detection for each bug
To validate the effectiveness of FuzzGPU’s moving parts, we during a 20-core-hour run, repeated 10 times with distinct
conducted an ablation study by disabling: (i) the generation random seeds. Notably, Ported Cascade and Ported DiveFuzz
stack (which inherently disables simulation guidance), (ii) the were used only for instruction generation. They cannot inde-
barrier planner, and (iii) taint propagation in interleaved warp pendently detect GPU bugs. We therefore use the proposed
generation. We profiled the execution of 5,000 test programs, differential testing framework as the sole bug oracle, using an
each generated with a fixed length of 10,000 instructions on identical checking configuration across all generators. This
the OpenGPGPU ISS. comparison isolates how well the generated tests trigger bugs.
The results indicate that FuzzGPU discovers bugs within
Impact on Program Metrics. Removing the generation stack 1,000 core-seconds in most cases and, overall, achieves a
reduces the median instruction completion rate from 97.3% shorter time to bug discovery than the baselines. FuzzGPU
to 5.6%. It also removes divergence: the median divergent- successfully triggered and identified 14 bugs in Figure 16,
branch ratio drops from 19.1% to 0%, and the median SIMT whereas the programs generated by Ported Cascade and Di-
stack depth decreases from 8 to 0. Without the barrier plan- veFuzz triggered 7 and 8 bugs, respectively, under our differ-
ner (i.e., with random barrier generation), the median number ential testing framework. This demonstrates that FuzzGPU’s
of completed barriers decreases from 35 to 15 per test. The generation techniques effectively synthesize programs that
median instruction completion rate also drops to 88.6%, sug- expose hardware bugs.
gesting that the barrier planner mitigates deadlocks. Disabling
taint propagation during interleaved warp generation reduces 5.6 Bug Analysis
the completion rate to 17.1% and lowers the median SIMT
stack depth from 8 to 6. This indicates that taint propagation is Memory Bugs. FuzzGPU identified a private memory ad-
important for high completion rates under non-determinism. dress calculation bug (V1). When active threads are fewer
Impact on Coverage. We measured final expression coverage than the hardware capacity, the GPU miscomputes physi-
on OpenGPGPU across 100 random seeds under the ablations cal addresses and triggers out-of-bounds accesses. FuzzGPU
above. The vanilla configuration reaches 59%. Disabling the also detected a flaw in atomic memory operations (V2): in-
generation stack reduces coverage to 37%. Removing taint structions such as amoadd and amoswap produce incorrect
propagation in interleaved warp generation reduces it to 41%. arithmetic or logical results. Finally, the hardware ignores the
Excluding the barrier planner reduces it to 58%. These results nfield parameter in vector load/store instructions (V3). It de-
show that each component contributes to higher coverage. faults to one field, truncating operations such as vlsseg2e32.
USENIX Association 35th USENIX Security Symposium 2475
Time (seconds)
Table 3: Summary of discovered bugs.
Design Bug Description CVE/#Issue/#PR CWE
(V1) Incorrect private memory address calculation. CVE-2025-70015 CWE-119
(V2) Atomic instructions yield incorrect results. PR #105 CWE-682
(V3) Vector load/store ignores nfield. CVE-2025-70011 CWE-682
(V4) FP ops corrupt integer registers. Issue #97 CWE-682
(V5) Vector compare reads vector instead of scalar. CVE-2025-70009 CWE-682
(V6) The MISA register was improperly initialized. CVE-2025-70014 CWE-665
OpenGPGPU (V7) Thread ID CSR yields invalid values when only a portion of threads are active. CVE-2025-70012 CWE-665
(V8) Inactive threads perform write-back. CVE-2025-70007 CWE-787
(V9) vfmax returns hardcoded zero on NaN. CVE-2025-70010 CWE-682
(V10) Multiply-high returns the low 32 bits. CVE-2025-70008 CWE-682
(V11) Spurious register write-back in tensor ops. CVE-2025-70016 CWE-119
(V12) I-Cache deadlock on specific sequences. PR #108 CWE-833
(V13) No write-back for certain vmv.x.s, causing SM hangs. Issue #98 CWE-833
(V14) Vector mask-register logical operations yield incorrect results. PR #106 CWE-682
Vortex 64 (X1) Scoreboard releases the freed entry. PR #356 CWE-672
(X2) Conditional zero ops show value mismatches against the ISS. PR #358 CWE-682
(X3) Tensor ops corrupt registers under BF16 and FP16 output formats. PR #359 CWE-682
OpenGPGPU-Spike (P1) Improper sign-extend for 32-bit results, misleading conditional branches. CVE-2025-70013 CWE-682
Vortex-Simx (S1) sra is implemented as a logical right shift. PR #320 CWE-682
(S2) jalr ignores LSB clearing. PR #339 CWE-682
Register Bugs. FuzzGPU discovered a register corruption bug vmulhsu) incorrectly yield the lower 32 bits of the product (V4) where floating-point results are incorrectly written to in- rather than the upper 32 bits (V10). Moreover, the vector teger registers sharing the same index when register extension mask-register logical operations such as vmand.mm yield in- prefixes are used, thereby corrupting subsequent integer oper- correct results that are inconsistent with Spike (V14). In Vor- ations. FuzzGPU also identified an operand fetch error (V5) tex, the generated conditional zero operations produce flags in vector comparison instructions (vmsgt.vx, vmsgtu.vx), that mismatch the ISS, causing erroneous behavior in subse- where the hardware erroneously reads from vector registers quent control-flow instructions that rely on this flag (X2). The instead of the specified scalar registers. Moreover, FuzzGPU root cause likely stems from inconsistent instruction decoding detected a use-after-free vulnerability in the Vortex score- between the RTL and the ISS. board logic (X1), occurring when the hardware attempts to Tensor Core Bugs. In Vortex, tensor multiply-accumulate release registers that are not currently in use. instructions corrupt the architectural state (X3) because FP16 Configuration and State Management Bugs. FuzzGPU un- and BF16 accumulator outputs are incorrectly generated. In covered multiple configuration and state management issues. OpenGPGPU, FuzzGPU detected erroneous register write- First, the MISA register in OpenGPGPU, which declares ISA backs caused by tensor core instructions that executed without extension support, is improperly initialized, causing undefined triggering an illegal-instruction fault, even though the current behaviors for valid instructions (V6). Second, when execut- open-source release (V11) does not include a tensor-core unit. ing with a partial thread mask, the CSR register tracking the Microarchitectural Deadlock Bugs. OpenGPGPU exhibits thread ID fails to update, leading to incorrect ID retrieval a microarchitectural deadlock where specific instruction se- and subsequent errors in thread-dependent computations and quences cause the instruction cache to hang the Streaming memory accesses (V7). Lastly, OpenGPGPU hardware erro- Multiprocessor (SM), halting all subsequent instruction retire- neously performs write-backs for inactive threads when the ment (V12). Additionally, the vmv.x.s instruction hangs the active count is below the maximum (V8). While this poses a entire SM, halting subsequent write-backs (V13). risk of context corruption, the current OpenCL stack does not Simulator Bugs. FuzzGPU also uncovered a discrepancy utilize this configuration. in the Vortex ISS, where the arithmetic right shift (sra) is Calculation Bugs. FuzzGPU detected several arithmetic er- incorrectly implemented as a logical right shift (S1). More- rors. In OpenGPGPU, the vfmax instruction violates the over, Vortex fails to clear the least significant bit (LSB) of RISC-V standard by returning a hardcoded zero instead of PC during register-indirect jumps, as required by the specifi- the valid operand when the other operand is NaN (V9). Addi- cation, causing PC misalignment and fetch errors (S2). The tionally, Multiply High instructions (mulhu, vmulhu, mulhsu, Vortex developers noted that this bug is triggerable only by
2476 35th USENIX Security Symposium USENIX Association
hand-written assembly. The compiler aligns function-entry 6 Discussion addresses, so normal kernels remain unaffected. In addition, the OpenGPGPU-Spike zero-extends boxed 32-bit floats into This section discusses the limitations of FuzzGPU and poten- 64-bit registers (P1), violating Spike’s required sign-extension tial directions for improvement. and causing incorrect signed comparisons when reinterpreted Open-Source GPU Targets. Open-source RISC-V hardware as integers. This bug can misdirect branches. has facilitated processor research and has recently enabled explorations in high-performance GPU designs. We evalu- 5.7 Security Analysis ated FuzzGPU on Vortex [41] and OpenGPGPU [27, 28], two full-stack open-source GPGPUs that support RTL val- In this section, we assess the security impact of the discovered idation, FPGA prototyping, and commodity GPU program- bugs. ming models. OpenGPGPU is the largest open-source GPU we identified by coverable lines. It has 26.7K coverable Out-of-Range Memory Access. Private memory address lines according to Verilator’s metric, while other available miscalculation caused by V1 can trigger out-of-bounds ac- designs are much smaller (e.g., MIAOW, Nyuzi, Vortex, and cesses, enabling memory corruption. We assume an attacker TinyGPU). The targets are also practical: Vortex supports can launch and execute a custom kernel on the target GPU. OpenCL, OpenGL, and FPGA deployment [41]. It has been Under this threat model, V1 can trigger out-of-bounds ac- integrated into a METASAT safety-critical RISC-V SoC plat- cesses to private memory, enabling reads or modifications of form [16]. OpenGPGPU also provides OpenCL support. sensitive data. Side Effects of Focusing on Completion Rates. FuzzGPU Data-Flow Integrity. Register corruption can taint critical improves testing efficiency by generating valid tests with computations, enabling malicious calculations or bypassing high completion rates. However, this design choice excludes safety checks (V4). For example, an attacker can craft ma- bugs that manifest only through fault-handling paths, such licious code that invokes regext to set the register-index as reserved opcodes, illegal CSRs, malformed SIMT state, extension to 32 and then writes to register 32 + R. Under ISA- or unaligned memory accesses. Future work could generate level formal checks, the program is expected not to modify invalid programs with illegal instruction encodings, invalid a critical register R. However, due to V4, the floating-point memory addresses, malformed SIMT control flow, and illegal result can be erroneously written into R at runtime, thereby hi- register accesses to explore these paths. jacking the data flow and affecting subsequent computations Scalability to Larger Designs. We evaluated FuzzGPU on that depend on R. two open-source GPUs, which are small relative to com- Control-Flow Hijack. In ROP-style attacks, attackers seek mercial designs. Scaling to larger GPUs is bottlenecked by gadgets ending in ret (i.e., jalr). If jalr leaves the LSB the RTL emulation time identified in Figure 14. Fortunately, uncleared, targets can point into the middle of instructions, hardware-accelerated emulators for large-scale RTL, such as greatly expanding gadget availability and power. Fortunately, Cadence Palladium and Synopsys ZeBu, may further improve bug S2 exists only in the ISS. The hardware automatically throughput by offloading from software simulators. truncates the low bits of the PC. Vector compare bug (V5) that Porting to Other ISAs. Although FuzzGPU follows a uni- deviates from the ISS may misdirect control-flow branches, fied GPU execution model, its current implementation targets causing them to follow paths unintended by programmers or two specific ISAs: Vortex and OpenGPGPU. The two ISAs the LLVM compiler. have different semantics. Adapting FuzzGPU to other pub- Denial of Service. An adversarial co-tenant can execute a licly released ISAs, such as AMD Southern Islands, would crafted instruction sequence to trigger an instruction-cache necessitate modifications to the ISS, differential testing har- deadlock, hanging the GPU hardware (currently observed in ness, and instruction generator. We anticipate that porting to our RTL simulator) and causing a denial-of-service condition proprietary ISA ecosystems, such as NVIDIA PTX and SASS, (V12). Moreover, incorrect MISA configuration may cause will be more challenging due to undocumented features and software to mis-detect ISA extensions and disable supported increased instruction set complexity. features (V6). Fuzzing Closed-Source GPUs. Many commercial GPU ven- Cryptographic Impact. Cryptographic libraries can be po- dors offer soft IP cores that are source-encrypted but permit tentially impacted by arithmetic errors (V10), because many RTL simulation. Applying FuzzGPU to these closed-source implementations rely on multi-precision arithmetic. Multiply- cores is technically feasible but requires vendor cooperation high instructions are commonly used to obtain the upper limb to expose the necessary DPI-C interfaces. We have designed of a product in big-integer multiplication and modular re- our DPI-C instrumentation to be minimal. For instance, im- duction (e.g., Montgomery/Barrett-style routines). With V10, plementing such instrumentation in OpenGPGPU required returning the low 32 bits instead of the high word can cor- only about 90 lines of Chisel code. However, instrumenting rupt carry propagation and reduction steps, yielding incorrect encrypted sources is impossible without vendor assistance. residues or intermediate values. Limited Baselines. To the best of our knowledge, no publicly
USENIX Association 35th USENIX Security Symposium 2477
available GPU hardware fuzzer exists for comparison. We available GPU hardware fuzzer exists that targets RTL designs therefore adapted Cascade [39] and DiveFuzz [10], which and supports SIMT-aware test generation and checking. generate instruction-level data flows and control flows. We also considered SIMD-oriented fuzzers. RVISmith [11], how- 8 ever, fuzzes compilers via SIMD data-flow sequences in Conclusion strip-mined loops, rather than the richer control-flow instruc- This paper presents FuzzGPU, the first RTL-level fuzzing tions needed for SIMT-aware generation. Future work could framework for GPU hardware. FuzzGPU addresses GPU- adapt RVISmith-style SIMD generation with SIMT-aware specific architectural challenges by generating valid Single- control, synchronization, and RTL-level differential checking Instruction, Multiple-Thread (SIMT) programs with diverse to broaden the set of GPU fuzzers available for comparison. data and control flows that conform to the execution model, en- abling effective GPU fuzzing where state-of-the-art processor 7 Related Work fuzzers fall short. We further design and implement a GPU- oriented differential testing mechanism that mitigates microar- GPU Hardware Assessment. Prior works on GPU reliabil- chitectural non-determinism and enables instruction-level vul- ity fundamentally fall under fault injection and stress cam- nerability detection and localization. Experimental evalua- paigns [8, 35, 36, 38,51]. They typically inject or stress known tions on two open-source GPUs, Vortex and OpenGPGPU, fault modes and then measure resulting error rates or miti- show that FuzzGPU uncovers 20 previously unknown bugs, gation effectiveness, rather than systematically searching for 10 of which have been assigned CVE identifiers. previously unknown functional bugs. Accordingly, such stud- ies are outside the scope of this paper. Concurrent with our 9 work, Ventus developed GVM, a UVM-like verification frame- Acknowledgement work for OpenGPGPU [27, 28]. However, it neither fuzzes This work was partially supported by the Beijing Key Lab- GPUs nor validates memory consistency models. oratory of Network Security Protection Technology (Grant GPU Software Fuzzing. GPU software fuzzers target drivers No. 2022YFB3103900) and E5YY07111K4. We gratefully and runtime libraries. Moneta [18] performs ex vivo driver acknowledge the constructive comments and helpful feedback fuzzing without physical hardware, while Fuzz4CUDA [54] provided by the anonymous reviewers. targets CUDA libraries and leverages runtime analysis to aid device-side coverage collection. Processor Fuzzing. Prior processor fuzzers predominantly Ethical Considerations target CPUs rather than GPUs [3–5, 7, 10, 12, 15, 19, 20, 23, 24, 39, 40, 42, 45–47]. Based on the bug-detection oracle they This section analyzes the ethical implications of FuzzGPU, employ, CPU fuzzers broadly fall into four categories: (i) including potential impacts on stakeholders, mitigations we assertion-based, (ii) Golden Reference Model (GRM)-based, implemented, and our rationale for publishing the work. (iii) equivalence-based, and (iv) termination-state-based. Stakeholder Analysis. We outline several major stakeholder Assertion-based fuzzers (e.g., RFuzz [24], DirectFuzz [4], groups that could be affected by the outcomes of this research. Hardware Fuzzing [42]) use user-provided assertions (e.g., in • Maintainers of Vortex, OpenGPGPU, Vortex-Simx, and SystemVerilog) as oracles, and often rely on coverage-guided OpenGPGPU-Spike. These stakeholders bore the most im- mutation to generate test stimuli. GRM-based methods com- mediate burden from our work: reproducing reports, triaging pare hardware states against a golden reference model. Sev- traces, prioritizing fixes, reviewing patches, and correcting eral fuzzers adopt instruction-level differential testing as an simulator behavior. They also benefit directly from the bug oracle [5,7,10,20,45,47], while RISCover [40] compares mul- reports and patches. We tried to reduce that burden by provid- tiple closed-source RISC-V cores to expose inconsistencies. ing traces, proof-of-concept inputs, and patches so they could Equivalence-based approaches (e.g., EPEX [22]) generate se- validate and remediate the problems more quickly, rather than mantically equivalent (or closely related) program variants forcing downstream adopters to discover and debug them to explore different data or control paths. Fuzzilicon [25] independently. further extends this technique toward microcode-level vulner- • Downstream developers and integrators. Those who reuse abilities. Termination-state-based approaches detect bugs via these open-source GPU or simulator components may need non-termination. Cascade [39] constructs entangled control- to reassess their systems in light of memory safety errors, and data-flow patterns and flags bugs by observing nontermi- arithmetic errors, invalid write-back, control-flow mistakes, nating executions. and hangs. Publication creates review and patching costs, but These fuzzers were originally designed for CPUs and do it also helps them avoid silently inheriting flawed components. not directly transfer to GPUs, because GPUs’ SIMT execution • Users of systems or research platforms built on such de- model and parallel thread organization differ substantially signs. These users may bear the downstream consequences of from those of CPUs. To the best of our knowledge, no publicly flawed GPU designs. If related designs are deployed, bugs of
2478 35th USENIX Security Symposium USENIX Association
the kind we found could affect system integrity, availability, tificity, reproducibility, and follow-on defensive work. We or memory safety. The benefit to this group is earlier identifi- therefore concluded that the expected defensive benefits are cation and remediation of such flaws before the designs are greater than the foreseeable harms. adopted more widely. nity.• Researchers and the broader hardware-security commu- Open Science The main benefit to this group is a concrete methodology for SIMT-aware GPU hardware fuzzing, plus artifacts and To support the reproducibility of our research, we provide the baselines for future verification. The main burden is that oth- complete implementation of FuzzGPU, including its SIMT ers may spend time porting the verification to other GPUs. We program generator, the differential testing harness, and the judged this tradeoff acceptable because the work fills a real hardware configurations for Vortex and OpenGPGPU. We gap without human-subject interaction, or privacy-invasive also provide fuzzing seed parameters and experimental scripts data collection.• to reproduce the fuzzing results. We provide the GPU-ported Potential malicious actors. An adversary might adapt versions of Cascade and DiveFuzz for baseline evaluation. our methodology to analyze related open-source or derivative The artifacts are available on Zenodo (https://doi.org/ GPU designs. This is a dual-use risk. The expertise threshold 10.5281/zenodo.20321391). is high, but technically capable actors could still benefit from publication. • Disclosure coordinators and CVE-assignment institutions. References These stakeholders incur coordination work, but also provide accountability and clearer communication. CVE IDs were [1] Jade Alglave, Luc Maranget, Susmit Sarkar, and Peter assigned by a CNA following our request through MITRE. Sewell. Litmus: Running tests against hardware. In Mitigations. We mitigated harm in three main ways. First, Parosh Aziz Abdulla and K. Rustan M. Leino, editors, we limited the research to isolated simulation of open-source Tools and Algorithms for the Construction and Analy- targets rather than probing deployed systems, because simu- sis of Systems, pages 41–44, Berlin, Heidelberg, 2011. lation sufficed for study verification while avoiding risks to Springer Berlin Heidelberg. uninvolved parties, including service disruption, operational costs, and unintended live-state changes. [2] Nils Bars, Moritz Schloegel, Tobias Scharnowski, Nico Second, we responsibly disclosed the vulnerabilities to Schiller, and Thorsten Holz. Fuzztruction: using fault the affected maintainers and shared traces and proof-of- injection-based fuzzing to leverage implicit domain concept inputs to support validation and remediation, because knowledge. In Proceedings of the 32nd USENIX Con- disclosure-first starts remediation before broader dissemina- ference on Security Symposium, SEC ’23, USA, 2023. tion and is less harmful than leaving maintainers or down- USENIX Association. stream users to encounter unresolved flaws without context. [3] Pallavi Borkar, Chen Chen, Mohamadreza Rostami, As of submission, the maintainers patched three vulnerabil- Nikhilesh Singh, Rahul Kande, Ahmad-Reza Sadeghi, ities and scheduled an additional fix. We contributed two Chester Rebeiro, and Jeyavijayan (JV) Rajendran. Whis- further patches. The remaining vulnerabilities are under eval- perfuzz: white-box fuzzing for detecting and locating uation. Ten CVE IDs were assigned. timing vulnerabilities in processors. In Proceedings of Third, we frame the paper around defensive methodology the 33rd USENIX Conference on Security Symposium, and verification lessons rather than exploitation guidance. SEC ’24, USA, 2024. USENIX Association. Decision. We weighed the publication decision as a balance of benefits and costs. On the benefit side, the work was con- [4] Sadullah Canakci, Leila Delshadtehrani, Furkan Eris, fined to local, isolated RTL simulation and ISA-level refer- Michael Bedford Taylor, Manuel Egele, and Ajay Joshi. ence models and did not target public infrastructure, which Directfuzz: Automated test generation for rtl designs us- reduced direct harm while preserving the main scientific ben- ing directed graybox fuzzing. In 2021 58th ACM/IEEE efit: finding flaws in GPU designs before they are adopted Design Automation Conference (DAC), pages 529–534, more widely. Publication also helps maintainers and future 2021. doi:10.1109/DAC18074.2021.9586289. researchers scrutinize, reproduce, and build on the verification methods and bug classes presented here. On the cost side, the [5] Sadullah Canakci, Chathura Rajapaksha, Leila main ethical risk is dual use. Publishing a GPU-verification Delshadtehrani, Anoop Nataraja, Michael Bedford methodology with concrete bug classes may lower the barrier Taylor, Manuel Egele, and Ajay Joshi. Processorfuzz: for others to study related open-source or derivative designs. Processor fuzzing with control and status registers We considered whether not publishing the work, or with- guidance. In 2023 IEEE International Symposium on holding vulnerability details, would be preferable. That might Hardware Oriented Security and Trust (HOST), pages 1– reduce some misuse risk, but it would also weaken scien- 12, 2023. doi:10.1109/HOST55118.2023.10133714.
USENIX Association 35th USENIX Security Symposium 2479
[6] Katharina Ceesay-Seitz, Flavien Solt, and Kaveh Razavi. [13] Heqing Huang, Anshunkang Zhou, Mathias Payer, and μCFI: Formal verification of microarchitectural control- Charles Zhang. Everything is Good for Some- flow integrity. In Proceedings of the 2024 on ACM thing: Counterexample-Guided Directed Fuzzing via SIGSAC Conference on Computer and Communications Likely Invariant Inference . In 2024 IEEE Symposium Security, CCS ’24, page 213–227, New York, NY, USA, on Security and Privacy (SP), pages 1956–1973, Los 2024. Association for Computing Machinery. doi:10. Alamitos, CA, USA, May 2024. IEEE Computer So- 1145/3658644.3690344. ciety. URL: https://doi.ieeecomputersociety. [7] Chen Chen, Rahul Kande, Nathan Nguyen, Flem- org/10.1109/SP54263.2024.00142, doi:10.1109/ ming Andersen, Aakash Tyagi, Ahmad-Reza Sadeghi, SP54263.2024.00142. and Jeyavijayan Rajendran. HyPFuzz: Formal- [14] Rodrigo Huerta, Mojtaba Abaie Shoushtary, José- Assisted processor fuzzing. In 32nd USENIX Secu- Lorenzo Cruz, and Antonio Gonzalez. Dissecting and rity Symposium (USENIX Security 23), pages 1361– modeling the architecture of modern gpu cores. In Pro- 1378, Anaheim, CA, August 2023. USENIX Associa- ceedings of the 58th IEEE/ACM International Sympo- tion. URL: https://www.usenix.org/conference/ sium on Microarchitecture, MICRO ’25, page 369–384, usenixsecurity23/presentation/chen-chen. New York, NY, USA, 2025. Association for Computing [8] Dimitris Gizopoulos, George Papadimitriou, and Machinery. doi:10.1145/3725843.3756041. Odysseas Chatzopoulos. Estimating the fail- [15] Jaewon Hur, Suhwan Song, Dongup Kwon, Eunjin Baek, ures and silent errors rates of cpus across isas Jangwoo Kim, and Byoungyoung Lee. Difuzzrtl: Dif- and microarchitectures. In 2023 IEEE Interna- ferential fuzz testing to find cpu bugs. In 2021 IEEE tional Test Conference (ITC), pages 377–382, 2023. Symposium on Security and Privacy (SP), pages 1286– doi:10.1109/ITC51656.2023.00056. 1303, 2021. doi:10.1109/SP40001.2021.00103. [9] Kevin Immanuel Gubbi, Mohammadnavid Tarighat, Arvind Sudarshan, Inderpreet Kaur, Pavan Dheeraj Kota, [16] Marc Soé i Bonet, Jannis Wolf, and Leonidas Kosmidis. Avesta Sasan, and Houman Homayoun. State of hard- A RISC-V multicore and GPU SoC platform with a ware fuzzing: Current methods and the potential of ma- qualifiable software stack for safety critical systems, chine learning and large language models. In 2025 2025. URL: https://arxiv.org/abs/2502.21027, 26th International Symposium on Quality Electronic arXiv:2502.21027. Design (ISQED), pages 1–7, 2025. doi:10.1109/ [17] Yuancheng Jiang, Chuqi Zhang, Bonan Ruan, Jiahao Liu, ISQED65160.2025.11014308. Manuel Rigger, Roland H. C. Yap, and Zhenkai Liang. [10] Zihui Guo, Miaomiao Yuan, Yanqi Yang, Liwei Chen, Fuzzing the php interpreter via dataflow fusion. In Pro- Gang Shi, and Dan Meng. Divefuzz: Enhancing cpu ceedings of the 34th USENIX Conference on Security fuzzing via diverse instruction construction. In Proceed- Symposium, SEC ’25, USA, 2025. USENIX Associa- ings of the 2025 ACM SIGSAC Conference on Com- tion. puter and Communications Security, CCS ’25, page 1964–1978, New York, NY, USA, 2025. Association [18] Joonkyo Jung, Jisoo Jang, Yongwan Jo, Jonas Vinck, for Computing Machinery. doi:10.1145/3719027. Alexios Voulimeneas, Stijn Volckaert, and Dokyung 3765167. Song. Moneta: Ex-vivo gpu driver fuzzing by recalling in-vivo execution states. In NDSS, 2025. [11] Yibo He, Cunjian Huang, Xianmiao Qu, Hongdeng Chen, Wei Yang, and Tao Xie. Rvismith: Fuzzing com- [19] Nursultan Kabylkas, Tommy Thorn, Shreesha Srinath, pilers for rvv intrinsics. In Proceedings of the 2025 Polychronis Xekalakis, and Jose Renau. Effective ACM SIGSAC Conference on Computer and Communi- processor verification with logic fuzzer enhanced co- cations Security, CCS ’25, page 768–782, New York, simulation. In MICRO-54: 54th Annual IEEE/ACM NY, USA, 2025. Association for Computing Machinery. International Symposium on Microarchitecture, MICRO doi:10.1145/3719027.3744790. ’21, page 667–678, New York, NY, USA, 2021. As- [12] Muhammad Monir Hossain, Arash Vafaei, Kimia Zamiri sociation for Computing Machinery. doi:10.1145/ Azar, Fahim Rahman, Farimah Farahmandi, and Mark 3466752.3480092. Tehranipoor. Socfuzzer: Soc vulnerability detection [20] Rahul Kande, Addison Crump, Garrett Persyn, Patrick using cost function enabled fuzz testing. In 2023 Design, Jauernig, Ahmad-Reza Sadeghi, Aakash Tyagi, and Automation & Test in Europe Conference & Exhibition Jeyavijayan Rajendran. TheHuzz: Instruction fuzzing of (DATE), pages 1–6, 2023. doi:10.23919/DATE56975. processors using Golden-Reference models for finding 2023.10137024. Software-Exploitable vulnerabilities. In 31st USENIX
2480 35th USENIX Security Symposium USENIX Association
Security Symposium (USENIX Security 22), pages 3219– risc-v and its vector extension. In 2024 IEEE 42nd In- 3236, Boston, MA, August 2022. USENIX Associa- ternational Conference on Computer Design (ICCD), tion. URL: https://www.usenix.org/conference/ pages 276–279, 2024. doi:10.1109/ICCD63220. usenixsecurity22/presentation/kande. 2024.00049.
[21] Gerwin Klein, Kevin Elphinstone, Gernot Heiser, June [28] Jingzhou Li, Fangfei Yu, Mingyuan Ma, Wei Liu, Yuhan Andronick, David Cock, Philip Derrin, Dhammika Elka- Wang, Hualin Wu, and Hu He. Risc-v-based gpgpu duwe, Kai Engelhardt, Rafal Kolanski, Michael Norrish, with vector capabilities for high-performance comput- Thomas Sewell, Harvey Tuch, and Simon Winwood. ing. IEEE Transactions on Very Large Scale Inte- sel4: formal verification of an os kernel. In Proceedings gration (VLSI) Systems, 33(8):2239–2251, 2025. doi: of the ACM SIGOPS 22nd Symposium on Operating Sys- 10.1109/TVLSI.2025.3574427. tems Principles, SOSP ’09, page 207–220, New York, NY, USA, 2009. Association for Computing Machinery. [29] Chris S Lin, Joyce Qu, and Gururaj Saileshwar. Gpuham- doi:10.1145/1629575.1629596. mer: Rowhammer attacks on gpu memories are practical. [22] Lucas Klemmer and Daniel Große. Epex: Proces- In USENIX Security, volume 25, 2025. sor verification by equivalent program execution. In [30] Hangtian Liu, Shuitao Gan, Chao Zhang, Zicong Gao, Proceedings of the 2021 Great Lakes Symposium on Hongqi Zhang, Xiangzhi Wang, and Guangming Gao. VLSI, GLSVLSI ’21, page 33–38, New York, NY, USA, Labrador: Response guided directed fuzzing for black- 2021. Association for Computing Machinery. doi: box iot devices. In 2024 IEEE Symposium on Security 10.1145/3453688.3461497. and Privacy (SP), pages 1920–1938, 2024. doi:10. [23] Kevin Laeufer, Vighnesh Iyer, David Biancolin, 1109/SP54263.2024.00127. Jonathan Bachrach, Borivoje Nikoli´c, and Koushik [31] Yuwei Liu, Junquan Deng, Xiangkun Jia, Yanhao Wang, Sen. Simulator independent coverage for rtl hard- Minghua Wang, Lin Huang, Tao Wei, and Purui Su. ware languages. In Proceedings of the 28th ACM Promefuzz: A knowledge-driven approach to fuzzing International Conference on Architectural Support harness generation with large language models. In Pro- for Programming Languages and Operating Systems, ceedings of the 2025 ACM SIGSAC Conference on Com- Volume 3, ASPLOS 2023, page 606–615, New York, puter and Communications Security, CCS ’25, page NY, USA, 2023. Association for Computing Machinery. 1559–1573, New York, NY, USA, 2025. Association doi:10.1145/3582016.3582019. for Computing Machinery. doi:10.1145/3719027. [24] Kevin Laeufer, Jack Koenig, Donggyu Kim, Jonathan 3765222. Bachrach, and Koushik Sen. Rfuzz: Coverage-directed [32] Daniel Lustig, Sameer Sahasrabuddhe, and Olivier fuzz testing of rtl on fpgas. In 2018 IEEE/ACM Interna- Giroux. A formal analysis of the NVIDIA PTX mem- tional Conference on Computer-Aided Design (ICCAD), ory consistency model. In Proceedings of the Twenty- pages 1–8, 2018. doi:10.1145/3240765.3240842. Fourth International Conference on Architectural Sup- [25] Johannes Lenzen, Mohamadreza Rostami, Lichao Wu, port for Programming Languages and Operating Sys- and Ahmad-Reza Sadeghi. Fuzzilicon: A post-silicon tems, ASPLOS ’19, page 257–270, New York, NY, USA, microcode-guided x86 cpu fuzzer. arXiv preprint 2019. Association for Computing Machinery. doi: arXiv:2512.23438, 2025. 10.1145/3297858.3304043. [26] Reese Levine, Mingun Cho, Devon McKee, Andrew [33] Yatin A. Manerkar, Daniel Lustig, Margaret Martonosi, Quinn, and Tyler Sorensen. Gpuharbor: Testing gpu and Michael Pellauer. Rtlcheck: verifying the mem- memory consistency at large (experience paper). In ory consistency of rtl designs. In Proceedings of the Proceedings of the 32nd ACM SIGSOFT International 50th Annual IEEE/ACM International Symposium on Symposium on Software Testing and Analysis, ISSTA Microarchitecture, MICRO-50 ’17, page 463–476, New 2023, page 779–791, New York, NY, USA, 2023. As- York, NY, USA, 2017. Association for Computing Ma- sociation for Computing Machinery. doi:10.1145/ chinery. doi:10.1145/3123939.3124536. 3597926.3598095. [34] Mahta Mayahinia and Mehdi Tahoori. Electromigration [27] Jingzhou Li, Kexiang Yang, Chufeng Jin, Xudong Liu, reliability analysis of sram-based register files in gpus Zexia Yang, Fangfei Yu, Yujie Shi, Mingyuan Ma, and ai accelerators. In 2025 IEEE 43rd VLSI Test Sympo- Li Kong, Jing Zhou, Hualin Wu, and Hu He. Ven- sium (VTS), pages 1–4, 2025. doi:10.1109/VTS65138. tus: A high-performance open-source gpgpu based on 2025.11022883.
USENIX Association 35th USENIX Security Symposium 2481
[35] Josie E. Rodriguez Condia, Paolo Rech, Fernando Fer- [43] Oreste Villa, Daniel Lustig, Zi Yan, Evgeny Bolotin, nandes dos Santos, Luigi Carro, and Matteo Sonza Re- Yaosheng Fu, Niladrish Chatterjee, Nan Jiang, and David orda. An effective method to identify microarchitec- Nellans. Need for speed: Experiences building a trust- tural vulnerabilities in gpus. IEEE Transactions on worthy system-level gpu simulator. In 2021 IEEE Device and Materials Reliability, 22(2):129–141, 2022. International Symposium on High-Performance Com- doi:10.1109/TDMR.2022.3166260. puter Architecture (HPCA), pages 868–880, 2021. doi: [36] Fernando F. dos Santos, Josie E. Rodriguez Condia, 10.1109/HPCA51647.2021.00077. Luigi Carro, Matteo Sonza Reorda, and Paolo Rech. [44] Yingchen Wang, Riccardo Paccagnella, Zhao Gang, Revealing gpus vulnerabilities by combining register- Willy R. Vasquez, David Kohlbrenner, Hovav Shacham, transfer and software-level fault injection. In 2021 51st and Christopher W. Fletcher. Gpu.zip: On the side- Annual IEEE/IFIP International Conference on Depend- channel implications of hardware-based graphical data able Systems and Networks (DSN), pages 292–304, 2021. compression. In 2024 IEEE Symposium on Secu- doi:10.1109/DSN48987.2021.00042. rity and Privacy (SP), pages 3716–3734, 2024. doi: [37] Shidong Shen, Jinyu Liu, Weizhi Feng, Fu Song, and 10.1109/SP54263.2024.00084. Zhilin Wu. Bmcfuzz: Hybrid verification of processors [45] Lichao Wu, Mohamadreza Rostami, Huimin Li, Jeyavi- by synergistic integration of bound model checking and jayan Rajendran, and Ahmad-Reza Sadeghi. GenHuzz: fuzzing. In 2025 IEEE/ACM International Conference an efficient generative hardware fuzzer. USENIX Asso- On Computer Aided Design (ICCAD), pages 1–9, 2025. ciation, USA, 2025. doi:10.1109/ICCAD66269.2025.11240887. [38] Adit Singh, Sreejit Chakravarty, George Papadimitriou, [46] Lichao Wu, Mohamadreza Rostami, Huimin Li, and and Dimitris Gizopoulos. Silent data errors: Sources, Ahmad-Reza Sadeghi. Hfl: Hardware fuzzing loop with detection, and modeling. In 2023 IEEE 41st VLSI Test reinforcement learning. In 2025 Design, Automation Symposium (VTS), pages 1–12, 2023. doi:10.1109/ & Test in Europe Conference (DATE), pages 1–7, 2025. VTS56346.2023.10139970. doi:10.23919/DATE64628.2025.10993080. [39] Flavien Solt, Katharina Ceesay-Seitz, and Kaveh [47] Jinyan Xu, Yiyuan Liu, Sirui He, Haoran Lin, Yajin Razavi. Cascade: CPU fuzzing via intricate pro- Zhou, and Cong Wang. MorFuzz: Fuzzing proces- gram generation. In 33rd USENIX Security Sym- sor via runtime instruction morphing enhanced syn- posium (USENIX Security 24), pages 5341–5358, chronizable co-simulation. In 32nd USENIX Secu- Philadelphia, PA, August 2024. USENIX Associa- rity Symposium (USENIX Security 23), pages 1307– tion. URL: https://www.usenix.org/conference/ 1324, Anaheim, CA, August 2023. USENIX Associa- usenixsecurity24/presentation/solt. tion. URL: https://www.usenix.org/conference/ [40] Fabian Thomas, Eric García Arribas, Lorenz Het- usenixsecurity23/presentation/xu-jinyan. terich, Daniel Weber, Lukas Gerlach, Ruiyi Zhang, and [48] Yinan Xu, Zihao Yu, Dan Tang, Guokai Chen, Lu Chen, Michael Schwarz. Riscover: Automatic discovery of Lingrui Gou, Yue Jin, Qianruo Li, Xin Li, Zuojun Li, Ji- user-exploitable architectural security vulnerabilities in awei Lin, Tong Liu, Zhigang Liu, Jiazhan Tan, Huaqiang closed-source risc-v cpus. In Proceedings of the 2025 Wang, Huizhe Wang, Kaifan Wang, Chuanqi Zhang, ACM SIGSAC Conference on Computer and Communi- Fawang Zhang, Linjuan Zhang, Zifei Zhang, Yangyang cations Security, CCS ’25, page 3326–3340, New York, Zhao, Yaoyang Zhou, Yike Zhou, Jiangrui Zou, Ye Cai, NY, USA, 2025. Association for Computing Machinery. Dandan Huan, Zusong Li, Jiye Zhao, Zihao Chen, Wei doi:10.1145/3719027.3765141. He, Qiyuan Quan, Xingwu Liu, Sa Wang, Kan Shi, [41] Blaise Tine, Krishna Praveen Yalamarthy, Fares Elsab- Ninghui Sun, and Yungang Bao. Towards developing bagh, and Kim Hyesoon. Vortex: Extending the risc-v high performance risc-v processors using agile method- isa for gpgpu and 3d-graphics. In MICRO-54: 54th An- ology. In 2022 55th IEEE/ACM International Sympo- nual IEEE/ACM International Symposium on Microar- sium on Microarchitecture (MICRO), pages 1178–1199, chitecture, pages 754–766, 2021. 2022. doi:10.1109/MICRO56248.2022.00080. [42] Timothy Trippel, Kang G Shin, Alex Chernyakhovsky, [49] Yiru Xu, Hao Sun, Jianzhong Liu, Yuheng Shen, and Garret Kelly, Dominic Rizzo, and Matthew Hicks. Yu Jiang. Saturn: Host-gadget synergistic usb driver Fuzzing hardware like software. In 31st USENIX Se- fuzzing. In 2024 IEEE Symposium on Security and curity Symposium (USENIX Security 22), pages 3237– Privacy (SP), pages 4646–4660, 2024. doi:10.1109/ 3254, 2022. SP54263.2024.00051.
2482 35th USENIX Security Symposium USENIX Association
[50] Kaiyuan Yang, Matthew Hicks, Qing Dong, Todd Austin, A deadlock implies a cycle in g-HB ∪ po. However, travers-
and Dennis Sylvester. A2: Analog malicious hardware. ing any edge in the cycle strictly increases the rank (for g-HB)
In 2016 IEEE Symposium on Security and Privacy (SP), or preserves it (for po). Since a cycle must contain at least
pages 18–37, 2016. doi:10.1109/SP.2016.10. one g-HB edge (intra-warp dependencies alone cannot form
[51] Lishan Yang, George Papadimitriou, Dimitrios Sartze- a cycle), traversing the cycle would imply ∃ix : R(ix) < R(ix),
takis, Adwait Jog, Evgenia Smirni, and Dimitris Gi- a contradiction. Therefore, the planner is deadlock-free.
zopoulos. Probing weaknesses in gpu reliability as-
sessment: A cross-layer approach. In 2024 IEEE In- B Target Hardware Details
ternational Symposium on Performance Analysis of
Systems and Software (ISPASS), pages 331–333, 2024. We evaluated the latest revisions of two open-source GPG-
doi:10.1109/ISPASS61541.2024.00048. PUs. Vortex was configured for 64-bit execution (commit
[52] Kunlin You, Yinan Xu, Kehan Feng, Luoshan Cai, 2189194), while OpenGPGPU is 32-bit only. We used the
Yaoyang Zhou, and Yungang Bao. DiffTest-H: To- official ventus-env repository at commit 7e97907.
ward Semantic-Aware Communication in Hardware- At submission, we used OpenGPGPU’s latest release (also
Accelerated Processor Verification, page 1462–1476. mainline) and Vortex’s latest mainline. Additional backtesting
Association for Computing Machinery, New York, on the Vortex v2.2 release reproduced 4 out of 5 bugs and
NY, USA, 2025. URL: https://doi.org/10.1145/ revealed a new CSR bug, confirming bug persistence.
3725843.3756108. Each test uses the maximum number of warps supported by
the DUT, except for OpenGPGPU, where we generate tests
[53] Zhenkai Zhang, Kunbei Cai, Yanan Guo, Fan Yao, using half of the warp slots per SM to avoid an unconfirmed
and Xing Gao. Invalidate+Compare: A Timer-Free writeback error observed under full load. However, the RTL
GPU cache attack primitive. In 33rd USENIX design remains in the default full-SM configuration.
Security Symposium (USENIX Security 24), pages Before the final fuzzing campaign, we patched V4, V7, P1,
2101–2118, Philadelphia, PA, August 2024. USENIX and S2 identified in a 30-second pilot run, to prevent generator
Association. URL: https://www.usenix.org/ crashes and recurring blockers from limiting exploration.
conference/usenixsecurity24/presentation/ Aligned with processor differential testing practices, we
zhang-zhenkai. considered two floating-point values comparable if their dif-
[54] Yuhao Zhou, Peng Jia, Jiayong Liu, and Ximing Fan. ference was within 2 Unit in the Last Place (ULP). Fuzz4cuda: Fuzzing your nvidia gpu libraries through debug interface. Computers & Security, 161:104754, C Fully Random Baseline 2026. URL: https://www.sciencedirect.com/ science/article/pii/S0167404825004432, The fully random OpenGPGPU baseline generates test cases doi:10.1016/j.cose.2025.104754. without any feedback or enforced structure. Its low comple- tion and coverage rates, measured under the 100-seed cover- A Deadlock-Free Proof of Barrier Planner age ablation setting (§5.4), provide additional context. We prove that the proposed barrier planner guarantees dead- Table 4: Fully random baseline for OpenGPGPU. lock freedom. Let B = (b1, b2, . . . , bN) be the global barrier sequence. For any instruction i executed by thread t, we de- Method Instruction Completion Rate Expr. Coverage fine its rank R(i) as the index of the next barrier t will en- FuzzGPU 97.3% 59% counter in B, i.e., R(i) = min{k | bk ∈ B ∧ t ∈ scope(bk) ∧ Cascade 19.8% 20% bk succeeds i in pot }. If no such barrier exists, R(i) = N + 1. DiveFuzz 44.7% 21% Since each thread generates barriers following the sorted or- Fully Random 0.1% 14% der of B, for any two instructions i po 1, i2 in thread t, i1 →t i2 =⇒ R(i1) ≤ R(i2). g-HB Consider the g-HB relation. If i − then by definition i1 precedes bk 1 −−→ i2 via barrier bk, D Porting Cascade to GPUs in some thread t1, and i2 succeeds bk in some thread t2. Because i1 precedes bk, its next This appendix presents the technical details of porting the barrier must bei2 bk or an earlier one, so R(i1) ≤ k. Because Cascade fuzzing framework to two open-source GPU archi- succeeds bk, its next barrier must be strictly later than bk tectures: Vortex and Ventus. Cascade was originally designed g in the sequence filtered for t2, so R(i -HB 2) > k. Thus, i −− 1 −→ for scalar RISC-V processors and generates semantically valid i2 =⇒ R(i1) < R(i2). random programs to detect implementation bugs.
USENIX Association 35th USENIX Security Symposium 2483
We ported Cascade to two GPU designs: (i) Ventus, a Rather than changing DiveFuzz’s core generation work- vector-oriented GPU based on RV32GCV with GPU-specific flow, we extended a small set of well-isolated subsystems extensions, and (ii) Vortex, a SIMT GPU that supports both to reflect GPU execution models and toolchain constraints: RV32 and RV64 with custom thread-management extensions. (1) architecture templates, (2) instruction specifications, (3) SIMT Execution Model Adaptation. The fundamental chal- register/operand handling, (4) compilation and linking, (5) lenge lies in initializing multiple warps and threads correctly. differential-testing backends, and (6) mutation rules. For Vortex, we inject a startup sequence at the beginning of Instruction Template Modifications. We extended Dive- each generated program: Fuzz’s template system with GPU-specific templates that Listing 1: Vortex SIMT Initialization Sequence encapsulate architecture-dependent prologue/epilogue se- quences and execution assumptions. Concretely, Ventus tem- addi x1 , x0 , NUM_WARPS plates account for vector execution and GPU-specific initial- l u i x2 , t a r g e t _ a d d r [ 3 1 : 1 2 ] ization requirements, whereas Vortex templates expose SIMT- addi x2 , x2 , t a r g e t _ a d d r [ 1 1 : 0 ] oriented control primitives (e.g., thread masking, divergence wspawn x1 , x2 management, and barriers). This preserves DiveFuzz’s orig- addi x1 , x0 , THREAD_MASK inal template and mutation design. We use a 1:3 ratio of tmc x1 template generation to mutation for each baseline testcase. For Ventus, we utilize the standard RVV vector length con- We expanded DiveFuzz’s instruction template. For Ventus, figuration: this primarily consists of RVV-based vector arithmetic and memory instructions, complemented by a small set of Ventus- Listing 2: Ventus RVV Initialization Sequence specific extensions. For Vortex, we added SIMT control in- addi x1 , x0 , NUM_THREADS structions (encoded using assembler directives) to model v s e t v l i x0 , x1 , e32 , m1 thread management, synchronization, and divergence. Over- all, the extension is declarative: new formats and operand Instruction Class Extensions. We extended Cascade’s in- constraints are registered without altering the higher-level struction classification system to accommodate GPU-specific generation pipeline. operations. Specifically, we introduce a Thread Control class Register File Names. GPU backends introduce operand con- that captures explicit SIMT thread management in Vortex ventions that differ from scalar cores. Ventus uses a unified (e.g., wspawn, tmc, split, join) and vector-length setup in register file with architectural mechanisms to select integer vs. Ventus (vsetvli, vsetvl). We also add a Predication class, floating-point banks. Consequently, our generator normalizes realized via dedicated predicate instructions in Vortex (pred, register naming and inserts the required prefixing where nec- pred.n) and via mask registers in Ventus. For inter-thread essary. We also added support for Ventus vector registers and coordination, a Synchronization class accounts for Vortex bar- their extended encodings. In contrast, Vortex largely follows riers (local vx.bar and a global barrier) as well as Ventus’s conventional integer/floating-point register naming, with the barrier instruction. Finally, we separate compute-focused main additions being SIMT-specific control operands. classes that are architecture-dependent: Ventus exposes a Vec- Compiler Configuration. Ventus relies on an tor ALU class (e.g., vadd, vmul, vfma), whereas Vortex in- LLVM/Clang-based toolchain with an explicit tar- cludes a Tensor Core class via wmma. WMMA and barrier get configuration (e.g., -target=riscv32 and instructions are suppressed in single-thread configurations to -mcpu=ventus-gpgpu for assembling, and -nostdlib prevent deadlock. Barrier number is limited by hardware. for linking). In contrast, Vortex uses a conventional Simulator Interface Abstraction. We implemented a Spike GNU toolchain (riscv32/64-unknown-elf-as/ld) command-compatible GPU simulator interface to seamlessly with architecture-specific -march settings integrate with Cascade’s program construction pipeline. To (-march=rv32ifmd_zicsr_zifencei or remain compatible with Cascade’s original CPU-centric algo- -march=rv64ifmd_zicsr_zifencei). This abstrac- rithmic design, we use the context of the first active thread as tion enables the same generator to emit assembly and produce a representative program state. executables across both GPU backends. GPU-Aware Spike Engine. To maintain compatibility with E Porting DiveFuzz to GPUs DiveFuzz’s original algorithmic framework, we integrated a GPU-capable ISS backend via a lightweight pybinding DiveFuzz is a differential fuzzing framework for hardware ver- adapter that provides a Spike-compatible control interface ification that combines template-driven program construction while exposing GPU-specific bootstrap parameters. We use with semantics-preserving mutations to generate valid test the context of the first active thread as a proxy for the program cases. While the original implementation targets scalar RISC- state. The session manager selects the appropriate ISS build V CPU cores (e.g., XiangShan, NutShell, Rocket Chip), our based on the target architecture and bit width, allowing a goal is to extend differential fuzzing to GPU-class execution. uniform harness to drive both Ventus and Vortex executions.
2484 35th USENIX Security Symposium USENIX Association