SOURCE ARCHIVE
SHA256: f16659faf129855eba1b3f5903469b37771893fd05565a2bddfe5037de54fd97
TYPE: application/pdf
SIZE: 765.3 KB
FETCHED: 6/18/2026, 10:06:17 PM
EXTRACTOR: liteparse
CHARS: 66,260
EXTRACTED CONTENT
66,260 chars2 sacs / Oy Journal of Advanced Computing Systems (JACS) 57 ISSN: 3066-3962 = Content Available at SciPublication SCIPUBLICATION
StimulusRL: A Universal Deep Reinforcement Learning Stimulus Agent for
Coverage-Driven Chip Design Verification
Jingyi Chen¹, Chenyao Zhu²
1Electrical and Computer Engineering, Carnegie Mellon University, PA, USA
2Industrial Engineering & Operations Research, UC Berkeley, CA, USA
jingyi.chen221@gmail.com
DOI: 10.69987/JACS.2026.60104
K e y w o r d s A b s t r a c t
design verification; Modern chip design verification (DV) relies heavily on constrained-random
stimulus generation; simulation and manual testcase engineering to close functional coverage. This
functional coverage; workflow is effective but increasingly expensive as designs scale and corner
deep reinforcement cases require long, protocol-valid stimulus sequences. We present StimulusRL,
learning; DQN; a universal deep reinforcement learning (RL) stimulus agent that learns to
coverage-guided generate cycle-accurate stimuli from coverage feedback and differential bug
fuzzing; differential oracles. StimulusRL formalizes stimulus generation as a Markov decision
testing; cocotb; process (MDP) and trains a Deep Q-Network (DQN) policy that maps partial
Verilator; UVM signal observations to legal stimulus actions. To support reproducible
evaluation, we introduce DVSBench, a compact benchmark suite of five
representative DUT families (FIFO, ALU, cache, arbiter, and SPI controller)
with explicit functional coverage models and three injected bug variants per
DUT. We conduct full experimental evaluations across 3 independent seeds
with a fixed 2000-cycle budget and compare StimulusRL against three
baselines: uniform random, constrained-random verification (CRV), and
coverage-guided mutation fuzzing (CGM-Fuzz). Across DVSBench,
StimulusRL matches baseline final coverage on four DUTs and achieves
comparable coverage AUC on three DUTs while providing a learnable
interface that can be integrated into cocotb/Verilator/UVM flows. In
differential bug-finding, StimulusRL reliably detects cache and arbiter defects
and discovers SPI waveform mismatches faster when successful, but exhibits
lower success rate on the SPI controller, motivating improved reward shaping
and hierarchical action modeling. All numbers, tables, and figures in this paper
are generated from deterministic scripts with released seeds.
Introduction Coverage as an optimization target. SystemVerilog
provides language-level constructs for functional
Simulation-based functional verification remains the coverage (covergroups, coverpoints, and cross
dominant pre-silicon validation approach for complex coverage) and for code-oriented metrics such as
digital systems, spanning IP blocks, SoCs, and statement and branch coverage [1]. Most industrial
accelerators. Industrial verification flows are built flows treat coverage as a measurable proxy for
around SystemVerilog and UVM testbenches [1], [2] verification completeness rather than a proof of
and quantify progress using functional and code correctness: coverage bins encode the verification
coverage. Despite decades of methodological advances, plan’s intent, while assertions and scoreboards encode
a recurring cost driver is the final portion of correctness properties. To support multi-tool
verification—functional coverage closure—where regressions, coverage is often exported into a unified
remaining unhit bins correspond to rare scenarios, deep database format such as UCIS, which standardizes
protocol sequences, or subtle interactions between coverage data exchange and aggregation across
internal states and input constraints. simulators and analysis tools [4]. These conventions
make coverage a natural learning signal for agents: it is
observable online, decomposes verification intent into
Vol. 6(1), pp. 45-60, January 2026 [45]
Journal of Advanced Computing Systems (JACS) ISSN: 3066-3962
discrete targets, and supports incremental closure over random baselines (legality-preserving) and a legality-
many short tests. aware coverage-guided fuzzer (CGM-Fuzz).
In the constrained-random paradigm, DV engineers Deep reinforcement learning [26-39] offers a
express stimulus spaces via random variables and complementary route: rather than hand-engineering bias
constraints, run massive regressions, then iteratively add heuristics, an agent can learn a stimulus policy that
directed tests or retune constraints to target uncovered maximizes a coverage-derived reward through
scenarios. This loop is effective but manual: it depends interaction with a simulator. DVCon work has shown
on engineering intuition, a detailed mental model of the that DQN-style agents can be integrated into verification
DUT and protocol, and repeated cycles of analysis and environments to accelerate coverage closure on specific
test authoring. As designs become more configurable designs such as compression encoders [11]. However,
and software-programmable, the action space of practical adoption requires generality (across DUT
possible tests grows faster than available human effort, types), compatibility with legality constraints, and
making testcase engineering a bottleneck. rigorous experimental evidence on multiple designs
Agentizing testcase engineering. A UVM verification with reproducible protocols.
environment typically decomposes stimulus into Challenges for deep RL in DV. Verification
sequences, drivers, monitors, and scoreboards that environments are partially observable and highly non-
operate on transaction-level abstractions [2], [3]. When stationary from the agent’s perspective: once a coverage
coverage gaps remain, engineers respond by (i) writing bin is hit, it no longer provides reward, and the agent
new sequences that realize specific protocol scenarios, must continually adapt its exploration strategy. Rare
(ii) adjusting constraint distributions, and (iii) adding bins often depend on multi-cycle temporal sequences
targeted error-injection or stress tests. Viewed through (e.g., cache conflict patterns or controller timing), which
an automation lens, this process is a sequential decision create long-horizon credit assignment problems.
problem: each new test provides feedback (coverage and Finally, simulation is expensive, so learning must be
failures), and the next test should be chosen to maximize sample-efficient and stable. These constraints favor off-
expected progress under constraints such as legality, policy methods with replay buffers and target networks,
runtime budget, and debug cost. StimulusRL targets such as the Deep Q-Network family [16], combined
precisely this loop by learning a stimulus policy that with legality masks to guarantee protocol-valid actions.
maps observations and coverage progress to legal
stimulus actions. This paper addresses these gaps by proposing
StimulusRL, a universal RL stimulus agent designed for
Coverage-driven generation (CDG) has long been the EDA tool layer of a DV platform. StimulusRL views
studied as a way to bias stimulus toward hard-to-reach the DUT as an environment that exposes signal-level
coverage bins. Early work used probabilistic models observations and accepts protocol-valid stimulus
such as Bayesian networks to relate random variables to actions. The agent learns from two feedback channels:
coverage outcomes and steer test generation [8]. Recent (i) functional coverage events and (ii) an oracle
DVCon contributions show renewed interest in indicating mismatches between a golden model and a
automating coverage closure with structural analyses buggy variant, enabling both coverage closure and
(e.g., coverage dependency graphs [9]) and RTL graph- corner-case bug discovery. To enable full, repeatable
based biasing (GraphCov [10]). In parallel, the security evaluation, we build DVSBench, a benchmark suite that
community has demonstrated that feedback-guided provides five representative DUTs, explicit coverage
fuzzing and differential testing can uncover RTL defects bins, and bug variants with deterministic seeds.
efficiently, exemplified by DifuzzRTL [12],
BugsBunny [14], and large-scale open-source hardware Our contributions are threefold. First, we define an end-
fuzzing pipelines [13]. to-end closed-loop architecture for RL-driven DV that
aligns with cocotb [5], Verilator coverage collection [6],
From fuzzing to DV-grade automation. Coverage- [7], and UCIS-style coverage databases [4]. Second, we
guided mutation has become a practical default for release DVSBench, including DUT models, bin
software fuzzing, where test inputs are unconstrained definitions, and bug variants, enabling controlled
byte streams and legality is implicit in the input format empirical comparisons. Third, we conduct full
[15]. In hardware DV, the stimulus space is often experimental evaluations and report detailed per-DUT
structured (handshakes, timing protocols, configuration coverage trajectories, AUC, runtime, and bug-finding
fields), and violating legality constraints can either be metrics for StimulusRL and three baseline generators.
useless noise or an explicit robustness objective.
Therefore, a stimulus agent must be able to incorporate
protocol legality, preserve transaction structure, and still
explore aggressively enough to reach deep states. This
motivates our comparison against both constrained-
Vol. 6(1), pp. 45-60, January 2026 [46]
Journal of Advanced Computing Systems (JACS) ISSN: 3066-3962
Materials and Methods
Fig. 1. Closed-loop DV automation with StimulusRL integrated at the EDA tool layer.
stimulus our
Agent (Deep RL) (RTLmodule) trales —>
Closed-loop DV automation: action simulation — coverage/oracle learning
Table 1. DVSBench benchmark suite used in this study.
DUT Description #Actions Obs. Dim. #Coverage #Bug
Bins Variants
FIFO8 8-depth FIFO 32 7
(sequential) 38 3
ALU32 32-bit ALU 56 6 23 3
(combination
al)
DMCache 4-line direct- | 64 17 24 3
mapped
cache
(sequential)
RRArb4 4-master 16 12 27 3
round-robin
arbiter
(sequential)
SPIM8 SPI master | 12 10 21 3
controller
FSM
(sequential)
Table 2. Stimulus generation methods compared in experiments.
Method Core idea Uses legality Uses coverage | Learns a policy
constraints feedback
Vol. 6(1), pp. 45-60, January 2026
[47]
Journal of Advanced Computing Systems (JACS) ISSN: 3066-3962
Random Uniform No No No
sampling over
action space
CRV Constrained- Yes No No
random
sampling over
legal actions
CGM-Fuzz Coverage- Optional Yes No
guided mutation (enabled here)
of action
sequences (AFL-
style)
StimulusRL Deep Q-learning Yes Yes Yes
(DQN) from
coverage/oracle
rewards
Table 3. Core hyperparameters (shared across DUTs unless stated otherwise).
Component Hyperparameter Value DQN Network 2-layer MLP, 64 hidden units per layer, ReLU DQN Discount factor (γ) 0.99 DQN Learning rate 1e-3 DQN Optimizer Adam DQN Replay buffer size 50,000 transitions DQN Batch size 64 DQN Target network update every 200 steps DQN Exploration ε linear 0.30 → 0.05 over 1500 steps CGM-Fuzz Initial corpus size 8 seeds CGM-Fuzz Mutations per test 1–3 random edits All Episode length 50 cycles All Budget per run 2000 cycles All Random seeds {0,1,2} (3 independent trials)
Table 4. Bug variants used for differential bug-finding evaluation (3 per DUT).
DUT Bug ID Injected defect Primary observable mismatch
Vol. 6(1), pp. 45-60, January 2026 [48]
Journal of Advanced Computing Systems (JACS) ISSN: 3066-3962
FIFO8 1 Off-by-one full flag flag divergence (full)
(full asserted early)
FIFO8 2 Write pointer wraps data ordering
too early (mod depth- divergence on pop
1)
FIFO8 3 Simultaneous flag/data divergence
push/pop updates after dual ops
count incorrectly
ALU32 1 SUB implemented as result mismatch
ADD
ALU32 2 Overflow flag flag mismatch
computed incorrectly
ALU32 3 Shift-right result mismatch
implemented as shift-
left
DMCache 1 Tag compare uses hit/miss + data
wrong bit mismatch
DMCache 2 Dirty bit not set on read-after-evict data
write (missing mismatch
writeback)
DMCache 3 Read miss does not hit/miss behavior
allocate/fill line mismatch
RRArb4 1 Pointer not updated grant sequence
after grant mismatch
RRArb4 2 Arbiter scans grant mismatch
priorities in reverse
order
RRArb4 3 Can grant two invalid multi-grant
requesters
simultaneously
SPIM8 1 Bit-order config mosi waveform
ignored (always mismatch
MSB-first)
SPIM8 2 CS deasserted one cs/mosi mismatch
cycle too early near end
SPIM8 3 CPHA handling mosi/sck mismatch
wrong in shift timing
Table 10. Metrics reported in this paper.
Metric Definition Higher is better
Final Coverage #unique bins hit / #bins after Yes
budget
Coverage AUC (1/T) Σ_t coverage(t) Yes
Vol. 6(1), pp. 45-60, January 2026 [49]
Journal of Advanced Computing Systems (JACS) ISSN: 3066-3962
Runtime Wall-clock seconds for 2000 No
steps
Bug Success Rate fraction of trials finding Yes
mismatch within budget
Steps-to-Bug first mismatch step index No
(successful trials)
This section defines the verification setting, the Benchmark dataset (DVSBench). We created
benchmark dataset, and the stimulus-generation DVSBench as a compact but representative suite of
algorithms evaluated. All experiments were executed DUT families frequently encountered in IP-level DV:
using deterministic scripts that drive each DUT for a FIFO, ALU, cache, arbiter, and controller FSM. Each
fixed number of simulated cycles and record functional DUT defines (i) a cycle-accurate golden behavioral
coverage and oracle mismatches. model, (ii) an observation function producing a numeric
Closed-loop DV architecture. Figure 1 shows the feature vector, (iii) a finite action space encoding legal
conceptual integration of StimulusRL into a standard and semi-legal stimulus combinations, (iv) a functional
simulation DV flow. In a production setting, the DUT is coverage model represented as a set of named bins, and
implemented as RTL and simulated using an event- (v) three injected bug variants. Table 1 summarizes the
driven simulator or a compiled simulator such as suite.
Verilator; cocotb embeds a Python interpreter into the Episode structure and reset semantics. Each evaluation
simulator process to enable Python-based testbenches run is organized as 40 episodes of 50 cycles (total 2000
and coroutine scheduling [5]. Coverage points are steps). At the start of every episode, the DUT is reset
collected from the simulator (e.g., via SystemVerilog into a deterministic initial state, mirroring common IP-
coverage or Verilator's coverage instrumentation [6]) level DV practice where tests begin from reset and then
and can be stored in a tool-interoperable format (e.g., apply a bounded transaction sequence. Coverage is
UCIS [4]). StimulusRL sits at the EDA tool layer: it accumulated across episodes, matching a regression
chooses stimulus actions, receives signal observations setting in which many short tests contribute to a shared
and coverage events, and learns a policy that maximizes coverage database. This structure makes the learning
long-horizon coverage gain. problem non-trivial: some bins require multi-cycle
Problem formulation. For each DUT, we define a temporal patterns within an episode, while other bins
Markov decision process (MDP) with: state s_t correspond to configuration choices that must be set
corresponding to the DUT's internal state at cycle t, early (e.g., SPI mode) to influence later behavior.
observation o_t corresponding to the externally visible Stimulus action encoding. To make the problem
signal vector (plus a scalar coverage progress feature), tractable for deep RL while preserving verification-
action a_t corresponding to a discretized stimulus relevant diversity, each DUT’s raw input space is
applied at that cycle, and reward r_t defined as the discretized into a finite action set (Table 1). FIFO8
number of newly covered functional bins at that step actions encode push/pop handshakes and one of eight
minus a small penalty for illegal FIFO operations. A data classes designed to exercise boundary values and
coverage database C is maintained across episodes distribution shifts. ALU32 actions encode one of seven
(tests), reflecting typical DV regressions where multiple operations combined with eight operand-pattern classes
tests contribute to a shared coverage closure goal. (zero/one, sign extremes, alternating bits, and
Reward definition. Let B denote the set of functional randomized subranges). DMCache actions encode
coverage bins and let H_t ⊆ B be the set hit up to cycle read/write, one of eight addresses (four indices × two
t. Each cycle produces a set of triggered bins T_t. The tags), and one of four write-data patterns, enabling
incremental coverage gain is Δ_t = |T_t \ H_t|, and the conflict misses and dirty evictions. RRArb4 actions
base reward is r_t = Δ_t. To discourage wasted illegal encode a 4-bit request vector. SPIM8 actions encode
operations in FIFO-style interfaces, we subtract a small configuration toggles (CPOL/CPHA and bit order),
penalty λ·I_illegal (λ=0.1 in our harness), yielding r_t = start/abort commands, and four data patterns.
Δ_t − λ·I_illegal. Because Δ_t becomes sparse as Observations and monitors. Observation vectors are
coverage saturates, we augment the observation with the built from externally visible signals that a UVM monitor
/ or cocotb coroutine can sample without accessing
policy replay to preserve informative transitions for internal RTL state [2], [5]. We represent boolean signals
learning. as {0,1} scalars and represent categorical signals using
one-hot encodings (e.g., arbiter pointer, SPI controller
Vol. 6(1), pp. 45-60, January 2026 [50]
Journal of Advanced Computing Systems (JACS) ISSN: 3066-3962
state). For sequential DUTs, the observation includes StimulusRL deployment, each simulation run can emit both current status (flags/state) and recent outcomes a UCIS database; the agent merges these databases to (e.g., cache hit/miss), which is typical of monitor-driven determine the global bin-hit set H_t and uses this scoreboarding. We also append the scalar coverage- information for reward computation and for selecting / follow-up tests. UCIS also preserves covergroup minimal regression context. hierarchy and cross coverage, enabling reward shaping Legality constraints and protocol masks. CRV and that prioritizes bins according to the verification plan StimulusRL apply a per-cycle legality mask that (e.g., safety-critical scenarios) rather than treating all removes actions violating protocol preconditions. bins equally. DVSBench uses an in-memory bitmap for FIFO8 forbids pop when empty and push when full; efficiency, but its API mirrors the UCIS/covergroup SPIM8 permits start only in IDLE and treats abort as workflow to ensure that the experimental results remain meaningful only during TRANSFER. Other DUTs representative of production DV environments. accept all enumerated actions. The legality mask reflects Bug oracle and corner-case detection. To evaluate bug realistic DV constraints and improves sample efficiency discovery, we follow the differential testing principle by reducing wasted simulator cycles. For fairness, used in RTL fuzzers such as DifuzzRTL [12]: the same CGM-Fuzz enforces legality at execution time by stimulus sequence is applied to a golden model and to a replacing any corpus action that becomes illegal in the buggy variant, and the oracle declares a failure when current state with a uniformly sampled legal action. their observable outputs diverge. For sequential DUTs, Transaction-level actions and UVM sequences. In full this setup captures bugs that only manifest after multi- UVM environments, stimulus is often expressed as step interactions (e.g., cache evictions or SPI waveform transactions generated by sequences and translated by timing). Table 4 lists the injected bug variants and their drivers into pin-level waveforms [2]. StimulusRL can primary mismatch signatures. operate at either level. At the pin level (as in Baselines. We compare four stimulus generators (Table DVSBench), actions are per-cycle drive values, and the 2). Random samples actions uniformly without policy must learn timing. At the transaction level, the respecting protocol legality. CRV samples uniformly action space is structured (bus reads/writes, bursts, over the subset of legal actions at each cycle, modeling configuration writes), legality is encoded by sequence typical constrained-random stimulus. CGM-Fuzz item constraints, and the driver enforces the protocol. A maintains a corpus of action sequences and mutates practical deployment therefore lets StimulusRL choose them using coverage feedback, similar in spirit to parameters for existing sequences—addresses, burst coverage-guided fuzzing systems such as AFL++ [15], lengths, ordering, and inter-transaction delays—while while enforcing legality during execution. StimulusRL the UVM infrastructure handles low-level signaling. (DQN) learns an action-value function and selects Our discrete action design can be viewed as a minimal actions using ε-greedy exploration. instantiation of this idea: each action corresponds to a coarse transaction or micro-operation that is valid under Coverage-guided mutation fuzzer (CGM-Fuzz). CGM- a cycle-level protocol mask. Fuzz maintains a corpus of candidate tests, each represented as a fixed-length action sequence. A test is Functional coverage models. For each DUT, we define executed from reset for 50 cycles, and any newly hit bins functional bins that reflect typical DV intent: state are recorded. If the test increases global coverage, it is reachability (e.g., FIFO occupancy levels), transition added to the corpus; otherwise, it is discarded. New tests events (e.g., arbiter grant rotations), data-pattern are generated by selecting a parent sequence uniformly sensitivity (e.g., ALU operands), and protocol from the corpus and applying 1–3 mutations that replace configuration combinations (e.g., SPI mode and bit individual actions. During execution, each action is order). Bins are triggered on each cycle based on the validated against the legality mask; if an action is illegal applied stimulus and observed outputs, and the global in the current state, CGM-Fuzz substitutes a uniformly coverage fraction is computed as the number of hit bins sampled legal action. This retains the core feedback loop divided by the total number of bins. For FIFO8, two bins of AFL-style fuzzing—corpus retention, mutation, and correspond to explicit illegal-operation attempts (push coverage feedback—while operating over structured when full, pop when empty), allowing evaluation of protocol actions rather than arbitrary byte arrays. robustness tests; constrained generators intentionally do not target these bins, which is reflected in their StimulusRL algorithm. StimulusRL implements a Deep achievable maximum coverage. Q-Network (DQN) agent [16] with experience replay Coverage database interoperability. Industrial coverage and a periodically updated target network. We use the Double DQN target to reduce overestimation bias [17]. closure often merges coverage over thousands of tests The Q-network is a two-layer multilayer perceptron and across tools. The Unified Coverage Interoperability with ReLU activations and outputs one Q-value per Standard (UCIS) defines a common data model and discrete action. At each step, the agent stores (o_t, a_t, exchange format for coverage databases [4], [25]. In a r_t, o_{t+1}, done) into a replay buffer and performs
Vol. 6(1), pp. 45-60, January 2026
[51]
Journal of Advanced Computing Systems (JACS) ISSN: 3066-3962
gradient updates using mini-batches sampled uniformly. making it easy to integrate into typical CI and regression
Optimization uses Adam [19]. We update the target infrastructure. The DQN forward and backward passes
network periodically and linearly anneal ε from 0.30 to are implemented in NumPy (rather than a GPU
0.05 over the first 1500 steps. Table 3 reports the full framework) to keep the artifact self-contained and
hyperparameter configuration used in all experiments. deterministic. Raw logs include per-step coverage, per-
Experimental protocol and metrics. For each DUT and run runtime, and DQN TD losses logged at fixed
method, we run a fixed budget of 2000 simulation steps intervals; all figures in this paper are regenerated
organized into episodes of 50 cycles with reset at directly from these logs.
episode boundaries. We repeat each run for three Scaling considerations. RL sample efficiency must be
independent random seeds {0,1,2}. We report (i) final evaluated relative to simulator throughput. In our
functional coverage, (ii) coverage area-under-curve Python harness, model stepping is fast and DQN
(AUC), (iii) wall-clock runtime in the Python evaluation learning overhead dominates runtime (Table 7). In RTL
harness, (iv) bug-finding success rate over 3 bug signoff flows, the balance can shift: large SoCs
variants, and (v) mean steps-to-bug for successful trials simulated with SystemVerilog DPI may execute orders
(Table 10). of magnitude slower than a small neural-network
Implementation and reproducibility. The entire evaluation. For compiled simulation, Verilator
benchmark and evaluation harness are implemented in translates RTL into C++ and can achieve high cycle
Python with deterministic pseudo-random number rates while still producing coverage information [6], [7].
generators seeded per run. All reported tables and plots This suggests a hybrid workflow: train StimulusRL on
are generated by scripts that read raw CSV logs fast compiled simulation or emulation targets, then
produced by the simulator loop. The evaluation harness apply the learned policy to slower signoff simulation to
is designed to mirror a cocotb-driven simulation loop [5] accelerate closure of specific bins. Furthermore, RL
while remaining lightweight enough to support full training can be distributed across parallel simulations by
repeated evaluations within this paper. collecting transitions from multiple workers and
updating a shared network, reducing wall-clock time
Software stack. DVSBench is implemented as a pure- without changing the total simulator cycle budget.
Python simulator harness with explicit cycle stepping, Results
Fig. 2. Coverage closure on FIFO8 (mean ± std, n=3).
Coverage closure on FIFO8 n=3)
1.0
=
73 0.6 IJ
8
®
304
2
=
02 =
dan
0.0
0 250 500 750 1000 1250 1500 1750 2000
Simulation steps
Vol. 6(1), pp. 45-60, January 2026 [52]
Journal of Advanced Computing Systems (JACS) ISSN: 3066-3962
Fig. 3. Coverage closure on ALU32 (mean ± std, n=3).
Coverage closure on ALU32 (meanzstd, n=3)
1.0 r=
08
g
306
g
2
Toa
==
dan
0.0
o 250 500 750 1000 1250 1500 1750 2000
Simulation steps
Fig. 4. Coverage closure on DMCache (mean ± std, n=3).
Coverage closure on DMCACHE n=3)
1.0 7
08
g
306
4
2
s
504
s
02 — cm
0.0
0 25 500 750 1000 1250 1500 1750 2000
Simulation steps
Fig. 5. Coverage closure on RRArb4 (mean ± std, n=3).
Coverage closure on RRARB4 (meanzstd, n=3)
10
08
9
3
g
306
2
gos
2
02 — an
— dn
0.0
0 250 500 750 1000 1250 1500 1750 2000
Simulation steps
Vol. 6(1), pp. 45-60, January 2026 [53]
Journal of Advanced Computing Systems (JACS) ISSN: 3066-3962
Fig. 6. Coverage closure on SPIM8 (mean ± std, n=3).
Coverage closure on SPIM8 (meanzstd, n=3)
10
Los [
02
0 250 500 750 1000 1250 1500 1750 2000
Simulation steps
This section reports full empirical results on sequential control problems relative to pure constrained
DVSBench. All metrics are averaged over three random sampling. For RRArb4, StimulusRL reaches
independent seeds unless stated otherwise. AUC 0.926 ± 0.004 versus 0.892 ± 0.008 for CRV,
Coverage closure trajectories. Figures 2–6 show per- indicating faster exploration of distinct grant-rotation
DUT coverage as a function of simulation steps (mean and request-pattern bins. For DMCache, all methods
± std). Across the suite, ALU32 and DMCache reach eventually hit all bins, but AUC separates the trajectory:
full coverage within the 2000-step budget for all StimulusRL achieves 0.971 ± 0.000 compared to 0.978
methods, indicating that the defined bins are relatively ± 0.000 for CGM-Fuzz and 0.975 ± 0.000 for CRV. For
easy to satisfy with broad stimulus diversity. RRArb4 SPIM8, StimulusRL underperforms with AUC 0.844 ±
also reaches full final coverage, but the AUC values 0.024 versus 0.900 ± 0.000 for CGM-Fuzz, consistent
differentiate how quickly each method reaches closure. with its reduced coverage of configuration-combination
bins within the fixed episode length.
On FIFO8, Random achieves a higher maximum Runtime. Table 7 reports wall-clock runtimes in the
coverage than CRV and StimulusRL because the evaluation harness. Random and CRV are fastest
coverage model includes two explicit illegal-operation because they only sample actions and step the model.
attempt bins. CRV and StimulusRL enforce legality and CGM-Fuzz adds sequence management but remains
therefore converge to 36/38 bins (0.947). This illustrates close to CRV. StimulusRL is slowest due to neural-
a practical design choice: whether the DV plan treats network forward passes and gradient updates. In a real
illegal stimulus as part of coverage closure or as a Verilator-based flow, this overhead may be amortized
separate robustness objective. by simulator cost and by reusing a trained policy across
On SPIM8, all baselines reach 0.952 ± 0.000 final regressions.
coverage within budget, while StimulusRL achieves Runtime magnitudes. On the heaviest model
0.921 ± 0.071. Inspection of the bin definitions shows (DMCache), Random completes a 2000-step run in
that missed bins correspond to configuration 0.378 ± 0.004 s, CRV in 0.537 ± 0.004 s, and
combinations (CPOL/CPHA and bit order) that require StimulusRL in 2.111 ± 0.005 s (Table 7). Across all
deliberate exploratory toggling early in an episode DUTs, StimulusRL requires 1.16–2.11 s per run,
before starting transfers. This result suggests that sparse whereas Random and CRV complete in 0.15–0.54 s.
coverage rewards can be insufficient for long-horizon These numbers quantify the learning overhead in a
configuration exploration without additional shaping or lightweight simulator and motivate two deployment
hierarchical actions. patterns: (i) use StimulusRL on long-running RTL
Tables 5 and 6 quantify final coverage and AUC. CGM- simulations where action-selection cost is negligible
Fuzz provides strong AUC on sequential DUTs, compared with simulator time, and (ii) pre-train policies
consistent with the effectiveness of coverage-guided on fast models and reuse them across nightly
mutation in software fuzzing [15]. StimulusRL provides regressions.
competitive AUC on DMCache and RRArb4 and Differential bug-finding. We evaluate corner-case
slightly lower AUC on SPIM8, reflecting the learning discovery by running each stimulus generator against 3
overhead and exploration schedule. injected bug variants per DUT with a differential oracle.
Quantitative coverage comparison. Table 6 highlights Tables 8 and 9 summarize success rates and steps-to-
that StimulusRL improves early coverage on some bug (successful trials only). Across FIFO8, DMCache,
Vol. 6(1), pp. 45-60, January 2026 [54]
Journal of Advanced Computing Systems (JACS) ISSN: 3066-3962
and RRArb4, all methods achieve near-perfect success, Training dynamics. Figure 8 reports the temporal-
but CGM-Fuzz and Random often find bugs in fewer difference (TD) loss trace for the DQN agent on
than 50 steps. StimulusRL detects these bugs DMCache (loss values logged every 50 training steps).
consistently but typically later, reflecting that its policy The loss decreases and stabilizes, indicating that the
is optimized for coverage gain rather than immediate agent learns consistent value estimates under the chosen
mismatch discovery. reward and exploration schedule.
The SPI controller presents a different pattern: Random, Quantifying value-learning convergence. Using the
CRV, and CGM-Fuzz find all three SPI bugs within the logged TD losses, the mean DMCache loss averaged
2000-step budget, with increasing average steps-to-bug across seeds drops from 0.386 over the first 500 steps to
(Table 9). StimulusRL finds SPI bugs very quickly 0.058 over the final 500 steps. SPIM8 exhibits a similar
when it succeeds (16.2 steps on average), but its success decay (0.164 to 0.037), which indicates that the
rate is lower (0.444), indicating that the learned policy comparatively lower SPIM8 coverage is not caused by
collapses to a subset of transaction patterns that do not divergence or optimizer instability, but by the policy's
exercise all waveform-sensitive corner cases. Figure 7 exploration and representation limits under the current
aggregates bug discovery as an empirical CDF across all action abstraction. Across DUTs, small loss spikes
DUTs and bugs. occur when a previously unseen coverage bin is first
reached, reflecting the non-stationary nature of
coverage-driven rewards.
Table 5. Final functional coverage after 2000 simulation steps (mean ± std, n=3).
DUT Random CRV CGM-Fuzz StimulusRL
(DQN)
FIFO8 1.000 ± 0.000 0.947 ± 0.000 0.947 ± 0.000 0.947 ± 0.000
ALU32 1.000 ± 0.000 1.000 ± 0.000 1.000 ± 0.000 1.000 ± 0.000
DMCACHE 1.000 ± 0.000 1.000 ± 0.000 1.000 ± 0.000 1.000 ± 0.000
RRARB4 0.963 ± 0.000 0.963 ± 0.000 0.963 ± 0.000 0.963 ± 0.000
SPIM8 0.952 ± 0.000 0.952 ± 0.000 0.952 ± 0.000 0.921 ± 0.045
Table 6. Normalized area-under-curve (AUC) of coverage trajectories (mean ± std, n=3).
DUT Random CRV CGM-Fuzz StimulusRL
(DQN)
FIFO8 0.984 ± 0.006 0.934 ± 0.005 0.919 ± 0.017 0.918 ± 0.007
ALU32 0.995 ± 0.001 0.995 ± 0.001 0.995 ± 0.002 0.990 ± 0.000
DMCACHE 0.997 ± 0.000 0.997 ± 0.000 0.997 ± 0.000 0.995 ± 0.002
RRARB4 0.958 ± 0.001 0.958 ± 0.001 0.958 ± 0.000 0.950 ± 0.003
SPIM8 0.941 ± 0.000 0.933 ± 0.006 0.913 ± 0.015 0.878 ± 0.030
Table 7. Wall-clock runtime per 2000-step run in the evaluation harness (seconds, mean ± std, n=3).
DUT Random CRV CGM-Fuzz StimulusRL
(DQN)
FIFO8 0.095 ± 0.008 0.133 ± 0.003 0.087 ± 0.005 1.217 ± 0.011
ALU32 0.052 ± 0.004 0.097 ± 0.005 0.047 ± 0.004 1.264 ± 0.092
Vol. 6(1), pp. 45-60, January 2026 [55]
Journal of Advanced Computing Systems (JACS) ISSN: 3066-3962
DMCACHE 0.079 ± 0.003 0.119 ± 0.005 0.071 ± 0.001 1.248 ± 0.011 RRARB4 0.064 ± 0.003 0.104 ± 0.003 0.046 ± 0.005 1.134 ± 0.019 SPIM8 0.065 ± 0.002 0.106 ± 0.001 0.087 ± 0.002 1.150 ± 0.018
Table 8. Bug-finding success rate within 2000 steps (averaged over 3 bug variants; n=9 trials per DUT×method). DUT Random CRV CGM-Fuzz StimulusRL (DQN) FIFO8 1.000 1.000 1.000 0.889 ALU32 1.000 1.000 1.000 1.000 DMCACHE 1.000 1.000 1.000 1.000 RRARB4 1.000 1.000 1.000 1.000 SPIM8 1.000 1.000 1.000 0.444
Table 9. Mean steps to first detected mismatch (successful trials only; averaged over 3 bug variants). DUT Random CRV CGM-Fuzz StimulusRL (DQN) FIFO8 30.4 30.4 55.7 84.8 ALU32 8.8 8.8 8.6 71.9 DMCACHE 4.3 4.3 4.1 11.2 RRARB4 2.0 2.0 1.8 3.1 SPIM8 32.8 112.6 235.7 16.2
Fig. 7. Empirical CDF of steps to first mismatch across all DUTs, bug variants, and seeds (censored trials included in
denominator).
Bug discovery CDF across all DUTs/bugs/seeds
1.0 — com (censored=0) a= z — random (censored=0) £os
06 5 Soa £ 02
0.0 0 250 500 750 1000 1250 1500 1750 2000 Steps to first detected mismatch
Vol. 6(1), pp. 45-60, January 2026
[56]
Journal of Advanced Computing Systems (JACS) ISSN: 3066-3962
Fig. 8. DQN temporal-difference loss during DMCache training (loss logged every 50 steps).
DQN TD-loss (all seeds) on DMCACHE
2.00
175
1.50
1.25
1.00
o
p
075
0.50
025
0.00
250 500 750 1000 1250 1500 1750 2000
Training step (logged every 50
Discussion discovery. For easy-to-trigger combinational bugs
(ALU32), random exploration finds mismatches
The results demonstrate that deep RL stimulus rapidly; RL may overfit to a coverage-efficient subset of
generation can be evaluated rigorously with a fixed- patterns and thus delay bug exposure. For SPI, the low
budget, multi-DUT benchmark and that StimulusRL is success rate indicates that sparse rewards can cause
competitive with classical generators on several policy collapse away from rare waveform-sensitive
representative designs. At the same time, the benchmark conditions. In real flows, this could be addressed with
reveals practical considerations that matter for reward shaping, curriculum learning, or hierarchical
deploying RL in real DV flows. options that separate configuration exploration from
transfer execution.
Coverage definition and legality. FIFO8 illustrates that Toolchain integration. While DVSBench uses cycle-
what counts as "coverage closure" depends on whether accurate Python models for repeatability, the
illegal stimulus is treated as a verification objective. In architecture maps directly onto cocotb-driven
practice, UVM environments often separate functional simulation [5]. Verilator can emit coverage files and
coverage for legal operation from error-injection and supports functional and code coverage instrumentation
robustness suites. StimulusRL supports either choice by [6], [7], and UCIS provides a standardized API for
changing the legal action mask and reward, and coverage interoperability [4]. StimulusRL can therefore
DVSBench makes this trade-off explicit by including be deployed as an external stimulus engine that
illegal-operation bins as separate targets. consumes coverage events and emits constrained
RL vs. coverage-guided fuzzing. CGM-Fuzz achieves stimulus sequences in the same way existing CDG tools
strong AUC on several sequential DUTs. This aligns bias random variables [8], [9].
with the broader literature that coverage-guided Toward a universal agent. The long-term promise of a
mutation is a highly effective exploration strategy when general-purpose DV stimulus agent is to amortize
coverage instrumentation is available [15]. RL brings learning across many DUTs and projects. Achieving this
different advantages: it can condition decisions on requires representations that capture DUT structure and
observations and learn context-dependent action semantics. GraphCov demonstrates that RTL graph
choices, enabling closed-loop control for protocols features can guide test biasing [10], and similar graph
where the next best stimulus depends on the DUT's embeddings could condition an RL policy. A universal
current state. This capability is crucial for controller-like agent could ingest a DUT embedding plus current
blocks and coherent buses, and is consistent with DQN- observations and output stimulus in a common action
based DVCon demonstrations [11]. schema (e.g., bus transactions, FIFO operations,
Bug finding as a byproduct. Because StimulusRL is configuration writes). Multi-task RL or meta-learning
trained to maximize coverage rather than to trigger could then learn transferable exploration strategies such
mismatches, it is not uniformly superior for bug as fill-then-drain for FIFOs, conflict-then-probe for
caches, and toggle-config-then-transfer for controllers.
Vol. 6(1), pp. 45-60, January 2026 [57]
Journal of Advanced Computing Systems (JACS) ISSN: 3066-3962
DVSBench is a starting point for this research because Conclusions
it includes diverse block archetypes and exposes both
coverage and bug oracles under a common runner. We introduced StimulusRL, a universal deep RL
Practical guardrails. Agent-driven stimulus must be safe stimulus agent that automates testcase engineering by
for regression use. First, constraints must prevent the learning stimulus policies from coverage and oracle
agent from producing illegal or destructive actions that feedback. We also presented DVSBench, a compact
break the testbench, particularly when the action space benchmark suite of five representative DUT families
includes protocol pins. Our legality masks are a minimal with explicit coverage models and injected bug variants.
example; in UVM, this corresponds to constraining Across full empirical evaluations with fixed budgets and
sequence items and respecting ready/valid semantics. three seeds, StimulusRL matched final coverage on four
Second, reproducibility is essential: regressions need DUTs and provided competitive coverage AUC on three
deterministic replays of failure-inducing stimuli, which DUTs, while exhibiting clear failure modes on SPI
requires logging action sequences and simulator seeds. configuration exploration. In differential bug-finding,
Third, reward design must align with verification intent; StimulusRL consistently detected cache and arbiter
naive coverage rewards can encourage superficial bin defects and found SPI bugs quickly when successful but
toggling without exercising meaningful end-to-end with reduced success rate, highlighting the need for
scenarios. Verification plans often specify weighted or improved reward shaping and hierarchical stimulus
staged goals, and integrating those priorities into reward abstractions. Overall, the results confirm that RL-based
shaping or curricula is an important engineering step for stimulus generation is a viable path toward agentized
industrial adoption. DV workflows, and DVSBench provides a reproducible
foundation for comparing future EDA+AI techniques.
Multi-objective verification goals and curricula.
Coverage is only one of several objectives in industrial References
verification; teams also prioritize specific bins,
assertion coverage, bug risk, and simulator throughput. [1] IEEE Standard for SystemVerilog—Unified
StimulusRL naturally extends to multi-objective Hardware Design, Specification, and Verification
rewards, for example r_t = w_cov·Δcov + w_bug·I_fail − Language, IEEE Std 1800-2017, Dec. 2017.
w_time·Δt, where I_fail indicates an oracle failure such
as an assertion trigger or differential mismatch, and Δt [2] Accellera Systems Initiative, "Available IEC/IEEE
captures simulation cost. UCIS coverage hierarchies can Standards (IEEE Get)". [Online]. Available:
provide bin-level weights that encode a verification https://www.accellera.org/downloads/ieee. Accessed:
plan [4], enabling the agent to focus on high-value Jan. 19, 2026.
holes rather than maximizing unweighted coverage.
Curriculum scheduling is another practical lever: start [3] Accellera Systems Initiative, "Download UVM
with short episodes and high exploration to cover (Universal Verification Methodology)". [Online].
shallow bins, then progressively increase episode Available:
length and shift reward toward the remaining hard-to- https://www.accellera.org/downloads/standards/uvm.
hit bins. In DVSBench, SPIM8 includes configuration Accessed: Jan. 19, 2026.
combinations and sequencing bins (e.g., abort and [4] Accellera Systems Initiative, "Download UCIS
back-to-back transfers); a curriculum that first closes (Unified Coverage Interoperability Standard)".
configuration bins and then optimizes transaction [Online]. Available:
sequencing would likely reduce variance and improve https://www.accellera.org/downloads/standards/ucis.
success rates. Accessed: Jan. 19, 2026.
Limitations and future work. This study evaluates per- [5] cocotb, "Welcome to cocotb’s Documentation".
DUT policies and does not attempt cross-DUT transfer. [Online]. Available: https://docs.cocotb.org. Accessed:
A next step is to learn a single universal policy Jan. 19, 2026.
conditioned on a DUT embedding (e.g., RTL graphs
[10]) or to use graph neural networks as in other EDA [6] Verilator, "Coverage Analysis" in "Simulating
learning tasks. Second, our action spaces are discrete; (Verilated-Model Runtime)". [Online]. Available:
scaling to realistic buses may require structured actions https://verilator.org/guide/latest/simulating.html.
and constraint solvers. Finally, integrating with full Accessed: Jan. 19, 2026.
UVM scoreboards and temporal assertions would allow
the oracle to express richer correctness properties [7] Verilator, "verilator_coverage" (coverage report
beyond differential mismatches. processing tool). [Online]. Available:
https://verilator.org/guide/latest/exe_verilator_coverag
e.html. Accessed: Jan. 19, 2026.
Vol. 6(1), pp. 45-60, January 2026 [58]
Journal of Advanced Computing Systems (JACS) ISSN: 3066-3962
[8] S. Fine and A. Ziv, "Coverage directed test [23] Accellera Systems Initiative, "UVM Reference
generation for functional verification using Bayesian Implementation Aligned with IEEE 1800.2-2020
networks," in Proc. 40th Design Automation Conf. Standard" (press release), Dec. 16, 2020.
(DAC), 2003, pp. 286–291. [24] cocotb contributors, "cocotb: Python-based chip
[9] A. Nazi, Q. Huang, H. Shojaei, H. A. Esfeden, A. (RTL) verification" (source repository). [Online].
Mirhosseini, and R. Ho, "Adaptive Test Generation for Available: https://github.com/cocotb/cocotb. Accessed:
Fast Functional Coverage Closure," in Proc. DVCon Jan. 19, 2026.
US, 2022. [25] A. Yehia, "UCIS Applications: Improving
[10] D. Chatterjee, S. Kachhadia, C. Luo, K. Kushal, Verification Productivity, Simulation Throughput and
and S. Dhodhi, "GraphCov: RTL Graph Based Test Coverage Closure Process," DVCon, 2013.
Biasing for Exploring Uncharted Coverage Landscape,"
in Proc. DVCon US, 2022. [26] Q. Xin, “Hybrid Cloud Architecture for Efficient
and Cost-Effective Large Language Model
[11] E. Ohana, "Closing Functional Coverage With Deployment”, journalisi, vol. 7, no. 3, pp. 2182-2195,
Deep Reinforcement Learning: A Compression Encoder Sep. 2025.
Example," in Proc. DVCon US, 2023. [27] Z. Ling, Q. Xin, Y. Lin, G. Su, and Z. Shui,
[12] J. Hur, S. Song, D. Kwon, H. Zhou, M. Kim, and “Optimization of autonomous driving image detection
B. Lee, "DifuzzRTL: Differential Fuzz Testing to Find based on RFAConv and triplet attention,” Proceedings
CPU Bugs," in Proc. IEEE Symp. Security and Privacy of the 2nd International Conference on Software
(SP), 2021, doi:10.1109/SP40001.2021.00103. Engineering and Machine Learning (SEML 2024),
[13] T. Trippel, K. G. Shin, A. Chernyakhovsky, G. 2024.
Kelly, D. Rizzo, and M. Hicks, "Fuzzing Hardware Like [28] B. Wang, Y. He, Z. Shui, Q. Xin, and H. Lei,
Software," in Proc. 31st USENIX Security Symp., 2022. “Predictive optimization of DDoS attack mitigation in
[14] H. Ragab, K. Koning, H. Bos, and C. Giuffrida, distributed systems using machine learning,”
"BugsBunny: Hopping to RTL Targets with a Directed Proceedings of the 6th International Conference on
Hardware-Design Fuzzer," in SILM Workshop, 2022. Computing and Data Science (CDS 2024), 2024, pp.
89–94.
[15] A. Fioraldi, D. Maier, H. Eißfeldt, and M. Heuse, [29] Q. Xin, Z. Xu, L. Guo, F. Zhao, and B. Wu, “IoT
"AFL++: Combining Incremental Steps of Fuzzing traffic classification and anomaly detection method
Research," in Proc. WOOT @ USENIX Security, 2020. based on deep autoencoders,” Proceedings of the 6th
[16] V. Mnih et al., "Human-level control through deep International Conference on Computing and Data
reinforcement learning," Nature, vol. 518, no. 7540, pp. Science (CDS 2024), 2024.
529–533, Feb. 2015, doi:10.1038/nature14236. [30] K. Xu, H. Zhou, H. Zheng, M. Zhu, and Q. Xin,
[17] H. van Hasselt, A. Guez, and D. Silver, "Deep “Intelligent classification and personalized
Reinforcement Learning with Double Q-learning," in recommendation of e-commerce products based on
Proc. AAAI Conf. Artificial Intelligence (AAAI), 2016. machine learning,” Proceedings of the 6th International
Conference on Computing and Data Science (ICCDS),
[18] T. Schaul, J. Quan, I. Antonoglou, and D. Silver, 2024.
"Prioritized Experience Replay," in Proc. ICLR, 2016. [31] Hanqi Zhang, “Prediction Markets as Calibration
[19] D. P. Kingma and J. Ba, "Adam: A Method for Teachers for Real-Time Bidding: Market Pricing Meets
Stochastic Optimization," in Proc. ICLR, 2015. Ad Auctions”, JACS, vol. 6, no. 1, pp. 1–18, Jan. 2026,
[20] K. A. Ismail and A. A. el-Moursy, "Survey on doi: 10.69987/JACS.2026.60101.
Machine Learning Algorithms Enhancing the [32] Hanqi Zhang, “Privacy-Preserving Bid
Functional Verification Process," Electronics, vol. 10, Optimization and Incrementality Estimation under
no. 21, 2021. Privacy Sandbox Constraints: A Reproducible Study of
[21] C. Bennett et al., "Review of Machine Learning for Differential Privacy, Aggregation, and Signal Loss”,
Micro-Electronic Design Automation," Journal of Computing Innovations and Applications,
arXiv:2503.11687, 2025. vol. 3, no. 2, pp. 51–65, Jul. 2025, doi:
10.63575/CIA.2025.30204.
[22] Siemens Digital Industries Software, "What’s next [33] Hanqi Zhang, “Counterfactual Learning-to-Rank
for SystemVerilog in the upcoming IEEE 1800 for Ads: Off-Policy Evaluation on the Open Bandit
standard" (whitepaper), 2023.
Vol. 6(1), pp. 45-60, January 2026 [59]
Journal of Advanced Computing Systems (JACS) ISSN: 3066-3962
Dataset”, JACS, vol. 5, no. 12, pp. 1–11, Dec. 2025, doi:
10.69987/JACS.2025.51201.
[34] Jubin Zhang, “Graph-based Knowledge Tracing for
Personalized MOOC Path Recommendation”, JACS,
vol. 5, no. 11, pp. 1–15, Nov. 2025, doi:
10.69987/JACS.2025.51101.
[35] T. Shirakawa, Y. Li, Y. Wu, S. Qiu, Y. Li, M. Zhao,
H. Iso, and M. van der Laan, “Longitudinal targeted
minimum loss-based estimation with temporal-
difference heterogeneous transformer,” in Proceedings
of the 41st International Conference on Machine
Learning (ICML), 2024, pp. 45097–45113, Art. no.
1836.
[36] Hanqi Zhang, “Risk-Aware Budget-Constrained
Auto-Bidding under First-Price RTB: A Distributional
Constrained Deep Reinforcement Learning
Framework”, JACS, vol. 4, no. 6, pp. 30–47, Jun. 2024,
doi: 10.69987/JACS.2024.40603.
[37] Xinzhuo Sun, Jing Chen, Binghua Zhou, and
Meng-Ju Kuo, “ConRAG: Contradiction-Aware
Retrieval-Augmented Generation under Multi-Source
Conflicting Evidence”, JACS, vol. 4, no. 7, pp. 50–64,
Jul. 2024, doi: 10.69987/JACS.2024.40705.
[38] Hanqi Zhang, “DriftGuard: Multi-Signal Drift
Early Warning and Safe Re-Training/Rollback for
CTR/CVR Models”, JACS, vol. 3, no. 7, pp. 24–40, Jul.
2023, doi: 10.69987/JACS.2023.30703.
[39] Xinzhuo Sun, Yifei Lu, and Jing Chen,
“Controllable Long-Term User Memory for Multi-
Session Dialogue: Confidence-Gated Writing, Time-
Aware Retrieval-Augmented Generation, and
Update/Forgetting”, JACS, vol. 3, no. 8, pp. 9–24, Aug.
2023, doi: 10.69987/JACS.2023.30802.
Vol. 6(1), pp. 45-60, January 2026 [60]