Abstract

Integrated Circuit (IC) verification consumes nearly 70% of the IC development cycle, and recent research has begun leveraging Large Language Models (LLMs) to automatically generate testbenches and reduce verification overhead. However, we found that LLMs have difficulty generating testbenches correctly. Unlike high-level programming languages, Hardware Description Languages (HDLs) are extremely rare in LLMs training data, leading LLMs to produce incorrect code.

To overcome challenges when using LLMs to generate Universal Verification Methodology (UVM) testbenches and sequences, we propose HAVEN (Hybrid Automated Verification ENgine) to prevent LLMs from writing HDL directly. For UVM testbench generation, HAVEN utilizes LLM agents to analyze design specifications to produce a structured architectural plan. The HAVEN Template Engine then combines with predefined and protocol-specific templates to generate all UVM components with correct bus-handshake timing. For UVM sequence generation, HAVEN introduces a Protocol-Aware Sequence Domain-Specific Language (DSL) that decomposes sequences into fine-grained step types. A set of predefined DSL patterns first establishes sequences that achieve a high coverage rate without LLM involvement. HAVEN continues to improve the coverage rate by iteratively leveraging LLM agents to analyze coverage gap reports and compose additional targeted DSL sequences.

Unlike previous works, HAVEN is the first system that utilizes pre-defined, protocol-specific Jinja2 templates to generate all UVM components and UVM sequences using our proposed Protocol-Aware DSL and rule-based code generator. Our experimental results on 19 open-source IP designs (180–11 k LOC) spanning three interface protocols (Direct, Wishbone, AXI4-Lite) show that HAVEN achieves 100% compilation success, 90.6% code coverage, and 87.9% functional coverage on average, and is SOTA among LLM-assisted testbench generation systems.

I Introduction

Integrated Circuit (IC) Verification, a key process in ensuring that the functional units in IC chips work correctly, consumes nearly 70% of the IC development cycle [6, 5]. Conventionally, verification engineers manually develop testbenches and sequences tailored to each IC design’s protocol and control logic, thereby incurring substantial verification overhead [2]. To reduce this overhead, recent research has begun leveraging Large Language Model (LLM) agents to automatically generate testbenches [38, 18, 29, 9, 36, 23, 13, 3, 31].

However, LLMs have difficulty generating correct testbenches. Unlike high-level programming languages such as Python, JavaScript, and C++, HDLs are extremely rare in LLM training data. For instance, LLMs find it hard to distinguish between blocking (=) and non-blocking (<=) assignments and to employ clock-edge synchronization correctly. These constructs have no analog in high-level languages, causing LLMs to frequently produce incorrect code [30, 35, 37, 24]. Recent approaches such as AutoBench, CorrectBench, and ConfiBench [20, 21, 19, 25, 32] leverage LLMs to directly write entire testbenches and rely on iterative self-correction to repair errors. However, this iterative debugging process is expensive in both time and token usage and may still not yield correct results.

As IC designs grow in complexity, hand-writing monolithic testbenches becomes increasingly difficult to maintain and reuse. The Universal Verification Methodology (UVM) [1], a widely adopted SystemVerilog-based industry framework, addresses this by decomposing a testbench into standardized, reusable components, such as drivers, monitors, scoreboards, and subscribers. Each UVM component has a well-defined syntax convention. When applying LLMs to automate UVM testbench generation, two specific challenges arise.

The first challenge is generating the structural UVM components. We observed that these components adhere to well-defined formats. When the configurations of the Device Under Test (DUT) are altered, we can only update the port list and protocol type in the testbench template. This template-based method is well-suited for generating UVM testbenches. The prior work, UVM2 [33], employed hand-crafted templates to assist an LLM in generating testbenches. However, the UVM2 template covers only five simple UVM components, without protocol information. Consequently, when DUTs contain bus protocols with clocking-sequence events, misusing a blocking assignment in a bus-handshake protocol often causes the entire verification process to fail.

Second, it is also challenging to generate UVM sequences that achieve a high coverage rate. UVM sequences define the test stimuli and their execution ordering to examine the DUT’s functional units. Unlike the structural UVM components, sequences are derived from design specifications and vary across DUTs, as each design has its own protocol and control flow. Previous work has applied reinforcement learning and Bayesian optimization to search for appropriate test stimuli [7, 4, 12, 28, 10]. However, a DUT can contain complex control flows or multi-step bus protocols. For instance, a Wishbone transaction requires a coordinated sequence of address setup, strobe assertion, and acknowledgment waiting. Although the exploration space grows significantly, the set of stimuli that actually reach uncovered states remains small. These methods, therefore, struggle to converge efficiently, resulting in a reduced IC verification coverage rate.

To address these challenges in automatic UVM testbench generation, we propose HAVEN (Hybrid Automated Verification ENgine), which prevents LLM agents from writing SystemVerilog code directly. Instead, HAVEN leverages LLMs for what they do best—extracting and organizing information from design specifications—and delegates all code generation to the rule-based code generator (CodeGen) that guarantees SystemVerilog syntactic correctness.

To generate correct UVM components, HAVEN employs LLM agents to analyze design specifications and produce a UVM architectural plan: a structured JSON document that captures the testbench structure (agent topology, component connectivity, interface protocol type) and signal-level data contracts. A Template Engine then combines this architectural plan with pre-defined, protocol-specific Jinja2 templates to generate all UVM components, including drivers with correct bus-handshake timing (Section III). Since the templates encode protocol-correct coding conventions, this approach eliminates SystemVerilog syntax errors when using LLMs to generate UVM components for the testbench.

To accurately generate UVM sequences and optimize the coverage rate, HAVEN introduces a Protocol-Aware Sequence Domain-Specific Language (DSL) that decomposes sequences into fine-grained step types (e.g., register write, poll, value sweep). A rule-based CodeGen translates each DSL step into protocol-correct UVM sequence code, ensuring SystemVerilog syntactic correctness (Section II). HAVEN uses the DSL in two stages. First, a set of predefined DSL sequence patterns, such as constrained random, field value sweeps, toggle patterns, and FIFO stress tests, is triggered by signal characteristics and protocol structure, without any LLM involvement. These predefined sequences alone have achieved a sufficient coverage rate. Second, to minimize remaining coverage gaps, HAVEN employs LLM agents to analyze coverage gap reports and compose additional, targeted DSL sequences (Section IV). This process can iterate: after each simulation, HAVEN feeds the updated coverage gaps back to the LLM, which generates new DSL sequences targeting uncovered behaviors, until coverage converges or a maximum iteration count is reached.

We evaluate HAVEN on 19 open-source IP designs spanning three interface protocols (Section V-A). Our experiments show the superiority of HAVEN in LLM-assisted testbench generation, specifically achieving the following:

1. High compile reliability: Our template-generated UVM components achieve 100% compilation success across all 19 IP designs, confirming that rule-based code generation eliminates the syntax errors observed in LLM-generated components.

2. High coverage: The DSL sequences achieve 90.6% average code coverage and 87.9% average functional coverage through iterative refinement, establishing state-of-the-art results among LLM-assisted UVM verification approaches.

3. Fast and low-cost: The full pipeline completes in a single run without per-design tuning, averaging 6 LLM calls, 68k tokens, and $0.38 per design.

The key insight behind these state-of-the-art results is that LLMs should not write HDL testbenches directly. LLMs excel at understanding design specifications, extracting structured information, and interpreting coverage gaps, but they struggle with the syntactic and timing conventions of HDL code. By restricting LLMs to structured information extraction and delegating all code generation to rule-based systems, HAVEN achieves both HDL syntax correctness and high coverage with low costs.

II HAVEN: An Overview

There are two stages in the HAVEN pipeline: Stage 1 generates all UVM components, including predefined sequences; Stage 2 iteratively improves the coverage rate through LLM-driven DSL sequence generation.

In Stage 1 (Section III), the goal is to generate a complete, compile-passing UVM testbench with predefined sequences. To achieve this, LLM agents extract structural and protocol information from the design specification into two structured JSON artifacts: the UVM Blueprint and Protocol Flows. A Template Engine then combines these artifacts with pre-defined, protocol-specific templates to generate all UVM components, including drivers with correct bus-handshake timing, peripheral Bus Functional Models (BFMs), and functional coverage subscribers; Fig. 1 illustrates this with the driver path as a worked example. Stage 1 also produces predefined sequences through rule-based strategy inference, without any LLM involvement. A compile-fix loop ensures the testbench compiles correctly before proceeding to Stage 2.

In Stage 2 (Section IV), the goal is to close the remaining coverage gaps that predefined sequences cannot reach. To do this, the LLM analyzes the coverage gap report from Stage 1’s simulation and expresses targeted sequences in the structured DSL JSON format (Fig. 2). A rule-based DSL CodeGen translates the JSON into protocol-correct UVM sequences, which accumulate with Stage 1’s predefined sequences. HAVEN can iterate Stage 2 after each simulation: the updated coverage gap report is fed back to the LLM, which generates additional DSL sequences targeting uncovered behaviors, until coverage converges or a maximum iteration count is reached.

Protocol-Aware Sequence DSL Both stages generate sequences through the same Protocol-Aware Sequence DSL. The DSL defines ten step types covering the primary space of verification operations (Table I), such as register writes, polling loops, value sweeps, and BFM actions. To produce a UVM sequence, the DSL source (whether generated by rule-based inference in Stage 1 or by the LLM in Stage 2) is expressed as a JSON document composed of these step types. A shared rule-based CodeGen then translates the JSON into protocol-correct SystemVerilog [27, 8]. This design ensures that the LLM never writes SystemVerilog directly; it only selects step types and fills in parameters (addresses, values, conditions), while the CodeGen guarantees syntactic and protocol correctness. Fig. 2 illustrates this translation with a concrete example: an ETHMAC MDIO register read expressed as two DSL steps (register_write and poll) and the protocol-correct UVM sequence code produced by the CodeGen. The translation mechanics are detailed in Section IV.

TABLE I: Protocol-Aware Sequence DSL: step types.

VII Discussion

Remaining coverage gaps. Six designs remain below 90% code coverage. DSL expressiveness limits affect SDRAM (84.0%), ETHMAC (81.8%), and HUF (76.5%), which require multi-phase initialization, DMA descriptor chains, or conditional branching beyond the current linear DSL. Multi-agent coordination limits affect I2C (75.5%), whose multi-master arbitration requires coordinated two-agent sequences. Large state space limits CAN (83.4%), where cross-domain synchronizer toggles remain undersampled.

LLM non-determinism. Compile success is deterministic (100%), but coverage can vary across runs because the sequence item and subscriber are LLM-generated [37]. Future work should characterize run-to-run variance.

Template engineering effort. Each protocol-specific template is a one-time development effort. In practice, we developed all templates in about one hour using LLM-assisted coding tools with manual verification of protocol-timing correctness. Once validated, the same template serves all designs under that protocol without per-design modification — e.g., the single Wishbone driver template covers all eight Wishbone designs from 352 to 11 k LOC. All other UVM components (monitor, scoreboard, subscriber) are protocol-agnostic and fully reused across all three protocols. Adding support for a new bus protocol requires only authoring and validating one new driver template and one new monitor template.

Threats to validity. Our benchmarks are open-source designs that may not reflect proprietary IP complexity [11, 26, 14, 15]. Section VI-E shows that HAVEN generalizes across five LLMs with only a 1–8 pp coverage gap. The DSL’s contribution is quantified by the Stage 1 vs. + Stage 2 columns in Table III (84.6% →\to 90.6% code, 79.8% →\to 87.9% functional), providing a built-in ablation.