Skip to content
STIMSMITH

Directed testing

Technique WIKI v4 · 9/30/2026

Directed testing is a hardware and digital-system verification technique in which engineers hand-craft specific test cases to exercise particular functionalities or corner cases of a design, rather than relying on randomly generated stimuli. It is one of the traditional 'workhorse' methodologies of digital design verification, valued for being precise and targeted but limited by substantial manual effort and scaling challenges as design complexity grows. In modern verification flows it is typically combined with constrained-random testing, property checking, reference-model step-and-compare flows, and benchmark-driven validation; coverage-directed selection, LLM-assisted generation, Python-UVM co-verification, and module-directed extensions build on or refine the core idea.

Overview

Directed testing is a verification methodology in which engineers deliberately construct specific test cases to validate targeted functionalities, corner cases, or microarchitectural scenarios of a design under test (DUT). It is described as one of the traditional 'workhorses' of digital-system verification, alongside constrained-random testing, and is characterized as precise and targeted but requiring significant manual effort to author comprehensive test suites [chunk:2525ccc3-a666-4b13-84b8-d765c09f7803]. In the processor verification literature, directed testing is characterized as an engineer-driven approach in which testers carefully craft specific scenarios to validate known functionalities and verify that features work as intended [chunk:3115804d-7529-488e-abc6-2a6eca425b72].

Definition and characteristics

In the context of digital hardware verification, directed testing involves:

  • Creating specific test cases to verify particular functionalities or corner cases of a design.
  • Requiring verification engineers to anticipate potential issues and create tests to expose them.
  • Being precise and targeted, but increasingly difficult to scale as system complexity grows.

The approach becomes challenging as design complexity grows, because engineers must anticipate all the corner cases and interactions worth probing [chunk:2525ccc3-a666-4b13-84b8-d765c09f7803]. For processor designs, FLAME's authors observe that as designs become more complex, directed testing alone becomes costly and insufficient to cover all scenarios, especially those at the periphery of expected behaviors [chunk:3115804d-7529-488e-abc6-2a6eca425b72].

Workflow and artifacts

A directed-testing flow typically begins with a black-box test specification written from the verification engineers' understanding of the DUT spec, followed by a test-bench description document reviewed by both design and verification teams [chunk:1ef82ebd-dbb2-486c-9ae1-cbb98633a2ef]. A directed test bench in classic industrial practice (e.g., the LFI bus-bridge case study) is built from internally supplied Bus Functional Models (BFMs) written in VHDL, driven by a simple transaction-level test language that performs read and write transactions plus idle cycles; the BFMs translate transactions into appropriate signal behavior on the bus, and a separate protocol checker (also in VHDL) monitors adherence to the protocol [chunk:1ef82ebd-dbb2-486c-9ae1-cbb98633a2ef]. The test specification process itself acts as a thorough review of the DUT specification and can uncover discrepancies or holes in it [chunk:1ef82ebd-dbb2-486c-9ae1-cbb98633a2ef].

For RISC-V designs, an Imperas-directed step-and-compare flow encapsulates an Instruction Set Simulator (OVP model) into a SystemVerilog module using DPI, exposing interfaces such as reset, step, address bus, data bus, and interrupts; the SystemVerilog testbench loads the same .elf program into both the DUT memory and the OVP reference memory, resets and steps both targets in lockstep, and compares dynamic state at each step rather than relying on a stored log [chunk:71769a57-023a-4d56-9a1f-6f8fda6567d8]. Functional coverage is collected with SystemVerilog covergroup/coverpoints and accumulated into the same results.log alongside compare results, allowing a single regression run to exercise a directed execution path against an instruction-accurate reference [chunk:71769a57-023a-4d56-9a1f-6f8fda6567d8].

Once directed tests have been written and run on the DUT, structural coverage is measured against pre-defined targets such as 100% justified statement coverage, >95% branch coverage, and 100% toggle coverage; these are treated as a necessary but not sufficient quality target — reaching them does not mean verification is complete [chunk:1ef82ebd-dbb2-486c-9ae1-cbb98633a2ef]. When structural coverage gaps appear, a "white-box" approach is used: coverage holes are examined with a designer and specific transaction sequences are specified to hit the previously uncovered holes, producing a set of very targeted tests [chunk:1ef82ebd-dbb2-486c-9ae1-cbb98633a2ef]. In addition to structural metrics, directed testing aims to demonstrate bus compliance via a compliance document with tick boxes listing the scenarios that must be performed correctly, each linked to the supplying test [chunk:1ef82ebd-dbb2-486c-9ae1-cbb98633a2ef].

Role in multi-layer verification flows

Directed testing is commonly embedded as the first phase of a layered verification strategy. In the NaxRiscv SoC verification flow, for instance, the methodology is described as a multi-layered approach that 'combines directed testing, real-world benchmarking, and constrained-random testing' to ensure comprehensive evaluation of both architectural and microarchitectural aspects of the design [chunk:e8c8a109-6fbe-4965-932d-6dd8a9020ec5]. The strategy begins with an automated Jenkins Continuous Integration (CI) pipeline that executes standard compliance test suites — specifically riscv-tests and riscv-arch-tests — as established benchmarks for verifying that the Instruction Set Architecture (ISA) is correctly implemented, including extensions (F, D, C) and proper interactions with system registers (CSRs). Following these directed tests, the methodology incorporates real-world benchmarking (CoreMark, Dhrystone, OS boot sequences) and constrained-random testing using RISCV-DV [chunk:e8c8a109-6fbe-4965-932d-6dd8a9020ec5].

The same layered structure was applied in the LFI bus-bridge case study at Infineon, where directed testing using VHDL BFMs was followed by Specman-based pseudo-random testing and GateProp formal property checking at the module level, plus in-system testing at the system level [chunk:3a3ac759-9de5-41ad-9114-484e550c9935]. Directed tests were completed about halfway through LFI-IBC32 but were continually run in regression and were enhanced to achieve the structural coverage targets; the same test bench and tests were then ported to LFI-S for regression and structural coverage there too [chunk:1ef82ebd-dbb2-486c-9ae1-cbb98633a2ef].

Comparison with constrained-random testing

Directed testing is positioned in contrast with constrained-random testing:

  • Directed testing creates specific test cases to verify particular functionalities or corner cases; it is precise and targeted but requires substantial manual effort [chunk:2525ccc3-a666-4b13-84b8-d765c09f7803].
  • Constrained-random testing (CRV) generates a large number of random test cases within specified constraints that limit the search to interesting scenarios, but may still miss rare corner cases and can be inefficient in the number of tests required to achieve adequate coverage [chunk:2525ccc3-a666-4b13-84b8-d765c09f7803]. In FLAME's terminology, CRV fills the gap left by directed testing by generating a large number of random inputs guided by specific constraints, allowing broader exploration of the design's functionality space and making it effective for complex designs [chunk:3115804d-7529-488e-abc6-2a6eca425b72].

The LFI case study reinforces that the techniques are complementary: in principle all bugs found by random testing could also be found by directed testing, but in practice there is insufficient resource to generate self-checking sequences for every large random sequence, and insufficient ingenuity to think of every corner case the random generator creates [chunk:0aa746c2-184c-448c-ab6e-be50dbda2205].

Coverage and bug-finding characteristics

Reported coverage figures from the LFI project show the practical strength of directed testing on a bus-bridge DUT: a directed test bench achieved 100% justified statement coverage, 98.4% branch coverage, 95.2% condition coverage, 84.2% trigger coverage, and 100% toggle coverage; random testing reached 100% FSM transition coverage but did not attain 100% cross-coverage of the FSMs, while property-checking coverage is hard to measure but can be assessed via team review or formal proof [chunk:3115804d-7529-488e-abc6-2a6eca425b72; chunk:1ef82ebd-dbb2-486c-9ae1-cbb98633a2ef; chunk:0d43471d-2ad7-433a-bb89-3e4037aed7c2]. Bug-type breakdowns across the LFI projects show that the three techniques stress different areas: in LFI-S, property checking found 18 bugs (notably all 11 bus-protocol violations), directed testing found 14 (notably all 5 livelock/deadlock bugs), and random testing found 22 (with the broadest spread across categories) [chunk:3115804d-7529-488e-abc6-2a6eca425b72; chunk:0d43471d-2ad7-433a-bb89-3e4037aed7c2]. Some bug classes — for example, bugs caused by filling up internal queues and then trying to add further transactions — require many simulated cycles to reach the failure point, which is inefficient in property checking but tractable in directed or random testing [chunk:0aa746c2-184c-448c-ab6e-be50dbda2205].

Sign-off criteria that include directed testing

In multi-methodology flows, sign-off typically combines criteria from all techniques. In the LFI case study, sign-off required that:

  • directed and random test bench and test specifications be reviewed and signed off by design and verification teams;
  • structural and functional coverage targets be met;
  • all directed tests pass;
  • a certain number of random transactions run successfully since the last bug found;
  • property specifications be reviewed and signed off;
  • all properties pass; and
  • the bug curve flatten off, with no bugs found for a given period of time [chunk:da3e8910-37ed-4e02-a1b1-b14133c6285d].

Practical considerations and reuse

A key practical issue for directed-testing reuse is regression repeatability: tests must be runnable deterministically across runs, which is naturally satisfied by a directed test bench and by listing Specman seeds per scenario for random runs, but requires the eVCs and the Specman version to be stable across the regression [chunk:da3e8910-37ed-4e02-a1b1-b14133c6285d]. Directed test elements built from VHDL BFMs are typically less reusable than the random test-bench elements (which were developed into reusable IP via the eVC approach) and the property-checking deliverables (where external bus constraints and transaction properties lend themselves to reuse across similar interfaces) [chunk:da3e8910-37ed-4e02-a1b1-b14133c6285d]. For RISC-V flows, the Imperas OVP step-and-compare approach is naturally deterministic: both the DUT and the encapsulated OVP reference execute the same .elf under the same step stimuli issued from SystemVerilog, so the comparison is reproducible run-to-run as long as the program and stimulus sequence are fixed [chunk:71769a57-023a-4d56-9a1f-6f8fda6567d8].

Motivations for automation

Industry surveys cited in the ConfigDrivenV paper report that verification engineers allocate more than 70% of total project time ensuring RTL designs behave correctly before fabrication, and that manual testbench development is error-prone, coverage-limited, and cannot scale to the billions of transistors present in modern SoC designs [chunk:ee7dd3e3-37ae-42a9-bc6f-567063340cb4]. The ConfigDrivenV authors explicitly characterize existing approaches as falling into two camps — SystemVerilog-internal randomization requiring commercial simulator licenses, and manually written directed tests that provide limited design space coverage — and position Python-UVM co-verification as filling the gap between them [chunk:ee7dd3e3-37ae-42a9-bc6f-567063340cb4].

Related and evolving techniques

Several adjacent or derivative approaches build on or refine directed testing:

  • Module-directed testing (e.g., MoDitector for autonomous driving systems) extends the directed idea to module-level fault attribution: it employs module-specific oracles, module-specific feedback, adaptive seed selection, and mutation to generate tests that provoke failures in specific subsystems rather than only detecting collisions in a black-box manner [source:arxiv:2502.08504v1].
  • Coverage-directed test selection uses supervised learning on coverage feedback to bias selection of tests toward those likely to increase functional coverage, reducing manual constraint writing and accelerating coverage closure in simulation-based verification [source:arxiv:2205.08524v3].
  • Python-UVM co-verification (ConfigDrivenV) addresses the manual-effort limitations of directed testing by using a Python golden reference model that generates random instruction sequences and computes mathematically correct expected register states, paired with a SystemVerilog UVM testbench that reads these test vectors and performs cycle-accurate comparison via a self-checking scoreboard; the framework is validated on a custom 16-bit RISC-style processor and a UART controller with reported 100% verification accuracy on correct RTL and 100% bug detection on deliberately corrupted RTL [chunk:ee7dd3e3-37ae-42a9-bc6f-567063340cb4].
  • Step-and-compare flows against an instruction-set reference (e.g., Imperas OVP encapsulated into SystemVerilog via DPI) provide a directed execution harness for RISC-V designs in which the SystemVerilog testbench loads the same .elf into DUT and reference memories, resets both, and steps them in lockstep, comparing dynamic state at each step [chunk:71769a57-023a-4d56-9a1f-6f8fda6567d8].
  • Reinforcement-learning-driven test generation has been explored for RISC-V functional verification, where an RL agent guides instruction generation based on coverage metrics and CPU state, with reported coverage improvements over random instruction generation [chunk:d4d5f66e-2a3c-4123-80b0-2be363bd78fe].
  • LLM-assisted test generation (FLAME) targets processor functional coverage by generating test cases whose prompts are augmented with previously failed counterexamples and refreshed uncovered functional points (FPs), iterating test generation round by round; FLAME's authors position their approach as complementary to directed and random testing, reporting 34.25%–220% more target FPs covered on average than five typical/state-of-the-art baselines across four RISC-V processor designs (IBEX, CVA6, CV32E40P), and a 28.13% improvement in test validity over the state of the art [chunk:3115804d-7529-488e-abc6-2a6eca425b72].
  • Directed Test Generation for Hardware Validation: A Survey (Jayasena and Mishra, ACM Computing Surveys, vol. 56, no. 5, pp. 136, Dec. 2023) provides a systematic survey of directed test generation techniques for hardware validation, and is cited as reference [11] in the ConfigDrivenV literature [chunk:af8b615c-d756-48ee-97a1-f5c1fe3aba2e].

RISC-V context

For RISC-V cores specifically, directed testing typically relies on established open compliance suites and may be embedded in UVM-based flows. The riscv-tests suite provides ISA-level compliance tests, while riscv-arch-tests provides architectural compliance tests; together they form the foundation of the directed-testing phase in many RISC-V verification flows [chunk:e8c8a109-6fbe-4965-932d-6dd8a9020ec5]. UVM-based RISC-V verification flows combine directed and constrained-random test sequences targeting functional scenarios such as ALU operations (ADD, SUB, AND, OR) and register read/write operations, with cover groups and SystemVerilog assertions in the monitor to track exercised scenarios and check protocol or control logic correctness [chunk:3e73fa49-536f-41e1-9527-de845a6726e5]. Academic and industry RISC-V UVM environments — including Spike-based reference-model flows at Politecnico di Milano and the open-source CORE-V-VERIF environment used to verify cores like CV32E40P — integrate Spike as a reference ISA model, random instruction generators (e.g., RISCV-DV), directed tests, and coverage collection [chunk:82d3e866-4604-4bf1-b3d3-f5418ebe39c6]. The Imperas OVP step-and-compare approach is offered as an alternative instruction-accurate reference model that can be encapsulated into SystemVerilog via DPI and stepped alongside the DUT RTL, with functional coverage accumulated into the same results.log as the compare results [chunk:71769a57-023a-4d56-9a1f-6f8fda6567d8].

Limitations

  • Requires significant manual effort to author comprehensive test suites [chunk:2525ccc3-a666-4b13-84b8-d765c09f7803].
  • Engineers must anticipate potential issues ahead of time, which becomes increasingly difficult as system complexity grows [chunk:2525ccc3-a666-4b13-84b8-d765c09f7803]; FLAME's authors explicitly note that directed testing alone becomes costly and insufficient to cover scenarios at the periphery of expected behaviors [chunk:3115804d-7529-488e-abc6-2a6eca425b72].
  • In RISC-V, the modular nature of the architecture — with many optional extensions and custom instructions — multiplies the number of configurations that need to be verified, making purely manual directed-testing approaches harder to scale [chunk:2525ccc3-a666-4b13-84b8-d765c09f7803].
  • Directed-testing elements are typically less reusable across projects than the eVC-based random test-bench elements or the external-constraint portions of property-checking environments [chunk:da3e8910-37ed-4e02-a1b1-b14133c6285d].
  • ConfigDrivenV's authors note that manually written directed tests provide limited design space coverage, motivating automation [chunk:ee7dd3e3-37ae-42a9-bc6f-567063340cb4].

CITATIONS

11 sources
11 citations
[1] Directed testing is one of the traditional workhorses of digital-system verification and is precise and targeted but requires significant manual effort. Supervised Learning for Coverage-Directed Test Selection in Simulation-Based Verification
[2] FLAME characterizes directed testing as engineer-driven and observes it becomes costly and insufficient for scenarios at the periphery of expected behaviors as designs grow. MoDitector: Module-Directed Testing for Autonomous Driving Systems
[3] The LFI directed testbench workflow uses a black-box test specification reviewed by design and verification teams and VHDL BFMs driving read/write/idle transactions with a separate protocol checker. Supervised Learning for Coverage-Directed Test Selection in Simulation-Based Verification
[4] Reported LFI coverage figures: 100% justified statement coverage, 98.4% branch, 95.2% condition, 84.2% trigger, 100% toggle for directed tests; bug-type breakdown shows property checking found all 11 bus-protocol violations, directed testing found all 5 livelock/deadlock bugs, and random testing found 22 bugs across categories. Supervised Learning for Coverage-Directed Test Selection in Simulation-Based Verification
[5] NaxRiscv verification flow combines directed testing with real-world benchmarking and constrained-random testing, beginning with riscv-tests and riscv-arch-tests in a Jenkins CI pipeline. Supervised Learning for Coverage-Directed Test Selection in Simulation-Based Verification
[6] ConfigDrivenV reports that verification engineers allocate more than 70% of project time to RTL verification and characterizes existing approaches as either SystemVerilog-internal randomization or manually written directed tests with limited design-space coverage. ConfigDrivenV: A Python-Integrated UVM Testbench for Automated RTL Verification with Self-Checking Scoreboard
[7] ConfigDrivenV validates a Python golden reference model plus UVM self-checking scoreboard on a 16-bit RISC processor and a UART, reporting 100% verification accuracy on correct RTL and 100% bug detection on corrupted RTL. ConfigDrivenV: A Python-Integrated UVM Testbench for Automated RTL Verification with Self-Checking Scoreboard
[8] Imperas OVP step-and-compare flow encapsulates an Instruction Set Simulator (OVP model) into a SystemVerilog module via DPI, loads the same .elf into DUT and OVP memories, and steps both targets in lockstep while comparing dynamic state and accumulating functional coverage into results.log. Imperas verification RISC-V Global Forum 2020 v3
[9] MoDitector introduces module-specific oracles, module-specific feedback, adaptive seed selection, and mutation to generate tests that provoke failures in specific ADS modules (perception, prediction, planning, control), extending directed testing to module-level fault attribution. MoDitector: Module-Directed Testing for Autonomous Driving Systems
[10] Coverage-directed test selection uses supervised learning on coverage feedback to bias selection of tests toward those likely to increase functional coverage, reducing manual constraint writing and accelerating coverage closure. Supervised Learning for Coverage-Directed Test Selection in Simulation-Based Verification
[11] Jayasena and Mishra, 'Directed Test Generation for Hardware Validation: A Survey', ACM Computing Surveys, vol. 56, no. 5, pp. 136, Dec. 2023, is cited as a systematic survey of directed test generation techniques for hardware validation. ConfigDrivenV: A Python-Integrated UVM Testbench for Automated RTL Verification with Self-Checking Scoreboard

VERSION HISTORY

v4 · 9/30/2026 · minimax/minimax-m3 (current)
v3 · 9/21/2026 · minimax/minimax-m3
v2 · 9/21/2026 · minimax/minimax-m3
v1 · 6/25/2026 · minimax/minimax-m3