Directed Test Generation

Overview

Directed test generation is the practice of authoring or producing stimulus that intentionally targets specific scenarios, architectural features, or known regression cases, in contrast to random or constrained-random stimulus that explores a broad state space. In microprocessor and hardware verification it is used alongside random exercisers as part of a hybrid verification strategy, because directed tests provide precise, scenario-specific coverage while random tests provide broad, unexpected exploration. [C1] [C2] [C3]

The same conceptual structure appears across multiple sub-domains of hardware verification:

• Microprocessor / micro-architectural validation, where coverage-driven functional test generation (Koo and Mishra) defines directed tests as one of three categories of test generation and uses formal methods to automate them. [C1]
• Industrial micro-architectural test generation, where a directed test generation technique targeting pipeline interactions and micro-architectural faults was demonstrated on an industrial Power Architecture processor using decompositional model checking. [C14] [C15]
• Architectural (ISA-conformance) testing, where RISC-V architectural tests are generated as controlled, directed tests from ISA coverpoints, with the explicit goal of ensuring the tests are signature-influencing (positive testing only). [C13]
• Stimulus-engine flows, where PPIGen and SGen provide directed-mode generation as a backend that hand-written or sequence-composed tests are routed through, sitting alongside random/exer mode. [C5]
• Golden-model / ISS-based verification flows, where the classical role of hand-written directed assembly or C/C++ tests is to drive simulation whose expected results are compared against a golden reference such as an instruction-set simulator (ISS) for an open-source CPU core (e.g., OR1200). [C16]

Test-generation categories

Coverage-driven test-generation literature (Koo and Mishra) categorizes stimulus-generation techniques for functional validation of processor designs into three types:

• Random — stimuli drawn broadly from the input space, used to stress large design state spaces.
• Directed — stimuli authored or synthesized to target specific scenarios, regression cases, or functional coverage goals. Directed tests can reduce overall validation effort because shorter tests can achieve the same coverage goal as random tests. [C1] [C15]
• Directed-random — hybrid techniques that combine random exploration with directed targeting.

Hand-written directed tests are laborious and time-consuming because they require verification engineers with deep knowledge of the design under verification, and it is infeasible to manually author every directed test needed for comprehensive coverage. This motivates automatic directed test generation. [C1] [C14]

Motivation

Functional verification is widely recognized as a major bottleneck in processor design, driven by:

• The exponential growth in design complexity (pre-silicon logic bugs in successive Intel IA-32 generations grew at a rate of 300-400% per generation). [C2]
• Decreasing time-to-market pressure, especially for embedded processors. [C2]
• The fact that functional verification accounts for up to 50% to 70% of overall design development time and resources. [C1] [C2] [C16]
• Adopting complicated micro-architectural mechanisms such as deeply pipelined superscalar designs, dynamic scheduling, and dynamic speculation, which makes verification complexity directly proportional to design complexity. [C14]

Simulation using test programs is the most widely used form of processor verification, because formal methods provide completeness only at the component level due to the state-explosion problem. [C2] [C14] This is the practical context in which directed test generation, and especially its automated variants, are studied.

Why micro-architectural directed testing is needed

Industrial practice typically relies on random and biased-random test generation at the architecture (ISA) level, because this is the most widely used approach for simulation-based validation, both for uncovering errors early in the design cycle and for simulating the entire processor design. [C14] However, ISA-level architectural test generation has difficulty in activating micro-architectural target artifacts and pipeline functionalities, because it is not possible to generate information regarding pipeline interactions or timing details from an ISA-only input specification. [C14] For example, it is very hard from an ISA specification alone to generate an architectural test program for micro-architectural design bugs such as a pipeline interaction error (e.g., "decode stage is not stalled even if Completion Queue is full") or a performance error (e.g., "data dependency is not resolved by forwarding path even if operand is available"). [C14]

These observations motivate the use of micro-architectural details during test generation, and motivate automation of directed test generation for micro-architectural faults rather than relying solely on hand-written expert tests. [C14]

Limitations of classical hand-written directed tests

In classical simulation-based processor verification, directed tests are written by hand in assembly or high-level languages such as C/C++ and simulated against a reference model whose results are known in advance and used for comparison. [C16] This approach has been proven insufficient for the verification of complex designs. [C16] The reason is that the results of directed tests must be known in advance for comparison, so the approach is not suitable when hand-authoring every needed test becomes infeasible. [C16] For open-source CPU cores such as OR1200, the corresponding reference is typically an instruction-set simulator (ISS) used as a golden model, on the basis that the ISS is usually implemented before the core for performance analysis on the instruction set; reusing the ISS as the golden model speeds up implementation and verification of the CPU core while saving resources. [C16]

These limitations motivate the move toward automated directed test generation and toward complementary golden-model-based comparison infrastructures. [C16]

Formal-methods-based directed test generation

The Coverage-driven Functional Test Generation paper presents an automatic directed-test-generation methodology for pipelined processors using formal methods. The core idea is to express each desired coverage target as a temporal-logic property, then ask a model checker to falsify the negated property, which produces a counterexample that becomes the directed test program.

Functional fault model and temporal properties

The approach defines pipeline-interaction behaviors using a graph-based model of the processor. The negated versions of desired behaviors, called functional faults, are converted into temporal-logic properties. Functional faults therefore define a functional coverage metric and drive test generation: each test is a counterexample for one functional fault. [C3]

Coverage-driven flow

The overall flow is iterative:

1. Generate a list of functional faults in the design under validation from the processor model and fault model.
2. Select one undetected fault for test generation.
3. A functional test generator (e.g., a model checker or SAT solver) produces a test program for that fault.
4. Fault simulation determines all the other faults detected by that test program; the detected faults are removed from the functional fault list.
5. Repeat from step 2 until all faults are detected. [C3]

Model checking as the test-generation engine

Model checking is used as the test-generation engine. The processor model is described in a temporal specification language and a desired behavior is expressed as a temporal-logic property. A model checker exhaustively searches all reachable states of the model (unbounded model checking, UMC) to verify or falsify the property. [C3]

For test generation, instead of the desired property, its negated version is applied to the model checker so that any counterexample produced is precisely a test program that exhibits the target fault. This falsification capability is what makes model checking directly usable for test generation. [C3]

Related work in the same family

The coverage-driven paper situates its approach against related work for architectural and micro-architectural validation of pipelined processors:

• Processor models described as finite state machines (FSMs) with reachable states and state transitions used to generate test programs based on FSM coverage. [C3]
• RTL abstraction that creates an abstract FSM model while preserving clock-accurate behaviors, used to handle the large size of FSMs for modern processors. [C3]
• Abstraction of processor models to generate test programs for micro-architectural validation of a superscalar PowerPC processor. [C3]
• Separation of the design model from the test-generation engine to avoid state explosion in formal methods and to facilitate test generation for modern processors. [C3]

Challenges: state explosion

Traditional model checking does not scale to modern processor designs because of the state-explosion problem. Bounded model checking (BMC) combined with satisfiability (SAT) solving has been proposed to mitigate this. [C3] [C4]

Solutions: bounding and decomposition

The methodology addresses scalability through three contributions:

• Functional-fault-based property specification — pipeline-interaction faults are converted into temporal-logic properties suitable for formal-method-based test generation. [C4]
• Bound determination per property — a procedure is developed for determining the bound for each individual property, so that BMC remains tractable. [C4]
• Decomposition — the formal processor model and its properties are partitioned into components to exploit the falsification capability of model checking on smaller sub-problems. Search-space restriction is also applied for large partitions. [C3] [C4]

The methodology has been successfully applied to MIPS processors. [C3]

Industrial case study: Power Architecture processor (MTV'06)

A follow-up case study (Koo, Mishra, Bhadra, Abadir, MTV'06) extends the formal-methods-based directed test generation approach to an industrial processor based on the Power Architecture Technology. The work targets micro-architectural-level directed test generation for functional validation of microprocessors.

Input specification and methodology

The methodology takes an architectural specification as input that contains both the structure (micro-architectural details) and the behavior (instruction set) of the processor. [C14] The micro-architectural features in the processor model include pipelined and clock-accurate behaviors that enable micro-architectural test generation. [C14]

Properties are automatically generated from the input specification based on a functional fault model such as pipeline (graph) coverage. [C14] Additional properties can be added based on interesting scenarios such as combined pipeline stage rules and corner cases. [C14] These properties are described in temporal logic. [C14]

Decompositional model checking

For automatic test generation, the methodology uses decompositional model checking: the processor model and the properties are decomposed, and model checking is applied on smaller partitions of the design using decomposed properties. [C14] The methodology introduces the notion of time steps to enable decomposition of the properties into smaller ones based on their clock cycles. [C14] The paper develops an efficient algorithm to merge the partial counterexamples generated by the decomposed properties to produce the global counterexample corresponding to the target property. [C14]

Scalability observation

The case study notes that prior approaches have difficulties when applied to industrial processors due to the high complexity of even a single pipeline path, and that those approaches mainly focus on the data path rather than the control path. [C14] While data (opcode and operands) is located in a single pipeline stage, control signals (functional unit status and buffer status) may spread across multiple pipeline stages and buffers, which makes model partitioning and counterexample merging more difficult. [C14] This motivates the time-step-based decomposition and merging algorithms that the paper introduces. [C14]

The experiments using a Power Architecture-based processor showed very promising results in terms of test generation time as well as test program length. [C14]

Directed tests in architectural (ISA-conformance) testing for RISC-V

The CARRV 2022 slides on automating RISC-V architectural test suites extend the directed-test-generation concept to ISA-conformance / architectural-compliance testing, where the design under verification is treated as a black box that must obey the RISC-V ISA specification rather than as a micro-architecture to be verified internally.

Objectives and scope of architectural testing

• Architectural (Compatibility) Testing involves ensuring that an implementation of the ISA meets all the requirements under all conditions. [C13]
• RISC-V is micro-architecture agnostic and extremely configurable; an exhaustive test suite for all implementations is impossible, but it is possible to build a suite that tests limited areas of the ISA. [C13]
• Architectural testing is a subset of design verification and is not sufficient by itself to ensure a functionally correct implementation; it only deals with verification of details specified in the ISA. [C13]

Challenges in maintaining a high-quality suite

The slides identify four challenges to producing and maintaining a high-quality architectural test suite for RISC-V:

1. Standardised test format for uniformity and maintenance — the RISC-V Architectural Test SIG Test Format Specification provides standard macros and mandates signature-based tests compared against a "golden" model signature. [C13]
2. A standard ISA-coverage specification format to indicate test quality and identify gaps. [C13]
3. A tool to measure coverage as per that specification. [C13]
4. A tool to generate efficient, directed tests as the ISA specification and testing scopes grow. [C13]

RISCV-CTG: directed test generation from coverpoints

The slides describe RISCV-CTG, which is positioned as a directed test generator for architectural tests. Its properties, drawn from the slides' "Contributions" slide:

• Low barriers to entry: simple ISA-coverage specification format; tool is independent of simulator/implementation; only basic Python and RISC-V knowledge required. [C13]
• Generated tests are controlled and directed — generated from coverpoints, ensuring that full coverage is achieved with minimal testing; tests runnable on any implementation/model; no negative testing (test only for features/behavior implemented). [C13]
• Data-propagation analysis ensures that the signatures generated are influenced by the tests (preventing tests that compute a constant signature regardless of input). [C13]
• Open source tools for coverage measurement and test generation. [C13]

In this framing, each test is a controlled, directed stimulus synthesized from one or more coverpoints in the ISA-coverage specification; the directed nature is what keeps the suite small while still covering the targeted ISA behavior, mirroring the "shorter tests achieve the same coverage" observation from the Koo and Mishra work but at the ISA level rather than the micro-architectural level.

Directed tests in stimulus-generation flows (PPIGen / SGen)

The PPIGen / SGen C++11 stimulus-engine paper offers a complementary, simulator-centric view of directed test generation, in which the same logical operation is achieved by hand-written sources plus a configuration rather than by formal-method synthesis.

PPIGen directed and random modes

The PPIGen tool supports two distinct modes of operation for generating test programs:

• Directed mode — the input is a C or assembly (.S) source file together with a configuration (.cfg) file containing configuration overrides. PPIGen reads the source file and the config file and generates appropriate initialization code and linker scripts. In directed mode, configuration registers assume default (reset) values unless the config file specifies otherwise. [C5]
• Random (exer) mode — the input is a single config file that optionally constrains random variables affecting program generation and system configuration. In random mode, configuration registers are randomized rather than left at reset values. [C5]

In both modes PPIGen performs the same two basic functions: generating an instruction stream (random mode) or processing a hand-written source (directed mode), and producing the initialization code and linker scripts driven by the configuration. [C5]

SGen piggybacking on directed mode

SGen was developed to leverage the configuration and linker-script generation logic already present in PPIGen. SGen produces an assembly program and an appropriately formatted config file that is fed through PPIGen's directed flow. In effect, SGen piggybacks onto the directed mode of PPIGen, even though SGen is itself used to drive randomized sequence-based generation. [C6]

The SGen architecture builds directed-style tests out of:

• An inst library containing a class hierarchy of instructions organized by type (load, store, SIMD, etc.).
• A seq library of sequences, each written with a specific intent.
• A tests library of tests that compose one or more sequences to achieve a specific verification goal.
• A randutils package that provides randomization utility classes (including a weighted_set for picking randomly from a set of objects). [C7]

C++11 features used heavily by SGen include lambda functions (for closures that capture scope, enabling randomization and user-defined overrides), the improved <random> library distributions and generators, std::function for storing references to callable objects, regular expressions, and the auto specifier. [C8]

A typical SGen test is a bare_sequence whose body is supplied as a lambda. The body registers instruction objects with a weighted_set (wset) so that, when SGen feeds the generated assembly through PPIGen's directed flow, the result is a test that mixes targeted instruction selection with controlled randomization. [C9]

Directed tests in regression practice

The evidence presents data from an RTL designer who runs a small sample of random exercisers as part of regression qualification for check-ins. The regression suites used by this designer are composed mostly of directed tests, with a small selection of PPIGen and SGen exercisers. [C10]

Two key observations come from this regression data:

• SGen catches RTL bugs more consistently than the knob-based PPIGen. Both tools caught RTL errors, but SGen produced a greater number of total failures in these regression runs, and the designer reported that SGen consistently caught bugs that the standard regression suite and PPIGen would have missed. [C10]
• Directed failures do not necessarily block releases. Because the regression suite is often not 100% clean, directed failures alone do not necessarily prevent changes from being released. This is one reason the designer began supplementing regressions with random exercisers. Across the reported runs, only one regression series (reg3, with 21 failures) had directed failures that would have prevented release. [C10]

This motivates the hybrid pattern: directed tests dominate the standard regression suite, but random exercisers (PPIGen and SGen) are layered on top to find bugs that directed tests miss.

Exerciser-scale evidence

Over a one-month period, 25,000 exercisers per generator (PPIGen and SGen) were run, for 50,000 total runs and 1,731 total failures. PPIGen accounted for 171 of those failures and SGen accounted for 1,560. The evidence notes that, on first glance, this could suggest SGen is over-failing or is otherwise distinguished from PPIGen at scale; the full interpretation is not contained in the available excerpts. [C10]

Hybrid pattern in practice

The combined evidence supports a verification flow in which:

1. A primarily directed regression suite forms the baseline qualification gate for RTL check-ins.
2. Random exercisers from PPIGen and SGen are layered onto the regression to catch classes of bugs the directed suite misses, with SGen reported as the more consistent catcher in the cited data.
3. Where formal methods can express specific coverage goals as temporal properties, automatic directed test generation via model checking (with bounding and decomposition to mitigate state explosion) complements hand-written and random stimulus. [C1] [C3] [C4]
4. For industrial micro-architectural designs (e.g., Power Architecture-based processors), decompositional model checking with time-step-based property decomposition and partial-counterexample merging enables micro-architectural directed test generation that targets pipeline interactions and corner cases. [C14]
5. At the ISA level, RISCV-CTG generates controlled, directed architectural tests from coverpoints with data-propagation analysis, producing a small, signature-influencing suite that runs against any RISC-V implementation. [C13]
6. When failures arise, they can be replayed deterministically using directed test inputs (a C or .S file plus a config file for PPIGen, or a sequence/library-driven test for SGen), giving debug a stable, hand-written entry point. [C5] [C10]
7. In classical ISS-as-golden-model verification of open-source cores such as OR1200, hand-written directed assembly/C/C++ tests drive simulation whose results are compared against an ISS reference, illustrating the historical role of directed tests and the reason they are insufficient alone for complex designs. [C16]

Adjacent domains and references

Although the primary evidence is in processor / hardware verification, the same conceptual structure — generation of stimulus targeted at specific scenarios, regression cases, or coverage goals — appears in adjacent domains:

• Database systems. CLOTHO is a directed-test-generation framework for detecting serializability violations in SQL database-backed Java applications executing on weakly-consistent storage. It combines a static analyzer and a model checker to generate abstract executions, discover serializability violations, and translate them back into concrete test inputs. [C11]
• LLM-aided hardware test generation. Coverage Directed Test Generation (CDG) has been augmented with a Large Language Model acting as a "Verilog Reader" that understands code logic and generates stimuli to reach unexplored code branches; experiments show this LLM-aided approach outperforms random testing on designs within the LLM's comprehension scope. [C12]

See also

• Coverage-driven Functional Test Generation for Processor Validation using Formal Methods — formal-methods-based directed test generation for pipelined processors using model checking and SAT-based BMC. [C3] [C4]
• Directed Micro-architectural Test Generation for an Industrial Processor: A Case Study — decompositional model checking with time-step-based property decomposition and partial-counterexample merging applied to a Power Architecture-based industrial processor. [C14]
• Automating Generation and Maintenance of a High-Quality Architectural Test Suite for RISC-V — RISCV-CTG as coverpoint-driven, controlled, directed test generation for RISC-V architectural (ISA-conformance) testing, with data-propagation analysis. [C13]
• Verifying Open Source CPU Cores using Instruction Set Simulators in OVM Environments — ISS-as-golden-model verification of OR1200, documenting the classical role and limits of hand-written directed assembly/C tests. [C16]
• Random Test Generation — the broad-state-space counterpart to directed test generation; together with directed tests forms the directed-random hybrid. [C1]
• Model Checking — used as the test-generation engine that produces counterexamples for negated temporal-logic properties (functional faults). [C1] [C3]
• PPIGen — implements directed and random test generation modes; provides init-code and linker-script generation reused by SGen. [C5]
• SGen — uses PPIGen's directed flow as its downstream backend; provides sequence-based, C++11-driven test generation. [C6] [C7] [C8] [C9]