Model-Based Test Generation Wiki

Overview

Model-based test generation is an automated test-generation technique in which a model, input-format specification, or constraint-based description guides the creation of test cases or processor-level stimuli. In the processor-verification literature, model-based test generation is identified as a notable direction that relies on a constraint-based specification to guide the test generation process, in contrast with classical randomized instruction-stream generation. [Model-based definition and motivation]

The technique is also discussed more broadly in software-engineering contexts as a way to support automated test generation with domain-specific concepts. Public SE work describes a framework in which a language expert builds test models using a full (Java) programming language augmented with simple but powerful modeling elements; based on these models, the toolset automatically forms a domain-specific modeling language that domain experts can use to constrain and guide test generation, generate large test sets, target constrained parts of a model, or manually define specific test cases on top of the model. [Domain-specific model guidance]

Generation mechanisms

The evidence identifies several model-based or specification-driven mechanisms:

Constraint-based specification: model-based generation relies on a constraint-based specification to guide the test generation process. [Constraint-based specification]
Cross-instruction constraint propagation: further optimizations propagate constraints among multiple instructions in a more effective way, as in the abstract-CSP-based instruction-stream generator. [Cross-instruction propagation]
Symbolic/explicit input-format specifications: related work in processor verification describes model-based generators that use input-format specifications as the model driving generation. [Input-format specifications]
Automatically obtained input models: processor manuals can be mined to obtain an input model automatically, reducing manual effort in constructing the specification. [Manual-derived input models]

Public software-engineering work also describes model-based generation with domain-specific concepts, in which a language expert builds test models using a Java-based framework, after which the toolset forms a domain-specific modeling language that domain experts can use to constrain and guide generation. [Domain-specific model guidance]

Application areas in the evidence

Processor and ISS verification

In the processor-verification setting, model-based generation appears as a way to produce processor-level stimuli more systematically than pure random generation. The cited symbolic-execution verification paper contrasts such approaches with coverage-guided fuzzing, noting that an effective coverage-guided fuzzing-based methodology can be obtained by using an instruction-set simulator (ISS) as a reference model for the RTL processor under test. [ISS fuzzing comparison]

The same paper notes that modern coverage-guided fuzzers work by mutating randomly created data and are guided by coverage, hence they do not require an input model specification, in contrast with model-based approaches. [Fuzzing as model-free alternative]

A frequently cited foundational reference for processor-level model-based test generation is the IAAI 1994 paper by Lichtenstein, Malka, and Aharon, titled "Model Based Test Generation for Processor Design Verification," which is also listed in the related-work section of the cited processor-implementation-verification methodology paper alongside other industrial verification generators such as those for the IBM RISC System/6000, PowerPC, HP 9000 PA-RISC, and AVPGEN. [Foundational processor MBTG paper]

Symbolic-execution-based verification of RISC-V

The cited DATE paper proposes a verification approach for the open-source RISC-V MicroRV32 processor that uses co-simulation between an RTL processor (translated via Verilator) and a C++ ISS, with the open-source C++ symbolic execution engine KLEE performing symbolic exploration according to the co-simulation setting; symbolic values represent the instructions and initialize the ISS and processor registers, and the ISS serves as a functional reference model. The paper situates model-based test generation as a notable direction in the broader landscape of processor-level stimulus generation techniques. [Symbolic execution approach]

Web-application regression testing

Public work on FRAGGEN presents automated model-based test generation as an alternative to costly manual regression-test creation for web applications. FRAGGEN uses page-fragment-based state abstraction to infer better web-app models and generate regression test suites and oracles; the reported evaluation found more near-duplicate pages, higher model precision and recall, and robust test actions/oracles compared with whole-page techniques. [Web-app model-based generation]

Relationship to adjacent techniques

Model-based generation is adjacent to other stimulus-generation methods. The cited symbolic-execution verification paper lists several other notable approaches including coverage-directed generation using symbolic techniques and additional research directions that apply symbolic-execution tools from the SW domain to RTL test-vector generation. [Adjacent stimulus-generation methods]

Compared with coverage-guided fuzzing as described in the same paper, model-based generation relies on a constraint-based specification or input model, while modern coverage-guided fuzzers work by mutating randomly created data and are guided by coverage feedback without requiring an input model specification. [Coverage-guided fuzzing contrast]

The same paper also notes a tradeoff in applying symbolic-execution techniques from the SW domain to the HW domain: applying symbolic-execution tools to generate test vectors for RTL designs has been evaluated on specific units such as a floating-point unit, and combining static analysis with symbolic execution on execution traces of the RTL design has been proposed to systematically drive up branch coverage. [Symbolic-execution on RTL]

Practical tradeoff

Across the cited evidence, the practical tradeoff is that model-based test generation can direct and constrain generation through explicit models, input specifications, and solver-backed constraints, but it requires significant effort to provide a respective input-format specification. Coverage-guided fuzzing is presented as a complementary model-free alternative that uses mutation and coverage feedback, and in the ISS setting can drive effective processor verification by using an ISS as a reference model. Symbolic execution techniques from the SW domain have been proposed as another approach that mitigates the model-construction effort of model-based generation. [Model effort tradeoff]

Historical context from processor verification

The cited Springer methodology chapter lists several historical processor-verification generators and references that provide broader context for model-based and biased-random test-program generation, including Aharon et al.'s 1991 work on the IBM RISC System/6000 dynamic biased pseudo-random test program generator, Aharon et al.'s 1995 DAC paper on test program generation for PowerPC processors in IBM, the 1992 Hewlett-Packard Journal article on design verification of the HP 9000 Series 7000 PA-RISC workstations, and the 1995 AVPGEN paper in IEEE Transactions on VLSI Systems. [Historical processor verification generators]

The same chapter also lists "Coverage Directed Generation Using Symbolic Techniques" (FMCAD 96) as a related direction that combines coverage feedback with symbolic techniques for test generation. [Coverage directed generation]