Model-based Test Generation Wiki

Overview

Model-based test generation is an automated test-generation technique in which a model or input-format specification guides the generation process. In processor-level verification literature, model-based test generators are described as using an input-format specification to guide generation, with optional constraints processed by CSP/SMT solvers. [Model-based generation definition]

The technique is discussed as one response to the limitations of unguided random generation of processor-level stimuli: the cited ISS-verification paper states that various approaches have been proposed to improve random generation, then lists model-based test generators among them. [Processor-stimulus context]

Generation mechanisms

The evidence identifies several model-based or specification-driven mechanisms:

Input-format specifications: model-based generators guide test generation from a specification of the input format. [Model-based generation definition]
Constraint-based generation: constraints can be integrated and processed by CSP/SMT solvers. [Constraint-based generation]
Cross-instruction constraint propagation: an optimized test-generation framework can propagate constraints among multiple instructions effectively. [Cross-instruction propagation]
Automatically obtained input models: processor manuals can be mined to obtain an input model automatically. [Manual-derived input models]

Public software-engineering work also describes model-based generation with domain-specific concepts: a language expert builds test models using a Java-based framework, after which the toolset forms a domain-specific modeling language that domain experts can use to constrain and guide generation, generate large test sets, target constrained parts of a model, or define specific tests on top of the model. [Domain-specific model guidance]

Application areas in the evidence

Processor and ISS verification

In the processor-verification setting, model-based generation appears as a way to produce processor-level stimuli more systematically than pure random generation. The cited ISS paper contrasts such approaches with coverage-guided fuzzing and reports an implementation on top of libFuzzer evaluated on three public RISC-V instruction-set simulators, finding new errors in every considered ISS, including one in the RISC-V reference simulator Spike. [ISS fuzzing comparison]

The same paper notes that coverage-guided fuzzing can operate at the binary level and is not constrained to a specific instruction subset, unlike a generator such as RISC-V Torture; this enabled checks for errors that may be masked by compiler or assembler behavior and helped reveal decoder-related issues. [Binary-level fuzzing contrast]

Web-application regression testing

Public work on FRAGGEN presents automated model-based test generation as an alternative to costly manual regression-test creation for web applications. FRAGGEN uses page-fragment-based state abstraction to infer better web-app models and generate regression test suites and oracles; the reported evaluation found more near-duplicate pages, higher model precision and recall, and robust test actions/oracles compared with whole-page techniques. [Web-app model-based generation]

Relationship to adjacent techniques

Model-based generation is adjacent to other stimulus-generation methods. The ISS-verification related-work discussion lists coverage-guided test generation based on Bayesian networks and other machine-learning techniques as other notable approaches. [Adjacent stimulus-generation methods]

Compared with mutation-based coverage-guided fuzzing as described in the ISS paper, model-based generation relies on an input model or specification, while libFuzzer-style fuzzing creates binary bytestream inputs, mutates them randomly, and uses code-coverage feedback from an instrumented device under test. [Coverage-guided fuzzing contrast]

Practical tradeoff

Across the cited evidence, the practical tradeoff is that model-based test generation can direct and constrain generation through explicit models, input specifications, and solver-backed constraints, but it requires a model or a way to derive one. Coverage-guided fuzzing is presented as an alternative that uses mutation and coverage feedback, and in the ISS setting can explore binary inputs that compilers or assemblers may not normally generate. [Model effort tradeoff]