Skip to content
STIMSMITH

Automated Test Generation

Concept WIKI v1 · 7/6/2026

Automated Test Generation is a verification technique that programmatically produces test cases for software and hardware systems, reducing manual effort and enabling coverage at scale. It is applied in domains such as RISC-V processor and SoC verification (e.g., Breker's Test Suite Synthesis) and in software testing for non-traditional artifacts such as Dockerfiles and Python code.

Automated Test Generation

Overview

Automated Test Generation refers to techniques that programmatically create test cases for validating software or hardware systems, replacing or augmenting manually authored tests. It is used where manual test creation is impractical due to the scale, complexity, or sheer volume of configurations that must be exercised. Approaches range from coverage-guided search-based methods, to result/layer-based analysis for declarative artifacts, to constraint-driven and AI-planning synthesis for hardware.

Hardware Verification Applications

RISC-V Core and SoC Verification

In the RISC-V ecosystem, automated test generation has been identified as a key technique for meeting verification quality expectations. Industry data presented at Verification Futures Reading 2024 indicates that Arm spends approximately $150 million per year on 10^15 verification cycles per core, illustrating the scale of effort required for processor verification [Breker 2024].

RISC-V verification must address several distinct challenges:

  • Custom instruction verification — RISC-V permits custom extensions that must each be validated.
  • Compliance assurance — RISC-V International is developing a certification committee to certify devices against architectural compliance tests.
  • Broad architecture range — Different processors have different needs spanning embedded cores, processor clusters, and application processors [Breker 2024].

Test Suite Synthesis for Processors and SoCs

Breker Verification Systems provides Test Suite Synthesis products that generate tests for RISC-V cores and SoCs, drawing on its prior experience in x86 and Arm verification. The Breker SystemVIP Library includes offerings such as Core Integrity FastApps, the RISC-V System Integrity TrekApp, ARM System Integrity TrekApp, Cache Coherency TrekApp 2.0, Firmware-First TrekApp, Power Management TrekApp, Security TrekApp, and Networking TrekApp [Breker 2024].

Synthesis Approaches

Two principal paradigms are used for test generation in this hardware domain:

  • Constrained Random — generating tests within user-defined constraints.
  • AI Planning Algorithm Synthesis — using planning algorithms to derive test scenarios.

These approaches are applied to crossing RISC-V core verification components and to concurrent test execution [Breker 2024].

Core-Level Test Generation Scenarios

Within a single RISC-V core, automated test generation is applied to several scenarios:

  • RV64 Core Instruction Generation
  • Instruction Coverage Analysis — measuring reachable opcodes (one example reported 27/103 reachable opcodes exercised in a register-only test, leaving atomics, loads, and stores unreachable)
  • RV64 Core Load/Store — including locality of writes
  • RV64 Core Exception Testing
  • Page-Based Virtual Memory Tests and Page-Based MMU Tests — for example, swapping MMU Page Table Entries and checking memory access [Breker 2024]

Software Testing Applications

Dockerfile Test Generation

Dockerfiles lack traditional control-flow branches, which makes conventional branch-coverage metrics — typically used as objective functions in search-based test generation for general-purpose languages — meaningless. To address this, an automated test generation method for Dockerfiles was proposed that operates on processing results rather than processing steps. The method analyzes Dockerfile instructions and Docker image layers to determine which files should be tested and generates the corresponding tests. In experiments, the approach reproduced over 80% of the tests created by developers [arXiv:2504.18150, 2025].

Python Test Generation Using LOC Heuristics

For property-based Python testing using the TSTL (Template Scripting Testing Language) tool, a heuristic based on relative Lines of Code (LOC) — comparing LOC across components within the same project — was shown to substantially improve the effectiveness of automated test generation. This is particularly valuable in languages where collecting code coverage data to guide testing has very high overhead. Reported experimental results include:

  • Branch and statement coverage improvements of often more than 20%, up to 40% or more.
  • Fault-detection improvements of usually more than 75%, up to 400% or more.
  • The LOC heuristic is easy to combine with other approaches and is comparable to, and possibly more effective than, two well-established approaches for guiding random testing [arXiv:2103.07006, 2021].

Techniques and Approaches Summary

The evidence covers several distinct automated test generation techniques:

  • Coverage-guided search-based generation — uses branch or statement coverage as the objective function; standard for languages such as Java but ineffective for non-branch artifacts such as Dockerfiles.
  • Result-based / layer-analysis generation — generates tests from analysis of what files should be produced (e.g., Dockerfile image layers), rather than from execution paths.
  • Heuristic-based generation — uses relative metrics such as LOC to guide test input selection when coverage collection is expensive.
  • Constrained-random and AI-planning synthesis — composes test scenarios from constraint models in hardware verification [Breker 2024; arXiv:2504.18150; arXiv:2103.07006].

See Also

LINKED ENTITIES

3 links
Breker Verification Systems IMPLEMENTS
RISC-V APPLIED_TO
Test Suite Synthesis RELATED_TO

CITATIONS

7 sources
7 citations
[1] Automated test generation is identified as a key technique to meet RISC-V verification quality expectations, given the scale at which established ISAs (e.g., Arm, ~$150M/year on 10^15 verification cycles per core) are verified. RISC-V Core & SoC Verification: A New Normal in Verification Techniques (Breker, Verification Futures Reading 2024)
[2] RISC-V verification challenges include custom instruction verification, compliance assurance via RISC-V International certification, and a broad range of target architectures (embedded cores, processor clusters, application processors). RISC-V Core & SoC Verification: A New Normal in Verification Techniques (Breker, Verification Futures Reading 2024)
[3] Breker Verification Systems delivers Test Suite Synthesis for RISC-V cores and SoCs, with a SystemVIP Library that includes Core Integrity FastApps, RISC-V System Integrity TrekApp, ARM System Integrity TrekApp, Cache Coherency TrekApp 2.0, Firmware-First TrekApp, Power Management TrekApp, Security TrekApp, and Networking TrekApp. RISC-V Core & SoC Verification: A New Normal in Verification Techniques (Breker, Verification Futures Reading 2024)
[4] Two principal automated test generation paradigms used in hardware verification are Constrained Random and AI Planning Algorithm Synthesis, applied to crossing RISC-V core verification components and concurrent test execution. RISC-V Core & SoC Verification: A New Normal in Verification Techniques (Breker, Verification Futures Reading 2024)
[5] Core-level automated test generation scenarios for RV64 include instruction generation, instruction coverage analysis (e.g., 27/103 reachable opcodes exercised in a register-only test, with atomics/loads/stores unreachable), load/store (with locality of writes), exception testing, and page-based virtual memory/MMU tests (e.g., swapping MMU PTEs and checking memory access). RISC-V Core & SoC Verification: A New Normal in Verification Techniques (Breker, Verification Futures Reading 2024)
[6] Because Dockerfiles contain no branches, conventional branch-coverage objective functions used in search-based automated test generation are meaningless for them; a result-based method that analyzes Dockerfile instructions and Docker image layers reproduced over 80% of developer-authored tests. Toward Automated Test Generation for Dockerfiles Based on Analysis of Docker Image Layers (arXiv:2504.18150v1, 2025)
[7] A relative Lines-of-Code (LOC) heuristic for property-based Python testing with TSTL improved branch and statement coverage by often >20% (up to 40%+), and improved fault detection by usually >75% (up to 400%+), and is comparable to or better than two established random-testing guidance approaches. Using Relative Lines of Code to Guide Automated Test Generation for Python (arXiv:2103.07006v1, 2021)