Automatic Test Generation

Automatic test generation (ATG) refers to the automated construction of test inputs, test cases, or test programs used to validate the correctness of software systems or hardware designs. Because manually authored tests are costly and rarely exhaustive, ATG techniques have been developed across multiple domains including software engineering, hardware design verification, and automated program repair.

Software Quality Assurance

In software engineering, ATG is treated as a fundamental quality assurance mechanism. Without effective ATG, developers risk introducing subtle data inconsistency bugs, compatibility issues, and security vulnerabilities that can affect large numbers of end users and cause severe financial losses. JSON libraries, which serve as fundamental toolkits for data exchange in the Java ecosystem, illustrate this concern: they power real-world applications such as web services, Android apps, and data management systems.

Recent research has explored the use of large language models (LLMs) for software ATG. General-purpose LLM-based methods such as TitanFuzz and YanHui have shown potential, but lack specialization for particular application domains. To address this, a multi-agent ATG system called JsonATG has been proposed for Java JSON libraries (JJLs). JsonATG uses historical bug information as seeds and introduces a code summarization agent and a test validation agent into the generation pipeline. It applies agent-generated program mutation rules tailored to the structural and semantic characteristics of JJLs, such as data streaming operations, serialization formats, and data-binding patterns, and refines the generated tests through post-processing to ensure syntactic and semantic correctness. In experiments, JsonATG achieved higher coverage than two state-of-the-art LLM-based test generation methods on a widely used JSON library, and with a $25 budget reported 59 bugs in fastjson (including non-crashing functional bugs), of which 47 were confirmed and 28 fixed.

Program Repair

ATG has also been studied as a means to alleviate overfitting in automated program repair. Test-suite-based repair techniques generate patches that may overfit to the available tests and fail to generalize. An approach called UnsatGuided strengthens the repair constraint used by synthesis-based repair techniques by injecting additional automatically generated tests. In a study over the 224 bugs of the Defects4J repository, UnsatGuided was shown to be effective in alleviating one kind of overfitting issue—regression introduction—while having minimal positive impact on the other kind—incomplete fixing—due to the oracle problem.

Hardware Design Verification

In the hardware domain, ATG has long been central to simulation-based functional verification of processors and digital integrated circuits. Functional verification has been driven by the increasing complexity of designs (as documented in the International Technology Roadmap for Semiconductors, Design Chapter, 2005 edition), motivating tools such as Genesys-Pro, which introduced innovations in test program generation for functional processor verification. Foundational references include Bergeron's Writing Testbenches: Functional Verification of HDL Models and the IBM patent on coverage-directed test by Fine, Levinger, and Ziv.

A range of algorithmic techniques have been applied to ATG in this setting:

• Evolutionary / genetic search. Corno et al. presented evolutionary test program induction for microprocessor design verification and a case study on automatic test program generation. Bose et al. proposed a genetic approach to automatic bias generation for biased random instruction generation. A genetic testing framework for digital integrated circuits was introduced by Yu, Fin, Fummi, and Rudnick.
• Coverage-directed generation using Bayesian networks. Fine and Ziv described coverage directed test generation for functional verification using Bayesian networks, and Braun, Fine, and Ziv enhanced the efficiency of Bayesian-network-based coverage-directed test generation.
• Activity monitors and stochastic approaches. A stochastic approach to test generation via activity monitors has also been investigated in the design automation literature.
• Integer linear programming (ILP). ILP-based methods have been pursued as a route toward automating simulation-based design verification.

These approaches illustrate that ATG in hardware verification is closely coupled with coverage metrics, biased random instruction generation, and feedback from coverage analysis, rather than relying purely on unconstrained random stimulus.

RISC-V Vector Verification

In the open-source RISC-V vector (RVV) ecosystem, ATG is one of several suite-generation paradigms (alongside manually authored "Tests" suites and randomized "RTG" suites) used to verify vector implementations. A comparative analysis of open-source RVV test suites distinguishes ATG-class generators such as the RIOS Labs Vector Test Generator and Yang's Vector Test Generator:

• RIOS Labs Vector Test Generator. Targets the RVV v1.0 specification and produces fully self-checking tests. Its configuration features allow users to select specific instructions and set configuration parameters (such as vector register length VLEN, register multiplier LMUL, and element width SEW) to generate targeted tests. It is fully compatible with the RISCOF infrastructure used widely in RISC-V testing and uses the RISCV-ISAC coverage tool against the Sail ISS.
• Yang's Vector Test Generator. An Automatic Test Generator (ATG) developed by Yang Liu of PLCT lab that verifies the RISC-V Vector implementation. It is written in GoLang and uses GNU Make to automate the test generation flow. Its generated tests are fully self-checking, so any verification environment for a core that already utilizes riscv-tests can use them directly; it is supported on the Spike ISS and integrated with the ARA core.

Other RVV suites classified in the same comparative analysis include the Imperas Vector Test Suite (a manually authored "Tests" suite with signature-based checking using riscvOVPsim), RISC-V Torture and RISC-V-DV (both RTG-class, signature-based, targeting RVV v0.9), FORCE-RISCV (an Instruction Sequence Generator using randomization over instructions, registers, addresses, and data to produce valid sequences, with advanced APIs for finer control, targeting RVV v1.0), and the Tenstorrent Vector Test Suite (a manually authored "Tests" suite with self-checking on the Whisper ISS and Ocelot core).

Cross-Cutting Themes

Across domains, ATG research shares several recurring themes: (1) the use of feedback signals such as code coverage or structural program features to guide generation, (2) the need to address the oracle problem when no reliable specification exists, and (3) the integration of domain knowledge (e.g., structural and semantic characteristics of a target library, configuration parameters of a hardware design, or coverage models of a vector ISA) to make generated tests both valid and effective at triggering faults.