Skip to content
STIMSMITH

LLM-based Test Generation

Technique WIKI v1 · 8/12/2026

LLM-based Test Generation uses large language models to synthesize tests or test inputs, often as seed programs that are compiled and executed by downstream testing workflows such as fuzzing, mutation-guided testing, and differential testing. Evidence from SearchSYS shows the technique applied to ARM simulator testing by generating C seed programs, fuzzing compiled binaries, and comparing executions on real hardware and gem5; broader studies report both strong baseline coverage and sensitivity to program evolution.

Overview

LLM-based Test Generation is a testing technique in which a large language model produces executable tests or test inputs. In the SearchSYS ARM-simulator workflow, the LLM is used for initial C code generation; the generated C programs are compiled, their binaries are used as inputs to a fuzzer, and the resulting test cases are checked with differential testing against real hardware and a simulator.[1]

The technique is commonly used as one stage in a larger automated testing pipeline rather than as a complete oracle by itself. The SearchSYS artifact combines LLM-generated seed programs, AFL++ fuzzing, and differential testing to test the gem5 ARM ISA simulator.[2]

SearchSYS-style workflow

A representative workflow from the ARM simulator artifact is:

  1. Generate a corpus of LLM-produced C programs.
  2. Compile each C program into a binary.
  3. Use the compiled binaries as seed material for fuzzing.
  4. Execute generated or fuzzed test cases on both real ARM hardware and the gem5 simulator.
  5. Use differential testing to identify mismatches between hardware and simulator behavior.[1]

The artifact documentation describes seven sets of LLM-generated C programs and includes setup steps for models such as TinyLlama, Magicoder, Phi, and GPT-3.5 variants.[3]

Reported results in ARM simulator testing

In the Search+LLM-based Testing for ARM Simulators artifact, SearchSYS is reported to create bug-activating test cases by combining LLMs, fuzzing, and differential testing. The artifact reports that using only LLM-generated tests identified 74 bug-activating test cases; adding fuzzing produced 93 additional bug-activating cases within 24 hours. Differential testing identified 624 bugs with LLM-generated test cases and 126 with fuzzed test inputs, and 4 unique bugs were reported and acknowledged by developers.[4]

Other reported uses and limitations

A large-scale empirical study of LLM-based test generation under software evolution reports strong baseline results on original programs, including 79% line coverage and 76% branch coverage with fully passing suites, but finds that generated-test performance degrades under both semantic-altering and semantic-preserving changes. The study reports that under semantic-altering changes, newly generated test pass rate drops to 66% and branch coverage to 60%; under semantic-preserving changes, pass rate drops to 79% and branch coverage to 69%. The authors conclude that current LLM-based test generation relies heavily on surface-level cues and struggles to maintain regression awareness as programs evolve.[5]

Mutation-guided LLM-based test generation has also been reported in Meta's ACH system. ACH generates relatively few issue-focused mutants, uses them to produce tests that kill currently uncaught faults, and applies an LLM-based equivalent-mutant detection agent. In Meta's reported deployment, ACH was applied to 10,795 Android Kotlin classes across 7 platforms and generated 571 privacy-hardening test cases.[6]

Practical considerations

For SearchSYS on ARM, the artifact requires an ARM machine, at least 80 GB of free disk space, and either 72 GB RAM for AFL++ fuzzing or 16 GB RAM for differential testing without building gem5 with AFL++ instrumentation.[7] The full ICSE-SEIP 2025 evaluation is documented as a long-running reproduction process, while the artifact also provides a shortened evaluation intended to run within a day or less.[8]

[1]: SearchSYS pipeline and differential-testing design. [2]: SearchSYS target and ARM ISA simulator context. [3]: LLM-generated C corpora and model-specific setup. [4]: Reported SearchSYS bug-activation and differential-testing results. [5]: Public arXiv summary, "Evaluating LLM-Based Test Generation Under Software Evolution". [6]: Public arXiv summary, "Mutation-Guided LLM-based Test Generation at Meta". [7]: ARM hardware requirements in the SearchSYS artifact. [8]: Artifact documentation structure and shortened evaluation guidance.

LINKED ENTITIES

8 links
Search+LLM-based Testing for ARM Simulators USES
SearchSYS IMPLEMENTS
TinyLlama IMPLEMENTS
Magicoder IMPLEMENTS
Phi IMPLEMENTS
GPT-3.5 IMPLEMENTS
C Program Generation USES
seed corpus USES

CITATIONS

8 sources
8 citations
[1] SearchSYS uses LLM-generated C code as initial seed programs, compiles them, feeds binaries to a fuzzer, and uses differential testing on real hardware and a simulator. Artifact of Search+LLM-based Testing for ARM Simulators - Zenodo
[2] SearchSYS is applied to testing the gem5 VLSI digital circuit simulator with a focus on ARM ISA simulation. Artifact of Search+LLM-based Testing for ARM Simulators - Zenodo
[3] The artifact describes seven sets of LLM-generated C programs and setup steps for model-specific corpora including TinyLlama, Magicoder, Phi, and GPT-3.5 variants. Artifact of Search+LLM-based Testing for ARM Simulators - Zenodo
[4] SearchSYS reported 74 bug-activating test cases from LLM-only testing, 93 additional bug-activating cases after 24 hours of fuzzing, 624 bugs found with LLM-generated test cases, 126 with fuzzed inputs, and 4 unique acknowledged bugs. Artifact of Search+LLM-based Testing for ARM Simulators - Zenodo
[5] The SearchSYS ARM artifact requires an ARM machine with 80 GB free disk space and either 72 GB RAM for AFL++ fuzzing or 16 GB RAM for differential testing without building gem5 with AFL++ instrumentation. Artifact of Search+LLM-based Testing for ARM Simulators - Zenodo
[6] The SearchSYS artifact is organized into requirements, full ARM reproduction instructions, and a shortened evaluation intended to run within a day or less. Artifact of Search+LLM-based Testing for ARM Simulators - Zenodo
[7] A 2026 empirical study reports that LLM-generated tests achieve 79% line coverage and 76% branch coverage on original programs, but degrade under semantic-altering and semantic-preserving changes and show sensitivity to surface-level cues. Evaluating LLM-Based Test Generation Under Software Evolution
[8] Meta's ACH system is a mutation-guided LLM-based test generation system that generated issue-focused tests from mutants and was applied to 10,795 Android Kotlin classes across 7 platforms, producing 571 privacy-hardening test cases. Mutation-Guided LLM-based Test Generation at Meta