LLM-based Test Generation

Technique

LLM-based Test Generation uses large language models to synthesize tests or test inputs, often as seed programs that are compiled and executed by downstream testing workflows such as fuzzing, mutation-guided testing, and differential testing. Evidence from SearchSYS shows the technique applied to ARM simulator testing by generating C seed programs, fuzzing compiled binaries, and comparing executions on real hardware and gem5; broader studies report both strong baseline coverage and sensitivity to program evolution.

First seen 8/12/2026

Last seen 8/17/2026

Evidence 11 chunks

Wiki v1

WIKI

Overview

LLM-based Test Generation is a testing technique in which a large language model produces executable tests or test inputs. In the SearchSYS ARM-simulator workflow, the LLM is used for initial C code generation; the generated C programs are compiled, their binaries are used as inputs to a fuzzer, and the resulting test cases are checked with differential testing against real hardware and a simulator.^[1]

The technique is commonly used as one stage in a larger automated testing pipeline rather than as a complete oracle by itself. The SearchSYS artifact combines LLM-generated seed programs, AFL++ fuzzing, and differential testing to test the gem5 ARM ISA simulator.^[2]

READ FULL ARTICLE →

NEIGHBORHOOD

No graph connections found for this entity yet. It may appear in future ingestion runs.

explore full graph →

RELATIONSHIPS

17 connections

LLM-based verification workflow ← implements 95% 2e

The LLM-based verification workflow implements LLM-based test generation as its core technique.

TinyLlama ← implements 90% 2e

TinyLlama is used as an LLM for generating C test programs.

seed corpus uses → 100% 2e

LLM-based test generation produces the seed corpus for fuzzing.

LLM-based Processor Verification: A Case Study for Neuromorphic Processor ← uses 98% 2e

The paper employs LLM-based test generation as its primary verification technique.

Large Language Model uses → 98% 2e

LLM-based test generation relies on large language models as its core technology.

assembly test generation implements → 90% 2e

LLM-based test generation implements assembly test generation for neuromorphic ISA.

C program test generation implements → 90% 2e

LLM-based test generation implements C program test generation for RISC-V.

constrained random generation ← compares with 82% 2e

The paper positions LLM-based test generation as an alternative to constrained random generation.

Search+LLM-based Testing for ARM Simulators ← uses 100% 2e

The paper uses LLMs for initial seed C code generation.

SearchSYS ← implements 100% 1e

SearchSYS leverages LLMs for initial seed generation.

coverage-directed test generation ← compares with 80% 1e

The paper positions LLM-based test generation as a complement or alternative to CDG.

Magicoder ← implements 90% 1e

Magicoder is used as an LLM for generating C test programs.

Phi ← implements 90% 1e

Phi is used as an LLM for generating C test programs.

GPT-3.5 ← implements 90% 1e

GPT-3.5 is used as an LLM for generating C test programs.

C Program Generation uses → 100% 1e

LLM-based test generation produces C programs as test seeds.

GPT-3.5 uses → 97% 1e

The LLM-based test generation technique uses GPT-3.5 as the underlying model.

hardware-aware test generation uses → 88% 1e

LLM-based test generation incorporates hardware-aware strategies targeting specific processor units.

LINKED ENTITIES

8 links

Search+LLM-based Testing for ARM Simulators USES Extracted graph relationship

SearchSYS IMPLEMENTS Extracted graph relationship

TinyLlama IMPLEMENTS Extracted graph relationship

Magicoder IMPLEMENTS Extracted graph relationship

Phi IMPLEMENTS Extracted graph relationship

GPT-3.5 IMPLEMENTS Extracted graph relationship

C Program Generation USES Extracted graph relationship

seed corpus USES Extracted graph relationship

CITATIONS

8 sources

8 citations — click to expand

[1] SearchSYS uses LLM-generated C code as initial seed programs, compiles them, feeds binaries to a fuzzer, and uses differential testing on real hardware and a simulator. Artifact of Search+LLM-based Testing for ARM Simulators - Zenodo

[2] SearchSYS is applied to testing the gem5 VLSI digital circuit simulator with a focus on ARM ISA simulation. Artifact of Search+LLM-based Testing for ARM Simulators - Zenodo

[3] The artifact describes seven sets of LLM-generated C programs and setup steps for model-specific corpora including TinyLlama, Magicoder, Phi, and GPT-3.5 variants. Artifact of Search+LLM-based Testing for ARM Simulators - Zenodo

[4] SearchSYS reported 74 bug-activating test cases from LLM-only testing, 93 additional bug-activating cases after 24 hours of fuzzing, 624 bugs found with LLM-generated test cases, 126 with fuzzed inputs, and 4 unique acknowledged bugs. Artifact of Search+LLM-based Testing for ARM Simulators - Zenodo

[5] The SearchSYS ARM artifact requires an ARM machine with 80 GB free disk space and either 72 GB RAM for AFL++ fuzzing or 16 GB RAM for differential testing without building gem5 with AFL++ instrumentation. Artifact of Search+LLM-based Testing for ARM Simulators - Zenodo

[6] The SearchSYS artifact is organized into requirements, full ARM reproduction instructions, and a shortened evaluation intended to run within a day or less. Artifact of Search+LLM-based Testing for ARM Simulators - Zenodo

[7] A 2026 empirical study reports that LLM-generated tests achieve 79% line coverage and 76% branch coverage on original programs, but degrade under semantic-altering and semantic-preserving changes and show sensitivity to surface-level cues. Evaluating LLM-Based Test Generation Under Software Evolution

[8] Meta's ACH system is a mutation-guided LLM-based test generation system that generated issue-focused tests from mutants and was applied to 10,795 Android Kotlin classes across 7 platforms, producing 571 privacy-hardening test cases. Mutation-Guided LLM-based Test Generation at Meta