Differential Testing
TechniqueDifferential testing is used in SearchSYS-based ARM simulator testing by running generated or fuzzed programs on real ARM hardware and on the gem5 ARM simulator, then inspecting mismatches between native and simulated behavior. The Search+LLM-based Testing for ARM Simulators artifact provides scripts for single-input comparisons and folder-level differential-testing campaigns, with manual log analysis required because some simulation failures can be expected outcomes of corrupted binaries rather than simulator bugs.
First seen 5/28/2026
Last seen 9/20/2026
Evidence 79 chunks
Wiki v3
WIKI
Overview
Differential testing is a comparison-based testing technique used in the SearchSYS ARM-simulator workflow to find mismatches between a real hardware execution and a simulator execution. In the Search+LLM-based Testing for ARM Simulators artifact, the authors describe using differential testing by running fuzzing outputs on real hardware and on a system simulator to identify mismatches. The evaluated simulator target is gem5, with a focus on simulation of the ARM silicon chip Instruction Set Architecture (ISA). [C1]
Role in SearchSYS
NEIGHBORHOOD
No graph connections found for this entity yet. It may appear in future ingestion runs.
explore full graph →RELATIONSHIPS
47 connectionsDifuzzRTL uses differential fuzz testing to find CPU bugs.
ProcessorFuzz uses differential testing with ISA simulation as a reference.
SearchSYS implements differential testing by running outputs on real hardware and simulator.
Differential testing compares processor output against a golden-reference model to detect vulnerabilities.
Differential testing uses ISA simulation as a reference model to compare against RTL simulation.
The paper identifies SiliFuzz as a kind of differential testing that cross-validates CPU cores.
Mu2 uses differential testing as its oracle for mutation testing within the fuzzing loop.
Differential testing is used by Mu2 as the oracle for determining mutant killing in the fuzzing loop.
Prelude uses differential testing to detect errors by comparing DUT against ISA simulator.
Examiner uses differential testing to compare instruction execution between emulators and real devices.
DiffSpec is a framework that realizes differential testing using LLMs and prompt chaining.
Differential testing compares results from running programs on ARM hardware versus gem5 with ARM ISA.
HiFuzz uses differential testing against Spike as the reference model to detect bugs.
Differential testing uses CPU state comparison to identify inconsistent instructions between emulators and real devices.
Differential testing uses mismatch detection to identify bugs and vulnerabilities.
GenHuzz uses differential testing to detect bugs by comparing RTL and GRM execution traces.
ReFuzz employs differential testing by comparing PUT outputs against a golden-reference model.
The differential testing engine includes the APSR status register as part of the CPU state for comparison.
Differential testing compares RTL simulation output against ISA simulation output.
Hardware fuzzing uses differential testing to compare RTL and ISA simulator outputs.
StimulusRL uses differential testing via bug oracles to detect design defects.
DifuzzRTL exemplifies differential testing applied to RTL fuzzing.
LiFU performs differential checking against ISS golden traces.
SiliFuzz cross-validates different CPU cores, which is a form of differential testing.
The paper uses differential testing to identify mismatches between simulator and real hardware.
1-run-diff-test.sh implements the differential testing process for a folder of inputs.
Agile co-verification uses differential testing with Spike as a reference model.
Differential testing uses an ISS as a reference for comparison.
DiffTest implements the differential testing technique for chip functional verification.
Differential testing in processor fuzzing compares ISA simulation results against RTL simulation results.
SpecDoctor utilizes differential testing to detect sensitive data leakage.
The Python harness implements differential testing by comparing golden model and buggy variant outputs.
TurboFuzz uses differential testing by comparing DUT results with an ISA emulator.
Differential testing uses a bug oracle by comparing outputs of two implementations.
DITWO leveraged differential testing to uncover missed Wasm optimization opportunities.
WADIFF is described as the first differential testing framework for Wasm.
Mokav is an LLM-guided differential testing technique targeting Python program versions.
GenHuzz uses differential testing by comparing DUT outputs against the Golden Reference Model.
HARTBREAKER, like other fuzzers, relies on differential testing using an ISS or reference hart.
Cascade relies on differential testing for verification.
Differential testing addresses the oracle problem by using real hardware as a reference.
2-test-input.sh implements a single differential test comparison between native and simulation.
Differential testing compares DUT against an ISA simulator as the golden reference model.
DiffTest-H is a hardware-accelerated processor verification tool using differential testing.
Differential testing compares DUT outputs against ISS golden traces.
Spike is used as a reference model for differential testing in hardware/software co-verification.
The paper uses differential testing by comparing execution results across multiple ISSs.
CITATIONS
7 sources7 citations — click to expand
[1] Differential testing in the SearchSYS ARM workflow runs fuzzing outputs on real hardware and on a system simulator to identify mismatches, targeting gem5 ARM ISA simulation. Artifact of Search+LLM-based Testing for ARM Simulators - Zenodo
[2] SearchSYS generates LLM seed C code, compiles it, feeds the resulting binary to a fuzzer, and then uses differential testing between real hardware and a simulator. Artifact of Search+LLM-based Testing for ARM Simulators - Zenodo
[3] The artifact reports that SearchSYS combines LLMs, fuzzing, and differential testing, identifying 624 bugs with LLM-generated test cases, 126 with fuzzed test inputs, and 4 unique bugs acknowledged by developers. Artifact of Search+LLM-based Testing for ARM Simulators - Zenodo
[4] A single native-versus-simulation comparison is run with 2-test-input.sh after building or obtaining the ARM gem5 binary. Artifact of Search+LLM-based Testing for ARM Simulators - Zenodo
[5] Batch differential testing is performed with 1-run-diff-test.sh over input folders, with logs used for later manual analysis; the shortened evaluation runs the script on TinyLlama LLM and AFL input sets. Artifact of Search+LLM-based Testing for ARM Simulators - Zenodo
[6] Differential-testing failures require manual analysis because fuzzing may corrupt a binary, so a failure to simulate is not necessarily a simulator bug. Artifact of Search+LLM-based Testing for ARM Simulators - Zenodo
[7] The artifact requires an ARM machine with 80 GB disk space and specifies 72 GB RAM for AFL++ fuzzing or 16 GB RAM for differential testing without AFL++-instrumented gem5, with Ubuntu and Red Hat listed as tested systems. Artifact of Search+LLM-based Testing for ARM Simulators - Zenodo