Differential Testing
TechniqueDifferential testing is used in SearchSYS-based ARM simulator testing by running generated or fuzzed programs on real ARM hardware and on the gem5 ARM simulator, then inspecting mismatches between native and simulated behavior. The Search+LLM-based Testing for ARM Simulators artifact provides scripts for single-input comparisons and folder-level differential-testing campaigns, with manual log analysis required because some simulation failures can be expected outcomes of corrupted binaries rather than simulator bugs.
First seen 5/28/2026
Last seen 9/1/2026
Evidence 76 chunks
Wiki v3
WIKI
Overview
Differential testing is a comparison-based testing technique used in the SearchSYS ARM-simulator workflow to find mismatches between a real hardware execution and a simulator execution. In the Search+LLM-based Testing for ARM Simulators artifact, the authors describe using differential testing by running fuzzing outputs on real hardware and on a system simulator to identify mismatches. The evaluated simulator target is gem5, with a focus on simulation of the ARM silicon chip Instruction Set Architecture (ISA). [C1]
Role in SearchSYS
NEIGHBORHOOD
2 nodes · 1 edgesgraph · differential testing · depth=1
RELATIONSHIPS
44 connectionsDifuzzRTL uses differential fuzz testing to find CPU bugs.
SearchSYS implements differential testing by running outputs on real hardware and simulator.
ProcessorFuzz uses differential testing with ISA simulation as a reference.
Hardware fuzzing uses differential testing to compare RTL and ISA simulator outputs.
Examiner uses differential testing to compare instruction execution between emulators and real devices.
Differential testing uses CPU state comparison to identify inconsistent instructions between emulators and real devices.
The differential testing engine includes the APSR status register as part of the CPU state for comparison.
Differential testing uses the Golden Reference Model to compare against DUT execution results.
Differential testing uses ISA simulation as a reference model to compare against RTL simulation.
Differential testing compares RTL simulation output against ISA simulation output.
StimulusRL uses differential testing via bug oracles to detect design defects.
DifuzzRTL exemplifies differential testing applied to RTL fuzzing.
DiffSpec is a framework that realizes differential testing using LLMs and prompt chaining.
HiFuzz uses differential testing against Spike as the reference model to detect bugs.
SiliFuzz cross-validates different CPU cores, which is a form of differential testing.
The paper uses differential testing to identify mismatches between simulator and real hardware.
1-run-diff-test.sh implements the differential testing process for a folder of inputs.
The paper identifies SiliFuzz as a kind of differential testing that cross-validates CPU cores.
Mu2 uses differential testing as its oracle for mutation testing within the fuzzing loop.
Differential testing is used by Mu2 as the oracle for determining mutant killing in the fuzzing loop.
Prelude uses differential testing to detect errors by comparing DUT against ISA simulator.
Differential testing compares results from running programs on ARM hardware versus gem5 with ARM ISA.
LiFU performs differential checking against ISS golden traces.
Differential testing uses mismatch detection to identify bugs and vulnerabilities.
GenHuzz uses differential testing to detect bugs by comparing RTL and GRM execution traces.
Differential testing addresses the oracle problem by using real hardware as a reference.
Differential testing uses an ISS as a reference for comparison.
Differential testing compares DUT outputs against ISS golden traces.
The paper uses differential testing by comparing execution results across multiple ISSs.
Differential testing in processor fuzzing compares ISA simulation results against RTL simulation results.
SpecDoctor utilizes differential testing to detect sensitive data leakage.
Differential testing compares DUT against an ISA simulator as the golden reference model.
2-test-input.sh implements a single differential test comparison between native and simulation.
DiffTest implements the differential testing technique for chip functional verification.
The Python harness implements differential testing by comparing golden model and buggy variant outputs.
TurboFuzz uses differential testing by comparing DUT results with an ISA emulator.
DiffTest-H is a hardware-accelerated processor verification tool using differential testing.
Differential testing uses a bug oracle by comparing outputs of two implementations.
DITWO leveraged differential testing to uncover missed Wasm optimization opportunities.
WADIFF is described as the first differential testing framework for Wasm.
Mokav is an LLM-guided differential testing technique targeting Python program versions.
GenHuzz uses differential testing by comparing DUT outputs against the Golden Reference Model.
HARTBREAKER, like other fuzzers, relies on differential testing using an ISS or reference hart.
Cascade relies on differential testing for verification.
CITATIONS
7 sources7 citations — click to expand
[1] Differential testing in the SearchSYS ARM workflow runs fuzzing outputs on real hardware and on a system simulator to identify mismatches, targeting gem5 ARM ISA simulation. Artifact of Search+LLM-based Testing for ARM Simulators - Zenodo
[2] SearchSYS generates LLM seed C code, compiles it, feeds the resulting binary to a fuzzer, and then uses differential testing between real hardware and a simulator. Artifact of Search+LLM-based Testing for ARM Simulators - Zenodo
[3] The artifact reports that SearchSYS combines LLMs, fuzzing, and differential testing, identifying 624 bugs with LLM-generated test cases, 126 with fuzzed test inputs, and 4 unique bugs acknowledged by developers. Artifact of Search+LLM-based Testing for ARM Simulators - Zenodo
[4] A single native-versus-simulation comparison is run with 2-test-input.sh after building or obtaining the ARM gem5 binary. Artifact of Search+LLM-based Testing for ARM Simulators - Zenodo
[5] Batch differential testing is performed with 1-run-diff-test.sh over input folders, with logs used for later manual analysis; the shortened evaluation runs the script on TinyLlama LLM and AFL input sets. Artifact of Search+LLM-based Testing for ARM Simulators - Zenodo
[6] Differential-testing failures require manual analysis because fuzzing may corrupt a binary, so a failure to simulate is not necessarily a simulator bug. Artifact of Search+LLM-based Testing for ARM Simulators - Zenodo
[7] The artifact requires an ARM machine with 80 GB disk space and specifies 72 GB RAM for AFL++ fuzzing or 16 GB RAM for differential testing without AFL++-instrumented gem5, with Ubuntu and Red Hat listed as tested systems. Artifact of Search+LLM-based Testing for ARM Simulators - Zenodo