Overview

Differential testing is a comparison-based testing technique used in the SearchSYS ARM-simulator workflow to find mismatches between a real hardware execution and a simulator execution. In the Search+LLM-based Testing for ARM Simulators artifact, the authors describe using differential testing by running fuzzing outputs on real hardware and on a system simulator to identify mismatches. The evaluated simulator target is gem5, with a focus on simulation of the ARM silicon chip Instruction Set Architecture (ISA). [C1]

Role in SearchSYS

In the documented SearchSYS workflow for ARM simulators, test inputs are produced from a combination of LLM-generated C code and fuzzing. SearchSYS generates initial seed C code with a large language model, compiles it, feeds the resulting binary to a fuzzer, and then applies differential testing to compare execution on real ARM hardware with execution in the simulator. [C2]

The artifact reports that SearchSYS combines LLMs, fuzzing, and differential testing to create test cases that activate bugs. In the reported evaluation, the authors state that differential testing identified 624 bugs with LLM-generated test cases and 126 with fuzzed test inputs; they also report that 4 unique bugs were reported and acknowledged by developers. [C3]

Execution pattern in the artifact

The artifact provides scripts for differential testing of input corpus seeds. A single gem5 simulation run can be launched with the ARM gem5 binary and a SearchGEM5 configuration script, passing an input file with --input. A single native-versus-simulation comparison is run with 2-test-input.sh on a selected test input. [C4]

For batch processing, the artifact uses 1-run-diff-test.sh on folders of inputs and redirects output to log files for later inspection. The documented examples include running this script on input folders for CodeBooga, Llama, Magicoder, Phi, TinyLlama, and GPT-3.5-derived corpora. The shortened evaluation runs differential testing on set A and provides commands for LLM-generated and AFL-generated TinyLlama inputs. [C5]

Result interpretation

The artifact explicitly notes that differential-testing logs require manual analysis. Not every failure to simulate a binary is considered a bug: fuzzing may corrupt a binary, and in that case the expected behavior may be that the simulation does not run successfully. The documentation suggests inspecting logs and error strings to triage issues found in ARM native versus simulation runs. [C6]

Environment assumptions

The ARM differential-testing setup is documented for ARM machines. The artifact states that reproducing the evaluation requires an ARM machine with 80 GB of free disk space, and either 72 GB RAM for AFL++ fuzzing or 16 GB RAM when only running differential testing without building gem5 with AFL++ instrumentation. The documented operating systems include Ubuntu and Red Hat-based systems. [C7]