SOURCE ARCHIVE
EXTRACTED CONTENT
98,340 chars2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) | 979-8-3315-3685-5/25/$31.00 ©2025 IEEE | DOI: 10.1109/ICSE-SEIP66354.2025.00047
2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)
Search+LLM-based Testing for ARM Simulators
1st Karine Even-Mendoza 2nd Héctor D. Menéndez 3rd W.B Langdon
Department of Informatics Department of Informatics Department of Computer Science King’s College London King’s College London University College London London, United Kingdom London, United Kingdom London, United Kingdom karine.even_mendoza@kcl.ac.uk hector.menendez@kcl.ac.uk w.langdon@cs.ucl.ac.uk
4th Aidan Dakhama 5th Justyna Petke 6th Bobby R. Bruce
Department of Informatics Department of Computer Science Department of Computer Science King’s College London University College London University of California, Davis London, United Kingdom London, United Kingdom United States of America aidan.dakhama@kcl.ac.uk j.petke@ucl.ac.uk bbruce@ucdavis.edu
Abstract—In order to aid quality assurance of large complex systems is gem5 [1], [2], an open-source modular platform for
hardware architectures, system simulators have been developed. computer system architecture research that includes system-
However, such system simulators do not always accurately mirror level architecture and processor microarchitecture. The ac-
what would happen on a real device. A significant challenge in
testing these simulators arises from the complexity of having to ~~ curacy and reliability of software architecture simulators are
model both the simulation and the infinite number of software paramount for optimizing development investment and ensur-
that could be run on such a device. ing a smooth lifecycle. However, software complexity of such
Our previous work introduced SearchSYS, a testing frame- simulators makes them difficult to test comprehensively for all
work for software simulators. SearchSYS leverages a large ~~ potential architectural purposes. Identifying mismatches, inter- language model for initial seed C code generation, which is then nal simulation errors, and performance issues in simulators can compiled, and the resultant binary is fed to a fuzzer. We then use differential testing by running the outputs of fuzzing on real lead to their improvement and make them more robust, thereby hardware and a system simulator to identify mismatches. maximizing the return on investment. We present and discuss our solution to the problem of testing Our previous work [3]–[6] addressed this problem by com- software simulators, using SearchSYS to test the gem5 VLSI bining large language models (LLMs), fuzzing, and differen- digital circuit simulator, employed by ARM to test their systems. In particular, we focus on the simulation of the ARM silicon chip ~~ tial testing. By employing OpenAI’s Large Language Model Instruction Set Architecture (ISA). (LLM) GPT-3.5-turbo to generate a baseline test suite SearchSYS can create test cases that activate bugs by from seeded programs, and our modified version of AFL++, combining LLMs, fuzzing, and differential testing. Using only our system discovered various implementation bugs in gem5 LLM, SearchSYS identified 74 test cases that activated bugs. related to the Intel 64 bit X86 architecture. Whereas fuzzers By incorporating fuzzing, this number increased by 93 additional traditionally rely on implicit test oracles, such as crashes, and bug-activating cases within 24 hours. Through differential testing, we identified 624 bugs with LLM-generated test cases and 126 timing out loops, differential testing strengthens the fuzzer with fuzzed test inputs. Out of the total number of bug-activating by allowing it to automatically check for non-fatal errors, test cases, 4 unique bugs have been reported and acknowledged such as differences between running a simulation and running by developers. Additionally, we provided developers with a test on actual hardware. We then extended this work to create suite and fuzzing statistics, and open-sourced SearchSYS1. SearchSYS [5], which fully automates our initial proof-of- Index Terms—Software System Simulation, SearchSYS, gem5, AFL++, Fuzzing, Fuzz Testing, Differential Testing, concept and introduces novel mutation operators which are Search-Based Software Testing, LLM, Language Models, Ollama, able to mutate not only the seed program itself, but also its CodeLlama, TinyLlama, Phi2, Llama2, Magicoder, CodeBooga, command line parameters. Using SearchSYS we were able GPT-3.5-turbo, prompt engineering to further increase the bug finding ability of our approach. I. INTRODUCTION Again, we focused on the X64 architecture. Creating and developing new system architectures is a To showcase generalizability and widen impact and signifi- challenging task that requires a significant investment in both cance of our contribution, we now conduct an empirical study human and physical resources. Architectural simulators sup- on the ARM family of CPUs. ARM is a $160 billion [7] port this process by providing environments where developers company which designs chips and licenses them to others can validate their architectural goals. A good example of these to manufacture and integrate them into devices. Although its CPU stretches the full gamut of today’s computing, from 1See https://zenodo.org/records/13450472 for initial seeds and supercomputing HPC, cloud computing, servers, desktops, and SearchSYS code, and https://zenodo.org/records/14721385 for adjustment ~~ laptops, to Raspberry Pi, its processors are by far the most to ARMv8 and fuzzed seeds. successful CPUs largely because almost all mobile telephones
2832-7659/25/$31.00 ©2025 IEEE 469
DOI 10.1109/ICSE-SEIP66354.2025.00047
Authorized licensed use limited to: Univ of Calif Davis. Downloaded on April 21,2026 at 03:18:06 UTC from IEEE Xplore. Restrictions apply.
and handheld smart devices are based on ARM CPUs. Alto- different types of bugs in gem5’s simulation of the ARM
gether more than 200 billion ARM chips have been sold [8]. architecture.
Naturally, such a company has considerable in-house expertise, • Delivery of 3 661 LLM-generated tests and 30 000+
nevertheless ARM makes use of the open source gem5 project fuzzed tests along with a detailed analysis of bug-
keeping its own git clone but also paying close attention to activating test cases to the gem5 development team.
fixes and developments to the public version [9], [10]. • An empirical investigation of six large language models
Despite being crucial to the quality of chip designs, at the for input generation for SearchSYS.
cost of only a few days of computer time, by systematically We provide SearchSYS, bug reports, test cases and details
using SearchSYS we were able to find and highlight 14 of fuzzing performance and statistics via [20], to facilitate
different types of issues with gem5’s simulation of the ARM reproducibility of this study and wider use of SearchSYS.
processor chips. Furthermore, SearchSYS has automatically Section III describes SearchSYS and how it integrates
created test cases for gem5’s simulation of the ARM In- multiple C code generating LLMs with fuzz testing and
struction Set Architecture (ISA), which previously would have extends fuzz testing with simulator specific mutations and
taken skilled engineers weeks to do by hand. differential testing. Our research questions, methodology to
Finally, by simultaneously providing the gem5 development answer those, and experimental setup are given in Section IV.
team with details of C++ source line coverage for each test, we Whilst Section V gives our results including contrasting the
open the way for future work whereby regression testing can effectiveness of six LLMs at testing gem5’s simulations of
be automatically targeted at immediate development changes. ARM hardware. The discussion (Section VI) and related work
We expect that by executing tests that run the just modified (Section VII) are followed by our conclusions in Section VIII,
code, there will be more chance of finding and addressing but first, we start with the background (Section II).
issues. Moreover, selecting which tests to run should speed up
testing, possibly allowing real-time continuous integration test- II. BACKGROUND
ing [11]–[13]. Experience with Meta [14]–[16] highlighted the We provide here a quick introduction to fuzz testing and
importance of giving individual developers immediate feed- large language models, as these underpin our approach imple-
back. Hopefully, this can reduce reliance on the current mix mented in SearchSYS.
of daily and weekly regression testing, which is undirected and
already consumes several weeks of computer time per week. A. Fuzzing with AFL
Fuzz testing used in SearchSYS gives us automated test- Fuzzing is a technique used to identify bugs in programs by
ing. It consumes computer time, rather than a test engineer’s running the program with a variety of test inputs. Originally
time. Over the last few years fuzzing has been extensively used the test inputs were generated at random. With the introduction
by Google to find many thousands of security related problems of feedback-based fuzzing techniques, new methods have been
and other bugs [17]. AFL++ [18] is the state of the art fuzzing developed to manipulate the initial test inputs. Typically these
tool. Our SearchSYS extends AFL++ with the addition aim to change (mutate) the test inputs in order to exercise
of automatically generated test seeds and domain specific new parts of the software under test (SUT) during the fuzzing
mutation operators and, as we shall see, leads to cost effective process. One of the most popular fuzzers is the American
testing of the state of the art in VLSI simulators, i.e. gem5, Fuzzy Lop (AFL) [21], and related to the AFL family is
for the most widely used general purpose CPU on the planet. AFL++ [18], which we are using for SearchSYS.
The purpose of this study is to evaluate the reliability and AFL begins by automatically instrumenting the software
accuracy of gem5 as an ARM simulator and to extend gem5’s under test (SUT). This instrumentation provides feedback to
existing test suite. To achieve this, we apply SearchSYS the fuzzing process and is performed at compilation time. The
to identify bugs in gem5, generate test cases, and measure compiler introduces various flags in the programs that are
gem5’s capabilities on ARM machines. related to the code branches visited during execution. During
We identify 14 different types of bugs, including panic the fuzzing process, AFL starts by running a set of predefined
errors (which are of particular interest to gem5 develop- inputs, called seeds. These inputs are then mutated as part of
ers [19]), performance bugs, and differential bugs, where we the fuzzing strategy. AFL retains those mutated inputs that
compare the simulation outcomes with those from physical explore new sections of the program that previous inputs have
ARM machines. We have already reported 4 of these bugs. not visited, and continues to mutate them to discover new
The selection of which bugs to report depends on whether paths. The inputs are maintained in a queue and are selected
they belong to the previously mentioned categories. We do by the fuzzer based on the strategy and the effectiveness of
not report regular crashes, such as system call issues, non- their mutations. The main goal of fuzzing is to improve the test
panic segmentation faults, and timeouts that are less than 24 coverage of the SUT. The fuzzer runs until specific termination
hours, as the developers find them less interesting [19]. conditions are met, usually a time limit.
To summarize, our contributions are: In the context of this work, the seeds are created by
• An extensive empirical study of SearchSYS applied to large language models (LLMs). The LLMs generate not only
the ARM architecture, using the gem5 system simulator. programs that exercise the SUT (the simulator), but also the
Our results reveal that we were able to identify 14 program’s inputs and their types.
470
Authorized licensed use limited to: Univ of Calif Davis. Downloaded on April 21,2026 at 03:18:06 UTC from IEEE Xplore. Restrictions apply.
B. Large Language Models weekly. Some tests depend on the specific configuration of
Like the original SearchSYS work [3], [5], we use large the host system, and Docker containers are often required to
language models to provide a set of seeds for the fuzzers. ensure these tests run properly.
Large language models have become prominent in recent III. SE A R C HSYS
years, especially after the popular introduction of GPT-3.5. Figure 1 shows the structure of each part of SearchSYS.
GPT LLMs are based on transformers [22] with attention Our approach combines large language models with fuzz
mechanisms to identify relevant parts during the learning testing to identify bugs within system simulators and to
process. Currently, several large language models are available, generate test suites. The main idea is to address a key challenge
both private and public. Notable private LLMs include Gemini with fuzzers: traditionally fuzzers are given a set of input
by Google and Llama 3 by Meta. In terms of public LLMs, values for the software they are to test (the SUT). (These
various communities have also created their own, such as are known as seeds.) Here we are testing simulators, whose
Dolphin and Mistral, which are available through platforms inputs are programs to be simulated. That is, instead of
like Huggingface or the Ollama interface. starting with numbers, etc., as inputs, the initial seeds are
Here we use a variety of LLMs of different sizes and programs. Until recently, automatically creating programs of
natures. We use GPT-3.5 and Phi-2, a Microsoft LLM aimed at interest was hard3. However, now there are LLMs dedicated to
software generation. We employ Llama2 and TinyLlama, two program source code generation. Therefore, we use LLMs to
general-purpose LLMs of different sizes. We use Magicoder, create an initial set of test seeds, which are of interest for
combining auto-encoders and transformers for program source the fuzzing process and can improve test coverage. Given
code generation. Additionally, we use CodeBooga, which inte- the complexity of the simulator software, it is crucial to
grates various LLMs, specifically Phind-CodeLlama-34B-v2, have useful seeds that can enhance coverage, especially when
which outperforms GPT-4 in code generation tasks [23] and testing a comprehensive architecture like ARM. Further, as
WizardCoder-Python-34B-V1.0. All of these public LLMs a traditional fuzzer runs, it changes the SUT’s test inputs to
are run through the Ollama framework, which provides the try and cover new branches within the SUT. As our SUTs
necessary infrastructure for LLM execution. are simulators, SearchSYS extends the fuzzer so that it
C. gem5 System Simulator can mutate both the input executable binary program and its
gem52 is a state-of-the-art discrete time simulator for logic command line parameters along with their data type.
circuits. It is often used to try out the logic design of We consider three types of bugs: 1) crashes 2) issues with
new electronic components such as memory cache systems, efficiency or performance, such as program hangs 3) mis-
FPGAs, and even CPUs. gem5 is a large open-source project matches between the simulator and the actual hardware. For
hosted on GitHub, written mostly in C++ and Python. It is used this last type of bug, it is essential to determine whether the
by companies such as ARM and Google to simulate hardware. simulator behaves as the real system. This is where differential
Including objects, shared library and images, gem5 occupies testing becomes crucial. SearchSYS provides the differential
over 28 GB of memory. It is composed of ∼ 1.34 million lines testing infrastructure to compare test cases within the simulator
of code, comprising more than a million lines of C++. and outside of it.
The gem5 simulator has a comprehensive testing framework Our testing approach is divided into three main components:
comprising C++ unit tests, Python unit tests, and TestLib 1) Test Input Generation System: This system uses Large
integration tests. Each test set focuses on specific aspects of Language Models (LLMs) to create a set of test cases
the system, ranging from low-level code validation to full-scale which serve as our testing baseline. Unlike traditional
simulation testing. fuzzing processes that require existing programs as seeds,
The unit tests in gem5 are designed to validate the func- we ask different large language models to generate
tionality of the core C++ components and are automatically the test cases (programs to be simulated) automatically.
executed as part of gem5’s continuous integration (CI) pro- (Note: in the context of the ARM Instruction Set Ar-
cess. These test much of the core C++ code and maintain chitecture it can take weeks for a skilled engineer to sit
code integrity and correctness after Git commits. In addition down with the ARM ISA hardware documentation and
to C++ unit tests, gem5 includes Python unit tests, which code programs to test each part of the ISA.)
naturally focus on verifying Python-based components. The 2) Coverage-Guided Fuzz Testing Tool (AFL++): AFL++
Python tests are quick to run and fewer in number than the focuses on identifying new uncovered code regions in the
C++ unit tests. simulation system and achieving higher coverage through
Testlib integration testing is more intensive in nature. Al- mutation-based testing. AFL++ employs a driver for the
most everything in Testlib runs a gem5 simulation. The tests simulation system, enabling the simulation/execution of
are categorized into three sets: “quick”, “long” and “very- any binary program with specific parameters. The type
long”. The “quick” tests are run during CI, the “long” tests of the parameters is specified by the LLM. The LLM
are executed nightly, and the “very-long” tests are conducted 3 E.g. (in the context of testing) automatically generating programs that
achieve high coverage, have specific patterns or formats, or contain edge cases
2https://www.gem5.org or a specific set of instructions.
471
Authorized licensed use limited to: Univ of Calif Davis. Downloaded on April 21,2026 at 03:18:06 UTC from IEEE Xplore. Restrictions apply.
SearchSYS where n can be arbitrarily large. In total, n + 1 prompts
generate n test programs. One prompt for setting the LLM’s
Large Language Model role plus n prompts to generate n test programs.
a) Setting the LLM role: Before generating the n test
programs, a single one-off LLM prompt initializes the process.
Generates Test Cases This one-off LLM prompt serves to set the LLM’s role. This
LLM prompt specifies the programs to be generated (e.g.
Parametrized Inputs Generation “generate C programs with arguments as input”). Since each
LLM has memory 4, we only need to set LLM’s role once.
Provides Initial Test Seeds b) Generating a test program: The LLM prompt starts
with a problem description that specifies the program tasks and
the context in which these programs are to be generated. The
Fuzzing with Customized AFL++ Fuzzing problem description contains specific gaps that will be filled
with the tokens. A total of 4089 tokens are available as part of
Mutates and Executes Tests SearchSYS. Table I describes the three token types across 4
categories of tokens with a few examples. The first and second
Differential Testing Module types of tokens are derived from compiler optimization names
and compiler parts. The third token is chosen at random from
Executes Test Cases Executes Test Cases general tokens related to programming languages, tutorials,
Collects Outputs Collects Outputs or standards. The fourth tokens category in Table I includes
phrases taken from the C17 standard [25], such as "initialize"
Simulator Real ARM Hardware and "pointer". For instance, we might ask for examples of dead
code elimination, handling of the Abstract Syntax Tree (AST),
Differential Testing and a beginner tutorial name in C programming examples,
corresponding to the first, second, and third tokens.
Fig. 1. SearchSYS diagram starts with the generation phase, where the The template prompt is shown in Listing 1.
LLMs generate C programs which take parameters via the command line.
It compiles them. Then it applies the fuzzing phase, using the customized " Coding t a s k : g i v e me a pr og ra m i n C w i t h a l l
version of AFL, which is able to mutate both the executable binary and its i n c l u d e s . I n p u t i s t a k e n v i a a r g v o n l y .
parameters. Finally, in the differential testing phase, it compares each mutated P l e a s e r e t u r n a p ro gr am (C p ro gr am ) and a c o n c r e t e
executable binary in both the simulator and the real ARM architecture. ex a mp le o f an i n p u t (BASH ) . The C p ro gr am w i l l be
w i t h c o d e t r i g g e r i n g " + <Token −1> + " o p t i m i s a t i o n s ,
c o v e r s t h i s p a r t o f t h e c o m p i l e r " + <Token −2> + " ,
not only provides the program to be simulated but also and e x e r c i s e s t h i s i d e a i n C : " + <Token −3> + " .
specific inputs for it, which our special version of AFL++ To r e c a p t h e c o d e c o n t a i n s t h e s e : " + <Token −1> +
" and " + <Token −2> + " and " + <Token −3 >;
search will modify along with the program itself. Listing 1. Template prompt for generating C test programs with random
3) Differential Testing Module: This module compares the tokens
outputs of the simulator with those of the real ARM The value of the <Token-n> comes from the three
architecture. We run the specific binary files and their different components and is chosen at random from within
inputs on a real ARM machine and check if the outputs each corpus. This process fully automates the fuzzer’s input
from it and the simulator match. Any discrepancies are generation and improves the diversity of the test inputs.
identified as mismatches. Additionally, it is independent of the specific LLM being used.
A. LLM-based test code generation B. Fuzzing
The first step of SearchSYS focuses on creating a corpus SearchSYS’s fuzzing process mutates both the binary
of test cases with parameterized inputs. To achieve this, we program and its inputs. When employing bit-flip mutation,
query a large language model (LLM). The LLM generates the as standard in AFL++, context consideration is crucial.
test cases as C source code and provides the types for the input Mutations causing binaries to fail to load or execute even a
parameters. The query prompt given to the LLM specifies single instruction lead to inefficient SUT testing, reducing
the tasks the program should achieve and also requests it to the likelihood that developers will invest time in identifying
provide the corresponding parameters and their types. or fixing bugs. For instance, binaries failing with errors
We employ a zero-shot prompting technique [24]. That is, like "error while loading shared libraries:
the LLM is not given any prior examples or context. Hence unsupported version 0 of Verneed record"
we have to provide all the information the LLM needs about due to bit-flips are unlikely to receive fixes from developers
the specifics of the problem in its query prompt (see shaded since the correct behaviour for faulty binaries is to crash. To
example in Listing 1). We construct a prompt to generate a
new test program, repeated n times to produce n programs, 4i.e. the LLM’s consistency within a session.
472
Authorized licensed use limited to: Univ of Calif Davis. Downloaded on April 21,2026 at 03:18:06 UTC from IEEE Xplore. Restrictions apply.
TABLE I
THE FOUR C TOKEN CATEGORIES WHICH ARE USED TO FILL GAPS IN THE TEMPLATE USED TO PROMPT THE LLMS AND A FEW EXAMPLES OF EACH
ID Type Category #Tokens Examples 1 Token-1 Compiler Optimizations 26 "Scalar Optimizations", "Dead Code Elimination", "Constant Folding" 2 Token-2 Compiler Parts 36 "Sema", "Serialization", "Parse", "Lex", "AST" 3 Token-3 Domain Problems 192 "C Program to Sort an Array using Merge Sort", "Calendar Year in Different Formats", "input includes several arguments" 4 Token-3 Compiler Standard Indices [25] 3480 "AND operators", "cimagl function", "EOF", "locale", "pow", "SCNiMAX"
address this, SearchSYS employs a custom bit-flip mutation This approach allows for more complex mutations, such as
operator that controls the number and frequency of bit-flips, binary file mutations.
applying them only to a program’s compiled binary file. This Another factor affecting the throughput of fuzzing is the
avoids applying bit-flips to arguments or type information, number of mutation operations that AFL++ performs in a
preserving the structure of the test input. single iteration. The parameter afl_custom_fuzz_count
SearchSYS applies three mutation operators for testing controls the number of times a test input should be mutated
system simulators: and executed against the target. A lower value reduces the
1) A bit-flip operator for modifying a program’s compiled risk of iteration failure but can lead to inefficient fuzzed input
binary file. generation due to iteration overhead. Following our results [5],
2) A range-enhanced operator for editing argument values we set afl_custom_fuzz_count to be 17, 84 and 66
within their specified type range. for Operator 1, 2 and 3 from Section III-B, respectively.
3) An operator for changing the value’s type.
Operator (2) uses type information to ensure arguments C. Differential Testing
remain valid, while Operator (3) randomly changes the type, Fuzzed test inputs can uncover crashes, hangs and mis-
such as from INT32 to LONG, potentially exposing memory matches between the architecture and the system simulator.
safety issues in the SUT. We support all integer types, float, However, AFL++ only identifies a bug if the test input leads to
double, and strings but have not yet implemented pointer a crash or a hang, degrading SearchSYS’s ability to detect
support. Figure 2 and Figure 3 provide examples of value missimulation issues. To address this, SearchSYS follows
(operator 2) and type mutations (operator 3), respectively. the approach of Even-Mendoza et al. [26] and separates
fuzzing and differential testing. That is, after fuzzing we
1 ./mutator_args.so test.o, 5:INT, 20:LONG , "Hi": perform differential testing by comparing runs using fuzzed
STRING test inputs (i.e. mutated executable binaries and their inputs)
2 After Mutation: test.o, 10:INT, 20:LONG, "House": on ARM hardware (native) with those on the simulator with
STRING ARM ISA.
Fig. 2. Example of mutation operator (2) changing argument values: first D. Implementation
argument: 5 to 10 and third: "Hi" to "House" (second argument unchanged).
SearchSYS is implemented using a mixture of languages
and Unix shell scripts. Specifically, we use (1) Java with
1 ./mutator_args.so test.o, 5:INT, 20:LONG , "Hi": Ollama for LLM-based test input generation, (2) C/C++ for
2 AfterSTRING the custom mutator and fuzzing-related code, and (3) a set of
Mutation: test.o, 5:LONG, "20":STRING, "Hi":
STRING scripts for configuring the testing environment [3], [5].
When adapting SearchSYS for ARM, we encountered
Fig. 3. Example of mutation operator (3) changing argument types: first minor script issues related to the linker and data from
argument from INT to LONG and second argument from LONG to STRING. TinyLlama, the latter caused an early termination in AFL++
fuzzing due to memory errors. Additionally, we had to pass
SearchSYS loads all three mutators (1-3) using the exist- ARM as the ISA parameter, though no modifications were
ing AFL++ option, allowing AFL++’s heuristics to select the needed for the Python configuration script. We continued to
next mutation operator. However, we decrease the probability use the example script provided by the SSBSE Challenge
of choosing (2) by setting it to only 99.5% of the times AFL++ Track 2023 organizers5. These adjustments required minor
selected it originally and replacing the remaining 0.5% with modifications to our scripts (setting the testing environment,
mutator (1), as AFL++ favours this operator due to its low fuzzing and differential testing parts).
failure risk, which is too conservative for fuzzing. The code in parts (1) + (2), required no further modifica-
SearchSYS extends AFL++ by evaluating new test in- tions. However, in between this version and [5], we made some
puts in the form of binary name, arguments list, bug-fixing edits in the C/C++ code.
types. Then it applies mutation operators directly to the
compiled binaries, their arguments, or their arguments’ types. 5hello-custom-binary.py
473
Authorized licensed use limited to: Univ of Calif Davis. Downloaded on April 21,2026 at 03:18:06 UTC from IEEE Xplore. Restrictions apply.
IV. EVALUATION architectures. To achieve this, we compared the bug-finding
To assess the quality of gem5 as a simulator for ARM effectiveness and fuzzing throughput of SearchSYS between
architectures, we aim to answer the following questions: the ARM ISA and X86 ISA, as previously explored in [5].
We followed the experimental procedure outlined in [5], but
RQ1: Considering that large language models (LLMs) gener- adapted it for the ARM ISA and ARMv8 hardware, running
ate test suites independently, how effective are these test suites without Docker. Furthermore, the experiments were repeated
at identifying bugs in gem5’s ARM simulation processes? 5 times (instead of 10), using the minimized input corpora, as
To answer this question, we will ask LLMs to generate these have been shown to be more stable during fuzzing and
test suites, which will later be used as seeds for the fuzzing in particular for SearchSYS [5], [29], [30].
process. Each test program (of a test input in the suite) will The minimized input corpora TinyLlama, Phi2,
be generated using a different prompt: SearchSYS generates Llama2, Magicoder, CodeBooga and GPT-3.5-turbo
random tokens (Section III-A), embeds the random tokens into were taken from [5], [6], while GPT-3.5-turbo (SSBSE
the template prompt in Listing 1 to form a complete prompt, 2023) from [3], [4].
and prompts the LLM with the constructed prompt to generate Each fuzzing campaign ran for 24 hours, with five indepen-
a new test program. These steps are part of SearchSYS’s test dent repetitions of the fuzzing process per minimized input
input generation system (Section III). corpus. The throughput results of the fuzzing experiments were
This research question will help determine which specific calculated as the mean value across these five repetitions,
language models are more effective at generating test suites ensuring consistency and robustness in the findings. The
that can uncover bugs in the simulator, under the current differential testing post-fuzzing (i.e. when gem5 differed from
prompting mechanism (Section III-A). In this evaluation, we the real hardware) was done using the last repetition, for all 7
will identify the best language model for finding bugs in corpora. In case of a mismatch, to determine if the mismatch
the ARM simulation process and uncover specific bugs. Our is a genuine bug, we compared the results from the gem5
process will also ensure a minimum set of test cases after simulation with ARM ISA to those obtained via the two ARM
reducing the test suite using afl-cmin6. hardware (Section IV-A).
RQ2: How effective is SearchSYS at identifying bugs in V. RESULTS
the simulation of the ARM architecture performed with gem5 In this section, we present the outcomes of our experiments,
after the fuzzing process? specifically, bugs found from the initial corpus of LLM-
To answer this research question, we will run the fuzzing generated test inputs, and those found as a result of fuzzing
strategy using the different test suites generated by each LLM campaigns. To detect bugs, we cross-validated our results
as seeds. The fuzzer will run against the simulator, with the between an X86 and two ARMv8 machines. One of these
generated inputs focusing not only on identifying crashes ARMv8 machines is being configured to be more strict, often
and hangs within the simulator but also on mismatches in initializing uninitialized local variables to zero.
differential testing by comparing the output of the generated In total, we reported 4 bugs from the LLM-generated test
tests with that of a real ARM machine. inputs [31]–[34]. We further identified a bug in the GNU
A. Hardware Multiple Precision Arithmetic Library (GMP 6.1.0) during
cross-system testing [35].
We ran tests on gem5 using two machines: (1) a single
CloudLab7 [27] m400 machine with 64 GB RAM, ARMv8 A. Bugs Detected from LLM-generated Test Inputs
64-bit architecture with a single socket, 2.4 GHz, 8 cores, To evaluate LLMs as sources of inputs for regression testing
and 1 thread per core, running Ubuntu 22.04 ARM, and of system simulators, we used 7 sets of LLM-generated test
(2) a single machine (UCL) with 224 CPU cores (Cavium inputs, created during 25-hour runs with different LLMs with
ThunderX2 CN9975, 2.0 GHz) and 130 GB RAM, ARMv8 qualitative and quantitative evaluation and analysis of LLM-
64-bit architecture with 2 sockets, 28 cores per socket, and generated test inputs detailed in [3], [5]. Here, we focus on
4 threads per core, running Red Hat Linux (aarch64-redhat- ARMv8, noting that LLM test program generation is agnostic
linux-gnu). We installed the same compiler versions, adapted to X86 or ARM CPU. We classified the bugs found for each of
for ARM, and set up the same tools as described in [5], on the 7 sets of simulations with ARM ISA against real ARMv8
both machines. The exact specification can be found in [28]. machines and compared the bug finding rate with our previous
B. Experiments results on the gem5 X86 backend [5].
For the experiments, we selected ARMv8 hardware and the Table II shows the bugs identified in our investigation
ARM ISA, given its relevance from the industry’s perspec- from the LLM-generated test inputs. Table II includes the
tive [19]. The primary objective was to provide a comprehen- bug description (“Bug”) with a bug number if already
sive evaluation of SearchSYS’s performance across different reported to the gem5 bug tracking system. Columns A
to G represent the number of test inputs triggering this
6afl-cmin and afl-cmin’s manpage bug category: A: TinyLlama, B: Phi2, C: Llama2,
7See https://docs.cloudlab.us/hardware.html D: Magicoder, E: CodeBooga, F: GPT-3.5-turbo and
474
Authorized licensed use limited to: Univ of Calif Davis. Downloaded on April 21,2026 at 03:18:06 UTC from IEEE Xplore. Restrictions apply.
TABLE II
BUGS FOUND IN G E M5 WITH LLM-GENERATED TEST INPUTS. COLUMNS A TO G HOLD THE NUMBER OF INSTANCES OF EACH BUG FOUND BY EACH
LLM: A: TI N YLL A M A, B: PH I2, C: LL A M A2, D: MA G I C O D E R, E: CO D EBO O G A, F: GPT-3.5-T U R B O AND G: GPT-3.5-T U R B O (SSBSE 2023).
BUG A B C D E F G
Bug #1527 (panic) 1 1 5 1 1 0 0
Bug #1544 (Missing support) 0 0 2 3 6 5 0
Bug #1547 (Missing support) 0 0 0 1 7 0 0
src/sim/syscall_emul.cc:67: fatal: syscall dup3 (#24) unimplemented 0 0 0 3 0 14 0
src/sim/syscall_emul.cc:67: fatal: syscall pipe2 (#59) unimplemented 0 0 0 1 1 0 0
src/sim/syscall_emul.cc:67: fatal: syscall clock_getres (#114) unimplemented 0 0 0 0 0 1 0
src/sim/syscall_emul.cc:67: fatal: syscall clock_nanosleep (#115) unimplemented 0 1 8 7 4 4 0
src/sim/syscall_emul.cc:67: fatal: syscall wait4 (#260) unimplemented 0 0 2 0 0 0 0
instruction ‘bti’ unimplemented 20 17 81 127 51 202 4
Bug #1629 (some time functionality unimplemented) 0 0 0 2 1 28 0
some thread functionality unimplemented in SE mode 0 0 0 1 0 10 0
Variable’s value is random in ARM but fixed in simulation 0 0 0 2 1 7 0
Timeout 1 2 4 27 15 15 1
Totals (of number of test inputs exposing a bug) 22 21 102 175 87 286 5
G: GPT-3.5-turbo (SSBSE 2023). The bugs found include expecting complex input or having large output, these can be panic errors, usually triggered by assertion violation, timeouts, grouped according to coverage and instructions triggered to and different outputs including fatal errors triggered by wrong target untested areas of the current regression tests of the gem5 or missing instruction implementation for ARM ISA, revealing test suite. Furthermore, test inputs exposing unimplemented gaps in the system’s support for ARMv8 functionalities. We features can be saved for future use. For example, tests ran the differential testing scripts comparing the result of the triggering unimplemented system calls can be saved for future native run on ARMv8 machines against the simulation with a use (e.g. to implement test-case-driven development methods). time out of 50 seconds and a memory limit of approximately 90 MB (stack size) for both the gem5 simulation and the native run. In most cases, unimplemented syscalls represent a gap in In total, GPT-3.5-turbo found the highest number gem5 functionality which the gem5 developers do not intend of issues (286), followed by Magicoder (175), Llama2 to plug, but the bug report itself is helpful since it can alert (102), CodeBooga (87), TinyLlama (22), and Phi2 (21). gem5 users to a now-known issue, saving them time trying GPT-3.5-turbo (SSBSE 2023) had the fewest, with only to resolve what was previously an issue known only to the 5 instances. Two of these bugs, panic and timeout, were developers. identified by executing the simulator, totalling 74 test cases (9 panic and 65 timeouts), while an additional 624 bug- In gem5 using SearchSYS, we found around 530 test activating test cases were identified using differential testing. inputs exposing missimulations and errors and 30 test inputs The differing results for GPT-3.5-turbo (SSBSE 2023) exposing optimization issues (timeouts) on X86 ISA [5] com- compared to the GPT-3.5-turbo set, even though both pared to around 630 and over 60 instances on ARM. Note, are using the same language model, can be attributed to that the ’530’ and ’30’ instances on X86 are not a subset of three factors: (1) GPT-3.5-turbo (SSBSE 2023) dataset the 630 and 60 instances on ARM and reflect different bugs. was generated in 2023, while GPT-3.5-turbo dataset was This suggests that contrary to expectations [19], ARM is no generated in 2024, (2) GPT-3.5-turbo (SSBSE 2023) more stable than X86 ISA in gem5, as given the same set of was trained on the LLVM test suite, likely overlapping with test inputs, more of them exposed an issue. tests already evaluated by gem5, and (3) GPT-3.5-turbo (SSBSE 2023) used a few-shots approach instead of zero- shots, with zero-shots generally providing better input diversity We reported four new bugs from the LLM-generated test and throughput [5]. We observed these differences for the bug inputs to the gem5 bug tracker [31]–[34] under “arch-arm” count results during the pre-fuzzing test input generation stage tag. These bugs were not previously identified during our X86 (Table II), but as well in the results from the fuzzing and post- fuzzing campaigns [3], [5], except for bug #1629 [34], which fuzzing stages (to be discussed in Section V-B). we expected to be properly implemented in ARM. While we These LLM test cases are a suitable contribution to gem5’s anticipated that ARM would be generally more stable than C++ unit tests (as regression tests). They have several lines of X86, the occurrence of this bug in ARM was unexpected, code, are efficient (as they commonly terminate in under 50 prompting us to report it immediately. We excluded unimple- seconds) and are human-readable. With some semi-manual fil- mented issues from bug reports, as these represent new test tering, to remove tests triggering undefined behaviour or those cases rather than valid bugs.
475
Authorized licensed use limited to: Univ of Calif Davis. Downloaded on April 21,2026 at 03:18:06 UTC from IEEE Xplore. Restrictions apply.
ning them on the real hardware. Table IV includes the
RQ1 Answer. All of the LLMs found bugs in gem5’s bug description (“Bug”) with a bug number if already
simulation of ARM CPUs. We found 13 bugs, six reported to the gem5 bug tracking system. Columns A
are unimplemented system calls functionality in the to G represent the number of test inputs triggering this
simulator. We have already reported four (#1527 #1544 bug category: A: TinyLlama, B: Phi2, C: Llama2,
#1547 #1629). Of these three have already received D: Magicoder, E: CodeBooga, F: GPT-3.5-turbo and
detailed consideration by the gem5 development team. G: GPT-3.5-turbo (SSBSE 2023). Note “Crash” and
“Hangs” from Table III: (1) Table IV does not include timed-
out fuzzed test inputs counters since these are already stated
B. Fuzzing as Part of the Testing Process in Table III in the “Hangs” column; and (2) not all instances
SearchSYS is an AFL++-based tool. After generating test counted in the “Crash” column of Table III represent gen-
inputs using LLMs and compiling them into a test case (binary, uine crashes; some are due to corrupted binaries or non-
input, and input type info), we fuzzed an instrumented version reproducible crashes. Since AFL++ categorizes all these as a
of gem5 for 24 hours to have sufficient time to explore the single “Crash” type, we included these in Table IV and refined
codebase of a specific gem5 version [36]. it for a more detailed breakdown.
Table III shows throughput during 24 hours of fuzzing, Table IV identifies hangs and 7 distinct issues discovered
detailing the number of new test inputs generated for each during a manual inspection of automatically flagged
of the 7 minimized input corpora (from each of the LLMs mismatches, warnings, and crashes. During 24-hour fuzzing,
used). The columns “Initial Corpus” indicates the size of each TinyLlama had the highest number of fuzzed test inputs
initial corpus at the start of the fuzzing, “Fuzzed Corpus (Std exposing issues (87), followed by GPT-3.5-turbo (31),
Dev)”, “Queue (Std Dev)”, “Crash (Std Dev)” and “Hangs (Std Magicoder (10), Phi2 and Llama2 (6 each), and
Dev)” are means over five trials per corpus with their observed CodeBooga (3). Most fuzzed sets identified 2-3 distinct
standard deviations. The “Fuzzed Corpus” column shows the issues, with GPT-3.5-turbo finding 4 distinct ones. As in
total number of test inputs generated (i.e. queue + crashes Table II, GPT-3.5-turbo (SSBSE 2023) had the lowest
+ hangs). At the end of each 24-hour fuzzing campaign, we bug-finding rate with only 2 instances, each of a different
recorded: (1) “Queue” (test inputs generated by AFL++ that issue category.
did not crash or hang and were therefore suitable for further Some previously known bugs were encountered during
mutation), (2) “Crash” (number of crashed test inputs), and fuzzing, generating further examples of the issue, which can
(3) “Hangs” (number of hanged test inputs). be useful for bug localization and debugging. We found one
The minimized input corpora vary in size, with approxi- “Out of Memory” instance. This indicates that fuzzing can be
mately 100 test inputs generally being recommended [29], beneficial, though it suggests that longer fuzzing runs may be
[30]. During fuzzing, the highest throughput (in total) necessary for uncovering additional bugs and a better under-
of fuzzed test inputs was achieved (on average) by standing of the codebase code coverage. During fuzzing, we
GPT-3.5-turbo (SSBSE 2023) with 997 fuzzed test also identified a fatal error in src/mem/port_proxy.hh
inputs, followed by Magicoder (986), Llama2 (948), readBlob, which at first, looked like a genuine error. How-
GPT-3.5-turbo (888) and CodeBooga (839). The smaller ever, comparing the results between the two ARMv8 ma-
LLMs had a lower throughput during fuzzing: TinyLlama chines, with the simulation failing only on the UCL machine
(776) and Phi2 (680). This trend was slightly different when (Section IV-A) led us to conclude that it is likely to be a
observing the queue size only (the fuzzed test inputs for configuration issue rather than an ARM ISA bug, and thus we
differential testing post fuzzing), with Magicoder achiev- excluded these from the tables.
ing the highest rate with 962 fuzzed test inputs, followed To recap, during the fuzzing campaign, three of these
by GPT-3.5-turbo (SSBSE 2023) with 945 fuzzed test bugs, panic, out-of-memory and timeout, were identified by
inputs. This is different from the pattern we already re- executing the simulator, totalling 93 test cases (19 panic, 1
ported in [5] for X86, where GPT-3.5-turbo (SSBSE out-of-memory and 73 timeouts), while an additional 126 bug-
2023) and TinyLlama achieved the best throughput while activating test cases were identified using differential testing.
GPT-3.5-turbo had extremely poor fuzzing throughput. Fuzzing is a time-intensive process, and as such is unsuit-
Lastly, CodeBooga showed a large standard deviation across able for regular runs of regression tests, where the developer
all measured outputs (Queue, Crash, and Hangs), while requires feedback quickly. However, due to its ability to
Llama2, Phi2, and TinyLlama had high deviations in produce diverse inputs covering unexpected branches, it could
some outputs, though smaller than CodeBooga’s. The re- be valuable for use in less common, but large test suites, such
maining LLMs exhibited generally smaller standard devia- as those run between new releases, where there is a larger
tions. Despite these variations, the minimized input corpora time budget. Fuzzing can then help identify obscure bugs
typically generated between 700 and 1,000 test inputs during which may not have been caught during the other stages of
24 hours of fuzzing. development. In the context of gem5, fuzzing of weekly or
Table IV: shows the bugs identified by comparing when release versions can be integrated into the TestLib process of
gem5 simulated the mutated binary programs with run- gem5, which can include using Docker containers to provide
476
Authorized licensed use limited to: Univ of Calif Davis. Downloaded on April 21,2026 at 03:18:06 UTC from IEEE Xplore. Restrictions apply.
TABLE III
NUMBER OF G E M5 TEST INPUTS GENERATED BY EACH INITIAL CORPUS (BY LLM, MEAN AND STANDARD DEVIATION OF 5 RUNS). SEE SECTION V-B
Initial Corpus Fuzzed Corpus (Std Dev) Queue (Std Dev) Crash (Std Dev) Hangs (Std Dev)
TinyLlama 206 776 (±59) 737 (±59) 29 (±4) 10 (±1)
Phi2 366 680 (±87) 651 (±82) 14 (±6) 15 (±3)
Llama2 613 948 (±62) 916 (±62) 16 (±6) 16 (±3)
Magicoder 719 986 (±41) 962 (±42) 13 (±2) 11 (±3)
CodeBooga 612 839 (±112) 816 (±106) 11 (±7) 12 (±6)
GPT-3.5-turbo 703 888 (±26) 871 (±24) 11 (±3) 6 (±3)
GPT-3.5-turbo 442 997 (±34) 945 (±34) 49 (±5) 3 (±3)
(SSBSE 2023)
TABLE IV
DIFFERENTIAL TESTING RESULTS. BUGS FOUND IN G E M5 DURING 24-HOUR FUZZING OF THE LAST REPETITION. COLUMNS A TO G HOLD THE NUMBER
OF INSTANCES OF EACH BUG FOUND BY INITIAL MINIMIZED INPUT CORPUS: A: TI N YLL A M A, B: PH I2, C: LL A M A2, D: MA G I C O D E R, E: CO D EBO O G A,
F: GPT-3.5-T U R B O AND G: GPT-3.5-T U R B O (SSBSE 2023). (SEE SECTION V-A FOR EXPLANATION OF DIFFERENCES BETWEEN F AND G.)
BUG A B C D E F G
src/sim/syscall_emul.cc:67: fatal: syscall dup3 (#24) unimplemented 0 0 0 0 0 4 0
src/sim/syscall_emul.cc:67: fatal: syscall pipe2 (#59) unimplemented 0 0 0 0 1 0 0
src/sim/syscall_emul.cc:67: fatal: syscall clock_nanosleep (#115) unimplemented 0 0 0 4 0 7 0
instruction ‘bti’ unimplemented 21 3 3 5 1 19 1
Out of Memory 0 0 0 0 1 0 0
Likely to be Bug #1544 (Missing support) 56 0 0 0 0 0 0
Bug #1527 (panic) 10 3 3 1 0 1 1
Totals (of number of test inputs exposing a bug) 87 6 6 10 3 31 2
a consistent environment for fuzzing as in [37], ensuring based on their occurrences in simulations. Notably, we discov-
reproducibility and isolating system dependencies. Further- ered a range of panic errors, timeouts, and fatal errors, predom-
more, SearchSYS, a coverage-directed fuzzer, can assist in inantly triggered by assertion violations or improper/missing
exploring newly added code by using partial instrumentation or instruction implementations for ARM ISA. This highlights
tailoring the mutators, enabling deeper testing of the codebase significant gaps in gem5’s support for ARMv8 functionalities.
of gem5. Our comparative analysis revealed that ARM is not neces-
Our fuzzing campaigns have uncovered further bugs and sarily more stable than the X86 ISA within gem5, contradict-
produced a larger corpus of test cases than LLM-generated ing previous expectations [19]. This higher incidence of issues
tests alone. We found crashes and hangs using all LLMs tested. on ARM suggests that there are still considerable challenges
Even with small LLMs such as TinyLlama, fuzzing was able to achieving parity between the support for ARM and X86
to grow the corpus size from 206 input tests to 776 input tests, within gem5.
including finding 29 crashes, and 10 hangs.
The efficiency and human readability of the LLM-generated
RQ2 Answer. SearchSYS generated 30 000+ test test cases make them excellent candidates for integration into
cases for gem5. The majority of bugs, except for panic gem5’s C++ unit tests as regression tests. They run quickly
errors, out-of-memory and timeouts, were identified and we can exclude tests that trigger undefined behaviour,
using our differential testing mechanism. gem5 devel- require specific input, or produce large outputs. This allows
opers’ feedback reveals that SearchSYS’s ability to for targeted testing of untested areas, improving the overall
tie test cases to particular features of the ARM ISA robustness of the gem5 test suite. In addition, tests that reveal
is of great importance, and could help with ongoing unimplemented features can be earmarked for future develop-
development of gem5 also for RISC V and other ISAs. ment, supporting a case-driven test development approach.
As mentioned in the next section, recently LLMs have
VI. DISCUSSION proved very popular in software engineering research and
their rapid development means any paper will lack recency,
The use of LLM-generated test inputs has proven to be a nonetheless our selection of LLMs provides good coverage.
valuable method for uncovering new bugs within the gem5 The analysis in [5] required approximately 3500 hours of
system, particularly when evaluating the ARMv8 ISA. During machine time. There is no need to repeat the LLM-based
our experiments, 7 distinct sets of LLM-generated inputs were test program generation as: (1) by the time the analysis is
used, and the bugs identified were systematically categorized completed, newer LLM versions would render it outdated, and
477
Authorized licensed use limited to: Univ of Calif Davis. Downloaded on April 21,2026 at 03:18:06 UTC from IEEE Xplore. Restrictions apply.
(2) it is CPU-agnostic8, relying on GPU or OpenAI platform. Hu et al. [48] show it can do better than vanilla AFL++. How-
Our fuzzing campaigns significantly expanded the corpus ever, they test only one LLM (ChatGPT), and test ChatFuzz on
size and identified numerous crashes and hangs across all three real-world benchmarks which require structured inputs
LLMs. For example, even with a smaller LLM such as rather than an open-source simulator used by industry whose
TinyLlama, the initial corpus size grew considerably, inputs are executable programs.
uncovering several crashes and hangs. This demonstrates Our approach leverages AFL++ (Section II-A), specializing
the ability of fuzzing to produce diverse inputs that explore its application to mutate binary files and their inputs. Even if
unexpected branches, making it a valuable complement to this approach is uncommon, similar approaches can be found
other testing methods. in compiler testing [49], [50] and fuzzing network protocol
For corpus seed generation, the primary objective was analysis [51],
achieving a high compilation rate for effective test program Simulation tools like gem5 provide rich information for
generation, leaving test input diversification to fuzzing. The evaluating architectures and have been widely used for dif-
compilation rate is critical [26], [38], for two fundamental ferent aspects of the ARM architectures [52]–[54]. Testing
reasons: (1) in grey-box fuzzing, the priority is on seeds complex systems, such as gem5, involves not only efforts to
enabling the fuzzer to explore less-covered areas effectively, verify its architectural compliance [55] and code integrity but
especially when the diversity metric relevant to the SUT is also deeper internal testing or verification methods, such as
unknown to the LLM, and (2) diverse but failed to compile SearchSYS, which are crucial for validating its numerous
code is useless, as it yields no executable for simulation. options [56]. Similarly Xia et al.’s Fuzz4All [38] also uses
This objective is unlikely to be fully achieved with prompt LLM to example code snippets for fuzz tester input. Although
engineering alone, and we leave this to future work. None- they consider differential testing in fact in [38] they used
theless, frameworks like ROCODE [39], which achieve higher traditional “fuzzing oracles, such as crashes”.
compilation rates via backtracking and code analysis, might be Serebryany et al. [57] discuss a hardware fault diagnosis
used with SearchSYS to enhance its bug-finding capabilities. tool employed in Google data centres, where test cases were
SearchSYS implements a differential testing approach to generated through software fuzzing of CPU simulators akin to
uncover missimulation bugs (Figure 1). Hence, it requires gem5. They highlight gem5 as a potential tool for future use.
access to physical hardware matching the simulated architec- On the other hand, Rajeev et al. [58] used gem5 to test their
ture, as the correct output of the test inputs may be difficult fuzz inputs rather than fuzzing gem5 itself.
to determine. This challenge is commonly known as the Our previous work [3], [5] stands out as the only instance
oracle problem [40]. We acknowledge that access to diverse which has been using fuzzing to test gem5 so far. Initially, we
physical hardware may not always be feasible. In such cases, integrated LLMs and SBSE to test system simulators and de-
alternative strategies may be explored, such as leveraging veloped a prototype of SearchSYS. With SearchSYS [5],
similar architectures or replacing the hardware with another we fully automate the process and enhance the tool with
different simulator. new mutation operators. Specifically, we advance the previous
work by having LLM generate an example program instead
VII. RELATED WORK of providing one ourselves, and by constructing independent
There has been extensive interest in applying Large Lan- mutators to give automated feedback to AFL++ for mutation
guage Models to Software Engineering. For example, Hou et selection. In addition, in this work, we show how this approach
al. [41] mention seven existing surveys (e.g. [42]) before their supports testing the ARM architecture by also identifying deep
own, which covers 2017 to the start of 2024 and includes 395 vulnerabilities in these kinds of systems.
papers. However, neither Wang et al. [42] or the earlier Fan VIII. CONCLUSIONS
et al. [43], nor the most recent Hou et al. [41] or [44] include In this work, we show how effective SearchSYS is at
work directly on fuzz testing of simulators.
LLMs have already made an impact in industry (e.g. work in generating tests for the gem5 software simulator. In particular,
Meta reported by Alshahwan et al. [16]). Other works include we focus on the simulation of the ARM silicon chip Instruction
[45], [46] where Feldt et al. argue for the use of testing to Set Architecture (ISA). SearchSYS uses LLM-generated
kerb LLMs’ enthusiasm to make up answers. Also missing programs as seeds in a fuzzing campaign with specialized
from both surveys is Le et al. [47], where they propose using mutation operators to generate test cases. Through differential
LLMs to assist testing of software following the RESTful web testing, we identified 624 bugs with LLM-generated test cases
API architecture. and 126 with fuzzed test inputs. Out of the total number of
ChatFuzz (Hu et al. [48]) aims to improve fuzz testing. Like bug-activating test cases, 4 unique bugs have been reported
our SearchSYS it uses an LLM to automatically generate and acknowledged by developers. In 24 hours of compute
seeds for fuzz testing which are similar to existing seeds but time SearchSYS typically generates about a thousand test
fit better to a given format. Where the format was simple cases with their line coverage. Taking specifically the problem
of testing gem5’s implementation of the ARM instruction set
8Yet, we adjusted the dataset [5] by re-compiling the test input binaries for architecture, it would have taken a skilled engineer weeks to
the ARMv8 target. generate test cases starting from the ARM ISA documentation.
478
Authorized licensed use limited to: Univ of Calif Davis. Downloaded on April 21,2026 at 03:18:06 UTC from IEEE Xplore. Restrictions apply.
ACKNOWLEDGMENT [16] N. Alshahwan, J. Chheda, A. Finogenova, B. Gokkaya, M. Harman,
I. Harper, A. Marginean, S. Sengupta, and E. Wang, “Automated unit
We thank CloudLab [27] for the platform and infrastructure test improvement using large language models at Meta,” in FSE. Porto
support that enabled the experiment resulting in Table II de Galinhas, Brazil: ACM, July 15-19 2024, pp. 185––196. [Online].
and reproducibility check of bugs when we classified them. Available: https://doi.org/10.1145/3663529.3663839
This work was supported by the UKRI TAS Hub grants no. [17] K. Serebryany, “OSS-Fuzz - Google’s continuous fuzzing service
for open source software,” in 26th USENIX Security Symposium.
EP/V00784X/1 and EP/V026801/2 and the Alan Turing grant Vancouver: USENIX Association, August 16-18 2017, retrived 8
G2027 - MuSE. October 2024. [Online]. Available: https://www.usenix.org/conference/
usenixsecurity17/technical-sessions/presentation/serebryany
[18] A. Fioraldi, D. Maier, H. Eißfeldt, and M. Heuse, “AFL++ : Combining
REFERENCES incremental steps of fuzzing research,” in USENIX Workshop at WOOT
20. online: USENIX Association, 11 August 2020, p. 12, https://www.
[1] N. Binkert, B. Beckmann, G. Black, S. K. Reinhardt, A. Saidi, A. Basu, usenix.org/conference/woot20/presentation/fioraldi. J. Hestness, D. R. Hower, T. Krishna, S. Sardashti, R. Sen, K. Sewell, [19] gem5 Developers’ Meeting: August 2024, “Discussion on bugs M. Shoaib, N. Vaish, M. D. Hill, and D. A. Wood, “The gem5 in gem5 from fuzzing with SearchSYS,” https://github.com/orgs/ simulator,” SIGARCH Comput. Archit. News, vol. 39, no. 2, p. 1–7, aug gem5/discussions/1398, 8 Aug 4pm 2024. [Online]. Available: 2011. [Online]. Available: https://doi.org/10.1145/2024716.2024718 https://www.youtube.com/watch?v=hEyhXJg-rbU [2] B. R. Bruce, A. Akram, H. Nguyen, K. Roarty, M. Samani, M. Fariborz, [20] K. Even-Mendoza, H. Menendez, W. Langdon, A. Dakhama, J. Petke, T. Reddy, M. D. Sinclair, and J. Lowe-Power, “Enabling reproducible and B. R. Bruce, “Artifact of Search+LLM-based Testing for ARM and agile full-system simulation,” in IEEE International Symposium on Simulators,” Jan. 2025. [Online]. Available: https://doi.org/10.5281/ Performance Analysis of Systems and Software, ISPASS. Stony Brook, zenodo.14721385 NY, USA: , March 28-30 2021, pp. 183–193. [Online]. Available: [21] Zalewski Michal, “Technical “whitepaper” for afl-fuzz,” http://lcamtuf. [3] https://doi.org/10.1109/ISPASS51385.2021.00035 coredump.cx/afl/technical_details.txt, Retrieved April 21, 2023. A. Dakhama, K. Even-Mendoza, W. B. Langdon, H. D. Menéndez, [22] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, and J. Petke, “Searchgem5: Towards reliable gem5 with search based L. Kaiser, and I. Polosukhin, “Attention is all you need,” in NEURIPS, software testing and large language models,” in SSBSE. Springer, 2017, pp. 5998–6008. [Online]. Available: https://proceedings.neurips. 2023, pp. 160–166, best challenge track paper. [Online]. Available: cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html [4] https://doi.org/10.1007/978-3-031-48796-5_14 [23] L. Yang, C. Yang, S. Gao, W. Wang, B. Wang, Q. Zhu, X. Chu, ——, “Artifact of SearchGEM5: Towards reliable gem5 with search J. Zhou, G. Liang, and J. Chen, “On the evaluation of large language based software testing and large language models,” Sep. 2023. [Online]. models in unit test generation,” in ASE, 2024, pp. 1607–1619. [Online]. [5] Available: https://doi.org/10.5281/zenodo.8316685 Available: https://doi.org/10.1145/3691620.3695529 ——, “Enhancing search-based testing with LLMs for finding bugs in [24] T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and system simulators,” Research Square, 18 September 2024. [Online]. Y. Iwasawa, “Large language models are zero-shot rea- [6] Available: https://doi.org/10.21203/rs.3.rs-5004178/v1 soners,” in NEURIPS, vol. 35, 2022, pp. 22 199–22 213. ——, “Artifact of enhancing search-based testing with LLM for [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/ finding bugs in system simulators,” Aug. 2024. [Online]. Available: 2022/file/8bb0d291acd4acf06ef112099c16f326-Paper-Conference.pdf [7] https://doi.org/10.5281/zenodo.13450472 [25] ISO C, Working Group SC22/WG14, “The c17 standard for the c R. Holway, “Monday 30 September 2024 Intel and ARM,” 30 programming language (draft iso/iec9899:2017 of iso/iec 9899:2018), September 2024, retrieved 7 Oct 2024. [Online]. Available: https://www. Index Section, pp. 476-515,” https://www.iso.org/standard/74528.html, [8] techmarketview.com/ukhotviews/archive/2024/09/30/intel-and-arm 2018. S. Segars, “Arm partners have shipped 200 billion chips,” Blog, [26] K. Even-Mendoza, A. Sharma, A. F. Donaldson, and C. Cadar, 18 Oct 2021, retrieved 7 Oct 2024. [Online]. Available: https: “GrayC: Greybox fuzzing of compilers and analysers for C,” in //newsroom.arm.com/blog/200bn-arm-chips [9] Arm Education, “Arm research starter kit on system modeling us- Proceedings of the 32nd ACM SIGSOFT International Symposium ing gem5,” https://github.com/arm-university/arm-gem5-rsk, 2024, ac- on Software Testing and Analysis, ser. ISSTA 2023. Seattle, WA, cessed: 10-Oct-2024. USA: ACM, July 17-21 2023, pp. 1219–1231. [Online]. Available: [10] gem5 developers, “Extending gem5 for ARM,” https://www.gem5.org/ https://doi.org/10.1145/3597926.3598130 documentation/learning_gem5/part1/extending_configs, 2024, accessed: [27] D. Duplyakin et al., “The design and operation of CloudLab,” 10-Oct-2024. in 2019 USENIX annual technical conference (USENIX ATC [11] M. Fowler, “Continuous integration,” Blog, 18 January 2024. [Online]. 19), Renton, WA, USA, 2019, pp. 1–14. [Online]. Available: Available: https://martinfowler.com/articles/continuousIntegration.html https://www.usenix.org/conference/atc19/presentation/duplyakin [12] T. E. J. Vos, P. Tonella, W. Prasetya, P. M. Kruse, A. Bagnato, [28] SearchGEM5, https://github.com/karineek/SearchGEM5/tree/ M. Harman, and O. Shehory, “FITTEST: A new continuous and ssbse2023challenge/, Sept 2023. automated testing process for future internet applications,” in 2014 [29] A. Herrera, H. Gunadi, S. Magrath, M. Norrish, M. Payer, and A. L. Software Evolution Week - IEEE Conference on Software Maintenance, Hosking, “Seed selection for successful fuzzing,” in Proceedings of Reengineering, and Reverse Engineering, CSMR-WCRE 2014, Antwerp, the 30th ACM SIGSOFT International Symposium on Software Testing Belgium, February 3-6, 2014, S. Demeyer, D. W. Binkley, and F. Ricca, and Analysis, ser. ISSTA 2021. Virtual, Denmark: ACM, 2021, pp. Eds. IEEE Computer Society, 2014, pp. 407–410. [Online]. Available: 230–243. [Online]. Available: https://doi.org/10.1145/3460319.3464795 https://doi.org/10.1109/CSMR-WCRE.2014.6747206 [30] “AFL Internals - Stats, Counters and the UI,” https://www.core.gen.tr/ [13] M. Greiler, M. D. Storey, and A. Noda, “An actionable framework posts/007-afl-stats-counters-and-ui/, Fri, May 27, 2022. for understanding and improving developer experience,” IEEE Trans. [31] gem5, “Panic page table fault error in ARM when accessing invalid ad- Software Eng., vol. 49, no. 4, pp. 1411–1425, 2023. [Online]. Available: dress in the simulated program,” https://github.com/gem5/gem5/issues/ https://doi.org/10.1109/TSE.2022.3175660 1527, accessed: 2024-09-17. [14] A. Marginean, J. Bader, S. Chandra, M. Harman, Y. Jia, K. Mao, [32] ——, “SIGABRT does not trigger its signal handler and skips outputs on A. Mols, and A. Scott, “SapFix: Automated end-to-end repair at scale,” ARM (won’t fix),” https://github.com/gem5/gem5/issues/1544, accessed: in 41st International Conference on Software Engineering, J. M. Atlee 2024-09-17. and T. Bultan, Eds. Montreal: ACM, 25-31 May 2019, pp. 269–278. [33] ——, “Missing locale support within gem5 simulator on arm,” https: [Online]. Available: https://doi.org/10.1109/ICSE-SEIP.2019.00039 //github.com/gem5/gem5/issues/1547, accessed: 2024-09-17. [15] N. Alshahwan, “Industrial experience of genetic improvement in [34] ——, “Sim of localtime of time_t in ARM is different than native run,” Facebook,” in GI-2019, ICSE workshops proceedings, J. Petke, https://github.com/gem5/gem5/issues/1629, 2024, accessed: 2024-10-05. S. H. Tan, W. B. Langdon, and W. Weimer, Eds. Montreal: [35] Personal Communication, “Bug in GMP 6.1.0 with ARMv8 (build and IEEE, 28 May 2019, p. 1, Invited Keynote. [Online]. Available: diff-tested with GCC-11 and CLANG-14),” October 2024, leading to https://doi.org/10.1109/GI.2019.00010 some tests failed with both clang-14 and gcc-11.
479
Authorized licensed use limited to: Univ of Calif Davis. Downloaded on April 21,2026 at 03:18:06 UTC from IEEE Xplore. Restrictions apply.
[36] G. Klees, A. Ruef, B. Cooper, S. Wei, and M. Hicks, “Evaluating [52] F. A. Endo, D. Couroussé, and H.-P. Charles, “Micro-architectural
fuzz testing,” in Proceedings of the 2018 ACM SIGSAC Conference on simulation of in-order and out-of-order ARM microprocessors with
Computer and Communications Security, ser. CCS ’18. New York, gem5,” in 2014 international conference on embedded computer
NY, USA: Association for Computing Machinery, 2018, p. 2123–2138. systems: Architectures, modeling, and simulation (SAMOS XIV).
[Online]. Available: https://doi.org/10.1145/3243734.3243804 IEEE, 2014, pp. 266–273. [Online]. Available: http://dx.doi.org/10.
[37] S. Saha, L. Sarker, M. Shafiuzzaman, C. Shou, A. Li, G. Sankaran, 1109/SAMOS.2014.6893220
and T. Bultan, “Rare path guided fuzzing,” in Proceedings of the [53] I. Wang, P. Chakraborty, Z. Y. Xue, and Y. F. Lin, “Evaluation of gem5
32nd ACM SIGSOFT International Symposium on Software Testing for performance modeling of ARM Cortex-R based embedded SoCs,”
and Analysis, ISSTA 2023, Seattle, WA, USA, July 17-21, 2023, R. Just Microprocessors and Microsystems, vol. 93, p. 104599, 2022. [Online].
and G. Fraser, Eds. ACM, 2023, pp. 1295–1306. [Online]. Available: Available: https://doi.org/10.1016/j.micpro.2022.104599
https://doi.org/10.1145/3597926.3598136 [54] Y. Qiu, S. Yi, M. Jing, X. Xiong, D. Xu, X. Zhu, X. Zeng,
[38] C. S. Xia, M. Paltenghi, J. Le Tian, M. Pradel, and L. Zhang, and Y. Fan, “Performance error evaluation of gem5 simulator for
“Fuzz4All: Universal fuzzing with large language models,” in ARM server,” in 2023 IEEE 15th International Conference on ASIC
Proceedings of the IEEE/ACM 46th International Conference on (ASICON). IEEE, 2023. [Online]. Available: http://dx.doi.org/10.1109/
Software Engineering, ser. ICSE ’24, Lisbon, 2024. [Online]. Available: ASICON58565.2023.10396046
https://doi.org/10.1145/3597503.3639121 [55] N. Bruns, V. Herdt, D. Große, and R. Drechsler, “Toward RISC-
[39] X. Jiang, Y. Dong, Y. Tao, H. Liu, Z. Jin, and G. Li, V CSR compliance testing,” IEEE Embedded Systems Letters,
“ROCODE: Integrating backtracking mechanism and program vol. 13, no. 4, pp. 202–205, 2021. [Online]. Available: https:
analysis in large language models for code generation,” //doi.org/10.1109/LES.2021.3077368
in Proceedings of the 47th International Conference on [56] L. Bossuet, V. Grosso, and C. A. Lara-Nino, “Emulating side
Software Engineering (ICSE). IEEE, 2025, accepted for publi- channel attacks on gem5: lessons learned,” in 2023 IEEE European
cation. [Online]. Available: https://conf.researchr.org/track/icse-2025/ Symposium on Security and Privacy Workshops (EuroS&PW), Delft,
icse-2025-research-track#Accepted-papers-First-and-Second-Cycle Netherlands, 3-7 July 2023, pp. 287–295. [Online]. Available:
[40] E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, “The http://dx.doi.org/10.1109/EuroSPW59978.2023.00036
oracle problem in software testing: A survey,” IEEE Transactions on [57] K. Serebryany, M. Lifantsev, K. Shtoyk, D. Kwan, and P. Hochschild,
Software Engineering, vol. 41, no. 5, pp. 507–525, 2015. [Online]. “SiliFuzz: Fuzzing CPUs by proxy,” arXiv 2110.11519, 2021.
Available: http://dx.doi.org/10.1109/TSE.2014.2372785 [58] R. Rajeev and X. Song, “An empirical study of fuzz stimuli generation
[41] X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, X. Luo, for asynchronous fifo and memory coherency verification,” Journal of
D. Lo, J. Grundy, and H. Wang, “Large language models for software Electrical Electronics Engineering, vol. 2, no. 3, pp. 302–306, 2023.
engineering: A systematic literature review,” ACM Transactions on
Software Engineering and Methodology, 2024, accepted on 27 August
2024. [Online]. Available: https://doi.org/10.1145/3695988
[42] J. Wang, Y. Huang, C. Chen, Z. Liu, S. Wang, and Q. Wang, “Software
testing with large language model: Survey, landscape, and vision,” IEEE
Transactions on Software Engineering, vol. 50, no. 4, pp. 911–936,
2024. [Online]. Available: http://dx.doi.org/10.1109/TSE.2024.3368208
[43] A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta,
S. Yoo, and J. M. Zhang, “Large language models for software
engineering: Survey and open problems,” in ICSE-FoSE, Melbourne,
Australia, 14-20 May 2023, pp. 31–53. [Online]. Available: http:
//dx.doi.org/10.1109/ICSE-FoSE59343.2023.00008
[44] Q. Zhang, C. Fang, Y. Xie, Y. Zhang, Y. Yang, W. Sun, S. Yu, and
Z. Chen, “A survey on large language models for software engineering,”
arXiv 2312.15223, 8 Sep 2024.
[45] R. Feldt, S. Kang, J. Yoon, and S. Yoo, “Towards autonomous
testing agents via conversational large language models,” in ASE,
Luxembourg, 11-15 September 2023, pp. 1688–1693. [Online].
Available: https://doi.org/10.1109/ASE56229.2023.00148
[46] S. Yoo, “Executing one’s way out of the Chinese Room,” in
13th International Workshop on Genetic Improvement @ICSE 2024,
G. An, A. Blot, V. Nowack, O. Krauss, and J. Petke, Eds. Lisbon:
ACM, 16 April 2024, p. viii, Invited Keynote. [Online]. Available:
http://dx.doi.org/10.1145/3643692
[47] T. Le, T. Tran, D. Cao, V. Le, T. N. Nguyen, and V. Nguyen, “KAT:
Dependency-aware automated API testing with large language models,”
in ICST, Toronto, 27-31 May 2024, pp. 82–92. [Online]. Available:
http://dx.doi.org/10.1109/ICST60714.2024.00017
[48] J. Hu, Q. Zhang, and H. Yin, “Augmenting greybox fuzzing with
generative AI,” arXiv 2306.06782, 2023.
[49] A. Groce, R. van Tonder, G. T. Kalburgi, and C. Le Goues, “Making
no-fuss compiler fuzzing effective,” in CC ’22: 31st ACM SIGPLAN
International Conference on Compiler Construction. Seoul, South
Korea: ACM, April 2-3 2022, pp. 194–204. [Online]. Available:
https://doi.org/10.1145/3497776.3517765
[50] C. Aschermann, T. Frassetto, T. Holz, P. Jauernig, A.-R. Sadeghi, and
D. Teuchert, “NAUTILUS: Fishing for deep bugs with grammars,”
in Network and Distributed Systems Security (NDSS) Symposium,
San Diego, CA, USA, 24-27 February 2019. [Online]. Available:
http://dx.doi.org/10.14722/ndss.2019.23412
[51] V.-T. Pham, M. Böhme, and A. Roychoudhury, “AFLNET: A greybox
fuzzer for network protocols,” in ICST, Porto, Portugal, 24-28 October
2020, pp. 460–465. [Online]. Available: http://dx.doi.org/10.1109/
ICST46399.2020.00062
480
Authorized licensed use limited to: Univ of Calif Davis. Downloaded on April 21,2026 at 03:18:06 UTC from IEEE Xplore. Restrictions apply.