SOURCE ARCHIVE
EXTRACTED CONTENT
87,625 charsSearch+LLM-based Testing for ARM Simulators
1st Karine Even-Mendoza 2nd Héctor D. Menéndez 3rd W.B Langdon
Department of Informatics Department of Informatics Department of Computer Science
King’s College London King’s College London University College London
London, United Kingdom London, United Kingdom London, United Kingdom
karine.even_mendoza@kcl.ac.uk hector.menendez@kcl.ac.uk w.langdon@cs.ucl.ac.uk
4th Aidan Dakhama 5th Justyna Petke 6th Bobby R. Bruce
Department of Informatics Department of Computer Science Department of Computer Science
King’s College London University College London University of California, Davis
London, United Kingdom London, United Kingdom United States of America
aidan.dakhama@kcl.ac.uk j.petke@ucl.ac.uk bbruce@ucdavis.edu
Abstract—In order to aid quality assurance of large complex systems is gem5 [1], [2], an open-source modular platform for
hardware architectures, system simulators have been developed. computer system architecture research that includes system-
However, such system simulators do not always accurately mirror level architecture and processor microarchitecture. The ac-
what would happen on a real device. A significant challenge in
testing these simulators arises from the complexity of having to curacy and reliability of software architecture simulators are
model both the simulation and the infinite number of software paramount for optimizing development investment and ensur-
that could be run on such a device. ing a smooth lifecycle. However, software complexity of such
Our previous work introduced SearchSYS, a testing frame- simulators makes them difficult to test comprehensively for all work for software simulators. SearchSYS leverages a large potential architectural purposes. Identifying mismatches, inter- language model for initial seed C code generation, which is then nal simulation errors, and performance issues in simulators can compiled, and the resultant binary is fed to a fuzzer. We then use differential testing by running the outputs of fuzzing on real lead to their improvement and make them more robust, thereby hardware and a system simulator to identify mismatches. maximizing the return on investment. We present and discuss our solution to the problem of testing Our previous work [3]–[6] addressed this problem by com- software simulators, using SearchSYS to test the gem5 VLSI bining large language models (LLMs), fuzzing, and differen- digital circuit simulator, employed by ARM to test their systems. tial testing. By employing OpenAI’s Large Language Model In particular, we focus on the simulation of the ARM silicon chip Instruction Set Architecture (ISA). (LLM) GPT-3.5-turbo to generate a baseline test suite SearchSYS can create test cases that activate bugs by from seeded programs, and our modified version of AFL++, combining LLMs, fuzzing, and differential testing. Using only our system discovered various implementation bugs in gem5 LLM, SearchSYS identified 74 test cases that activated bugs. related to the Intel 64 bit X86 architecture. Whereas fuzzers By incorporating fuzzing, this number increased by 93 additional traditionally rely on implicit test oracles, such as crashes, and bug-activating cases within 24 hours. Through differential testing, we identified 624 bugs with LLM-generated test cases and 126 timing out loops, differential testing strengthens the fuzzer with fuzzed test inputs. Out of the total number of bug-activating by allowing it to automatically check for non-fatal errors, test cases, 4 unique bugs have been reported and acknowledged such as differences between running a simulation and running by developers. Additionally, we provided developers with a test on actual hardware. We then extended this work to create suite and fuzzing statistics, and open-sourced SearchSYS¹. SearchSYS [5], which fully automates our initial proof-of- Index Terms—Software System Simulation, SearchSYS, gem5, AFL++, Fuzzing, Fuzz Testing, Differential Testing, concept and introduces novel mutation operators which are Search-Based Software Testing, LLM, Language Models, Ollama, able to mutate not only the seed program itself, but also its CodeLlama, TinyLlama, Phi2, Llama2, Magicoder, CodeBooga, command line parameters. Using SearchSYS we were able GPT-3.5-turbo, prompt engineering to further increase the bug finding ability of our approach. I. INTRODUCTION Again, we focused on the X64 architecture. Creating and developing new system architectures is a To showcase generalizability and widen impact and signifi- challenging task that requires a significant investment in both cance of our contribution, we now conduct an empirical study human and physical resources. Architectural simulators sup- on the ARM family of CPUs. ARM is a $160 billion [7] port this process by providing environments where developers company which designs chips and licenses them to others can validate their architectural goals. A good example of these to manufacture and integrate them into devices. Although its CPU stretches the full gamut of today’s computing, from 1See https://zenodo.org/records/13450472 for initial seeds and supercomputing HPC, cloud computing, servers, desktops, and SearchSYS code, and https://zenodo.org/records/14721385 for adjustment laptops, to Raspberry Pi, its processors are by far the most to ARMv8 and fuzzed seeds. successful CPUs largely because almost all mobile telephones
and handheld smart devices are based on ARM CPUs. Alto- different types of bugs in gem5’s simulation of the ARM
gether more than 200 billion ARM chips have been sold [8]. architecture.
Naturally, such a company has considerable in-house expertise, • Delivery of 3 661 LLM-generated tests and 30 000+ nevertheless ARM makes use of the open source gem5 project fuzzed tests along with a detailed analysis of bug- keeping its own git clone but also paying close attention to activating test cases to the gem5 development team. fixes and developments to the public version [9], [10]. • An empirical investigation of six large language models Despite being crucial to the quality of chip designs, at the for input generation for SearchSYS. cost of only a few days of computer time, by systematically We provide SearchSYS, bug reports, test cases and details using SearchSYS we were able to find and highlight 14 of fuzzing performance and statistics via [20], to facilitate different types of issues with gem5’s simulation of the ARM reproducibility of this study and wider use of SearchSYS. processor chips. Furthermore, SearchSYS has automatically Section III describes SearchSYS and how it integrates created test cases for gem5’s simulation of the ARM In- multiple C code generating LLMs with fuzz testing and struction Set Architecture (ISA), which previously would have extends fuzz testing with simulator specific mutations and taken skilled engineers weeks to do by hand. differential testing. Our research questions, methodology to Finally, by simultaneously providing the gem5 development answer those, and experimental setup are given in Section IV. team with details of C++ source line coverage for each test, we Whilst Section V gives our results including contrasting the open the way for future work whereby regression testing can effectiveness of six LLMs at testing gem5’s simulations of be automatically targeted at immediate development changes. ARM hardware. The discussion (Section VI) and related work We expect that by executing tests that run the just modified (Section VII) are followed by our conclusions in Section VIII, code, there will be more chance of finding and addressing but first, we start with the background (Section II). issues. Moreover, selecting which tests to run should speed up testing, possibly allowing real-time continuous integration test- II. BACKGROUND ing [11]–[13]. Experience with Meta [14]–[16] highlighted the We provide here a quick introduction to fuzz testing and importance of giving individual developers immediate feed- large language models, as these underpin our approach imple- back. Hopefully, this can reduce reliance on the current mix mented in SearchSYS. of daily and weekly regression testing, which is undirected and already consumes several weeks of computer time per week. A. Fuzzing with AFL Fuzz testing used in SearchSYS gives us automated test- Fuzzing is a technique used to identify bugs in programs by ing. It consumes computer time, rather than a test engineer’s running the program with a variety of test inputs. Originally time. Over the last few years fuzzing has been extensively used the test inputs were generated at random. With the introduction by Google to find many thousands of security related problems of feedback-based fuzzing techniques, new methods have been and other bugs [17]. AFL++ [18] is the state of the art fuzzing developed to manipulate the initial test inputs. Typically these tool. Our SearchSYS extends AFL++ with the addition aim to change (mutate) the test inputs in order to exercise of automatically generated test seeds and domain specific new parts of the software under test (SUT) during the fuzzing mutation operators and, as we shall see, leads to cost effective process. One of the most popular fuzzers is the American testing of the state of the art in VLSI simulators, i.e. gem5, Fuzzy Lop (AFL) [21], and related to the AFL family is for the most widely used general purpose CPU on the planet. AFL++ [18], which we are using for SearchSYS. The purpose of this study is to evaluate the reliability and AFL begins by automatically instrumenting the software accuracy of gem5 as an ARM simulator and to extend gem5’s under test (SUT). This instrumentation provides feedback to existing test suite. To achieve this, we apply SearchSYS the fuzzing process and is performed at compilation time. The to identify bugs in gem5, generate test cases, and measure compiler introduces various flags in the programs that are gem5’s capabilities on ARM machines. related to the code branches visited during execution. During We identify 14 different types of bugs, including panic the fuzzing process, AFL starts by running a set of predefined errors (which are of particular interest to gem5 develop- inputs, called seeds. These inputs are then mutated as part of ers [19]), performance bugs, and differential bugs, where we the fuzzing strategy. AFL retains those mutated inputs that compare the simulation outcomes with those from physical explore new sections of the program that previous inputs have ARM machines. We have already reported 4 of these bugs. not visited, and continues to mutate them to discover new The selection of which bugs to report depends on whether paths. The inputs are maintained in a queue and are selected they belong to the previously mentioned categories. We do by the fuzzer based on the strategy and the effectiveness of not report regular crashes, such as system call issues, non- their mutations. The main goal of fuzzing is to improve the test panic segmentation faults, and timeouts that are less than 24 coverage of the SUT. The fuzzer runs until specific termination hours, as the developers find them less interesting [19]. conditions are met, usually a time limit. To summarize, our contributions are: In the context of this work, the seeds are created by • An extensive empirical study of SearchSYS applied to large language models (LLMs). The LLMs generate not only the ARM architecture, using the gem5 system simulator. programs that exercise the SUT (the simulator), but also the Our results reveal that we were able to identify 14 program’s inputs and their types.
B. Large Language Models weekly. Some tests depend on the specific configuration of
Like the original SearchSYS work [3], [5], we use large the host system, and Docker containers are often required to
language models to provide a set of seeds for the fuzzers. ensure these tests run properly.
Large language models have become prominent in recent III. SE A R C HSYS
years, especially after the popular introduction of GPT-3.5. Figure 1 shows the structure of each part of SearchSYS.
GPT LLMs are based on transformers [22] with attention Our approach combines large language models with fuzz mechanisms to identify relevant parts during the learning testing to identify bugs within system simulators and to process. Currently, several large language models are available, generate test suites. The main idea is to address a key challenge both private and public. Notable private LLMs include Gemini with fuzzers: traditionally fuzzers are given a set of input by Google and Llama 3 by Meta. In terms of public LLMs, values for the software they are to test (the SUT). (These various communities have also created their own, such as are known as seeds.) Here we are testing simulators, whose Dolphin and Mistral, which are available through platforms inputs are programs to be simulated. That is, instead of like Huggingface or the Ollama interface. starting with numbers, etc., as inputs, the initial seeds are Here we use a variety of LLMs of different sizes and programs. Until recently, automatically creating programs of natures. We use GPT-3.5 and Phi-2, a Microsoft LLM aimed at interest was hard3. However, now there are LLMs dedicated to software generation. We employ Llama2 and TinyLlama, two program source code generation. Therefore, we use LLMs to general-purpose LLMs of different sizes. We use Magicoder, create an initial set of test seeds, which are of interest for combining auto-encoders and transformers for program source the fuzzing process and can improve test coverage. Given code generation. Additionally, we use CodeBooga, which inte- the complexity of the simulator software, it is crucial to grates various LLMs, specifically Phind-CodeLlama-34B-v2, have useful seeds that can enhance coverage, especially when which outperforms GPT-4 in code generation tasks [23] and testing a comprehensive architecture like ARM. Further, as WizardCoder-Python-34B-V1.0. All of these public LLMs a traditional fuzzer runs, it changes the SUT’s test inputs to are run through the Ollama framework, which provides the try and cover new branches within the SUT. As our SUTs necessary infrastructure for LLM execution. are simulators, SearchSYS extends the fuzzer so that it C. gem5 System Simulator can mutate both the input executable binary program and its gem52 is a state-of-the-art discrete time simulator for logic command line parameters along with their data type. circuits. It is often used to try out the logic design of We consider three types of bugs: 1) crashes 2) issues with new electronic components such as memory cache systems, efficiency or performance, such as program hangs 3) mis- FPGAs, and even CPUs. gem5 is a large open-source project matches between the simulator and the actual hardware. For hosted on GitHub, written mostly in C++ and Python. It is used this last type of bug, it is essential to determine whether the by companies such as ARM and Google to simulate hardware. simulator behaves as the real system. This is where differential Including objects, shared library and images, gem5 occupies testing becomes crucial. SearchSYS provides the differential over 28 GB of memory. It is composed of ∼ 1.34 million lines testing infrastructure to compare test cases within the simulator of code, comprising more than a million lines of C++. and outside of it. The gem5 simulator has a comprehensive testing framework Our testing approach is divided into three main components: comprising C++ unit tests, Python unit tests, and TestLib 1) Test Input Generation System: This system uses Large integration tests. Each test set focuses on specific aspects of Language Models (LLMs) to create a set of test cases the system, ranging from low-level code validation to full-scale which serve as our testing baseline. Unlike traditional simulation testing. fuzzing processes that require existing programs as seeds, The unit tests in gem5 are designed to validate the func- we ask different large language models to generate tionality of the core C++ components and are automatically the test cases (programs to be simulated) automatically. executed as part of gem5’s continuous integration (CI) pro- (Note: in the context of the ARM Instruction Set Ar- cess. These test much of the core C++ code and maintain chitecture it can take weeks for a skilled engineer to sit code integrity and correctness after Git commits. In addition down with the ARM ISA hardware documentation and to C++ unit tests, gem5 includes Python unit tests, which code programs to test each part of the ISA.) naturally focus on verifying Python-based components. The 2) Coverage-Guided Fuzz Testing Tool (AFL++): AFL++ Python tests are quick to run and fewer in number than the focuses on identifying new uncovered code regions in the C++ unit tests. simulation system and achieving higher coverage through Testlib integration testing is more intensive in nature. Al- mutation-based testing. AFL++ employs a driver for the most everything in Testlib runs a gem5 simulation. The tests simulation system, enabling the simulation/execution of are categorized into three sets: “quick”, “long” and “very- any binary program with specific parameters. The type long”. The “quick” tests are run during CI, the “long” tests of the parameters is specified by the LLM. The LLM are executed nightly, and the “very-long” tests are conducted 3E.g. (in the context of testing) automatically generating programs that achieve high coverage, have specific patterns or formats, or contain edge cases 2https://www.gem5.org or a specific set of instructions.
SearchSYS where n can be arbitrarily large. In total, n + 1 prompts
generate n test programs. One prompt for setting the LLM’s
Large Language Model role plus n prompts to generate n test programs.
a) Setting the LLM role: Before generating the n test
programs, a single one-off LLM prompt initializes the process.
Generates Test Cases This one-off LLM prompt serves to set the LLM’s role. This
LLM prompt specifies the programs to be generated (e.g.
Parametrized Inputs Generation “generate C programs with arguments as input”). Since each
LLM has memory 4, we only need to set LLM’s role once.
Provides Initial Test Seeds b) Generating a test program: The LLM prompt starts
with a problem description that specifies the program tasks and
the context in which these programs are to be generated. The
Fuzzing with Customized AFL++ Fuzzing problem description contains specific gaps that will be filled
with the tokens. A total of 4089 tokens are available as part of
Mutates and Executes Tests SearchSYS. Table I describes the three token types across 4
categories of tokens with a few examples. The first and second
Differential Testing Module types of tokens are derived from compiler optimization names
and compiler parts. The third token is chosen at random from
Executes Test Cases Executes Test Cases general tokens related to programming languages, tutorials,
Collects Outputs Collects Outputs or standards. The fourth tokens category in Table I includes
phrases taken from the C17 standard [25], such as "initialize"
Simulator Real ARM Hardware and "pointer". For instance, we might ask for examples of dead
code elimination, handling of the Abstract Syntax Tree (AST),
Differential Testing and a beginner tutorial name in C programming examples,
corresponding to the first, second, and third tokens.
Fig. 1. SearchSYS diagram starts with the generation phase, where the The template prompt is shown in Listing 1.
LLMs generate C programs which take parameters via the command line.
It compiles them. Then it applies the fuzzing phase, using the customized " Coding t a s k : g i v e me a program i n C w i t h a l l
version of AFL, which is able to mutate both the executable binary and its i n c l u d e s . I n p u t i s t a k e n v i a a r g v o n l y .
parameters. Finally, in the differential testing phase, it compares each mutated P l e a s e r e t u r n a program (C program ) and a c o n c r e t e
executable binary in both the simulator and the real ARM architecture. example o f an i n p u t (BASH ) . The C program w i l l be
w i t h co de t r i g g e r i n g " + <Token −1> + " o p t i m i s a t i o n s ,
c o v e r s t h i s p a r t o f t h e c o m p i l e r " + <Token −2> + " ,
not only provides the program to be simulated but also and e x e r c i s e s t h i s i d e a i n C : " + <Token −3> + " .
specific inputs for it, which our special version of AFL++ To" r e c a p t h e co de c o n t a i n s t h e s e : " + <Token −1> +
and " + <Token −2> + " and " + <Token −3 >;
search will modify along with the program itself. Listing 1. Template prompt for generating C test programs with random
3) Differential Testing Module: This module compares the tokens
outputs of the simulator with those of the real ARM The value of the
TABLE I
THE FOUR C TOKEN CATEGORIES WHICH ARE USED TO FILL GAPS IN THE TEMPLATE USED TO PROMPT THE LLMS AND A FEW EXAMPLES OF EACH
ID Type Category #Tokens Examples 1 Token-1 Compiler Optimizations 26 "Scalar Optimizations", "Dead Code Elimination", "Constant Folding" 2 Token-2 Compiler Parts 36 "Sema", "Serialization", "Parse", "Lex", "AST" 3 Token-3 Domain Problems 192 "C Program to Sort an Array using Merge Sort", "Calendar Year in Different Formats", "input includes several arguments" 4 Token-3 Compiler Standard Indices [25] 3480 "AND operators", "cimagl function", "EOF", "locale", "pow", "SCNiMAX"
address this, SearchSYS employs a custom bit-flip mutation This approach allows for more complex mutations, such as operator that controls the number and frequency of bit-flips, binary file mutations. applying them only to a program’s compiled binary file. This Another factor affecting the throughput of fuzzing is the avoids applying bit-flips to arguments or type information, number of mutation operations that AFL++ performs in a preserving the structure of the test input. single iteration. The parameter afl_custom_fuzz_count SearchSYS applies three mutation operators for testing controls the number of times a test input should be mutated system simulators: and executed against the target. A lower value reduces the 1) A bit-flip operator for modifying a program’s compiled risk of iteration failure but can lead to inefficient fuzzed input binary file. generation due to iteration overhead. Following our results [5], 2) A range-enhanced operator for editing argument values we set afl_custom_fuzz_count to be 17, 84 and 66 within their specified type range. for Operator 1, 2 and 3 from Section III-B, respectively. 3) An operator for changing the value’s type. Operator (2) uses type information to ensure arguments C. Differential Testing remain valid, while Operator (3) randomly changes the type, Fuzzed test inputs can uncover crashes, hangs and mis- such as from INT32 to LONG, potentially exposing memory matches between the architecture and the system simulator. safety issues in the SUT. We support all integer types, float, However, AFL++ only identifies a bug if the test input leads to double, and strings but have not yet implemented pointer a crash or a hang, degrading SearchSYS’s ability to detect support. Figure 2 and Figure 3 provide examples of value missimulation issues. To address this, SearchSYS follows (operator 2) and type mutations (operator 3), respectively. the approach of Even-Mendoza et al. [26] and separates fuzzing and differential testing. That is, after fuzzing we 1 ./mutator_args.so test.o, 5:INT, 20:LONG , "Hi": perform differential testing by comparing runs using fuzzed 2 AfterSTRING test inputs (i.e. mutated executable binaries and their inputs) Mutation: test.o, 10:INT, 20:LONG, "House": on ARM hardware (native) with those on the simulator with STRING ARM ISA. Fig. 2. Example of mutation operator (2) changing argument values: first D. Implementation argument: 5 to 10 and third: "Hi" to "House" (second argument unchanged). SearchSYS is implemented using a mixture of languages and Unix shell scripts. Specifically, we use (1) Java with 1 ./mutator_args.so test.o, 5:INT, 20:LONG , "Hi": Ollama for LLM-based test input generation, (2) C/C++ for 2 AfterSTRING the custom mutator and fuzzing-related code, and (3) a set of Mutation: test.o, 5:LONG, "20":STRING, "Hi": STRING scripts for configuring the testing environment [3], [5]. When adapting SearchSYS for ARM, we encountered Fig. 3. Example of mutation operator (3) changing argument types: first minor script issues related to the linker and data from argument from INT to LONG and second argument from LONG to STRING. TinyLlama, the latter caused an early termination in AFL++ fuzzing due to memory errors. Additionally, we had to pass SearchSYS loads all three mutators (1-3) using the exist- ARM as the ISA parameter, though no modifications were ing AFL++ option, allowing AFL++’s heuristics to select the needed for the Python configuration script. We continued to next mutation operator. However, we decrease the probability use the example script provided by the SSBSE Challenge of choosing (2) by setting it to only 99.5% of the times AFL++ Track 2023 organizers5. These adjustments required minor selected it originally and replacing the remaining 0.5% with modifications to our scripts (setting the testing environment, mutator (1), as AFL++ favours this operator due to its low fuzzing and differential testing parts). failure risk, which is too conservative for fuzzing. The code in parts (1) + (2), required no further modifica- SearchSYS extends AFL++ by evaluating new test in- tions. However, in between this version and [5], we made some puts in the form of binary name, arguments list, bug-fixing edits in the C/C++ code. types. Then it applies mutation operators directly to the compiled binaries, their arguments, or their arguments’ types. 5 hello-custom-binary.py
IV. EVALUATION architectures. To achieve this, we compared the bug-finding
To assess the quality of gem5 as a simulator for ARM effectiveness and fuzzing throughput of SearchSYS between architectures, we aim to answer the following questions: the ARM ISA and X86 ISA, as previously explored in [5]. We followed the experimental procedure outlined in [5], but RQ1: Considering that large language models (LLMs) gener- adapted it for the ARM ISA and ARMv8 hardware, running ate test suites independently, how effective are these test suites without Docker. Furthermore, the experiments were repeated at identifying bugs in gem5’s ARM simulation processes? 5 times (instead of 10), using the minimized input corpora, as To answer this question, we will ask LLMs to generate these have been shown to be more stable during fuzzing and test suites, which will later be used as seeds for the fuzzing in particular for SearchSYS [5], [29], [30]. process. Each test program (of a test input in the suite) will The minimized input corpora TinyLlama, Phi2, be generated using a different prompt: SearchSYS generates Llama2, Magicoder, CodeBooga and GPT-3.5-turbo random tokens (Section III-A), embeds the random tokens into were taken from [5], [6], while GPT-3.5-turbo (SSBSE the template prompt in Listing 1 to form a complete prompt, 2023) from [3], [4]. and prompts the LLM with the constructed prompt to generate Each fuzzing campaign ran for 24 hours, with five indepen- a new test program. These steps are part of SearchSYS’s test dent repetitions of the fuzzing process per minimized input input generation system (Section III). corpus. The throughput results of the fuzzing experiments were This research question will help determine which specific calculated as the mean value across these five repetitions, language models are more effective at generating test suites ensuring consistency and robustness in the findings. The that can uncover bugs in the simulator, under the current differential testing post-fuzzing (i.e. when gem5 differed from prompting mechanism (Section III-A). In this evaluation, we the real hardware) was done using the last repetition, for all 7 will identify the best language model for finding bugs in corpora. In case of a mismatch, to determine if the mismatch the ARM simulation process and uncover specific bugs. Our is a genuine bug, we compared the results from the gem5 process will also ensure a minimum set of test cases after simulation with ARM ISA to those obtained via the two ARM reducing the test suite using afl-cmin6. hardware (Section IV-A). RQ2: How effective is SearchSYS at identifying bugs in V. RESULTS the simulation of the ARM architecture performed with gem5 In this section, we present the outcomes of our experiments, after the fuzzing process? specifically, bugs found from the initial corpus of LLM- To answer this research question, we will run the fuzzing generated test inputs, and those found as a result of fuzzing strategy using the different test suites generated by each LLM campaigns. To detect bugs, we cross-validated our results as seeds. The fuzzer will run against the simulator, with the between an X86 and two ARMv8 machines. One of these generated inputs focusing not only on identifying crashes ARMv8 machines is being configured to be more strict, often and hangs within the simulator but also on mismatches in initializing uninitialized local variables to zero. differential testing by comparing the output of the generated In total, we reported 4 bugs from the LLM-generated test tests with that of a real ARM machine. inputs [31]–[34]. We further identified a bug in the GNU A. Hardware Multiple Precision Arithmetic Library (GMP 6.1.0) during cross-system testing [35]. We ran tests on gem5 using two machines: (1) a single CloudLab7 [27] m400 machine with 64 GB RAM, ARMv8 A. Bugs Detected from LLM-generated Test Inputs 64-bit architecture with a single socket, 2.4 GHz, 8 cores, To evaluate LLMs as sources of inputs for regression testing and 1 thread per core, running Ubuntu 22.04 ARM, and of system simulators, we used 7 sets of LLM-generated test (2) a single machine (UCL) with 224 CPU cores (Cavium inputs, created during 25-hour runs with different LLMs with ThunderX2 CN9975, 2.0 GHz) and 130 GB RAM, ARMv8 qualitative and quantitative evaluation and analysis of LLM- 64-bit architecture with 2 sockets, 28 cores per socket, and generated test inputs detailed in [3], [5]. Here, we focus on 4 threads per core, running Red Hat Linux (aarch64-redhat- ARMv8, noting that LLM test program generation is agnostic linux-gnu). We installed the same compiler versions, adapted to X86 or ARM CPU. We classified the bugs found for each of for ARM, and set up the same tools as described in [5], on the 7 sets of simulations with ARM ISA against real ARMv8 both machines. The exact specification can be found in [28]. machines and compared the bug finding rate with our previous B. Experiments results on the gem5 X86 backend [5]. For the experiments, we selected ARMv8 hardware and the Table II shows the bugs identified in our investigation ARM ISA, given its relevance from the industry’s perspec- from the LLM-generated test inputs. Table II includes the tive [19]. The primary objective was to provide a comprehen- bug description (“Bug”) with a bug number if already sive evaluation of SearchSYS’s performance across different reported to the gem5 bug tracking system. Columns A to G represent the number of test inputs triggering this 6afl-cmin and afl-cmin’s manpage bug category: A: TinyLlama, B: Phi2, C: Llama2, 7See https://docs.cloudlab.us/hardware.html D: Magicoder, E: CodeBooga, F: GPT-3.5-turbo and
TABLE II
BUGS FOUND IN G E M5 WITH LLM-GENERATED TEST INPUTS. COLUMNS A TO G HOLD THE NUMBER OF INSTANCES OF EACH BUG FOUND BY EACH
LLM: A: TI N YLL A M A, B: PH I2, C: LL A M A2, D: MA G I C O D E R, E: CO D EBO O G A, F: GPT-3.5-T U R B O AND G: GPT-3.5-T U R B O (SSBSE 2023).
BUG A B C D E F G
Bug #1527 (panic) 1 1 5 1 1 0 0
Bug #1544 (Missing support) 0 0 2 3 6 5 0
Bug #1547 (Missing support) 0 0 0 1 7 0 0
src/sim/syscall_emul.cc:67: fatal: syscall dup3 (#24) unimplemented 0 0 0 3 0 14 0
src/sim/syscall_emul.cc:67: fatal: syscall pipe2 (#59) unimplemented 0 0 0 1 1 0 0
src/sim/syscall_emul.cc:67: fatal: syscall clock_getres (#114) unimplemented 0 0 0 0 0 1 0
src/sim/syscall_emul.cc:67: fatal: syscall clock_nanosleep (#115) unimplemented 0 1 8 7 4 4 0
src/sim/syscall_emul.cc:67: fatal: syscall wait4 (#260) unimplemented 0 0 2 0 0 0 0
instruction ‘bti’ unimplemented 20 17 81 127 51 202 4
Bug #1629 (some time functionality unimplemented) 0 0 0 2 1 28 0
some thread functionality unimplemented in SE mode 0 0 0 1 0 10 0
Variable’s value is random in ARM but fixed in simulation 0 0 0 2 1 7 0
Timeout 1 2 4 27 15 15 1
Totals (of number of test inputs exposing a bug) 22 21 102 175 87 286 5
G: GPT-3.5-turbo (SSBSE 2023). The bugs found include expecting complex input or having large output, these can be panic errors, usually triggered by assertion violation, timeouts, grouped according to coverage and instructions triggered to and different outputs including fatal errors triggered by wrong target untested areas of the current regression tests of the gem5 or missing instruction implementation for ARM ISA, revealing test suite. Furthermore, test inputs exposing unimplemented gaps in the system’s support for ARMv8 functionalities. We features can be saved for future use. For example, tests ran the differential testing scripts comparing the result of the triggering unimplemented system calls can be saved for future native run on ARMv8 machines against the simulation with a use (e.g. to implement test-case-driven development methods). time out of 50 seconds and a memory limit of approximately 90 MB (stack size) for both the gem5 simulation and the native run. In most cases, unimplemented syscalls represent a gap in In total, GPT-3.5-turbo found the highest number gem5 functionality which the gem5 developers do not intend of issues (286), followed by Magicoder (175), Llama2 to plug, but the bug report itself is helpful since it can alert (102), CodeBooga (87), TinyLlama (22), and Phi2 (21). gem5 users to a now-known issue, saving them time trying GPT-3.5-turbo (SSBSE 2023) had the fewest, with only to resolve what was previously an issue known only to the 5 instances. Two of these bugs, panic and timeout, were developers. identified by executing the simulator, totalling 74 test cases (9 panic and 65 timeouts), while an additional 624 bug- In gem5 using SearchSYS, we found around 530 test activating test cases were identified using differential testing. inputs exposing missimulations and errors and 30 test inputs The differing results for GPT-3.5-turbo (SSBSE 2023) exposing optimization issues (timeouts) on X86 ISA [5] com- compared to the GPT-3.5-turbo set, even though both pared to around 630 and over 60 instances on ARM. Note, are using the same language model, can be attributed to that the ’530’ and ’30’ instances on X86 are not a subset of three factors: (1) GPT-3.5-turbo (SSBSE 2023) dataset the 630 and 60 instances on ARM and reflect different bugs. was generated in 2023, while GPT-3.5-turbo dataset was This suggests that contrary to expectations [19], ARM is no generated in 2024, (2) GPT-3.5-turbo (SSBSE 2023) more stable than X86 ISA in gem5, as given the same set of was trained on the LLVM test suite, likely overlapping with test inputs, more of them exposed an issue. tests already evaluated by gem5, and (3) GPT-3.5-turbo (SSBSE 2023) used a few-shots approach instead of zero- shots, with zero-shots generally providing better input diversity We reported four new bugs from the LLM-generated test and throughput [5]. We observed these differences for the bug inputs to the gem5 bug tracker [31]–[34] under “arch-arm” count results during the pre-fuzzing test input generation stage tag. These bugs were not previously identified during our X86 (Table II), but as well in the results from the fuzzing and post- fuzzing campaigns [3], [5], except for bug #1629 [34], which fuzzing stages (to be discussed in Section V-B). we expected to be properly implemented in ARM. While we These LLM test cases are a suitable contribution to gem5’s anticipated that ARM would be generally more stable than C++ unit tests (as regression tests). They have several lines of X86, the occurrence of this bug in ARM was unexpected, code, are efficient (as they commonly terminate in under 50 prompting us to report it immediately. We excluded unimple- seconds) and are human-readable. With some semi-manual fil- mented issues from bug reports, as these represent new test tering, to remove tests triggering undefined behaviour or those cases rather than valid bugs.
RQ1 Answer. All of the LLMs found bugs in gem5’s ning them on the real hardware. Table IV includes the
simulation of ARM CPUs. We found 13 bugs, six bug description (“Bug”) with a bug number if already
are unimplemented system calls functionality in the reported to the gem5 bug tracking system. Columns A
simulator. We have already reported four (#1527 #1544 to G represent the number of test inputs triggering this
#1547 #1629). Of these three have already received bug category: A: TinyLlama, B: Phi2, C: Llama2,
detailed consideration by the gem5 development team. D: Magicoder, E: CodeBooga, F: GPT-3.5-turbo and
G: GPT-3.5-turbo (SSBSE 2023). Note “Crash” and
“Hangs” from Table III: (1) Table IV does not include timed-
B. Fuzzing as Part of the Testing Process out fuzzed test inputs counters since these are already stated
in Table III in the “Hangs” column; and (2) not all instances
SearchSYS is an AFL++-based tool. After generating test counted in the “Crash” column of Table III represent gen- inputs using LLMs and compiling them into a test case (binary, uine crashes; some are due to corrupted binaries or non- input, and input type info), we fuzzed an instrumented version reproducible crashes. Since AFL++ categorizes all these as a of gem5 for 24 hours to have sufficient time to explore the single “Crash” type, we included these in Table IV and refined codebase of a specific gem5 version [36]. it for a more detailed breakdown. Table III shows throughput during 24 hours of fuzzing, Table IV identifies hangs and 7 distinct issues discovered detailing the number of new test inputs generated for each during a manual inspection of automatically flagged of the 7 minimized input corpora (from each of the LLMs mismatches, warnings, and crashes. During 24-hour fuzzing, used). The columns “Initial Corpus” indicates the size of each TinyLlama had the highest number of fuzzed test inputs initial corpus at the start of the fuzzing, “Fuzzed Corpus (Std exposing issues (87), followed by GPT-3.5-turbo (31), Dev)”, “Queue (Std Dev)”, “Crash (Std Dev)” and “Hangs (Std Magicoder (10), Phi2 and Llama2 (6 each), and Dev)” are means over five trials per corpus with their observed CodeBooga (3). Most fuzzed sets identified 2-3 distinct standard deviations. The “Fuzzed Corpus” column shows the issues, with GPT-3.5-turbo finding 4 distinct ones. As in total number of test inputs generated (i.e. queue + crashes Table II, GPT-3.5-turbo (SSBSE 2023) had the lowest
- hangs). At the end of each 24-hour fuzzing campaign, we bug-finding rate with only 2 instances, each of a different recorded: (1) “Queue” (test inputs generated by AFL++ that issue category. did not crash or hang and were therefore suitable for further Some previously known bugs were encountered during mutation), (2) “Crash” (number of crashed test inputs), and fuzzing, generating further examples of the issue, which can (3) “Hangs” (number of hanged test inputs). be useful for bug localization and debugging. We found one The minimized input corpora vary in size, with approxi- “Out of Memory” instance. This indicates that fuzzing can be mately 100 test inputs generally being recommended [29], beneficial, though it suggests that longer fuzzing runs may be [30]. During fuzzing, the highest throughput (in total) necessary for uncovering additional bugs and a better under- of fuzzed test inputs was achieved (on average) by standing of the codebase code coverage. During fuzzing, we GPT-3.5-turbo (SSBSE 2023) with 997 fuzzed test also identified a fatal error in src/mem/port_proxy.hh inputs, followed by Magicoder (986), Llama2 (948), readBlob, which at first, looked like a genuine error. How- GPT-3.5-turbo (888) and CodeBooga (839). The smaller ever, comparing the results between the two ARMv8 ma- LLMs had a lower throughput during fuzzing: TinyLlama chines, with the simulation failing only on the UCL machine (776) and Phi2 (680). This trend was slightly different when (Section IV-A) led us to conclude that it is likely to be a observing the queue size only (the fuzzed test inputs for configuration issue rather than an ARM ISA bug, and thus we differential testing post fuzzing), with Magicoder achiev- excluded these from the tables. ing the highest rate with 962 fuzzed test inputs, followed To recap, during the fuzzing campaign, three of these by GPT-3.5-turbo (SSBSE 2023) with 945 fuzzed test bugs, panic, out-of-memory and timeout, were identified by inputs. This is different from the pattern we already re- executing the simulator, totalling 93 test cases (19 panic, 1 ported in [5] for X86, where GPT-3.5-turbo (SSBSE out-of-memory and 73 timeouts), while an additional 126 bug-
and TinyLlama achieved the best throughput while activating test cases were identified using differential testing. GPT-3.5-turbo had extremely poor fuzzing throughput. Fuzzing is a time-intensive process, and as such is unsuit- Lastly, CodeBooga showed a large standard deviation across able for regular runs of regression tests, where the developer all measured outputs (Queue, Crash, and Hangs), while requires feedback quickly. However, due to its ability to Llama2, Phi2, and TinyLlama had high deviations in produce diverse inputs covering unexpected branches, it could some outputs, though smaller than CodeBooga’s. The re- be valuable for use in less common, but large test suites, such maining LLMs exhibited generally smaller standard devia- as those run between new releases, where there is a larger tions. Despite these variations, the minimized input corpora time budget. Fuzzing can then help identify obscure bugs typically generated between 700 and 1,000 test inputs during which may not have been caught during the other stages of 24 hours of fuzzing. development. In the context of gem5, fuzzing of weekly or Table IV: shows the bugs identified by comparing when release versions can be integrated into the TestLib process of gem5 simulated the mutated binary programs with run- gem5, which can include using Docker containers to provide
TABLE III NUMBER OF G E M5 TEST INPUTS GENERATED BY EACH INITIAL CORPUS (BY LLM, MEAN AND STANDARD DEVIATION OF 5 RUNS). SEE SECTION V-B
Initial Corpus Fuzzed Corpus (Std Dev) Queue (Std Dev) Crash (Std Dev) Hangs (Std Dev)
TinyLlama 206 776 (±59) 737 (±59) 29 (±4) 10 (±1) Phi2 366 680 (±87) 651 (±82) 14 (±6) 15 (±3) Llama2 613 948 (±62) 916 (±62) 16 (±6) 16 (±3) Magicoder 719 986 (±41) 962 (±42) 13 (±2) 11 (±3) CodeBooga 612 839 (±112) 816 (±106) 11 (±7) 12 (±6) GPT-3.5-turbo 703 888 (±26) 871 (±24) 11 (±3) 6 (±3) GPT-3.5-turbo 442 997 (±34) 945 (±34) 49 (±5) 3 (±3) (SSBSE 2023)
TABLE IV
DIFFERENTIAL TESTING RESULTS. BUGS FOUND IN G E M5 DURING 24-HOUR FUZZING OF THE LAST REPETITION. COLUMNS A TO G HOLD THE NUMBER
OF INSTANCES OF EACH BUG FOUND BY INITIAL MINIMIZED INPUT CORPUS: A: TI N YLL A M A, B: PH I2, C: LL A M A2, D: MA G I C O D E R, E: CO D EBO O G A,
F: GPT-3.5-T U R B O AND G: GPT-3.5-T U R B O (SSBSE 2023). (SEE SECTION V-A FOR EXPLANATION OF DIFFERENCES BETWEEN F AND G.)
BUG A B C D E F G
src/sim/syscall_emul.cc:67: fatal: syscall dup3 (#24) unimplemented 0 0 0 0 0 4 0
src/sim/syscall_emul.cc:67: fatal: syscall pipe2 (#59) unimplemented 0 0 0 0 1 0 0
src/sim/syscall_emul.cc:67: fatal: syscall clock_nanosleep (#115) unimplemented 0 0 0 4 0 7 0
instruction ‘bti’ unimplemented 21 3 3 5 1 19 1
Out of Memory 0 0 0 0 1 0 0
Likely to be Bug #1544 (Missing support) 56 0 0 0 0 0 0
Bug #1527 (panic) 10 3 3 1 0 1 1
Totals (of number of test inputs exposing a bug) 87 6 6 10 3 31 2
a consistent environment for fuzzing as in [37], ensuring based on their occurrences in simulations. Notably, we discov- reproducibility and isolating system dependencies. Further- ered a range of panic errors, timeouts, and fatal errors, predom- more, SearchSYS, a coverage-directed fuzzer, can assist in inantly triggered by assertion violations or improper/missing exploring newly added code by using partial instrumentation or instruction implementations for ARM ISA. This highlights tailoring the mutators, enabling deeper testing of the codebase significant gaps in gem5’s support for ARMv8 functionalities. of gem5. Our comparative analysis revealed that ARM is not neces- Our fuzzing campaigns have uncovered further bugs and sarily more stable than the X86 ISA within gem5, contradict- produced a larger corpus of test cases than LLM-generated ing previous expectations [19]. This higher incidence of issues tests alone. We found crashes and hangs using all LLMs tested. on ARM suggests that there are still considerable challenges Even with small LLMs such as TinyLlama, fuzzing was able to achieving parity between the support for ARM and X86 to grow the corpus size from 206 input tests to 776 input tests, within gem5. including finding 29 crashes, and 10 hangs. The efficiency and human readability of the LLM-generated RQ2 Answer. SearchSYS generated 30 000+ test test cases make them excellent candidates for integration into cases for gem5. The majority of bugs, except for panic gem5’s C++ unit tests as regression tests. They run quickly errors, out-of-memory and timeouts, were identified and we can exclude tests that trigger undefined behaviour, using our differential testing mechanism. gem5 devel- require specific input, or produce large outputs. This allows opers’ feedback reveals that SearchSYS’s ability to for targeted testing of untested areas, improving the overall tie test cases to particular features of the ARM ISA robustness of the gem5 test suite. In addition, tests that reveal is of great importance, and could help with ongoing unimplemented features can be earmarked for future develop- development of gem5 also for RISC V and other ISAs. ment, supporting a case-driven test development approach. As mentioned in the next section, recently LLMs have VI. DISCUSSION proved very popular in software engineering research and their rapid development means any paper will lack recency, The use of LLM-generated test inputs has proven to be a nonetheless our selection of LLMs provides good coverage. valuable method for uncovering new bugs within the gem5 The analysis in [5] required approximately 3500 hours of system, particularly when evaluating the ARMv8 ISA. During machine time. There is no need to repeat the LLM-based our experiments, 7 distinct sets of LLM-generated inputs were test program generation as: (1) by the time the analysis is used, and the bugs identified were systematically categorized completed, newer LLM versions would render it outdated, and
(2) it is CPU-agnostic8, relying on GPU or OpenAI platform. Hu et al. [48] show it can do better than vanilla AFL++. How- Our fuzzing campaigns significantly expanded the corpus ever, they test only one LLM (ChatGPT), and test ChatFuzz on size and identified numerous crashes and hangs across all three real-world benchmarks which require structured inputs LLMs. For example, even with a smaller LLM such as rather than an open-source simulator used by industry whose TinyLlama, the initial corpus size grew considerably, inputs are executable programs. uncovering several crashes and hangs. This demonstrates Our approach leverages AFL++ (Section II-A), specializing the ability of fuzzing to produce diverse inputs that explore its application to mutate binary files and their inputs. Even if unexpected branches, making it a valuable complement to this approach is uncommon, similar approaches can be found other testing methods. in compiler testing [49], [50] and fuzzing network protocol For corpus seed generation, the primary objective was analysis [51], achieving a high compilation rate for effective test program Simulation tools like gem5 provide rich information for generation, leaving test input diversification to fuzzing. The evaluating architectures and have been widely used for dif- compilation rate is critical [26], [38], for two fundamental ferent aspects of the ARM architectures [52]–[54]. Testing reasons: (1) in grey-box fuzzing, the priority is on seeds complex systems, such as gem5, involves not only efforts to enabling the fuzzer to explore less-covered areas effectively, verify its architectural compliance [55] and code integrity but especially when the diversity metric relevant to the SUT is also deeper internal testing or verification methods, such as unknown to the LLM, and (2) diverse but failed to compile SearchSYS, which are crucial for validating its numerous code is useless, as it yields no executable for simulation. options [56]. Similarly Xia et al.’s Fuzz4All [38] also uses This objective is unlikely to be fully achieved with prompt LLM to example code snippets for fuzz tester input. Although engineering alone, and we leave this to future work. None- they consider differential testing in fact in [38] they used theless, frameworks like ROCODE [39], which achieve higher traditional “fuzzing oracles, such as crashes”. compilation rates via backtracking and code analysis, might be Serebryany et al. [57] discuss a hardware fault diagnosis used with SearchSYS to enhance its bug-finding capabilities. tool employed in Google data centres, where test cases were SearchSYS implements a differential testing approach to generated through software fuzzing of CPU simulators akin to uncover missimulation bugs (Figure 1). Hence, it requires gem5. They highlight gem5 as a potential tool for future use. access to physical hardware matching the simulated architec- On the other hand, Rajeev et al. [58] used gem5 to test their ture, as the correct output of the test inputs may be difficult fuzz inputs rather than fuzzing gem5 itself. to determine. This challenge is commonly known as the Our previous work [3], [5] stands out as the only instance oracle problem [40]. We acknowledge that access to diverse which has been using fuzzing to test gem5 so far. Initially, we physical hardware may not always be feasible. In such cases, integrated LLMs and SBSE to test system simulators and de- alternative strategies may be explored, such as leveraging veloped a prototype of SearchSYS. With SearchSYS [5], similar architectures or replacing the hardware with another we fully automate the process and enhance the tool with different simulator. new mutation operators. Specifically, we advance the previous work by having LLM generate an example program instead VII. RELATED WORK of providing one ourselves, and by constructing independent There has been extensive interest in applying Large Lan- mutators to give automated feedback to AFL++ for mutation guage Models to Software Engineering. For example, Hou et selection. In addition, in this work, we show how this approach al. [41] mention seven existing surveys (e.g. [42]) before their supports testing the ARM architecture by also identifying deep own, which covers 2017 to the start of 2024 and includes 395 vulnerabilities in these kinds of systems. papers. However, neither Wang et al. [42] or the earlier Fan VIII. CONCLUSIONS et al. [43], nor the most recent Hou et al. [41] or [44] include work directly on fuzz testing of simulators. In this work, we show how effective SearchSYS is at LLMs have already made an impact in industry (e.g. work in generating tests for the gem5 software simulator. In particular, Meta reported by Alshahwan et al. [16]). Other works include we focus on the simulation of the ARM silicon chip Instruction [45], [46] where Feldt et al. argue for the use of testing to Set Architecture (ISA). SearchSYS uses LLM-generated kerb LLMs’ enthusiasm to make up answers. Also missing programs as seeds in a fuzzing campaign with specialized from both surveys is Le et al. [47], where they propose using mutation operators to generate test cases. Through differential LLMs to assist testing of software following the RESTful web testing, we identified 624 bugs with LLM-generated test cases API architecture. and 126 with fuzzed test inputs. Out of the total number of ChatFuzz (Hu et al. [48]) aims to improve fuzz testing. Like bug-activating test cases, 4 unique bugs have been reported our SearchSYS it uses an LLM to automatically generate and acknowledged by developers. In 24 hours of compute seeds for fuzz testing which are similar to existing seeds but time SearchSYS typically generates about a thousand test fit better to a given format. Where the format was simple cases with their line coverage. Taking specifically the problem of testing gem5’s implementation of the ARM instruction set 8Yet, we adjusted the dataset [5] by re-compiling the test input binaries for architecture, it would have taken a skilled engineer weeks to the ARMv8 target. generate test cases starting from the ARM ISA documentation.
ACKNOWLEDGMENT [16] N. Alshahwan, J. Chheda, A. Finogenova, B. Gokkaya, M. Harman,
We thank CloudLab [27] for the platform and infrastructure I. Harper, A. Marginean, S. Sengupta, and E. Wang, “Automated unit
test improvement using large language models at Meta,” in FSE. Porto
support that enabled the experiment resulting in Table II de Galinhas, Brazil: ACM, July 15-19 2024, pp. 185––196. [Online].
and reproducibility check of bugs when we classified them. [17] Available: https://doi.org/10.1145/3663529.3663839 This work was supported by the UKRI TAS Hub grants no. K. Serebryany, “OSS-Fuzz - Google’s continuous fuzzing service for open source software,” in 26th USENIX Security Symposium. EP/V00784X/1 and EP/V026801/2 and the Alan Turing grant Vancouver: USENIX Association, August 16-18 2017, retrived 8 G2027 - MuSE. October 2024. [Online]. Available: https://www.usenix.org/conference/ usenixsecurity17/technical-sessions/presentation/serebryany REFERENCES [18] A. Fioraldi, D. Maier, H. Eißfeldt, and M. Heuse, “AFL++ : Combining incremental steps of fuzzing research,” in USENIX Workshop at WOOT [1] N. Binkert, B. Beckmann, G. Black, S. K. Reinhardt, A. Saidi, A. Basu, 20. online: USENIX Association, 11 August 2020, p. 12, https://www. J. Hestness, D. R. Hower, T. Krishna, S. Sardashti, R. Sen, K. Sewell, [19] usenix.org/conference/woot20/presentation/fioraldi. M. Shoaib, N. Vaish, M. D. Hill, and D. A. Wood, “The gem5 gem5 Developers’ Meeting: August 2024, “Discussion on bugs simulator,” SIGARCH Comput. Archit. News, vol. 39, no. 2, p. 1–7, aug in gem5 from fuzzing with SearchSYS,” https://github.com/orgs/ 2011. [Online]. Available: https://doi.org/10.1145/2024716.2024718 gem5/discussions/1398, 8 Aug 4pm 2024. [Online]. Available: [2] B. R. Bruce, A. Akram, H. Nguyen, K. Roarty, M. Samani, M. Fariborz, [20] https://www.youtube.com/watch?v=hEyhXJg-rbU T. Reddy, M. D. Sinclair, and J. Lowe-Power, “Enabling reproducible K. Even-Mendoza, H. Menendez, W. Langdon, A. Dakhama, J. Petke, and agile full-system simulation,” in IEEE International Symposium on and B. R. Bruce, “Artifact of Search+LLM-based Testing for ARM Performance Analysis of Systems and Software, ISPASS. Stony Brook, Simulators,” Jan. 2025. [Online]. Available: https://doi.org/10.5281/ NY, USA: , March 28-30 2021, pp. 183–193. [Online]. Available: [21] zenodo.14721385 https://doi.org/10.1109/ISPASS51385.2021.00035 Zalewski Michal, “Technical “whitepaper” for afl-fuzz,” http://lcamtuf. [3] A. Dakhama, K. Even-Mendoza, W. B. Langdon, H. D. Menéndez, [22] coredump.cx/afl/technical_details.txt, Retrieved April 21, 2023. and J. Petke, “Searchgem5: Towards reliable gem5 with search based A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, software testing and large language models,” in SSBSE. Springer, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in NEURIPS, 2023, pp. 160–166, best challenge track paper. [Online]. Available: 2017, pp. 5998–6008. [Online]. Available: https://proceedings.neurips. https://doi.org/10.1007/978-3-031-48796-5_14 [23] cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html [4] ——, “Artifact of SearchGEM5: Towards reliable gem5 with search L. Yang, C. Yang, S. Gao, W. Wang, B. Wang, Q. Zhu, X. Chu, based software testing and large language models,” Sep. 2023. [Online]. J. Zhou, G. Liang, and J. Chen, “On the evaluation of large language Available: https://doi.org/10.5281/zenodo.8316685 models in unit test generation,” in ASE, 2024, pp. 1607–1619. [Online]. [5] ——, “Enhancing search-based testing with LLMs for finding bugs in Available: https://doi.org/10.1145/3691620.3695529 system simulators,” Research Square, 18 September 2024. [Online]. [24] T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Available: https://doi.org/10.21203/rs.3.rs-5004178/v1 Y. Iwasawa, “Large language models are zero-shot rea- [6] ——, “Artifact of enhancing search-based testing with LLM for soners,” in NEURIPS, vol. 35, 2022, pp. 22 199–22 213. finding bugs in system simulators,” Aug. 2024. [Online]. Available: [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/ https://doi.org/10.5281/zenodo.13450472 2022/file/8bb0d291acd4acf06ef112099c16f326-Paper-Conference.pdf [7] R. Holway, “Monday 30 September 2024 Intel and ARM,” 30 [25] ISO C, Working Group SC22/WG14, “The c17 standard for the c September 2024, retrieved 7 Oct 2024. [Online]. Available: https://www. programming language (draft iso/iec9899:2017 of iso/iec 9899:2018), techmarketview.com/ukhotviews/archive/2024/09/30/intel-and-arm Index Section, pp. 476-515,” https://www.iso.org/standard/74528.html, [8] S. Segars, “Arm partners have shipped 200 billion chips,” Blog, 2018. 18 Oct 2021, retrieved 7 Oct 2024. [Online]. Available: https: [26] K. Even-Mendoza, A. Sharma, A. F. Donaldson, and C. Cadar, //newsroom.arm.com/blog/200bn-arm-chips “GrayC: Greybox fuzzing of compilers and analysers for C,” in [9] Arm Education, “Arm research starter kit on system modeling us- Proceedings of the 32nd ACM SIGSOFT International Symposium ing gem5,” https://github.com/arm-university/arm-gem5-rsk, 2024, ac- on Software Testing and Analysis, ser. ISSTA 2023. Seattle, WA, cessed: 10-Oct-2024. USA: ACM, July 17-21 2023, pp. 1219–1231. [Online]. Available: [10] gem5 developers, “Extending gem5 for ARM,” https://www.gem5.org/ https://doi.org/10.1145/3597926.3598130 documentation/learning_gem5/part1/extending_configs, 2024, accessed: [27] D. Duplyakin et al., “The design and operation of CloudLab,” 10-Oct-2024. in 2019 USENIX annual technical conference (USENIX ATC [11] M. Fowler, “Continuous integration,” Blog, 18 January 2024. [Online]. 19), Renton, WA, USA, 2019, pp. 1–14. [Online]. Available: Available: https://martinfowler.com/articles/continuousIntegration.html https://www.usenix.org/conference/atc19/presentation/duplyakin [12] T. E. J. Vos, P. Tonella, W. Prasetya, P. M. Kruse, A. Bagnato, [28] SearchGEM5, https://github.com/karineek/SearchGEM5/tree/ M. Harman, and O. Shehory, “FITTEST: A new continuous and ssbse2023challenge/, Sept 2023. automated testing process for future internet applications,” in 2014 [29] A. Herrera, H. Gunadi, S. Magrath, M. Norrish, M. Payer, and A. L. Software Evolution Week - IEEE Conference on Software Maintenance, Hosking, “Seed selection for successful fuzzing,” in Proceedings of Reengineering, and Reverse Engineering, CSMR-WCRE 2014, Antwerp, the 30th ACM SIGSOFT International Symposium on Software Testing Belgium, February 3-6, 2014, S. Demeyer, D. W. Binkley, and F. Ricca, and Analysis, ser. ISSTA 2021. Virtual, Denmark: ACM, 2021, pp. Eds. IEEE Computer Society, 2014, pp. 407–410. [Online]. Available: 230–243. [Online]. Available: https://doi.org/10.1145/3460319.3464795 https://doi.org/10.1109/CSMR-WCRE.2014.6747206 [30] “AFL Internals - Stats, Counters and the UI,” https://www.core.gen.tr/ [13] M. Greiler, M. D. Storey, and A. Noda, “An actionable framework posts/007-afl-stats-counters-and-ui/, Fri, May 27, 2022. for understanding and improving developer experience,” IEEE Trans. [31] gem5, “Panic page table fault error in ARM when accessing invalid ad- Software Eng., vol. 49, no. 4, pp. 1411–1425, 2023. [Online]. Available: dress in the simulated program,” https://github.com/gem5/gem5/issues/ https://doi.org/10.1109/TSE.2022.3175660 1527, accessed: 2024-09-17. [14] A. Marginean, J. Bader, S. Chandra, M. Harman, Y. Jia, K. Mao, [32] ——, “SIGABRT does not trigger its signal handler and skips outputs on A. Mols, and A. Scott, “SapFix: Automated end-to-end repair at scale,” ARM (won’t fix),” https://github.com/gem5/gem5/issues/1544, accessed: in 41st International Conference on Software Engineering, J. M. Atlee 2024-09-17. and T. Bultan, Eds. Montreal: ACM, 25-31 May 2019, pp. 269–278. [33] ——, “Missing locale support within gem5 simulator on arm,” https: [Online]. Available: https://doi.org/10.1109/ICSE-SEIP.2019.00039 //github.com/gem5/gem5/issues/1547, accessed: 2024-09-17. [15] N. Alshahwan, “Industrial experience of genetic improvement in [34] ——, “Sim of localtime of time_t in ARM is different than native run,” Facebook,” in GI-2019, ICSE workshops proceedings, J. Petke, https://github.com/gem5/gem5/issues/1629, 2024, accessed: 2024-10-05. S. H. Tan, W. B. Langdon, and W. Weimer, Eds. Montreal: [35] Personal Communication, “Bug in GMP 6.1.0 with ARMv8 (build and IEEE, 28 May 2019, p. 1, Invited Keynote. [Online]. Available: diff-tested with GCC-11 and CLANG-14),” October 2024, leading to https://doi.org/10.1109/GI.2019.00010 some tests failed with both clang-14 and gcc-11.
[36] G. Klees, A. Ruef, B. Cooper, S. Wei, and M. Hicks, “Evaluating [52] F. A. Endo, D. Couroussé, and H.-P. Charles, “Micro-architectural fuzz testing,” in Proceedings of the 2018 ACM SIGSAC Conference on simulation of in-order and out-of-order ARM microprocessors with Computer and Communications Security, ser. CCS ’18. New York, gem5,” in 2014 international conference on embedded computer NY, USA: Association for Computing Machinery, 2018, p. 2123–2138. systems: Architectures, modeling, and simulation (SAMOS XIV). [Online]. Available: https://doi.org/10.1145/3243734.3243804 IEEE, 2014, pp. 266–273. [Online]. Available: http://dx.doi.org/10. [37] S. Saha, L. Sarker, M. Shafiuzzaman, C. Shou, A. Li, G. Sankaran, 1109/SAMOS.2014.6893220 and T. Bultan, “Rare path guided fuzzing,” in Proceedings of the [53] I. Wang, P. Chakraborty, Z. Y. Xue, and Y. F. Lin, “Evaluation of gem5 32nd ACM SIGSOFT International Symposium on Software Testing for performance modeling of ARM Cortex-R based embedded SoCs,” and Analysis, ISSTA 2023, Seattle, WA, USA, July 17-21, 2023, R. Just Microprocessors and Microsystems, vol. 93, p. 104599, 2022. [Online]. and G. Fraser, Eds. ACM, 2023, pp. 1295–1306. [Online]. Available: Available: https://doi.org/10.1016/j.micpro.2022.104599 https://doi.org/10.1145/3597926.3598136 [54] Y. Qiu, S. Yi, M. Jing, X. Xiong, D. Xu, X. Zhu, X. Zeng, [38] C. S. Xia, M. Paltenghi, J. Le Tian, M. Pradel, and L. Zhang, and Y. Fan, “Performance error evaluation of gem5 simulator for “Fuzz4All: Universal fuzzing with large language models,” in ARM server,” in 2023 IEEE 15th International Conference on ASIC Proceedings of the IEEE/ACM 46th International Conference on (ASICON). IEEE, 2023. [Online]. Available: http://dx.doi.org/10.1109/ Software Engineering, ser. ICSE ’24, Lisbon, 2024. [Online]. Available: ASICON58565.2023.10396046 https://doi.org/10.1145/3597503.3639121 [55] N. Bruns, V. Herdt, D. Große, and R. Drechsler, “Toward RISC- [39] X. Jiang, Y. Dong, Y. Tao, H. Liu, Z. Jin, and G. Li, V CSR compliance testing,” IEEE Embedded Systems Letters, “ROCODE: Integrating backtracking mechanism and program vol. 13, no. 4, pp. 202–205, 2021. [Online]. Available: https: analysis in large language models for code generation,” //doi.org/10.1109/LES.2021.3077368 in Proceedings of the 47th International Conference on [56] L. Bossuet, V. Grosso, and C. A. Lara-Nino, “Emulating side Software Engineering (ICSE). IEEE, 2025, accepted for publi- channel attacks on gem5: lessons learned,” in 2023 IEEE European cation. [Online]. Available: https://conf.researchr.org/track/icse-2025/ Symposium on Security and Privacy Workshops (EuroS&PW), Delft, icse-2025-research-track#Accepted-papers-First-and-Second-Cycle Netherlands, 3-7 July 2023, pp. 287–295. [Online]. Available: [40] E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, “The http://dx.doi.org/10.1109/EuroSPW59978.2023.00036 oracle problem in software testing: A survey,” IEEE Transactions on [57] K. Serebryany, M. Lifantsev, K. Shtoyk, D. Kwan, and P. Hochschild, Software Engineering, vol. 41, no. 5, pp. 507–525, 2015. [Online]. “SiliFuzz: Fuzzing CPUs by proxy,” arXiv 2110.11519, 2021. Available: http://dx.doi.org/10.1109/TSE.2014.2372785 [58] R. Rajeev and X. Song, “An empirical study of fuzz stimuli generation [41] X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, X. Luo, for asynchronous fifo and memory coherency verification,” Journal of D. Lo, J. Grundy, and H. Wang, “Large language models for software Electrical Electronics Engineering, vol. 2, no. 3, pp. 302–306, 2023. engineering: A systematic literature review,” ACM Transactions on Software Engineering and Methodology, 2024, accepted on 27 August 2024. [Online]. Available: https://doi.org/10.1145/3695988 [42] J. Wang, Y. Huang, C. Chen, Z. Liu, S. Wang, and Q. Wang, “Software testing with large language model: Survey, landscape, and vision,” IEEE Transactions on Software Engineering, vol. 50, no. 4, pp. 911–936, 2024. [Online]. Available: http://dx.doi.org/10.1109/TSE.2024.3368208 [43] A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang, “Large language models for software engineering: Survey and open problems,” in ICSE-FoSE, Melbourne, Australia, 14-20 May 2023, pp. 31–53. [Online]. Available: http: //dx.doi.org/10.1109/ICSE-FoSE59343.2023.00008 [44] Q. Zhang, C. Fang, Y. Xie, Y. Zhang, Y. Yang, W. Sun, S. Yu, and Z. Chen, “A survey on large language models for software engineering,” arXiv 2312.15223, 8 Sep 2024. [45] R. Feldt, S. Kang, J. Yoon, and S. Yoo, “Towards autonomous testing agents via conversational large language models,” in ASE, Luxembourg, 11-15 September 2023, pp. 1688–1693. [Online]. Available: https://doi.org/10.1109/ASE56229.2023.00148 [46] S. Yoo, “Executing one’s way out of the Chinese Room,” in 13th International Workshop on Genetic Improvement @ICSE 2024, G. An, A. Blot, V. Nowack, O. Krauss, and J. Petke, Eds. Lisbon: ACM, 16 April 2024, p. viii, Invited Keynote. [Online]. Available: http://dx.doi.org/10.1145/3643692 [47] T. Le, T. Tran, D. Cao, V. Le, T. N. Nguyen, and V. Nguyen, “KAT: Dependency-aware automated API testing with large language models,” in ICST, Toronto, 27-31 May 2024, pp. 82–92. [Online]. Available: http://dx.doi.org/10.1109/ICST60714.2024.00017 [48] J. Hu, Q. Zhang, and H. Yin, “Augmenting greybox fuzzing with generative AI,” arXiv 2306.06782, 2023. [49] A. Groce, R. van Tonder, G. T. Kalburgi, and C. Le Goues, “Making no-fuss compiler fuzzing effective,” in CC ’22: 31st ACM SIGPLAN International Conference on Compiler Construction. Seoul, South Korea: ACM, April 2-3 2022, pp. 194–204. [Online]. Available: https://doi.org/10.1145/3497776.3517765 [50] C. Aschermann, T. Frassetto, T. Holz, P. Jauernig, A.-R. Sadeghi, and D. Teuchert, “NAUTILUS: Fishing for deep bugs with grammars,” in Network and Distributed Systems Security (NDSS) Symposium, San Diego, CA, USA, 24-27 February 2019. [Online]. Available: http://dx.doi.org/10.14722/ndss.2019.23412 [51] V.-T. Pham, M. Böhme, and A. Roychoudhury, “AFLNET: A greybox fuzzer for network protocols,” in ICST, Porto, Portugal, 24-28 October 2020, pp. 460–465. [Online]. Available: http://dx.doi.org/10.1109/ ICST46399.2020.00062