ARMv8

Overview

ARMv8 (also referred to as Armv8-A in its application-profile form) is the 64-bit instruction set architecture (ISA) that succeeds the ARMv7 32-bit ISA. It introduces the AArch64 execution state alongside the legacy AArch32 state and is used in a wide range of devices from smartphones to server-class machines. The SearchSYS paper explicitly motivates targeting ARMv8 hardware by noting ARM ISA's relevance from industry's perspective.

Security Research on ARMv8

Because ARMv8 underpins large classes of mobile and embedded devices, the security properties of code running under this ISA have been actively studied. A 2016 arXiv paper "ARMv8 Shellcodes from 'A' to 'Z'" describes a methodology to automatically transform arbitrary ARMv8 programs into alphanumeric executable polymorphic shellcodes. Shellcodes generated in this way can evade detection and bypass filters, broadening the attack surface of ARM-powered devices such as smartphones.

Architectural Semantics: Relaxed Virtual Memory

Virtual memory in Armv8-A has relaxed-memory concurrency semantics that had not previously been investigated in detail. An ESOP 2022 paper (extended version, arXiv 2203.00642) explores this design space in collaboration with Arm to support future system-software verification, including use cases drawn from the pKVM production hypervisor under development by Google. The work:

• Develops a test suite (litmus tests) for relaxed virtual-memory behavior.
• Delimits the design space with axiomatic-style concurrency models.
• Proves that under simple stable configurations the architectural model collapses to previous "user" models.
• Develops tooling to compute allowed behaviors in the model integrated with the full Armv8-A ISA semantics.
• Provides a hardware test harness.

This brings security-critical systems phenomena such as virtual-memory management into the domain of programming-language semantics and foundational architecture verification.

ARMv8 as a Testing Target for gem5

The gem5 system simulator supports ARM CPUs. A 2025 ICSE-SEIP paper, "Search LLM-Based Testing for ARM Simulators" (SearchSYS), evaluates the correctness of gem5's ARM ISA implementation using LLM-generated test programs combined with AFL++-based fuzzing and differential testing against real ARMv8 hardware.

Research Questions

• RQ1: How effective are LLM-generated test suites at identifying bugs in gem5's ARM simulation?
• RQ2: How effective is SearchSYS at identifying bugs in the simulation of the ARM architecture performed with gem5 after fuzzing?

Hardware Setup

Experiments were run on two ARMv8 machines:

1. A CloudLab m400 machine with 64 GB RAM, ARMv8 64-bit architecture with a single socket, 2.4 GHz, 8 cores, 1 thread per core, running Ubuntu 22.04 ARM.
2. A UCL machine with 224 CPU cores (Cavium ThunderX2 CN9975, 2.0 GHz) and 130 GB RAM, ARMv8 64-bit architecture with 2 sockets, 28 cores per socket, and 4 threads per core, running Red Hat Linux (aarch64-redhat-linux-gnu). One ARMv8 machine is configured to be more strict, often initializing uninitialized local variables to zero.

Both machines used the same compiler versions adapted for ARM and the same tools as described in prior work.

Adapting SearchSYS to ARM

When adapting SearchSYS for ARM, minor script issues were encountered related to the linker and data from TinyLlama; the latter caused an early termination in AFL++ fuzzing due to memory errors. The ISA parameter was set to ARM. The example script provided by the SSBSE Challenge Track 2023 organizers was continued to be used. The Python configuration script required no modifications. The LLM test program generation step itself is CPU-agnostic, relying on GPU or OpenAI platform.

Bugs Detected from LLM-Generated Test Inputs

13 bugs were identified using LLM-generated inputs, six of which are unimplemented system-call functionality in the simulator. Four bugs were reported to the gem5 bug tracker under the "arch-arm" tag (#1527, #1544, #1547, #1629), three of which had already received detailed consideration by the gem5 development team. Bug #1629 was unexpected because it was previously believed to be properly implemented in ARM. Unimplemented issues were excluded from bug reports since they represent new test cases rather than valid bugs.

GPT-3.5-turbo found the highest number of issues (286), followed by Magicoder (175), Llama2 (102), CodeBooga (87), TinyLlama (22), and Phi2 (21). GPT-3.5-turbo (SSBSE 2023) had the fewest with only 5 instances. Two bug categories, panic and timeout, were identified by executing the simulator, totalling 74 test cases (9 panic and 65 timeouts), while an additional 624 bug-activating test cases were identified using differential testing.

The differences between GPT-3.5-turbo (SSBSE 2023) and the later GPT-3.5-turbo set are attributed to three factors: (1) the SSBSE 2023 dataset was generated in 2023 while the later set was generated in 2024, (2) the SSBSE 2023 set was trained on the LLVM test suite, likely overlapping with tests already evaluated by gem5, and (3) the SSBSE 2023 set used a few-shots approach instead of zero-shots, with zero-shots generally providing better input diversity and throughput.

(Columns: A=TinyLlama, B=Phi2, C=Llama2, D=Magicoder, E=CodeBooga, F=GPT-3.5-turbo, G=GPT-3.5-turbo (SSBSE 2023).)

Fuzzing Campaign Results

SearchSYS is an AFL++-based tool. Each fuzzing campaign ran for 24 hours, with five independent repetitions per minimized input corpus. The throughput results were calculated as the mean value across these five repetitions. The differential testing post-fuzzing (when gem5 differed from the real hardware) was done using the last repetition, comparing the gem5 simulation with ARM ISA to results obtained via the two ARMv8 hardware machines.

During 24-hour fuzzing, TinyLlama had the highest number of fuzzed test inputs exposing issues (87), followed by GPT-3.5-turbo (31), Magicoder (10), Phi2 and Llama2 (6 each), and CodeBooga (3). Most fuzzed sets identified 2-3 distinct issues, with GPT-3.5-turbo finding 4 distinct ones. GPT-3.5-turbo (SSBSE 2023) had the lowest bug-finding rate with only 2 instances, each of a different issue category. Three bugs (panic, out-of-memory, and timeout) were identified by executing the simulator, totalling 93 test cases (19 panic, 1 out-of-memory, and 73 timeouts), while an additional 126 bug-activating test cases were identified using differential testing. One fatal error in src/mem/port_proxy.hh readBlob initially appeared genuine but, after comparing results between the two ARMv8 machines, was determined to be a configuration issue on the UCL machine rather than an ARM ISA bug, and was therefore excluded from bug reports.

The highest throughput (in total) of fuzzed test inputs was achieved (on average) by GPT-3.5-turbo (SSBSE 2023) with 997 fuzzed test inputs, followed by Magicoder (986), Llama2 (948), GPT-3.5-turbo (888) and CodeBooga (839). The smaller LLMs had a lower throughput: TinyLlama (776) and Phi2 (680). Observing only the queue size (the fuzzed test inputs suitable for differential testing post fuzzing), Magicoder achieved the highest rate with 962 fuzzed test inputs, followed by GPT-3.5-turbo (SSBSE 2023) with 945 fuzzed test inputs.

ARMv8 vs X86 Stability in gem5

Contrary to expectations, ARM is no more stable than the X86 ISA in gem5: given the same set of test inputs, more of them exposed an issue on ARM (around 630 missimulation/error instances and over 60 timeout instances) than on X86 (around 530 missimulation/errors and 30 timeouts). The '530' and '30' instances on X86 are not a subset of the 630 and 60 instances on ARM and reflect different bugs. This higher incidence of issues on ARM suggests considerable challenges to achieving parity between ARM and X86 support within gem5.

Differential Testing Process

Differential testing scripts compared the result of the native run on ARMv8 machines against the simulation with a time out of 50 seconds and a memory limit of approximately 90 MB (stack size) for both the gem5 simulation and the native run. Mismatches between gem5 simulation with ARM ISA and real ARMv8 hardware were then manually inspected.

Test Inputs as Regression Tests

LLM test cases form a suitable contribution to gem5's C++ unit tests (regression tests): they have several lines of code, are efficient (commonly terminate in under 50 seconds), and are human-readable. With semi-manual filtering to remove tests triggering undefined behavior or those requiring complex input or producing large output, they can be grouped according to coverage and instructions triggered to target untested areas of the gem5 test suite. Test inputs exposing unimplemented features can be saved for future use, supporting test-case-driven development. According to gem5 developer feedback, SearchSYS's ability to tie test cases to particular features of the ARM ISA is of great importance, and could help with ongoing development of gem5 also for RISC-V and other ISAs.

Related Entities

• SearchSYS: An LLM-based testing tool for ARM simulators that evaluates ARMv8 simulation by generating test programs via LLMs and applying AFL++-based fuzzing and differential testing against gem5.
• gem5: A system simulator that supports ARM CPUs, whose ARMv8 ISA implementation was the target of SearchSYS evaluation.