SOURCE ARCHIVE
EXTRACTED CONTENT
120,442 chars RISCVuzz: Discovering Architectural CPU
Vulnerabilities via Differential Hardware Fuzzing
Fabian Thomas∗, Lorenz Hetterich∗, Ruiyi Zhang∗, Daniel Weber∗, Lukas Gerlach∗, Michael Schwarz∗
∗CISPA Helmholtz Center for Information Security
{fabian.thomas, lorenz.hetterich, ruiyi.zhang, daniel.weber, lukas.gerlach, michael.schwarz}@cispa.de
Abstract—The open and extensible RISC-V instruction set The available RISC-V CPUs implement the base ISA and
architecture marks a significant advancement in the CPU in- typically an additional selection of finalized ISA extensions,
dustry by enabling new vendors to enter the CPU market. such as compressed instructions or instructions for handling
RISC-V is quickly gaining popularity, as demonstrated by its floating-point numbers [16]–[18], [61]. Ongoing work also
support in the Linux kernel and its presence in consumer devices focuses on thoroughly testing the implementation of the fi-
and even cloud platforms. However, the flexibility of RISC-V nalized ISA extensions to ensure that CPUs implement them
has resulted in a diverse range of hardware implementations,
which differ in features and security measures. Additionally, no correctly [20]. However, RISC-V also supports vendor-specific
automated approach exists currently to assess the security of these custom ISA extensions that are already used, e.g., to imple-
implementations. ment cache-maintenance instructions in the T-Head XuanTie
C906 [22]. Worse, for most available high-end cores, no source
thatIn this paper, we introduce a novel framework, RISCVuzz, code is available. Thus, previous approaches that focus on
leverages this diversity of RISC-V implementations to finding vulnerabilities in RISC-V cores on an RTL level [23],
automatically detect vulnerabilities in hardware CPUs with- [26], [51] cannot be applied by researchers or other third
out the need for source code or emulators. RISCVuzz uses parties. Consequently, while crucial, it remains challenging to
a differential CPU fuzzing approach to compare architectural analyze the security of high-end RISC-V cores due to missing
behaviors across different vendors and CPU models. We evaluate
RISCVuzz using all 5 currently available consumer-grade RISC- documentation and source code.
V CPUs and identify 3 severe security vulnerabilities along with In this paper, we ask the following research question:
numerous bugs. Notably, RISCVuzz identifies GhostWrite, an
unprivileged instruction sequence to write attacker-controlled Can we leverage the inhomogeneity of RISC-V implemen-
bytes to attacker-chosen physical memory locations, including tations to automatically find architectural CPU vulnerabilities
attached devices. In 3 end-to-end attacks, we demonstrate how
GhostWrite can be transformed to read physical memory and without requiring the source code of the CPU?
lead to arbitrary machine-mode code execution, even in cloud
environments. Additionally, RISCVuzz exposes 2 unprivileged To answer this question, we build RISCVuzz, a differ-
“halt-and-catch-fire” instruction sequences that result in an ential CPU fuzzing framework for analyzing RISC-V CPUs.
irrecoverable CPU halt. RISCVuzz relies on the assumption that the architectural result
of every instruction has to be the same across different CPUs if
the instruction is supported. Additionally, ISA extensions can
I. INTRODUCTION be detected using the same approach, independent of whether
they are standardized or vendor-specific. An instruction from
RISC-V is still a young instruction set architecture (ISA). an extension manifests itself by showing different architectural
Nevertheless, there is considerable support for this ISA: The behavior across different CPUs. RISCVuzz executes instruc-
Linux kernel supports RISC-V CPUs in the upstream code, tion sequences with different parameters on different RISC-
and major compilers, such as GCC and Clang, support RISC- V CPUs, including emulators. These instruction sequences
V [39]. Furthermore, mainstream Linux distributions support can be as simple as single instructions to complex instruction
the RISC-V architecture, e.g., Ubuntu and Debian [41]. Be- chains consisting of multiple thousands instructions as shown
sides software support, there is a steadily rising number of in prior work [51]. Any deviation from the majority vote
RISC-V CPUs. While the first CPUs were mainly softcores concerning the output or side effect, i.e., system crash, is
designed for emulators and FPGAs [1], [8], there are already a reported as a potentially misbehaving instruction sequence.
small number of hardware cores available on the market [47], These instruction sequences require further manual inspection.
[54], [55]. These cores are used in single-board computers We evaluate RISCVuzz on 5 RISC-V hardware CPUs: T-
(SBCs) similar to the Raspberry Pi, as well as laptops [50], Head XuanTie C906/C908/C910, and SiFive U54/U74. These
[63], mobile phones [49], servers [42], [46], and gaming are all currently consumer-available hardware RISC-V CPUs
consoles [50]. running a 64-bit Linux operating system, and are used in
various devices. For the evaluation, we use 10 different devices
featuring these CPUs. Additionally, we evaluate RISCVuzz on
4 emulators. In total, RISCVuzz discovers 3 architectural CPU
vulnerabilities and numerous bugs. On the T-Head XuanTie
C910 CPU, RISCVuzz discovers a successfully-executing in-
struction that leads to a segmentation fault on other CPUs.
Further manual analysis reveals a CPU vulnerability that with previous work on software fuzzing [33]. We emphasize fully breaks integrity by providing a write-anything-anywhere that our approach is orthogonal to RTL fuzzing, covering primitive for physical memory to unprivileged users. We dub scenarios RTL fuzzing cannot. However, even if the source this vulnerability GhostWrite for easy reference, as it fully code is available, RISCVuzz might find bugs introduced by circumvents virtual memory and caches, making it also in- the synthesis that are invisible in the RTL. Hence, we argue visible in performance counters. GhostWrite is in the non- that post-silicon fuzzing is a valuable extension of existing compliant implementation of the high-order strided vector- pre-silicon fuzzers [23], [26], [43], [51]. Another insight is store instructions (vse128.v to vse1024.v). The other that the simplicity of RISC-V does not prevent bugs, but 2 vulnerabilities belong to the class of “halt-and-catch-fire” prevents mitigating them as done in x86 CPUs using microcode CPU vulnerabilities [10] that can be used for unprivileged updates [5], [34], [35], [38], [66]. denial-of-service attacks on the CPU, one on each of T- Head XuanTie C906 and C908. One of these vulnerabilities Contributions. We summarize our contributions as follows. is in the vendor-specific XTheadMemIdx extension, which • We present RISCVuzz, a differential CPU-fuzzing frame- provides additional memory operations such as increment- work for finding CPU vulnerabilities on hardware RISC-V address-before-load (th.lbib). The other is a range of CPUs that does not require access to the CPU source or broken vector instructions that halt the CPU core. RISCVuzz an emulator, enabling vulnerability discovery on high-end discovers these vulnerabilities fully automated by observing cores. hangs of these CPUs while the instruction sequence succeeds • We automatically test 5 different off-the-shelf RISC-V on other CPUs. Additionally, we discover numerous architec- CPUs and 4 emulators and discover 3 severe CPU vul- tural bugs in CPUs and emulators, most of them within seconds nerabilities on real-world-deployed CPUs: an unprivileged of fuzzing. These include address-handling bugs, decoder bugs, arbitrary physical write primitive we dub GhostWrite and ISA incompatibilities, and fault-reporting issues, as well as 2 unprivileged “halt-and-catch-fire instructions”. segmentation faults in the two latest major versions of QEMU. • In 4 end-to-end attacks, we demonstrate the impact of the We demonstrate the security impact of our findings in vulnerabilities by reading and writing arbitrary memory 4 case studies. On the T-Head XuanTie C910, we build an and executing code with kernel- and machine-mode priv- end-to-end attack with GhostWrite that allows unprivileged ileges, fully breaking the confidentiality and integrity of users to read and write arbitrary memory, including machine- • these systems, including cloud setups. mode code and devices mapped via MMIO. Additionally, we We discover numerous additional architectural bugs in the build two end-to-end attacks escalating privileges to root and tested CPUs as well as in emulators within seconds of machine mode by using GhostWrite to inject and execute fuzzing. code in supervisor- and machine mode. As a third end-to-end Responsible Disclosure. We reported all the security-critical attack, we show how GhostWrite can be used on cryptographic vulnerabilities on the C906, C908, and C910 to T-Head. keys to mount an ineffective fault attack [9], [11], [65], fully They acknowledged and reproduced GhostWrite and the C906 recovering a 2048-bit RSA key within 30 min. We demonstrate CPU-halting instruction sequence. We have no answer yet for that this vulnerability can also be exploited by unprivileged the C908 CPU-halting instructions. Further, we also reported users in the cloud by successfully testing it on the Scaleway GhostWrite to Scaleway since they offer C910-based bare- TH1520 instances. For the instructions that halt the CPU, metal machines in the cloud. Scaleway reproduced our findings we demonstrate that they can be used by any unprivileged and is currently in the process of giving out instructions to application and also work from inside Docker containers. customers for manually rolling out kernel patches that disable The only mitigation we identify for the bugs in the vec- the vector extension, mitigating GhostWrite. Additionally, we tor extension, e.g., GhostWrite, is disabling the extension, reported a segmentation fault that is present in the latest which breaks applications using it. In a benchmark using version of QEMU. rvv-bench, we measure an overhead of up to 77 % when Availability. We will open source the RISCVuzz framework this mitigation is active. Thus, in spite of preventing exploita- and our reproducers with acceptance of the paper. We provide a tion, it is not a practical solution for entities relying on the preliminary version of these artifacts at the following address: vector extension. For the C906 CPU-halting bug, we find no https://anonymous.4open.science/r/riscvuzz-artifacts-116D mitigation since the responsible vendor extension cannot be disabled. II. BACKGROUND Our results provide interesting insights into the current state This section covers the relevant background required for of hardware RISC-V CPUs. Vendor extensions and “rushed” the remainder of the paper. implementations of non-finalized extensions do not only lead to bugs but also to exploitable security vulnerabilities that A. RISC-V are difficult to mitigate. Our results indicate that the base instruction set is tested significantly better than complex RISC-V is an open instruction set architecture (ISA) de- extensions, such as vector extensions. An additional insight veloped by the RISC-V foundation. The RISC-V ISA consists is that even for open-source cores, such as the C910, the of a core instruction set that must be implemented by all hardware implementation differs from the released source. RISC-V CPUs and extensions that can be implemented as RISCVuzz discovers most bugs and vulnerabilities within sec- needed [59], [60]. An example of such an extension is the onds, showing the efficacy of our approach. This is even true vector extension [17], which is implemented in the XuanTie for rather simple instruction-sequence generation, which aligns C908. In addition, multiple vendor-specific extensions, some
2
only vaguely documented, are added to RISC-V cores to equip to recover confidential information. In contrast to transient- them with additional functionality [16], [22], [56]. ISA variants execution attacks, architectural bugs are mismatches between for 32 bit, 64 bit, and plans for 128 bit addressing [59] are the CPU specification and implementation. For example, the available, making RISC-V suitable for a wide range of devices. Pentium F00F bug allows an unprivileged attacker to lock up Nowadays, RISC-V CPUs are not only available as soft cores an affected system by executing a specific instruction even for FPGAs but are already used in embedded devices [40], though the instruction encoding is invalid and should raise an SBCs [29], laptops [50], [63], and cloud computing [42]. exception according to the specification [10]. More recently, RISC-V devices are supported by upstream Linux [15] and these architectural bugs gained traction, with multiple CPU can run a variety of distributions, e.g., Ubuntu or Debian. bugs causing architectural data leakage [6], [35], [36], [66]. a) Privilege Levels: RISC-V has three privilege lev- III. M ETHODOLOGY els: User (U) for unprivileged applications, Supervisor (S) for operating systems, and Machine (M) to manage trusted This section describes the methodology of our work. We execution environments [60]. While only M is required to be introduce the main idea in Section III-A and outline challenges implemented, most non-embedded RISC-V CPUs implement in Section III-B. In Section III-C, we introduce the fuzzing all three privilege levels. Access to Control and Status Reg- targets used in the remainder of the paper. We provide imple- isters (CSRs) and privileged instructions is limited depending mentation details in Section IV. on the current privilege level. b) Paging: RISC-V supports virtual memory through A. Idea paging. With virtual memory, the single physical address space We rely on the basic assumption that the architectural for DRAM and memory-mapped I/O is isolated through virtual result of every instruction has to be the same across different address spaces. The mapping between virtual and physical CPUs. This assumption ensures that different CPUs have to addresses is defined in per-process page tables. A page table adhere to the ISA specification. However, this assumption is is a sparse tree-like data structure mapping virtual memory only valid for instructions supported on all the tested CPUs. blocks of fixed size called memory pages to physical memory Thus, we consider any architectural effect that differs between pages. The size of such memory pages is typically 4 kB [60]. In CPUs an instruction anomaly that has to be investigated addition to the physical address, page tables can store metadata further, excluding instructions only supported on one CPU. for each memory page, for instance, whether a memory page These instruction anomalies are likely due to a bug, or worse, is present as opposed to swapped to disk, whether it can be a security vulnerability in the CPU. accessed from userspace, or whether it is writable. The main advantage of this approach compared to previous CPU fuzzers searching for architectural bugs [23], [26], [51] is B. Fuzzing that we require neither source code nor golden models. Thus, Fuzzing is a soft- and hardware testing technique that this is the first approach that can automatically find vulner- provides randomly generated inputs to hardware or software abilities on closed-source CPUs, such as the T-Head C908. targets and checks that they behave in an expected way. While Moreover, running code on hardware CPUs is significantly fuzzing cannot prove the absence of bugs, it has been shown faster than emulating them. to be effective in finding bugs in soft- and hardware [23], [26], In addition to instruction differences, ISA extensions can be [43], [51]. A special case is differential fuzzing [32], where detected using our approach. This is true for both documented multiple targets implementing the same specification are tested and undocumented extensions if two CPUs implement differ- against each other. Each difference resulting from the execution ing extensions. An instruction from an extension manifests it- of identical inputs by different targets is considered a violation self by showing different architectural behavior across different of the specification by one of the targets. Differential fuzzing CPUs, e.g., by throwing an illegal-instruction exception on a has the benefit that no golden reference model, which correctly subset of CPUs. implements the specification, is required, as bugs manifest Finally, this approach can be extended from single in- through differing behavior. structions to instruction sequences. Although many previous CPU bugs do not require the interaction between multiple C. CPU Vulnerabilities instructions [6], [35], [66], some do [36], [51]. Thus, we While software-based side-channel attacks have been also compare the architectural effects of instruction sequences. known for decades [28], recent years have proven that more As our differential approach is generic, this addition does critical CPU vulnerabilities exist [27], [31], [35], [36]. A not result in conceptual changes but only in additional engi- prominent class of attacks are transient-execution attacks [7], neering effort. However, unlike single instructions, we cannot [27], [31], which abuse performance optimizations of mod- exhaustively test all instruction sequences. Thus we have to ern CPUs, such as out-of-order and speculative execution. select instructions pseudo-randomly, i.e., we rely on fuzzing While transient-execution attacks allow for leaking data from techniques for generating instruction sequences. various security boundaries [27], [58], they are restricted to B. Challenges read primitives. Transient-execution attacks do not violate the specification of a CPU but allow an attacker to execute While the idea of differential CPU testing (or fuzzing) is security-boundary-crossing instructions that are never com- quite intuitive, we identify multiple challenges in both design mitted to the architectural state of a CPU. However, those and implementation. Conceptually, there are the following 3 instructions leave microarchitectural traces that attackers use challenges:
3
a) C1: Sequence Generation: Executing every possible TABLE I: Overview of tested RISC-V boards and emulators.
instruction encoding in the 32-bit encoding space already We test 5 CPUs and 4 emulators. The CPUs come from 2
results in a large search space. This search space grows expo- different vendors.
nentially with the length of the instruction sequence. Hence,
randomly executing bitstreams is inefficient for exploring “in- Board CPU CPU Vendor Relevant Extensions teresting” effects. Additionally, if there is a difference between AB U54 SiFive - two CPUs in one instruction, such an instruction is reported multiple times if it encodes an immediate. As the difference U74 - is likely independent of the immediate, testing all possible C, D, EF C906 v0p7, zfh, XTheadMemIdx immediates wastes resources and inflates the result set, which G, H, I C908 T-Head v, zfh, XTheadMemIdx has to be checked manually. Hence, the challenge is finding C910 v0p7, zfh, XTheadMemIdx a trade-off between coverage of the encoding space and the Emulator Version Relevant Extensions number of tested instructions. In Section IV-C1, we describe A QEMU 6.2.0 - how our proof-of-concept implementation RISCVuzz solves B QEMU 7.2.0 v that challenge by using a bottom-up approach to gradually C QEMU 8.2.2 v increase the search space using instruction types inferred from D QEMU 9.0.0 v instruction encodings. b) C2: Non-deterministic Effects: Comparing the archi- C908 supports the ratified RISC-V vector extension (v), while tectural effects of instructions requires that these effects only C906 and C910 use a pre-ratified draft version 0.7.1 (v0p7) depend on factors we can control, e.g., memory and register supported by vendor-provided kernels and toolchains. Further, content. Unfortunately, this is not the case for all instructions. we test on different QEMU versions from version 6 (default Some instructions provide internal values of the CPU, such on Ubuntu 22.04) to 9 (newest) (cf. Table I). as performance counters. These values often depend on the CPU state and previous instructions executed on the core and IV. RISCVUZZ FRAMEWORK can thus not be controlled. Finally, memory reads from certain addresses, such as Linux vDSO [21] return values that are out In this section, we describe RISCVuzz, our proof-of- of the control of the testing framework. All these cases have to concept implementation of the methodology described in Sec- be considered to avoid false positives, i.e., reporting different tion III. Specifically, Section IV-A describes the design of behavior across CPUs even though the instructions have the RISCVuzz. Section IV-B and Section IV-C discuss relevant same behavior. In Section IV-B1, we describe how we prevent design and implementation details of the client and server the reporting of non-deterministic effects by minimizing the components of RISCVuzz, respectively. sources of non-determinism and ignoring the results of the remaining non-deterministic instructions. A. Design Overview c) C3: Test-framework Integrity: The test framework RISCVuzz uses a centralized design: A server orchestrating has to record the architectural effects of instruction sequences. the testing, and the clients, i.e., RISC-V CPUs, are connected Thus, from a high-level perspective, architectural states, such to the server. We reduce the task of the clients to a minimum as register and memory content, must be saved before and after due to resource constraints on the clients. A client receives test executing the sequence. While this is relatively easy for many cases, i.e., instruction sequences plus input, over the network, instructions, some instructions need special care to ensure the runs them, and reports back the resulting state e.g., register integrity of the test framework. These instructions include values and changed memory contents. The server is responsible those that change the control flow, e.g., calls and (conditional) for generating the test cases, distributing the cases to the jumps, those that change the CPU behavior, e.g., CSR writes, clients, collecting the results, and analyzing the results. and those that change the stack pointer or stack content. The server-based approach has different advantages over a Hence, the test framework implementation has to handle all decentralized approach. First, the clients typically have limited corner cases that would change the saved architectural states storage, making storing the architectural effects of all tested or the internal state of the test framework. We describe the instructions difficult. With a 32-bit search space for single implementation details of this approach in Section IV-B2. instructions, the resulting states require multiple gigabytes of C. Fuzzing Targets storage. Second, the CPU processing power of the clients is also constrained, leading to non-negligible overhead for While the number of silicon RISC-V cores is still limited, generating the test cases in addition to executing them. As we test all widespread commercially available 64-bit RISC-V we aim to test instructions as fast as possible, we delegate all cores that support booting a Linux distribution. At the time of resource-intensive tasks to more powerful CPUs. Finally, with writing, there are 2 manufacturers of silicon RISC-V CPUs— a central server-based solution, we can compare the results of SiFive and T-Head Semiconductors. We test on 2 SiFive and the tests in parallel and do not have to wait for the completion 3 T-Head CPU models as listed in Table I. All tested CPUs of the test runs on all machines. are mounted on single-board computers. We also use the C910 in the Scaleway cloud [42]. The targets run various operating B. Client systems such as Ubuntu or Debian (cf. Table III in Section A). The client is a runner of server-provided test cases. It is All tested cores support at least the base ISA, the standard implemented in a mixture of C and RISC-V assembly. To extensions, and compressed instructions, i.e., RV64GC. The execute test cases, the client sets the registers as specified,
4
C906: SIGSEGV
C910: NOSIG
user input vsetvli x0, x0, ... C906 SIGSEGV instr seq: C908, U54, U74: SIGILL
ext=rv64gcv0p7 vsetvli x0, x0, ... vse128.v t0, 0(t0) t0: 0x4000 ... run on other
seq len=3 vse128.v t0, 0(t0) li t0, 2 distribute collect machines
li t0, 2 NOSIG regs:
t0: 0x4000 t0: 2 t0: 0x4000 init
load regs from repro
generate instruction generate register C910 if diff, C reproducer:
sequence inputs generate repro run sequence
print register diffs
Fig. 1: Overview of RISCVuzz. The user selects the enabled extensions and sequence length. The server generates an instruction
sequence and register inputs for the clients and compares the results. Differences are logged as a reproducer file.
runs the provided instruction sequence, and reports the results RISCVuzz has to ensure the integrity of its internally used to the server. In case of instructions that access arbitrary registers, memory regions, and control flow. Figure 2 illustrates unmapped memory, execution is interrupted by a segmentation the design of the integrity-providing sandbox we discuss in the fault. Since RISCVuzz should compare memory differences following. too, RISCVuzz incorporates a routine for mapping these un- available pages. When the client detects a segmentation fault, a) Registers: For the integrity of registers, we employ it tries to map two pages, the page causing the fault and the two routines that ensure the register state is saved and restored subsequent page to handle corner cases where accesses might correctly. When a target instruction triggers a signal, such as span two pages. This lazy mapping of pages has the advantage an illegal instruction, the kernel saves the architectural state of that only a small amount of memory has to be scanned for the CPU. We simply copy this state and return to the execution modifications caused by the instruction sequence. The mapped loop with a longjmp, which restores the register state. When pages are filled once with all ‘0’s and once with all ‘1’s. This no signal is triggered, we use an instruction sequence similar is to detect writes of only ‘0’s or only ‘1’s respectively too. to the one used by the kernel when handling an interrupt. This This procedure is repeated until a threshold is reached1 or until sequence saves/restores every register to/from memory before no segmentation fault is triggered anymore by the instruction and after executing the instruction sequence. sequence. After mapping the pages, the testcase is restarted. The differences on all memory pages are recorded and included b) Memory: Memory state is the second dimension to in the results. protect. While single instructions can only access memory As discussed in Section III-B, RISCVuzz faces two chal- close to the current register contents defined by the inputs, in- lenges: non-deterministic instruction effects (C2) and the in- struction sequences, and non-RISC instructions can arbitrarily tegrity of the client (C3). In the following, we discuss how we shift these inputs and access any memory. Further, we want to tackle these challenges in the design of RISCVuzz. be able to provide various inputs, such as ‘-1’, via the registers.
C2: Noise Removal: Since we assume every difference Shifting the value ‘-1’, which results in the register value in register values to be a bug, we must ensure this does not 0xffff..., results in addresses potentially pointing to the happen randomly. The first step in this direction is using static program stack. We leverage the large virtual address space to compilation. Static compilation removes noise from differences “hide” the internal data in a region that is difficult to overwrite that occur in shared libraries and ensures that we can run the accidentally. The kernels we run on our CPUs all use the Sv39 same binary on all machines. We use Nix [13] for the static paging mode [60], i.e., 512 GB of virtual space is available for building to ensure that our builds stay reproducible across “hiding”. Consequently, we move the data section to a “safe” different machines employed for compilation. Reproducibility region of memory that we experimentally determined. Further, is essential since any change in the binary layout of the we switch to a new stack that we set to this region as well. client implementation could introduce new noise or break While this approach is heuristic, it works well in practice. framework integrity (C3). Additionally, we unmap the vdso, c) Control Flow: The third dimension is control flow. vdso_data, and vvar sections since loading from them No instruction sequence must jump out of RISCVuzz’s logic or results in differing values between kernels. Another noise infinitely lock up the client. We embed placeholder instructions source are instructions that naturally introduce noise, such as into a padding of ebreak instructions to ensure that control the rdcycle instruction. To remove these noise sources auto- flow is restricted to our path. These placeholder instructions are matically, we execute each instruction in the space of selected replaced in memory by the runner code. The ebreak padding extensions twice and exclude those that do not produce the ensures that the generated signal data is accurate. Padding with same result. nops would hide the target of relative jumps and branches
C3: State Protection: As RISCVuzz executes arbitrary since the runner would fall through to the last instruction after code sequences, these code sequences can modify its internal the nop-sled. We again rely on the size of the virtual address state. Thus, RISCVuzz has to protect its internal state to space to “hide” the client logic. We move the sandbox to its ensure a correct reporting of instruction effects. Specifically, own memory range and use absolute jumps to get into and out of the runner sandbox. Two instructions in the sandbox load 1For our tests, we use 10 as a threshold which we experimentally found to the last two registers, which are needed for the absolute jump yield good results. and as the base for loading the registers.
5
patch code abs. jump break chain TABLE II: Distribution of 4-byte RISC-V instruction space prime alarm load regs as documented by RISC-V Opcodes. All ratified (official) save registers <nop> save perf counters ... seq len parts of the ISA cover 84.03 % of the instruction space. The load registers <nop> vector extensions and T-Head extension cover only small parts on signal break chain of the instruction space. Overall, 85.51 % of instructions are abs. jump specified, the rest (14.49 %) are unknown or not specified. unprime alarm save registers copy registers save perf counters ISA part Percentage store signal meta restore registers unprime alarm Ratified + unratified 85.02 % return results Ratified 84.03 %
Fig. 2: The runner sandbox includes nop placeholders sur- Vector extension (v) 1.05 %
rounded by ebreak instructions. The runner patches the Vector extension (v0p7) 0.81 %
placeholder instructions, primes the alarm, saves registers, T-Head vendor extension 0.39 %
loads the supplied fuzzing registers, and jumps into the sand- Overall known 85.51 %
box. The executed instruction sequence either returns via the
runner, or a signal, and returns the results.
first fuzz only the undocumented space ( 3 ). This drastically
Instruction Space shrinks the search space by 85.51 % (Table II). Note that we
3 RISC-V Opcodes do not have to separate ISA extensions exactly. The encoding-
based filtering is only a rough but deterministic guidance
2 Vendor unratified 1 ext. C technique to prevent the fuzzer from wasting resources on
instruction encodings that mainly consist of instructions with
documented large immediate encodings. We still strive to cover the entire
ext. V ext. Zfh 4-byte instruction space of RISC-V but want to focus on more
C906 halting GhostWrite promising parts first. In the following, we describe in more
instructions C908 halting instructions detail how this bottom-up approach works.
Fig. 3: Overview of the RISC-V instruction space. RISC-V a) Instruction Classification: We use the official RISC-
Opcodes ( 1 ) covers most parts of the RISC-V ISA specifica- V Opcodes repository [19] for building our filters since it en-
tion. The specification reserves parts of the address space for codes all standard RISC-V instructions in a machine-parsable
custom vendor extensions ( 2 ). Other parts are either reserved format. Further, it clusters the instructions into their respec-
for future use or unclaimed ( 3 ). The dots describe where our tive extension. Moreover, it includes some of the unratified
discovered bugs are in the instruction space. extensions.
b) Instruction Exclusion: Encoding-based filtering also
allows for excluding instructions or entire instruction classes,
We handle infinite loops by priming an alarm before such as CSR-based instructions. As these instructions can
jumping into the sandbox. This alarm interrupts the client change the behavior of instructions on the current core, they
after a configurable timeout, breaking out of any loop. The require extra handling to avoid introducing false positives in
ebreak padding further minimizes the chance of infinite the differences. We leave the coverage of these CSR-based
loops compared to padding with nops. instructions to future work.
c) Instruction Selection: The server randomly chooses
C. Server instructions from the instructions used for fuzzing and assem-
The server is the pivot of our setup. Figure 1 illustrates the bles them based on the documented encoding. We generally
logic of the server. It generates instruction sequences and input initialize immediate parts of the instruction with values that
registers and sends these to the clients (Section IV-C1). The might lead to corner cases, e.g., ‘-1’ or ‘0’. To ensure that
clients run the test cases and report the resulting architectural any instruction encoding can be achieved, though, we mix in
states to the server (Section IV-C2). The server compares these a random immediate in 1 out of 8 cases. For the rest of the
states and logs differences as simple reproducer files, which fields, including register fields, we provide the required number
can be used for further analysis in a simple C program or by of random bits.
running the reproducer on other machines (Section IV-C3). Not defined or missing instructions in RISC-V Opcodes
1) C1: Sequence Generation: Instead of generating random are chosen by generating a random 4-byte value that cannot
4-byte sequences, we use a bottom-up approach based on the be decoded to a valid instruction. For any non-documented
instruction encoding to gradually increase the covered instruc- instruction, RISCVuzz does not have to fill any bitfields, as
tion space. Based on specific bits in the instruction encoding, a full instruction encoding with all bits set is already chosen.
RISC-V allows to classify the type of instruction and whether Thus, RISCVuzz covers the entire instruction space with this
an instruction is a standard instruction ( 1 ) or a vendor-specific approach.
instruction ( 2 ), leading to groups as illustrated in Figure 3. 2) Input Distribution: The server distributes the generated
This approach allows selectively including and excluding ISA fuzzing inputs and collects the results (cf. Figure 1). Since
extensions in our tests. Consequently, this makes it easy to we want to achieve high throughput fuzzing on the clients (cf.
6
sandbox
Section V), we implement 3 optimizations for the transfer. U54
First, we only send back registers and memory contents that U74
changed during the execution of the instruction sequence. C906
Second, we restrict ourselves to a list of values of register C910
contents. This list of values is shared between client and server. C908
The server then transmits one byte to select the respective value
on the client side. This reduces data consumption from 4 to 0 10 20 30 40 50 60 1000
1 byte per register for general-purpose registers and saves 15 s
bytes per register for vector registers. Third, we send batches of Fig. 4: Relative performance of each tested CPU. The C908 is
fuzzing inputs and ensure enough fuzzing inputs are buffered by far the fastest CPU in our test. The C910 and U74 perform
in the client. These optimizations ensure that the clients are similarly. C906 and U54 are the slowest CPUs.
never idle and the network is efficiently used.
3) Logging Differences: In the next step, the server com-
pares the collected architectural states. If it finds a difference, FPU, or reset performance counters to a shared state. Running
it logs a reproducer file for the fuzzing input. This reproducer these binaries on a Linux host is not possible by design
can then rerun the fuzzing input on selected machines or and requires significant modifications to the design of ELF
automatically create a simple C-based program consisting of generation, e.g., adapting the exception handling approach to
the instruction sequence and the register inputs (cf. Figure 1). userspace logic. Further, running these bare-metal ELF binaries
This reproducer file can then be compiled and executed as one by one on a CPU is an entirely new problem. For example,
a standalone binary on any RISC-V CPU for further manual on some boards this would require removing the microSD card,
analysis. copying the new ELF binary onto it, inserting it, and powering
While RISCVuzz only adds the necessary code to the on the board again for every testcase.
reproducer, i.e., the instruction sequence and architectural state Takeaway Testing on hardware cores is orders of magnitude
initialization, the reproducer is not necessarily minimal. How- faster than on emulated cores.
ever, in practice, the reproducer is typically small enough for
an analysis. Still, if necessary, program reduction techniques, 2) Multi-Core Scaling: Fuzzing throughput can be im-
such as those discussed by Solt et al. [51], can be used to proved by deploying fuzzing clients to more than one CPU
reduce the reproducer further. core. Such parallelization relies on the fact that each fuzzing
corpus should not impact the others. Thus, they can be
V. EVALUATION arbitrarily distributed to cores. Switching to 2 cores on the
s
In this section, we evaluate RISCVuzz. We evaluate the C910 nearly doubles performance from 39 994 to 71 825 instr. ,
while employing one more core again improves performance
general performance of testing instruction sequences (Sec- but only by half of the increase we see when going from 1
tion V-A), summarize the findings of RISCVuzz (Section V-B), to 2 cores (92 943 instr. ). Adding the last core only marginally
including the most severe finding, GhostWrite, and evaluate improves performance because the server tops ats 97 207 instr. .
how long it takes RISCVuzz to discover our findings (Sec- As the C906 is a single-core CPU, we can instead use multiples
tion V-C). CPUs, which comes close to a linear increase in throughput.
A. Fuzzing Performance nearly resembling a linear increase in throughput.
This section focuses on different performance metrics of Takeaway Multiple cores can be used to increase fuzzing
RISCVuzz. All experiments use 1 core of an Intel Core i9- throughput. For single-core machines, multiple machines can
13900K as the server. be joined to achieve the same effect.
1) General Throughput: To assess the general throughput 3) Sequence-Length Scaling: We evaluate the impact of the
of RISCVuzz, we benchmark a fuzzing run with only the sequence length on the performance. If there is no exception,
base ISA with 1 client instance on each of the target CPUs we rely on the retired instructions counter to collect the number
(cf. Section III-C). We test 18 194 (C906) to 59 205 (C908) of executed instructions. Otherwise, we infer the number from
instructions per second on average. Figure 4 provides the the program counter at which the signal is triggered. When the
results for all CPUs. Comparing the results to the fastest state- trapped program counter is not in bounds of the sandbox, we
of-the-art RISC-V RTL fuzzer Cascade [51] shows that fuzzing assume that only 1 instruction was executed. This can happen
on hardware cores is orders of magnitude faster. Cascade when a jump or branch instruction is executed.
achieves 2181 = 9 instructions per second. We divide the
256 Figure 5 shows the result of increasing the sequence length
reported throughput by 256 since the Cascade evaluation uses on the C906. The performance increases up to a sequence
a 512-core machine while we only use 2 cores (client and length of 5 and then gradually decreases with further increasing
server). However, Cascade already uses a sequence length of sequence length. The results are as expected since, at some
10 000 to achieve this throughput, while we use a sequence point, increasing the sequence length only rarely leads to more
length of 1. instructions executed per iteration, as some earlier instruction
Note that we cannot evaluate the bare-metal performance of might already raise an exception (cf. Section C). The added
Cascade. RTL fuzzers such as Cascade [51] or DifuzzRTL [23] network overhead of sending one more instruction absorbs
generate bare-metal ELF binaries, i.e., binaries that use privi- this slight increase and decreases performance overall. A
leged instructions to setup the interrupt vector table, enable the sequence length of 3 is a good tradeoff since adding more
7
CPU
30 a hard reset. On the C906, C908, and C910, a load to a
1000 20 non-canonical address is stuck until an interrupt arrives if the s 10 canonical part of the address is a valid address. RISCVuzz 0 generates such addresses if any upper bits of a valid virtual 1 2 3 4 5 6 7 8 9 seq. len address are modified by an instruction, e.g., by an xor, before Fig. 5: The fuzzing performance on the C906 increases up to the load happens. sequences of 5 instructions, then falls gradually with increasing b) Decoder Bugs: RISCVuzz discovers decoding bugs sequence length. on different hardware CPUs and in emulators. We find CPU- halting instruction sequences on the C906 and C908 that we suspect to be decoder issues (cf. Section VII). On the instructions only slightly improves performance while causing C906 and C910, RISCVuzz discovers fence and fence.i more congestion on the network, potentially hindering other instructions that raise an illegal-instruction exception, although clients from receiving data. they are valid according to the ISA specification. The RISC- Note that Cascade uses advanced techniques for generating V standard reserves these instructions for “finer-grain fences longer valid instruction sequences [51]. Although we suspect in future extensions” and dictates that “implementations shall such techniques would improve the throughput of RISCVuzz, ignore these fields” [59]. Conversely, RISCVuzz discovers we leave improving the sequence generation for future work. instructions that do not raise such an exception although they are invalid. For example, the C906 and C910 execute B. Findings the half-precision floating-point instructions fsqrt.h and fmv.x.h even when the rs2 field is = 0 [18]. Finally, for In this section, we summarize the findings of RISCVuzz. the latest versions QEMU 9.0.0 and QEMU 8.2.2 (Emulator We categorize the findings into address-handling bugs, decoder D and C), RISCVuzz discovers that cache-block management bugs, ISA incompatibilities, and fault-reporting issues. We instructions such as cbo.inval crash QEMU with a seg- discuss the findings with the highest security impact, i.e., mentation fault. For QEMU 7.2.0 (Emulator B), RISCVuzz GhostWrite and the C906 and C908 CPU-halting vulnerabili- discovers that truncating vector conversion instructions such as ties, in more detail in Section VI and Section VII, respectively. vfncvt.rtz.x.f.w crash QEMU. However, as the crash Figure 3 visualizes where RISCVuzz finds the most severe is due to an assertion, we do not expect that this is further bugs. GhostWrite is in the vector extension. The C908 halting exploitable. instructions are illegally encoded vector instructions close to c) ISA Incompatibility: The C906 and C910 are not but outside the vector extension. The C906 halting instructions fully compatible with the ISA specifications. These CPUs do are on the edge of the documented vendor extension since they not ignore writes to bits 8 to 10 of the fcsr register. Both the use an edge case in the instruction encoding. C910 and the C908 support a subset of the vector extension. Note that RISCVuzz automatically finds architectural dif- This manifests itself in some of the instructions doing nothing, ferences that are in most cases bugs. The analysis whether others doing unexpected things (cf. Section VI), and some these differences are security vulnerabilities still requires a not being implemented at all. Interestingly, the subset of manual analysis. However, due to the small reproducers created instructions also differs between the two CPUs. by RISCVuzz, the manual analysis is in many cases relatively d) Fault-reporting Issues: On all tested CPUs, quick. GhostWrite produces differences when fuzzing the 0.7.1 RISCVuzz discovers bugs during fault reporting. Overall, vector extension between C906 and C910. While the illegally- there are various inconsistencies in the raised signal for encoded vector-store instructions generate a segmentation fault exceptions. SiFive CPUs tend to raise bus faults, whereas on illegal memory addresses, the C910 generates no exception. T-Head CPUs raise segmentation faults. Additionally, the We provide an example of such a difference logged into reported program counter of the fault and the faulting address a reproducer file in Section D. During manual inspection are not always correct. On the C910, the reported address for of the instruction behavior, by varying register values, we faults is rounded up to the next multiple of 16 if the address observe kernel crashes when passing addresses in the physical modulo 16 is larger than 8. The C908 reports segmentation kernel range which motivates further analysis of these faulty faults for valid non-aligned addresses, where the correct instructions (cf. Section VI). The C906 and C908 CPU-halting behavior is to raise a bus error. instruction sequences directly crash the fuzzing client, there- fore no further analysis of such reproducers is needed as such a C. Time to Bug denial of service can always be considered a security problem. The other bugs we summarize below generate similar patterns, In line with other papers on fuzzing [43], [51], we provide i.e., either hangs or differences, which motivate further manual the fuzzing time to find the bugs. We use all extensions, i.e., inspection. all ratified and unratified extensions, during fuzzing. This is a) Address-handling: RISCVuzz finds different bugs the worst case for finding the bugs, since the only restriction around virtual address handling. The vse128.v instruction we pose on the fuzzing space is to use only documented on the C910 does not translate the provided virtual address instructions. We further test on a single core. Thus, the to a physical address but instead interprets it directly as a numbers from Section V-A1 apply. physical address, giving attackers a physical write primitive We find GhostWrite within the first second of fuzzing as no (cf. Section VI). Additionally, on the C910, reading from special encoding in the broken instruction is needed to make physically-backed virtual address ‘0’ locks the CPU, requiring the bug visible. Thus, the fuzzer only needs to select one of the
8
1 ; t0 = physical address, a0 = byte to be written yet, but it is “expected to be used to encode expanded memory 2 vsetvli zero, zero, e8, m1 sizes” [17]. The encoded effective element width contrasts what 34 vmv.v.x; v0, a0 we observe in practice, i.e., only one-byte stores. We further encoded: 0x10028027 5 vse128.v v0, 0(t0) test the 256-, 512- and 1024-bit encodings of the instruction and find that they behave the same, i.e., write only one byte. Listing 1: Code of GhostWrite. vsetvli and vmv.v.x set We further test the nf (number of fields) encoding of up the vector engine’s internal state and the byte to be written. the instruction, which controls how many fields are stored to The non-standardized vse128.v instruction (provided as memory. Increasing nf shifts the used source vector register machine code 0x10028027) performs the physical write. for the written byte by that exact amount. Thus, we only see the value of the last vector register in the group of fields. We suspect that the buggy instruction always writes to the 8 broken instructions out of 1283 possible instructions when same physical address, thereby dropping intermediate writes enabling all extensions as outlined above. of other field values that should normally be continuously We find the C906 halting instruction sequence bug within visible in memory. To further test this hypothesis, we measure the first 10 s. The slightly longer time to bug can be explained the cycles the instruction takes to execute while varying the by lower fuzzing throughput on the C906 (cf. Section V-A1) nf field and find that the instruction takes linearly more time and by slightly more involved conditions that the broken to execute. This observation strengthens the hypothesis that instruction needs to satisfy, e.g., using the same registers in multiple writes are scheduled, one for each field, but only the the encoding of the instruction. On the C908, RISCVuzz finds last one is visible since every write goes to the same address. the undocumented CPU-halting instruction in under 15 min of 2) Memory Interaction: Based on the observed effects, iterating over the undocumented space. we hypothesize that the instruction entirely circumvents the For the other documented instruction findings, the time to cache, directly writing to memory. We back this hypothesis bug is typically below 1 s. However, for some findings, fuzzing using a series of experiments. We set up a base experiment times of up to 30 s are required to reveal them. in which we initialize a memory address V, backed by the physical memory address P, with a known value x1. Next, we VI. GHOSTWRITE: WRITING ARBITRARY PHYSICAL perform operations to ensure that the target memory address, MEMORY i.e., V, is in a specific state before overwriting P with value In this section, we analyze GhostWrite, the arbitrary phys- x2 using GhostWrite. Afterward, we check whether a memory ical write primitive RISCVuzz finds on the C910. In Sec- read from V returns the original value x1 or whether it was tion VI-A, we reverse-engineer the prerequisites and microar- overwritten by the write gadget returning x2. Our experiment shows that if V is flushed or evicted from the CPU cache before chitectural properties, showing that GhostWrite can determin- the write gadget is executed, the primitive works in 100 % of istically write attacker-chosen values to attacker-chosen phys- the test cases (n = 10 000). We observe that if the memory ical addresses with byte granularity. Sections VI-B to VI-D at V is cached in a non-dirty cache line before we use the demonstrate 3 end-to-end attacks as case studies, using write gadget, we need to evict or flush it from memory to GhostWrite for reading and writing arbitrary physical memory make x2 visible. Once the memory is no longer cached, we and executing arbitrary code with kernel- and machine-mode successfully read x2 in 100 % of the tests. If the memory at privileges, fully circumventing virtual memory. V is cached in a dirty cache line, after flushing or evicting the cache line, the previous value x1 remains in memory. These A. Analysis observations lead to the hypothesis that GhostWrite does not Listing 1 shows the assembly code of GhostWrite. write through the cache hierarchy but directly to the physical A vsetvli instruction sets up the vector engine state. memory without interfering with the cache state at all. This vmv.v.x moves the byte to be written to a vector register. hypothesis explains why dirty cache lines can reset the state The vse128.v instruction performs the actual write with a to x1, as their value is written back to main memory. To target address in a general-purpose register. further strengthen this hypothesis, we investigate the hardware performance monitor counters available on the C910. First, the 1) Instruction Encoding: The instruction disassembles to counter that keeps track of dTLB misses (mhpmcounter6) a vector unit-stride store instruction from the unsupported does not count any event during the execution of the write vector extension 1.0. This instruction should operate on virtual primitive. Thus, we conclude that no virtual mapping is being memory and store vector registers continuously to target virtual resolved upon execution of the write primitive. Second, even addresses stored in a general purpose register. We reduce the if V is uncached, the counter keeping track of memory writes instruction’s encoding to its minimal form and perform tests that miss the L1d cache (mhpmcounter17) and the L2d on each component of the instruction encoding. cache (mhpmcounter21) do not count events for our write The source registers encoded in the instruction work as primitive. This further strengthens our hypothesis that the write intended, though only 1 byte is written. The destination primitive does not interact with the cache hierarchy. register encoding also works as intended, besides interpreting 3) MMIO: GhostWrite can write values to any address in the address as a physical instead of a virtual address. The the physical address space, including memory-mapped input- encoding of the effective element width is 128-bit. Thus, the output (MMIO). We use GhostWrite to write values of ‘0’ and vector registers should be handled as 16-byte registers. Note ‘0xff’ to the first 8 bytes of the MMIO range on Board I [48]. that this encoding of 128-bit and higher is not standardized With a voltmeter, we verify that this changes the state of the
9
GPIO pins. This demonstrates that the instruction bypasses any C. Kernel and Machine Mode Attacks: From Write to Execute virtual memory mechanics and has full privileges. Our second case study demonstrates how an attacker can 4) Simulation: Although the C910 sources are released as use GhostWrite to gain arbitrary code execution in the kernel openC910 [54], the vector extension cannot be enabled, as it is and machine mode, thus elevating privileges. not part of the source. Any attempt to execute GhostWrite in the simulator fails. This aligns with discussions in the GitHub a) Threat Model: We assume that the attacker has repository mentioning that the “openC910 didn’t include the unprivileged native code execution on the target. We assume V extension because it wasn’t officially final yet” [54]. the attacker knows the physical memory layout of the kernel. Takeaway Even for open-source cores, the published source This assumption is viable in practice, as the physical memory is not necessarily the code used for synthesizing the hard- layout is highly predictable. Alternatively, an attacker can ware. Thus, opaque-box testing techniques are needed even employ an arbitrary read gadget (e.g., Section VI-B) to scan when the sources are public. for the kernel. For escalating privileges to root, our end-to- end exploit assumes the presence of a setuid binary such as sudo or su that uses the getuid syscall to determine B. Page-Table Attack: From Write to Read whether a user is already root and, therefore, does not need Our first case study turns GhostWrite into an arbitrary read to authenticate. To gain code execution in machine mode, we gadget by rewriting page-table entries. assume the presence of OpenSBI with a known version and physical memory location. a) Threat Model: We assume that the physical memory b) Attack: To gain code execution in the kernel, the and kernel configuration is unknown to the attacker. We only attacker uses GhostWrite to overwrite the code of a system assume unprivileged native code execution on the target. call handler. Then, the attacker triggers the corresponding b) Attack: Our attack is inspired by the first privilege syscall to execute the injected payload. Since virtual memory escalation using Rowhammer [44]. We fill the entire available is completely bypassed by GhostWrite, the same principle physical memory with page tables by allocating large amounts can be used from containers or virtualized environments to of virtual memory. Specifically, we map the same file numer- attack a known host. In our proof-of-concept implementation, ous times until the physical memory is exhausted. We need the attacker overwrites the getuid syscall to always return to spawn multiple processes to exhaust physical memory with ‘0’. On Linux systems, the user ID ‘0’ is reserved for the page tables, as current RISC-V kernels use the Sv39 virtual privileged root user. Then, the exploit executes the su setuid addressing mode, where virtual addresses only use 39 bit. binary. If getuid returns ‘0’, su assumes a user is already Filling the entire 512 GB of virtual address space only spawns root and skips authentication, leading to privilege escalation to 1 + 29 + 29 2 = 262 657 page tables, which is only around the root user. Once a privileged shell is obtained, the exploit 1 GB of physical memory. We use GhostWrite to overwrite one uses GhostWrite to restore the original getuid to not break of the two least significant bytes of the page frame number of any legitimate functionality of other applications and to purge a potential page-table entry (PTE). Given that the memory is any traces of the attack. filled with page tables, we choose a random address in the For code execution in machine mode, our second half of physical memory, in line with our analysis of proof-of-concept overwrites parts of the function the distribution of page tables in physical memory (Section E). sbi_ecall_base_handler, which handles ecalls in If, after the modification, one of our page mappings does not OpenSBI. Since our previous attack enables code execution map to the initially mapped file anymore, we know that we in the kernel, we assume an attacker can trigger arbitrary successfully overwrote a PTE. We verify that by reading from SBI ecalls. In our proof of concept, we patch the ecall every mapping and comparing the read value to a fixed marker handler for SBI_EXT_BASE_GET_MVENDORID to return value. Note that we need to evict the TLB before employing 42 and verify the return value using a kernel module. In a this scan since the TLB might shadow our PTE modification. real-world scenario, an attacker could place arbitrary code at When no such virtual address is found, we write the PFN byte any physical address and patch a jump to their payload into on a different physical page. the ecall handler. Once such a virtual page is found, we know that we have c) Evaluation: We evaluate all attacks successfully on full control over a PTE and its corresponding virtual address. the 3 C910 boards (G, H, and I). The attacks take less than Thus, we can rewrite the PFN to any physical address to read 1 s, since the addresses are known and only a physical write the content of the address. We can also change the permission with GhostWrite is needed. The OpenSBI binaries, implement- bits in the PTE to provide write access if required. ing the machine-mode functionality for RISC-V systems, are c) Evaluation: We run the attack successfully on 3 mapped at physical address ‘0’ on the C910-based systems. different CPUs, Board G, Board H, and Board I, with 3 The kernel code and data follow at 0x200000. We verify different DRAM configurations, 4 GB, 8 GB, and 16 GB. The that this layout is stable across reboots. As GhostWrite always attack also works from within a Docker container. We evaluate works (Section VI-A), and the physical layouts of kernel and the attack on Board H with 8 GB of memory. We reboot OpenSBI do not change, the exploit’s success rate is 100 %. the machine between each run of the attack. Our attack is successful in all 20 tries, resulting in a success rate of 100 %. D. Ineffective Faults: From Write to Indirect Read The attack takes 32 to 94 s. 14 out of 20 times, overwriting the first randomly-selected address leads to a successful attack. In In our third case study, we demonstrate how GhostWrite the other cases, up to 3 addresses have to be tried. can be combined with ineffective fault attacks [9], [11], [65]
10
to read secrets such as cryptographic keys indirectly. We 1 th.lbib t0, (t0), 0, 0 demonstrate this by recovering TLS signing keys via an oracle 2 frcsr t0 that only exposes whether a signature was successful. 3 li t0, 0 a) Threat Model: We assume the victim runs a server Listing 2: The interaction of the th.lbib instruction with using OpenSSL with TLS 1.3 on the same machine as the the same register as source and destination, a CSR read, and attacker. The attacker acts as a malicious client, repeatedly an unrelated operation on the register halts the C906. connecting to the TLS server and triggering the handshake process. This scenario is practical when the attacker runs inside virtual machines or containers. The attacker can easily A. Analysis exhaust the physical memory, as described in Section VI-B. Consequently, we assume that the attacker can influence the a) C906: Listing 2 shows the instruction sequence physical address of the private key and one of the RSA-CRT halting the C906. The core of the sequence is the th.lbib parameters using memory massaging [30]. instruction from the custom XTheadMemIdx extension. This b) Attack: We target the TLS signing step in the vendor extension provides additional memory operations such OpenSSL library, explicitly focusing on scenarios where the as increment-address-before-load (th.lbib). The halt occurs server selects the RSA algorithm for digital signatures. In in combination with using the same register for source and des- TLS 1.3, the server must sign a hash that encapsulates the tination operand, a subsequent CSR read, and any subsequent handshake messages exchanged with the client before the interaction with the register provided to the instruction. In the handshake is finished. With such a signature, the client can example code, we read a CSR using the unprivileged frcsr verify the server’s authenticity. OpenSSL’s TLS implementa- instruction. However, any other instruction reading a CSR, tion employs the RSA-CRT (Chinese Remainder Theorem) op- such as rdcycle, can also be used. While the example uses timizations for efficiency. If faults happen during the signing, the load immediate instruction (li), the last instruction can leading to a wrong signature, the server attempts to re-sign be any instruction interacting with the used register. Unrelated using the traditional textbook RSA method. This additional instructions can be part of the sequence if they do not read verification step effectively mitigates Bellcore attacks [2], [4]. from or write to the used register (t0 in the example). To introduce a faulty signature, an attacker has to corrupt The T-Head vendor extension docs [56] mention that using both of the two signing algorithms. The attacker first corrupts the same register as source and destination is not a valid the parameters used for RSA-CRT employing GhostWrite. encoding. Compilers that support the vendor extension do That forces the server to start using the textbook RSA algo- not support compiling assembly code with this faulty encod- rithm. Then, the attacker corrupts the private key byte-by-byte ing. However, without the subsequent operations involving in physical memory. For each byte, the attacker attempts to a CSR and another operation on the register, no CPU halt establish an SSL connection 256 times, each with a different occurs. We further discover that 13 other instructions from the possible value written to that specific byte. Only if the writ- XTheadMemIdx extension are vulnerable (cf. Listing 3 in ten value is correct, the attacker observes a successful SSL Section B) in the same way. Surprisingly, the variants with 3 connection. The attacker uses this feedback as a side channel source registers are not vulnerable. to recover the private key. Similarly, the attacker can recover Takeaway CPU vulnerabilities exist for both single instruc- the parameters used for RSA-CRT by switching to iteratively tions and instruction sequences. modifying the content of the parameters. Using statistical ineffective fault attacks to recover the b) C908: The instructions discovered by RISCVuzz on private key typically involves a prolonged correlation analy- the C908 correspond to the vector mask store/load instructions sis [11], due to the difficulty of analyzing the biased distribu- vsm.v and vlm.v. Setting any of the bits 29 to 31 in the tion of each fault. In contrast, our attack uses a physical write encoding of these instructions crashes the machine. Note that that allows the attacker to modify the private key byte by byte. these bits should be all unset, i.e., zero, when the instruction This capability significantly simplifies the attack by reducing is assembled correctly. Other bits seem unaffected. the complexity of determining the correct bits of the key. Takeaway Testing the entire possible encoding space is c) Evaluation: We successfully mount the attack on necessary, as vulnerabilities are in the documented and Board H. The victim uses an unmodified OpenSSL version undocumented range. 3.0.9. We observe a leakage rate of 1.10 bit/s over 10 trials. On average, it takes the attacker 30 minutes to fully recover the private 2048-bit RSA key or half of the time to recover B. Bug Reproduction in Simulator the 1024-bit parameters of RSA-CRT. In contrast to the other tested CPUs, the source code of VII. CPU-HALTING INSTRUCTION SEQUENCES the C906 is available, and the source contains the same vul- nerability as the hardware CPU. Thus, we can also reproduce In this section, we analyze the instruction sequences the vulnerability in the simulation of the Verilog source. We RISCVuzz finds for halting the C906 and C908, requiring a run the source from the official T-Head repository of the hard reset. We analyze the sequences on hardware, reproduce C906 [53] using the ICARUS Verilog compilation system [62]. the findings in the simulation of the C906, and present an The simulation machine is an Intel Core i9-13900K with end-to-end denial-of-service attack from within Docker. 16 GB RAM running Ubuntu 22.04.
11
We reduce the instruction sequence to a minimal 24 B Takeaway Optional hardware features should have the ca- bare-metal binary containing only 6 instructions. Running this pability to be deactivated. sequence reliably stops the simulator with the error message that the CPU is stuck and no instructions are retiring anymore. C908 Halting Instructions. The C908 halting instructions This happens after 11.5 μs CPU time. It takes the simulation are in the vector-extension range. Disabling the vector exten- 2.5 min to reach this point. sion fortunately prevents exploitation. Recompiling the Linux kernel without vector support leads to an illegal-instruction C. Case Study exception when executing the instruction, mitigating the DoS of these instructions. To assess the impact of the vulnerability, we evaluate We perform the same benchmark as in Section VIII since in which contexts it can be executed to halt the CPU. The the mitigation is the same. We run rvv-bench on Board straightforward scenario is an unrestricted native environment F and find that the performance of memcpy and memset as has been used by RISCVuzz. Executing the instruction decreases by up to 77 % and 2 %, respectively. sequence as a privileged or unprivileged user immediately results in the CPU being halted. We verify the C906 behavior on two different boards, Board D and E, using two different IX. RELATED WORK operating systems, Debian 11 and Debian 12. Additionally, we verify the C908 behavior on Board F with Debian 13. Undocumented Instructions. Armshaker [52] is an approach While executing the instruction in machine mode also halts the that iterates over the entire 32-bit ARM instruction space to CPU, there is no realistic threat model where this is relevant. find undocumented instructions. Similarly, Dofferhoff et al. However, attackers can also use the instruction sequence in [12] propose a tool to find undocumented instructions for RISC more restricted environments. We verify that executing the architectures like ARMv8 and RISC-V using disassemblers instruction in an unprivileged Docker container also halts as a ground truth. Both works discovered multiple emulator the CPU. Thus, sandboxing mechanisms that work on the bugs and inconsistencies in the respective ISA standards. operating-system level cannot prevent an attacker from halting DifuzzRTL [23] and Morfuzz [64] also fuzz undocumented the CPU. Furthermore, given that all involved instructions are instructions on RISC-V. However, in contrast to RISCVuzz, unprivileged instructions, we also expect that sequence to work DifuzzRTL and Morfuzz require a perfect simulator to detect from a virtual machine. Unfortunately, we cannot verify that, misbehaviors in the executed code. Sandsifter [14] shows that as no current hypervisor supports the C906 or C908. it is feasible to search for undocumented x86 instructions by exploiting a side channel to infer up to which byte an instruction was successfully decoded, dealing with the 15-byte VIII. MITIGATIONS instruction space. In contrast to previous work, RISCVuzz does In this section, we discuss mitigations for GhostWrite not require any ground truth and can exhaustively test the entire (Section VI) and the CPU-halting sequences (Section VII). instruction space. GhostWrite. Disabling the vector extension is a viable miti- Differential CPU Fuzzing. Differential fuzzing is a well- gation for GhostWrite. We use a kernel module to verify this known software fuzzing technique that was recently applied to mitigation on the C910. The CPU throws an illegal instruction CPU fuzzing. Tavis Ormandy found critical security vulnera- exception when executing the instruction [60], making the bilities by comparing code generated by their fuzzer against a gadget unusable for an attacker. serialized variant of the code [35], [36]. For this so-called Ora- cle Serialization, the initial code is modified by adding memory We benchmark the impact of this mitigation on standard fences between the individual instructions. Such techniques memory operations such as memcpy and memset. We use the make it possible to find differences that depend on speculative RISC-V vector benchmarking suite rvv-bench [3] on Board or out-of-order execution. Bugs that do not depend on such H. We compare the fastest vector implementation of memcpy optimizations are invisible to this technique. For example, and memset to the fastest of glibc and musl libc. We observe GhostWrite behaves the same when fences are added and a performance hit of up to 33 % for memcpy and 8 % for hence cannot be detected by this approach. Further, cross- memset. Benchmarking the mitigation on a full-system level vendor or cross-generation bugs cannot be discovered with this is currently not possible, since no distribution uses the vector technique, since results are only compared to other cores on extension in the kernel and standard libraries. the same CPU. RISCVuzz finds these classes of bugs too, as it compares the behavior against different CPUs from potentially C906 Halting Sequence. There is no mitigation for the different vendors. SiliFuzz [45] targets x86 CPUs with the goal C906 halting instruction sequence that can be used for DoS of finding electrical defects on single cores instead of bugs. As from an unprivileged process. Since no special condition is SiliFuzz does not compare different CPUs but different cores needed to execute the C906 halting instruction sequence, we on one CPU, the found bugs of SiliFuzz and RISCVuzz are argue that the only option for mitigating the vulnerability is different. For our tested CPUs, all cores on the same CPU to disable one of the instructions. Unfortunately, the T-Head behave the same, as the differences are due to implementation vendor extension that includes the broken instructions cannot errors of the CPU and not due to electrical defects of single be disabled: “The th.sxstatus.THEADISAEE bit is not cores. Qin et al. [37] and Jiang et al. [24] compare CPUs expected to be cleared. The behavior of clearing this bit is with emulators and disassemblers to build stealthy malware by undefined” [56]. We verify that we cannot clear this bit from exploiting different runtime behaviors. In contrast to our paper, a kernel module. they discover and exploit bugs in software, not in CPUs.
12
Model Fuzzing. TheHuzz [26] and DifuzzRTL [23] are feedback channel. Similarly, evaluating diversity graphs of our fuzzers that target the register-transfer level (RTL) model of sequences is not possible, as we have no coverage information. CPUs. The benefit of targeting the CPU’s RTL is that these However, the results of MorFuzz [64] suggest that more fuzzers can be used during development and that emulation complex sequence generation approaches might be beneficial. of the cores can provide coverage information to guide the Note that these problems are inherent to opaque-box fuzzing fuzzer’s search. Solt et al. [51] improve upon that approach and not a particular weakness of our approach. However, even by generating more complex instruction sequences to improve with our simple sequence generation, we uncover most of the fuzzing throughput and discover bugs with more complex bugs in seconds due to the high throughput of running them conditions. The major drawback of all RTL-based fuzzers on hardware. Recent work on software fuzzing also suggests is that they require a complete RTL to work in the first that even simple input generation can be efficient [33]. We place. In contrast, RISCVuzz targets the orthogonal problem leave improving upon our simple sequence generation or using of finding bugs in opaque-box hardware CPUs with the ad- hardware side channels for coverage for future work. vantage of faster fuzzing throughput and testing the actual deployed CPUs. Due to its speed, RISCVuzz fully explores d) Compliance: The findings from RISCVuzz demon- all undocumented instructions with a bottom-up approach strate numerous architectural differences in instructions across (Section IV-C1) rather than a mutaion approach. Guidance [25] CPU vendors and even across CPUs of the same vendor. These or fuzzing by proxy [45] could also be used for RISCVuzz. findings also suggest that several of these differences violate However, even without this guidance, RISCVuzz produces the ISA specification, making it difficult to write applications many results, leaving this guidance for future work. that run correctly on all RISC-V CPUs. Thus, we advocate an extensive compliance-testing framework for RISC-V. While X. DISCUSSION there is an architectural test suite [20], many parts are not In this section, we discuss the impact of our findings, covered, including the vector extensions we exploited. coverage and ground truth, an outlook on future problems, e) Outlook: At the time of writing, only T-Head and and the need for compliance testing. SiFive have commercially available off-the-shelf machines a) Impact: While there are still not many 64-bit RISC- with general-purpose 64-bit RISC-V CPUs. Still, even with V boards on the market, they already gain traction. Scaleway this limited selection of vendors and CPUs, RISCVuzz finds provides cloud instances with the TH1520 SoC [42], which a large number of bugs and inconsistencies. In the future, contains a C910 CPU. We verified that the used CPU is we expect to see more vendors building CPUs based on indeed vulnerable, and we reported our findings to Scaleway. custom designs. Combined with the unregulated use of the ISA The Shandong University in China also has a RISC-V cluster and the possibility of creating custom vendor extensions, we using a variant of the C910 CPU [46]. Unfortunately, we do expect this state to worsen. Architectural inconsistencies will not have access to this system to test if this C910 variant is become especially relevant for trusted-execution environments also vulnerable. Mitigating GhostWrite is only possible by dis- and virtualization, as current vendor extensions might not abling the entire vector extension, resulting in a reduced feature consider the future-proofness of their extensions concerning set and lower performance for specific workloads. Worse, the potentially different privilege levels. Similarly, with RISC-V CPU halting sequence on the C906 has no mitigation, making support in Linux and Android, vendor customizations might these CPUs essentially unsuitable for running untrusted code undermine security guarantees or lead to unstable systems. or deployment in multi-user systems. We assume that with a microcode layer, as on x86, XI. CONCLUSION GhostWrite could be mitigated. An x86 microcode update can hook and patch instructions [5], which could have been used to In this paper, we introduced RISCVuzz, a differential CPU hook the broken vector instruction and simply raise an illegal- fuzzing framework for RISC-V hardware CPUs for automat- instruction exception. Given the increasing complexity of ically discovering architectural CPU bugs. RISCVuzz com- RISC-V CPUs, we advocate such a microcode layer on RISC- pares the architectural results of instruction sequences without V to have the possibility of mitigating CPU vulnerabilities. relying on CPU source code or any emulator. RISCVuzz b) Ground Truth: RISCVuzz can be used in scenarios discovered 3 severe security vulnerabilities and numerous other where no ground truth, i.e., an emulator or another reference bugs on 5 different CPUs. On the T-Head C910, RISCVuzz implementation, is available because it does not require a discovered GhostWrite, an instruction sequence that allows golden model. Cases where no ground truth is available include unprivileged attackers to write arbitrary values directly to (custom) ISA extensions like T-Head’s custom draft vector physical memory, entirely circumventing virtual memory and extension [57], CPUs where the source code is not (fully) its protection. We demonstrate that GhostWrite can also be available like with GhostWrite, or where no source code used to read memory and to inject attacker code into kernel and nor documentation about a feature is available at all as with machine mode. Further, we investigated two “halt-and-catch- the CPU-halting instructions on the C908. Even if emulators fire instructions” on two different CPUs, the T-Head XuanTie are available, they might be inconsistent [24] or based on a C906 and T-Head XuanTie C908, and showed how they lead different code base, as we see for the C910. to unprivileged denial of service. RISCVuzz discovered most bugs and vulnerabilities within seconds, showing the efficacy c) Coverage & Sequence Complexity: Generating com- of our post-silicon fuzzing approach. We outperform state-of- plex instruction sequences that increase coverage over time is the-art RTL-based fuzzers in instruction execution by orders difficult for opaque-box fuzzing since we have no adequate of magnitude, making it a valuable extension to these fuzzers.
13
ACKNOWLEDGMENT [24] M. Jiang, T. Xu, Y. Zhou, Y. Hu, M. Zhong, L. Wu, X. Luo, and
This work was supported in part by Semiconductor Re- K. Ren, “Examiner: Automatically locating inconsistent instructions
between real devices and cpu emulators for arm,” in ASPLOS, 2022.
search Corporation (SRC) Hardware Security Program (HWS) [25] N. Kabylkas, T. Thorn, S. Srinath, P. Xekalakis, and J. Renau, “Effective and by a Google Research Scholar award. Any opinions, processor verification with logic fuzzer enhanced co-simulation,” in findings, conclusions, or recommendations expressed in this MICRO, 2021. paper are those of the authors and do not necessarily reflect [26] R. Kande, A. Crump, G. Persyn, P. Jauernig, A.-R. Sadeghi, A. Tyagi, the views of the funding parties. and J. Rajendran, “TheHuzz: Instruction Fuzzing of Processors Using Golden-Reference Models for Finding Software-Exploitable Vulnera- bilities,” in USENIX Security Symposium, 2022. REFERENCES [27] P. Kocher, J. Horn, A. Fogh, D. Genkin, D. Gruss, W. Haas, M. Ham- [1] K. Asanovic, R. Avizienis, J. Bachrach, S. Beamer, D. Biancolin, burg, M. Lipp, S. Mangard, T. Prescher, M. Schwarz, and Y. Yarom, C. Celio, H. Cook, D. Dabbelt, J. Hauser, A. Izraelevitz et al., “The “Spectre Attacks: Exploiting Speculative Execution,” in S&P, 2019. rocket chip generator,” EECS Berkley, 2016. [28] P. C. Kocher, “Timing Attacks on Implementations of Diffe-Hellman, [2] C. Aum¨uller, P. Bier, W. Fischer, P. Hofreiter, and J.-P. Seifert, “Fault RSA, DSS, and Other Systems,” in CRYPTO, 1996. attacks on RSA with CRT: Concrete results and practical countermea- [29] Krimsky, “RISC-V Single Board Computers,” 2023. [Online]. sures,” in CHES, 2002. Available: http://krimsky.net/articles/riscvsbc.html [3] O. Bernstein, “rvv-bench: Risc-v vector benchmark,” 2023. [Online]. [30] A. Kwong, D. Genkin, D. Gruss, and Y. Yarom, “RAMBleed: Reading Available: https://github.com/camel-cdr/rvv-bench Bits in Memory Without Accessing Them,” in S&P, 2020. [4] D. Boneh, R. A. DeMillo, and R. J. Lipton, “On the importance of [31] M. Lipp, M. Schwarz, D. Gruss, T. Prescher, W. Haas, A. Fogh, eliminating errors in cryptographic computations,” 2001. J. Horn, S. Mangard, P. Kocher, D. Genkin, Y. Yarom, and M. Hamburg, [5] P. Borrello, C. Easdon, M. Schwarzl, R. Czerny, and M. Schwarz, “Meltdown: Reading Kernel Memory from User Space,” in USENIX “CustomProcessingUnit: Reverse Engineering and Customization of Security Symposium, 2018. Intel Microcode,” in WOOT, 2023. [32] W. M. McKeeman, “Differential testing for software,” 1998. [6] P. Borrello, A. Kogler, M. Schwarzl, M. Lipp, D. Gruss, and [33] B. P. Miller, M. Zhang, and E. R. Heymann, “The relevance of classic M. Schwarz, “ÆPIC Leak: Architecturally Leaking Uninitialized Data fuzz testing: Have we solved this one?” IEEE Transactions on Software from the Microarchitecture,” in USENIX Security, 2022. Engineering, vol. 48, 2022. [7] C. Canella, J. Van Bulck, M. Schwarz, M. Lipp, B. von Berg, P. Ortner, [34] K. Murdock, D. Oswald, F. D. Garcia, J. Van Bulck, D. Gruss, F. Piessens, D. Evtyushkin, and D. Gruss, “A Systematic Evaluation of and F. Piessens, “Plundervolt: Software-based Fault Injection Attacks Transient Execution Attacks and Defenses,” in USENIX Security, 2019, against Intel SGX,” in S&P, 2020. extended classification tree and PoCs at https://transient.fail/. [35] T. Ormandy, “Reptar,” 2023. [Online]. Available: https: [8] C. Celio, D. A. Patterson, and K. Asanovi´c, “The Berkeley Out- //lock.cmpxchg8b.com/reptar.html of-Order Machine (BOOM): An Industry-Competitive, Synthesizable, [36] ——, “Zenbleed,” 2023. [Online]. Available: https://lock.cmpxchg8b. Parameterized RISC-V Processor,” Tech. Rep., 2015. com/zenbleed.html [9] C. Clavier, “Secret external encodings do not prevent transient fault [37] S. Qin, C. Zhang, K. Chen, and Z. Li, “iDEV: Exploring and exploiting analysis,” in CHES, 2007. semantic deviations in arm instruction processing,” in ISSTA, 2021. [10] R. R. Collins, “The Pentium F00F Bug,” 1998. [Online]. Available: [38] P. Qiu, D. Wang, Y. Lyu, and G. Qu, “VoltJockey: Breaking SGX by http://www.rcollins.org/ddj/May98/F00FBug.html Software-Controlled Voltage-Induced Hardware Faults,” in AsianHOST, [11] C. Dobraunig, M. Eichlseder, T. Korak, S. Mangard, F. Mendel, and 2019. R. Primas, “SIFA: Exploiting Ineffective Fault Inductions on Symmetric [39] RISC-V Collaboration, “riscv-gnu-toolchain,” 2024. [Online]. Cryptography,” in IACR Transactions on Cryptographic Hardware and Available: https://github.com/riscv-collab/riscv-gnu-toolchain [12] Embedded Systems, 2018. [40] RISC-V Foundation, “RISC-V Exchange,” 2023. [Online]. Available: R. Dofferhoff, M. G¨oebel, K. Rietveld, and E. Van Der Kouwe, https://riscv.org/exchange/ “IScanU: A portable scanner for undocumented instructions on risc processors,” in International Conference on Dependable Systems and [41] RISC-V International, “Operating Systems,” 2024. [Online]. Available: Networks, 2020. https://wiki.riscv.org/display/HOME/Operating+Systems [13] E. Dolstra, A. L¨oh, and N. Pierron, “Nixos: A purely functional linux [42] Scaleway. (2024) The world’s first RISC-V servers available in the distribution,” in Journal of Functional Programming, 2010. cloud. [Online]. Available: https://labs.scaleway.com/en/em-rv1/ [14] C. Domas, “Hardware Backdoors in x86 CPUs,” Black Hat US, 2018. [43] T. Scharnowski, N. Bars, M. Schloegel, E. Gustafson, M. Muench, [15] L. Foundation. (2023). [Online]. Available: https://git.kernel.org/pub/ G. Vigna, C. Kruegel, T. Holz, and A. Abbasi, “Fuzzware: Using precise scm/linux/kernel/git/next/linux-next.git/tree/arch/riscv/boot/dts/thead MMIO modeling for effective firmware fuzzing,” in USENIX Security, 2022. [16] R.-V. Foundation. (2019) Risc-v ”v” vector extension 0.7.1. [Online]. [44] M. Seaborn, “Exploiting the DRAM rowhammer bug to gain Available: https://github.com/riscv/riscv-v-spec/releases/tag/0.7.1 kernel privileges,” March 2015, retrieved on June 26, 2015. [17] ——. (2021) Risc-v ”v” vector extension 1.0. [Online]. Available: [Online]. Available: http://googleprojectzero.blogspot.com/2015/03/ https://wiki.riscv.org/display/HOME/Ratified+Extensions exploiting-dram-rowhammer-bug-to-gain.html [18] ——. (2021) Risc-v “zfh” and “zfhmin” standard extensions [45] K. Serebryany, M. Lifantsev, K. Shtoyk, D. Kwan, and P. Hochschild, for half-precision floating-point, version 1.0. [Online]. Available: “Silifuzz: Fuzzing cpus by proxy,” arXiv:2110.11519, 2021. [19] https://wiki.riscv.org/display/HOME/Recently+Ratified+Extensions [46] A. Shah. (2023) China deploys massive risc-v server in ——, “riscv-opcodes,” 2022. [Online]. Available: https://github.com/ commercial cloud. [Online]. Available: https://www.hpcwire.com/ riscv/riscv-opcodes 2023/11/08/china-deploys-massive-risc-v-server-in-commercial-cloud/ [20] ——, “RISC-V Architecture Test SIG,” 2023. [Online]. Available: [47] SiFive, “HF105 Datasheet,” 2022. [Online]. Available: https:// https://github.com/riscv-non-isa/riscv-arch-test sifive.cdn.prismic.io/sifive/d0556df9-55c6-47a8-b0f2-4b1521546543 [21] M. Frysinger, “vdso(7) — linux manual page,” 2024. hifive-unmatched-datasheet.pdf [22] L. Gerlach, D. Weber, R. Zhang, and M. Schwarz, “A Security RISC: [48] Sipeed, “Sipeed wiki,” 2021. [Online]. Available: https://wiki.sipeed. Microarchitectural Attacks on Hardware RISC-V CPUs,” in S&P, 2023. com/en/index.html [23] J. Hur, S. Song, D. Kwon, E. Baek, J. Kim, and B. Lee, “Difuzzrtl: [49] ——, “RISC-V 64bit chip (C910) run Android 10,” 2022. [Online]. Differential fuzz testing to find cpu bugs,” in S&P, 2021. Available: https://twitter.com/SipeedIO/status/1457529282134089734
14
[50] ——. (2023) Lichee Console 4A. [Online]. Available: https: ISA throw a signal when run in RISCVuzz. Consequently, ev- //sipeed.com/licheepi4a ery further instruction in the sequence decreases the probability [51] F. Solt, K. Ceesay-Seitz, and K. Razavi, “Cascade: Cpu fuzzing via of a longer sequence by ∼ 40 %. This means the percentage [52] intricate program generation,” 2024. of actually executed sequence lengths is roughly given by the F. Strupe and R. Kumar, “Uncovering hidden instructions in Armv8-A function prob exec(n) = 0.4n. That further points out why implementations,” in Hardware and Architectural Support for Security increasing the sequence length leads to diminishing returns. and Privacy, 2020. [53] T-Head, “openC906,” 2021. [Online]. Available: https://github.com/ th.lbib th.lbia th.lwuib th.lbuib [54] T-head-Semi/openc906 th.lwia th.ldia th.lwib th.lbuia ——, “openC910,” 2021. [Online]. Available: https://github.com/ th.lhuia th.lhia th.ldib th.lhib [55] T-head-Semi/openc910 th.lwuia th.lhuib ——, “C906,” 2022. [Online]. Available: https://www.t-head.cn/ [56] product/c906 Listing 3: List of instructions from the XTheadMemIdx ——, “T-Head Extension Spec,” 2022. [Online]. Available: https: //github.com/T-head-Semi/thead-extension-spec extension that can be used in Listing 2 to halt the C906 CPU. [57] T-Head, “Xtheadvector,” 2022. [Online]. Avail- able: https://github.com/XUANTIE-RV/thead-extension-spec/blob/ master/xtheadvector/intrinsics.adoc APPENDIX D [58] J. Van Bulck, M. Minkin, O. Weisse, D. Genkin, B. Kasikci, F. Piessens, GHOSTWRITE SAMPLE REPRODUCER M. Silberstein, T. F. Wenisch, Y. Yarom, and R. Strackx, “Foreshadow: Listing 4 shows an example of a reproducer file generated Extracting the Keys to the Intel SGX Kingdom with Transient Out-of- Order Execution,” in USENIX Security Symposium, 2018. by RISCVuzz when comparing the results of vector instruc- [59] A. Waterman and K. Asanovi´c, “The RISC-V Instruction Set Manual, tions between C906 and C910, which both implement the Vol. I: Unprivileged ISA, Version 20191213,” 2019. 0.7.1 draft vector extension. The C906 generates a fault, while [60] A. Waterman, K. Asanovi´c, and J. Hauser, “The RISC-V Instruction the instruction executes just fine on the C910. This hints at Set Manual Volume II: Privileged Architecture, Document Version GhostWrite. 20211203,” 2021. [61] A. Waterman, Y. Lee, D. A. Patterson, and K. Asanovi´c, “The risc- # signum differs v compressed instruction set manual, version 1.7,” EECS Department, # si_addr differs University of California, Berkeley, 2015. # si_pc differs [62] S. Williams, “Icarus verilog,” 2024. [Online]. Available: https: # si_code differs //steveicarus.github.io/iverilog/ # [63] Xcalibyte, “Roma Laptop Pre-order,” 2022. [Online]. Available: ## base: C910 (lab46) https://xcalibyte.com.cn/en/roma-preorder/ # signum: OK -------------------------------- [64] J. Xu, Y. Liu, S. He, H. Lin, Y. Zhou, and C. Wang, “{MorFuzz}: # other: C906 (lab50) Fuzzing processor via runtime instruction morphing enhanced synchro- # signum: SIGSEGV nizable co-simulation,” in USENIX Security, 2023. # si_addr: 0x8000000000000000 [65] S.-M. Yen and M. Joye, “Checking before output may not be enough # si_pc: 0xe100178 against fault-based cryptanalysis,” IEEE Transactions on computers, # si_code: 0x1 [66] vol. 49, 2000. instr_seq: R. Zhang, L. Gerlach, D. Weber, L. Hetterich, Y. L¨u, A. Kogler, - 0x5201f0a7 and M. Schwarz, “CacheWarp: Software-based Fault Injection using dis: ’’ Selective State Reset,” in USENIX Security, 2024. dis_opcodes: - vse1024.v APPENDIX A regs: DETAILS ON USED BOARDS gp:gp: 0x8000000000000000 Table III extends Table I by the board models, memory flags: configuration, kernel, and OS version. - "-DVECTOR" APPENDIX B Listing 4: Example of a reproducer that hints at GhostWrite. C906 CPU-HALTING INSTRUCTIONS The C910 executes the vector-store instruction just fine, while the C906 generates a fault. Listing 3 lists the other broken instructions from the XTheadMemIdx vendor extension on the C906. Any of these instructions can be used instead of the th.lbib instruction APPENDIX E in Listing 2 to halt the CPU. PHYSICAL PAGE TABLE DISTRIBUTION In Section VI-B, we use GhostWrite to overwrite page APPENDIX C frame numbers in physical memory to transform it into an DISTRIBUTION OF EXECUTED INSTRUCTIONS IN arbitrary physical read primitive. We fill the entire physical SEQUENCE memory with page tables so that the probability of hitting such To further reason about the scaling of increasing the se- a page table when writing at a random address in memory is quence length (cf. Figure 5), we collect the distribution of the high. In this section, we experimentally verify this. number of executed instructions per sequence length. ∼ 40 % We create 1 500 000 last-level page tables, filling up the of the executed sequences stop right at the first instruction. entire 8 GB of memory on Board H by mapping a file Equivalently, this means that ∼ 40 % of instructions in the base repetitively to memory in multiple processes. We then record
15
TABLE III: Overview of tested RISC-V boards. We use CPUs from 2 vendors with varying extensions.
ID Board Model CPU CPU Vendor Relevant Extensions Memory OS Kernel
A BeagleV Fire U54 SiFive - 1.5 GB Ubuntu 23.04 6.1.33
B StarFive VisionFive2 U74 SiFive - 8 GB Ubuntu 22.04.1 6.5.0
C Sipeed Nezha 1 GB Debian 13 5.14.0
D Lichee RV Dock C906 T-Head v0p7, zfh, XTheadMemIdx 512 MB Debian 11 5.4.61
E Lichee RV Dock 512 MB Debian 12 5.14.0
F CanMV Kendryte K230 C908 T-Head v, zfh, XTheadMemIdx 512 MB Debian 13 5.10.4
G BeagleV Ahead 4 GB Ubuntu 23.04 5.10.113
HI LicheePi4A C910 T-Head v0p7, zfh, XTheadMemIdx 8 GB Debian 12 5.10.113
LicheePi4A 16 GB NixOS 5.10.113
J Milk-V Meles 8 GB Debian 12 5.10.113
Run 0 ·104 6
Run 1 4
Run 2 20
Bucket adresses
Fig. 6: Physical distribution of page tables when filling 8 GB of memory with page tables over 3 runs. Accumulated into 32
buckets of 256 MB.
the physical addresses of these last-level page tables in 3 runs.
Figure 6 visualizes the distribution of the collected addresses.
Page tables are nearly uniformly distributed over the entire
physical memory, with a gap in the middle of memory and at
the start, where kernel and OpenSBI reside.
16
Run index
0000000000100000000200000000300000000400000000500000000600000000700000000800000000900000000a00000000b00000000c00000000d00000000e00000000f00000001000000001100000001200000001300000001400000001500000001600000001700000001800000001900000001a00000001b00000001c00000001d00000001e00000001f0000000