SOURCE ARCHIVE
EXTRACTED CONTENT
53,815 chars Closing the RISC-V Compliance Gap:
Looking from the Negative Testing Side?
Vladimir Herdt1 Daniel Große1,2 Rolf Drechsler1,2
1Cyber-Physical Systems, DFKI GmbH, 28359 Bremen, Germany
2Institute of Computer Science, University of Bremen, 28359 Bremen, Germany
{vherdt,grosse,drechsle}@informatik.uni-bremen.de
Abstract—Compliance testing for RISC-V is very important. that represents the output of the test result and is dumped at the end
Therefore, an official hand-written compliance test-suite is being of the test execution. For compliance testing, these signatures are actively developed. However, besides requiring significant manual compared against golden reference signatures (obtained by running effort, it focuses on positive testing (the implemented instructions the test-suite on a reference simulator). A separate sub test-suite is work as expected) only and neglects negative testing (consider developed for the RISC-V base ISA as well as for each standard ISA illegal instructions to also ensure that no additional/unexpected behavior is accidentally added). This leaves a large gap in compli- extension. Besides the significant manual effort for the maintenance, ance testing. the compliance test-suite focuses on positive testing only, i.e. to In this paper we propose a fuzzing-based test-suite generation show that the implemented instructions work as expected. However, approach to close this gap. We found new bugs in several RISC-V it neglects negative testing, i.e. to consider illegal instructions to also simulators including riscvOVPsim from Imperas which is the official ensure that no additional/unexpected behavior is accidentally added. reference simulator for compliance testing. This leaves open a large gap in compliance testing. I. INTRODUCTION Contribution: In this paper we propose a fuzzing-based test-suite An Instruction Set Architecture (ISA) defines the interface be- generation approach to close this gap. We leverage state-of-the- tween the Hardware (HW) of a processor and the Software (SW). art fuzzing techniques (based on LLVM libFuzzer) to iteratively While, as a consequence, the format of a SW binary running on generate test-cases which are executed on a RISC-V simulator and a processor is clearly defined by the ISA, nothing is specified on guide the fuzzing process through the observed code coverage of how to implement the processor1. An ISA which has become very the simulator. A filter is integrated between fuzzer and simulator popular is the RISC-V ISA [1]. Driven by the ideas from open to conservatively remove test-cases with infinite loops and platform source SW, the RISC-V ISA is open, royalty-free, and maintained specific details, to avoid spurious signature mismatches and to en- by the non-profit RISC-V foundation [2]. The major goal of the able automated compliance testing. To further improve the fuzzing RISC-V ISA is to provide a path to a new era of processor innovation effectiveness, we incorporate a custom coverage metric and fuzzing via open standard collaboration. Around RISC-V an ecosystem is mutator. Our approach is very effective for negative testing and thus rapidly emerging. Staring from the base ISA, a big plus of the complements the official compliance test-suite. We found new bugs RISC-V ISA is the availability of modular standard extensions. In in several RISC-V simulators including riscvOVPsim from Imperas, addition, extensibility has been designed into the ISA allowing for which is the official reference simulator for compliance testing (i.e. custom instructions. While this flexibility offers significant advan- used to generate reference signatures).2 tages (free selection of what is needed from the standard extensions II. RELATED WORK and addition of dedicated custom instructions for optimization of the For the purpose of verification, several approaches to test pro- target application), also a major challenge is posed: fragmentation. gram generation have been proposed. In particular model-based The above mentioned cooperation driving the ecosystem will fail, if approaches, which separate the test generator from the architecture different RISC-V CPU implementations do not comply with the ISA description, have a long history. Prominent examples using constraint specification. Therefore, the compliance of each RISC-V CPU to the solving techniques are [6], [7]. An optimized test generation frame- ISA specification has to be validated. This is the task of compliance work is presented in [8]. It propagates constraints among multiple testing. More precisely, compliance testing checks whether registers instructions in an effective manner. The test program generator of [9] are missing, modes are not there, instructions are absent, as well includes a coverage model that holds constraints describing exe- as the presence of only those instructions which are part of the cution paths of individual instructions. Other approaches integrate selected ISA [3], [4]. If the compliance test passes for a CPU, coverage-guided test generation based on bayesian networks [10] and the HW/SW contract is maintained and the SW will be portable other machine learning techniques [11] as well as fuzzing [12]. between implementations. Note that compliance testing is not design Recently, test generation approaches specifically targeting verification. In contrast to compliance testing, the goal of verification RISC-V have emerged [13]–[15]. The Scala-based Torture Test gen- is to find errors in the CPU implementation. erator [13] generates tests by integrating pre-defined randomized test The importance of compliance testing has been recognized very sequences and supports several RISC-V ISA extensions. However, early by the RISC-V foundation and therefore the compliance task it has two major drawbacks: it does not build upon the official group has been formed [5]. The compliance task group actively compliance testing format and only performs positive testing, i.e. il- develops the official hand-written compliance test-suite. The indi- legal instructions are not considered. Another approach is RISCV- vidual test-cases are designed to compute an in-memory signature, DV [14]. It leverages SystemVerilog in combination with UVM ?This work was supported in part by the German Federal Ministry of (Universal Verification Methodology) to generate RISC-V instruc- Education and Research (BMBF) within the project VerSys under contract tion streams based on constrained-random descriptions. However, no. 01IW19001 and within the project SATiSFy under contract no. 16KIS0821K. 1Such an implementation is referred to as micro architecture and the most 2Visit http://www.systemc-verification.org/risc-v for our most recent RISC-V famous ones for the x86 ISA are the processors from AMD and Intel. related approaches.
RISCV-DV offers only very limited support for generation of illegal Phase A: coverage information (instrumented by clang compiler)
instructions and thus is not suitable for comprehensive negative test- Test-suite mutate bytestream
ing. In addition, the approach does not support the compliance testing Generation called Custom Mutators Drop Bytestream
by (No New Coverage)
format and requires a commercial RTL simulator providing Sys- Coverage fuzzer remove 3) Simulator
temVerilog (constrained-random features) as well as UVM support. 1) Fuzzer Bytestream 2) Filter ok Inject Bytestream
(libFuzzer) (Instructions) (Static Analysis) and Execute
[15] proposed coverage-guided fuzzing for verification of instruction collect on clone prior
set simulators. However, the approach is not compatible with the new coverage execution
Bytestream Test-case Template Test-case Template Sim. Engine+Memory
Bytestream
Bytestream
compliance testing format, since it generates platform dependent (Instructions) (Source File) (Pre-compiled ELF) (Initial State)
(Instructions)
(Instructions) pre-load into memory notify at
tests in ELF format instead of providing platform independent tests combine execution
written in assembler (ASM), which also significantly reduces its Custom Coverage Custom Coverage
applicability to different platforms. Furthermore, the approach does compile and run Specification Encoding
Test-case Reference generate
not support automated testing, as it requires manual inspection to (Source File) Simulator Phase B: Compliance Testing
avoid false negatives due to platform specific details. Test-suite compile and run compare signatures generate once (per RISC-V ISA config)
In addition, there are also formal verification approaches for Simulators/Cores e.g. RV32I seperate set
Under Test Test Output =? Reference Output RV32IMC of ref. outputs
RISC-V based on model checking. Notable are riscv-formal [16] and use specific ISA config. (Signature) (Signature) RV32GC,... per ISA config.
the OneSpin 360 DV RISC-V verification app [17]. However, both Fig. 1. Overview: fuzzer-based approach for RISC-V compliance testing
approaches clearly target the verification of an implementation.
Finally, [18] specifically considers compliance testing of RISC-V. functions. For example, the SUT receives inputs through the LLVM-
It defines a test-suite specification mechanism and leverages con- FuzzerTestOneInput(const uint8 t *Data, size t Size) function.
straint solving techniques to generate a comprehensive compliance IV. F UZZING-BASED RISC-V COMPLIANCE TESTING
test-suite as counterpart to the hand-written official compliance test-
suite. However, it also only focuses on positive testing and does not This section presents our fuzzer-based approach for RISC-V com-
consider negative testing aspects, such as illegal instructions. pliance testing. We start with an overview.
III. PRELIMINARIES A. Approach Overview
A. RISC-V Fig. 1 shows an overview on our approach. Essentially, it consists
The RISC-V ISA consists of a mandatory base integer instruction of two subsequent phases: first a fuzzer-based test-suite is generated
set, denoted RV32I, RV64I or RV128I with corresponding register (Phase A, shown on top of Fig. 1), then the test-suite is leveraged for
widths, and various optional extensions denoted as single letters, e.g. compliance testing (Phase B, shown on bottom of Fig. 1). Our gen-
M (integer multiplication and division), A (atomic instructions), C erated test-suite follows the same format as the official compliance
(compressed, i.e. 2 byte instructions), F and D (single and double test-suite and thus also generates signatures for compliance testing.
precision floating point) etc. Thus, RV32IMC denotes a 32 bit core However, in contrast to the official suite, which has a dedicated sub
with M and C extensions. G denotes the IMAFD instruction set, suite for each RISC-V ISA extension, we generate a single suite
hence RV32GC=RV32IMAFDC. Each core has 32 general purpose that can be compiled and executed with any supported RISC-V
registers x0 to x31 (with x0 being hardwired to zero) and the floating ISA (currently we support any configuration of RV32GC), since
point (FP) extensions add additional 32 FP registers. Instructions unsupported instructions should be considered illegal and result in
access registers (source: RS1 and RS2, destination: RD) and imme- an exception. Furthermore, due to the randomness of the fuzzing
diates to do their operation. Format and semantics (for the base ISA process, both phases can be continuously repeated, to achieve an even
and extensions) are defined in the unprivileged ISA specification [1]. more comprehensive testing.
In addition, the privileged (architecture) specification [19] cov- Test-suite generation involves three main steps: 1) fuzzer, 2) filter
ers further important functionality that is required for environment and 3) simulator, that are repeated until the specified time (or mem-
interaction and operating system execution. It includes different ory) limit is reached. The fuzzer generates (random) bytestreams,
execution modes, in particular the mandatory Machine mode as well which are interpreted as RISC-V instruction sequences, and passes
as the Supervisor and User mode extensions with corresponding them to the filter that decides whether the bytestream is further
Control and Status Registers (CSRs) descriptions. CSRs are registers processed or dropped. Essentially, the filter conservatively drops
serving a special purpose, that form the backbone of the privileged bytestreams with infinite loops and platform specific details (test-
architecture description, such as MTVEC (stores the trap/interrupt cases are available as source files and are compiled separately for
handler address), MHARTID (read-only core id) and MSTATUS each target platform with custom definitions), to avoid spurious
(main control and status register for the core). signature mismatches. This is very important to enable a continu-
ous and automated testing process, because the potential presence
B. LLVM libFuzzer of spurious mismatches would require manual analysis to confirm
that they are indeed spurious (to avoid missing bugs). In case the
libFuzzer is an LLVM-based state-of-the-art coverage-guided bytestream is dropped, no coverage information is returned to the fuzzing engine that proved very effective in finding several SW fuzzer and hence the fuzzer considers that bytestream uninteresting bugs [20]. It aims to create input data (binary bytestreams) in order to and does not collect it. Otherwise, the bytestream is executed on the maximize the code coverage of the SUT (SW Under Test). Therefore, simulator and coverage information is returned to the fuzzer. This the SUT is instrumented by Clang compiler to report code coverage happens automatically by compiling the simulator with Clang and to libFuzzer. Input data is transformed by applying a set of pre- using the -fuzzer sanitizer (because we use LLVM libFuzzer, which defined mutations (shuffle bytes, insert bit, etc.) randomly. Input size is compatible with Clang). For simulation, we provide a test-case is gradually increased (when coverage starts to saturate). template, as RISC-V assembler (ASM) source file. As optimization, Technically, libFuzzer is linked with the SUT, hence performs so the test-case template is pre-compiled into an ELF and pre-loaded called in-process fuzzing, and allows to pass inputs to the SUT as into the simulator memory. Before each bytestream execution the well as receive coverage information back through specific interface simulator is cloned to preserve the initial state.
To improve the fuzzing process we use a custom mutator and is not allowed to leave the local bounds of the bytestream (due to coverage specification. The coverage specification is automatically a jump/branch). A branch instruction forks the execution path, by transformed into a source file that is embedded into the simulator cloning the abstract execution state S into ST and SF . The PC of and updated on every instruction execution. ST is updated with the branch offset, which is relative to the current Next, we present more details on the test-case format (Sec- PC and hence platform independent, accordingly (the PC of SF is tion IV-B) and filter (Section IV-C) as well as the custom mutator already set correctly to fallthrough to the next instruction). ST and (Section IV-D) and coverage encoding (Section IV-E). SF are processed independently. B. Test-case Template The instructions JALR, [M,S,U]RET, WFI, EBREAK and SFENCE.VMA are forbidden (the bytestream is dropped if they are Our test-case template builds on the RISC-V compliance testing reachable on any path). The reason is that JALR and [M,S,U]RET format [5] to ensure that the generated test-suite is directly applicable perform a register/CSR based jump. WFI (Wait For Interrupt) might to all platforms that support this standard format. It performs a halt a processor causing non-termination (since no interrupt is com- generic system initialization sequence (initialize core CSRs and ing). EBREAK can have a special semantic and SFENCE.VMA is a register a trap handler) and then enters the actual test-case body. privileged instruction that is often not implemented (which is not a Macros are used to mark the begin/end of code and data as well as bug by itself but a deliberate decision). All (six) CSR instructions are halt execution. The macros as well as compilation flags are platform forbidden too, due to highly platform dependent behavior of CSRs specific, thus we cannot rely on hardcoded absolute addresses to (we provide more details on the problem and potential solutions access memory or use as jump target (because code and data may in Section VI). be stored at different addresses per platform). Any instruction writing to a register RD, marks RD dirty. A The test-case body starts by initializing all registers: x0 to x29 load/store instruction is forbidden if its address register is dirty. are loaded from hardcoded memory values, x30 and x31 (chosen In addition, we also require that the immediate (which will be arbitrarily) are set to point into the middle of the data memory by added to the register address to obtain the final access address) using a label. Thus, x0 to x29 have equal values among all platforms is properly aligned, because the RISC-V ISA allows both aligned and hence can be used for comparable computations while x30 and and unaligned load/store instructions (which would lead to spurious x31 are platform specific but can be used as address for memory signature mismatches). accesses. The data memory is large enough to support any additional A path passes when reaching an illegal instruction (since the next immediate offset, i.e. [-2048, +2047]. instructions will not be reached due to the exception) or the end of The test-case body ends by first incrementing x26 (an arbitrary the bytestream. register to distinguish between cases where the test code executes For illustration Fig. 2 shows an example. The left side shows with/without exceptions) and then initiates the shutdown sequence an ASM program, that represents the bytestream. Each instruction that will write back all register values (except x30 and x31 since they is prepended by it’s (local) address (for simplicity we assume all have platform specific values) to the data memory and halts execution instructions are non-compressed, i.e. are 4 byte long). The right side (causing a signature dump). In case of an illegal instruction in the shows the three possible control flow paths through this program, bytestream, control is transferred to the trap handler, which initiates starting from the initial state. Each instruction execution (annotated the shutdown sequence (but bypasses the x26 increment). above a state transition) results in a new state. The current PC In-between start/end of the test-case body, the fuzzer generated and set of registers marked clear are shown below each state. The bytestream is injected. The template provides a list of jump instruc- ASM program (bytestream) is accepted by the filter because all tions (to the body end) at this point that will simply be overwritten paths are accepted. Please note, that the program contains a WFI with raw memory declarations, e.g. .word 0x12345678, for each instruction which is in the forbidden category. However, the WFI word in the bytestream. The number of jump instructions in the has no influence, since it is never reached on any path. Similarly, the template is large enough for the bytestream to not exceed it. ADD instruction at address 12 that marks x30 dirty is not reached and Please note, we also load and store the content of floating point hence the LW at address 28 succeeds. BLT and BEQ fork the active (FP) registers alongside the normal registers. However, we condi- path to continue at P CT =28, P CF =20 and P CT =16, P CF =28, tionally guard it with the definition of riscv fdiv, which is set by respectively. GCC when selecting a RISC-V ISA (-march flag) with FP support. Our filter currently supports the RV32GC ISA. Hence, the gener- C. Filter ated test-suite can be executed on any sub-ISA of RV32GC, such as RV32I, RV32IMC etc. To add a new instruction extension, the filter The filter works by performing an abstract local execution of needs to be extended as well. Otherwise, the filter will consider them the bytestream that traverses the local control flow and checks as illegal instructions and let them pass unconditionally. the reachable instructions alongside. The abstract execution state consists of a program counter (PC), a mark (clean/dirty) for each D. Custom Mutator register that indicates whether the register can be used as address for We integrate a custom mutator to provide the fuzzer with valid in- a memory access, and data structures to keep track of the control struction (opcode) patterns to increase the number and length of valid flow to avoid loops. At the beginning PC is set to zero (i.e. pointing instructions. Our mutator is attached through the libFuzzer provided to the beginning of the bytestream) and all registers are marked dirty interface and is called with equal probability to the existing mutators. except for x30 and x31 (since they are initialized with a label to the Basically, the mutator moves through the bytestream instruction by data memory by the test-case, recall Section IV-B). instruction (we use a 4 byte format) and injects valid opcodes, while The filter then repeats a fetch, decode and execute loop. Thus, it keeping all other parameters randomized by the fuzzer. Please note, checks whether the next instruction (based on PC) is compressed (the we only inject instructions that pass our filter (since the bytestream two least significant bits are not 11). Then, it decodes the instruction, will be dropped otherwise). Besides avoiding instructions from the increments PC by 4 (normal) or 2 (compressed) accordingly, and (ab- forbidden category, we only use small offsets for branch and jump stractly) executes the decoded instruction. To avoid loops, the filter instructions (might still be rejected by the filter but the probability essentially checks that the same PC is not revisited. Furthermore, PC is much smaller) and only inject load/store instructions that use
1 0: ADD x31, x2, x3 //mark x31 dirty LW x30,-16(x30)
23 4: JAL x2, 20 //mark x2 dirty, to PC=24 initial state intermediate state BEQ x30,x31,12 T pass
8: WFI //fobidden, drop bytestream PC=28, {x30}
4 12: ADD x30, x2, x3 //mark x30 dirty final state (of a path) Legend PC=16, F
5 16: BLT x30, x31, 12 //fork, to PC=28:20 T illegal pass
67 20: illegal //accept path ADD x31,x2,x3 JAL x2,20 BEQ x1,x2,-8 {x30} PC=20, {x30}
24: BEQ x1, x2, -8 //fork, to PC=16:28 LW x5,-16(x30)
8 28: LW x5, -16(x30) //require x30 clean PC=0, PC=4, PC=24, F pass
9 //accept path {x30,x31} {x30} {x30} PC=28, {x30}
Fig. 2. Left side shows an example RISC-V ASM program (for a bytestream with 32 byte) and right side shows the corresponding control flow paths
31 25 24 20 19 15 14 12 11 7 6 0 31 20 19 15 14 12 11 7 6 0
0000000 xxxxx xxxxx 000 xxxxx 0110011 xxxxxxxxxxxx x31/x30 010 xxxxx 0000011
opcode RS2 RS1 opcode RD opcode I_imm RS1 opcode RD opcode
ADD: Regs[RD] = Regs[RS1] + Regs[RS2] LW (Load Word): Regs[RD] = Mem[Regs[RS1] + I_imm]
Fig. 3. Format and semantic for the ADD and LW instruction. The opcodes are injected by our mutator, the other fields remain randomized (each random bit is
denoted with an x). Special constraints are used (such as setting RS1 to x30 or x31 for LW) to pass our filter.
x30 or x31 as address register. For illustration Fig. 3 shows the bit (instruction set) simulator of the open source RISC-V VP [21],
instruction format and semantic for ADD and LW. When injecting [22]. Next, we first provide more information on the fuzzer-based
the ADD instruction, the RS1, RS2 and RD fields remain random, test-suite generation process (Section V-A) and then present results
but all opcode fields are overwritten by the mutator, thus making the on the compliance testing evaluation (Section V-B). All experiments
instruction a valid (though randomized) ADD. Similarly, for the LW have been performed on a Linux system with an Intel Core i5-7200U
instruction, though here rs1 is always set to either x30 or x31 register processor.
to pass the filter. A. Test-suite Generation
E. Custom Coverage Fig. 4 shows execution information for four different fuzzing
By default the fuzzing process is guided by code coverage emitted configurations (v0 to v3) that use different coverage metrics: v0
by the simulator that executes the bytestreams. We consider two uses only code coverage of the ISS; v1 adds the custom coverage
additional coverage metrics. rules (structural and value metrics) to v0 (additional 2281 coverage
The first is a hash-based coverage that is simple, generic and scal- points); v2 and v3 add hash coverage with 4096 and 16384 coverage
able. Basically it computes a small hash value of the instruction word points to v1, respectively. Fig. 4 shows how the number of test-
and considers every different hash value as new coverage. This adds cases grows compared to the number of fuzzer executions (i.e. over
a significant amount of variance and randomness to the generated time). The runtime is fixed to 30 minutes for each configuration. We
test-suite. Technically, we use a C++ std::hash<uint32 t> fn hash use a 64 byte input length limit for the fuzzer and configured it to
function. Then, every fetched instruction is passed through a (large) increase the input length more slowly (-len control=10000). It can
switch statement: switch (fn(fetched word) % N). N is the config- be observed that the number of test-cases grows very rapidly in the
urable number of hashes to use. Inside the switch statement we gener- first quarter and then gradually saturates (please note the logarithmic
ate N cases, for i ∈ {0, ..., N }, as case i: asm volatile (””); scale on the X axis). The average executions per second are at 45,873
break;. The asm volatile statement ensure that the cases are with the minimum at 12,302 and maximum at 68,873. To achieve
not removed by the compiler. this high performance, it has been particularly important to pre-
The second coverage reasons about structure and values of compile and pre-load the test-case template and use a small simulator RISC-V instructions. It is provided through an external specification memory size (32 KB). The highest measured memory consumption file. It can strengthen the fuzzer in the field of positive testing on our evaluation system has been 1063 MB for configuration v3. by collecting further test-cases with valid instructions. Basically, The coverage metric is very important since it has immediate impact we use a small set of rules such as: 1) RD=x0, 2) RD=x0, 3) on the fuzzing process. First, on the performance, since the coverage RD=RS1, and 4) RD=RS1. Each rule is applicable to instructions needs to be tracked (which costs time) and it influences how fast that have the corresponding fields and defines a coverage point with the fuzzer increases the input size (every time the coverage starts to the rules condition, for example if (decoded instruction.opcode == saturate), which in turn increases the probability that our filter drops ADD && decoded instruction.RD == x0) asm volatile ; more inputs. Second, on the number of generated test-cases, since for rule 1 and opcode ADD (all matching opcodes are enumerated)3. the fuzzer only collects test-cases that increase coverage. For the The first and second rule are due to the RISC-V hardwired x0 following compliance testing evaluation we use the v3 configuration. register. The third and fourth are useful to check for effects where the update order is not correct. Similarly, we have a rule for three B. Compliance Testing registers (all equal, all not equal, etc). Finally, we use value rules We consider five different RISC-V simulators, which all sup- Reg[RS1] OP Reg[RS2] with OP ∈ {=, =, <, >} and Reg[RS*] port the RISC-V compliance testing format, in this evaluation4: ∈ {M IN, M AX, −1, 0, 1}, and similar rules for immediates. riscvOVPsim, Spike, VP, GRIFT and sail-riscv. riscvOVPsim [5] (see V. EXPERIMENTS the riscv-ovpsim folder) is the reference simulator for compliance We have implemented our fuzzer-based approach for RISC-V 4We also briefly evaluated the rocket and Ibex cores, since they are listed as compliance testing and evaluated its effectiveness on a set of RISC-V targets in the compliance testing repository. Both (RTL) cores can be compiled simulators. As foundation for the fuzzing process we use the 32 into a (C++) simulator using verilator. However, the rocket simulator had problems with the compliance testing format (it failed every basic RV32I test) and the Ibex simulator stopped on the first exception (e.g. illegal instruction) 3We use a slightly optimized implementation by using switch case statement without dumping a signature, which makes it not applicable in combination with for the opcode and grouping all rules below the opcode. negative testing. Thus, we omitted these cores from the evaluation.
14000 TABLE I
v3: number test-cases=13540 NUMBER OF SIGNATURE MISMATCHES AGAINST riscvOVPsim
v2: number test-cases=8531
12000 v1: number test-cases=4066 RISC-V ISA Spike VP sail-riscv GRIFT
v0: number test-cases=689 RV32I 7 5 crash 124
10000 RV32IMC 9 32 crash 1047
RV32GC 9 / / 141
8000
• VP uses a wrong mask for the ECALL instruction in the
6000 decoder which allows an invalid instruction to be decoded and
executed as an ECALL. In addition, reserved non-hint com-
4000 pressed instructions, e.g. ”c.lwsp x0, 0(sp)”, are erroneously
normally expanded and executed without causing an illegal
2000 instruction exception5.
• GRIFT updates the RA register on an invalid jump (target
0 address not 32 bit aligned on RV32I) before triggering an
10⁰ 10¹ 10² 10³ 10⁴ 10⁵ 10⁶ 10⁷ 10⁸ illegal instruction exception (which is incorrect, since ille-
number of fuzzer executions gal instruction should have no side effects). Furthermore, the
Fig. 4. Fuzzer execution information for different settings (30 min. runtime) RV32IMC compliance testing target has been incorrectly con-
figured to RV32GC, thus floating point and atomic instructions
testing, i.e. it is used to generate the reference signatures by the are erroneously accepted as well. In addition, similar to VP,
compliance task group. Spike [23] is the official reference simulator reserved non-hint compressed instructions are also erroneously
for RISC-V from UC Berkeley that aims to be an executable golden accepted as legal instructions. Finally, we also found the bug
model for the RISC-V ISA specification. VP [21] is a RISC-V based that SC.W instruction performs memory access even without
Virtual Prototype implemented in SystemC TLM. GRIFT [24] is a pending LR.W reservation (which was the only bug found by
Haskell-based RISC-V formalization that aims to provide the foun- the official compliance test-suite).
dation for several analysis techniques for RISC-V. sail-riscv [25] is • sail-riscv has several incomplete decoder checks that cause
implemented in Sail, a special language for describing ISAs with invalid instructions to be accepted as valid ones. Some in-
support for generation of simulator backends as well as theorem- puts crashed sail-riscv, others led to non-termination (which
prover definitions, and aims to become another (executable) formal- indicates that an invalid instructions has been interpreted as a
ization of the RISC-V ISA. backward branch/jump).
Except for GRIFT which currently fails one test-case of the • riscvOVPsim accepts opcodes reserved for custom (non-
compliance test-suite (in the A extension), all simulators pass all compressed) instruction extensions which should cause an il-
applicable compliance test-suites (some test-suites are not applica- legal instruction exception on the base ISA configuration (to
ble, because the simulator simply does not implement the respective trigger this error on riscvOVPsim an additional special bit
RISC-V ISA extension support). pattern must be set as well in the instruction besides the
Following the compliance testing convention, we also use opcode). All other simulators that we tested (including Spike),
riscvOVPsim to generate reference signatures. In addition, we con- correctly performed a jump to the trap handler due to an illegal
sider three different RISC-V RV32 ISA configurations, namely: instruction exception in this case.
RV32I, RV32IMC and RV32GC. With our approach we observed Our evaluation clearly shows that a fuzzing-based approach is an
several errors and inconsistencies on every simulator (across all enrichment for compliance testing. In contrast to the existing com- ISA configurations) that we used in this experiment. Most errors pliance test-suite, which focuses on positive testing (i.e. use hand- are related to incorrect decoding of instructions, which can be written well-defined tests) to check that the required instructions are particularly well detected with fuzzing based approaches, but we correctly implemented, our fuzzing-based approach is complemen- also found logical execution errors in valid instructions and some tary by focusing on negative testing (i.e. to check that no additional other issues in the (internal) simulator implementation. Table I shows functionality is accidentally added). It is very well suited for decoder a summary of the results for an exemplary fuzzer generated test- checking, in particular that no illegal instruction passes as a legal suite (the v3 configuration test-suite as discussed in Section V-A). It one and that no additional instructions have been accidentally imple- shows the number of signature mismatches that we observed between mented, and testing against unexpected cases. Such error cases are riscvOVPsim and the respective simulator (shown in Column 2 very hard to detect, because the compiler does not generate illegal to 5) for each of the three ISA configurations (Rows 2 to 4) by instructions and the existing RISC-V testing frameworks provide running the test-suite (/ means not supported by simulator). It takes only very limited support. In addition, the inherent randomness of 10 to 20 minutes, depending on the simulator, to run (and also fuzzing enables it to be re-run continuously. In combination with compare the generated signatures) the whole test-suite for a single our carefully designed and fully compliance testing compatible test- ISA configuration. Please note, that the results can vary slightly, case format and static analysis based filter, a fully automated and due to the randomness in the fuzzer. However, we consistently comprehensive negative testing is enabled. Finally, our approach is observed several mismatches for each fuzzer run. We consider this not fixed to a specific RISC-V ISA but applicable to different ISA randomness actually a strength of our approach, since it can also be configurations (currently any combination of RV32GC) and is easily used for continuous negative testing to obtain more comprehensive extendable to support additional instruction set extensions. results. Next, we present our findings in more detail, grouped by the respective simulator: 5RISC-V distinguishes different cases of reserved compressed instructions, some are merely hints which execute as NOP, others are reserved opcodes for • Spike dumps an incorrect test signature in case of an ECALL internal and custom extension and should trigger an illegal instruction exception instruction inside of the test body. when attempted to be executed on an ISA without the respective extension.
number of generated test-cases
VI. DISCUSSION AND FUTURE WORK: more accurately. 2) Step 1 is a good foundation to develop suitable
THE CSR CHALLENGE coverage metrics for CSR testing to quantify the testing effort. 3)
Our experiments demonstrated the effectiveness of our fuzzer- Extend the compliance testing signature with don’t care values to
based approach for finding errors and inconsistencies through neg- deal with conditional behavior, i.e. store a second file alongside the
ative testing. It is complementary to the positive testing approach of signature that describes which parts can be ignored in the comparison
the official compliance testing test-suite and in combination provides on which condition (e.g. ignore the reference output of MTVAL if the
a strong compliance testing framework for the unprivileged RISC-V test output assigned a zero to MTVAL).
ISA specification. However, one large and important open challenge VII. CONCLUSION
is compliance testing of the privileged ISA, in particular the CSRs. In this paper we proposed a fuzzing-based approach to provide
In contrast to the instruction set specifications, the CSR behavior is strong negative testing capabilities for RISC-V compliance testing,
much less clearly and unambiguously defined. Next, we exemplarily which complements the existing official test-suite that focuses on
discuss the most relevant Machine mode CSRs: positive testing (of the unprivileged RISC-V ISA). We found new
MTVAL provides exception specific information (for example, bugs in several RISC-V simulators including riscvOVPsim from the instruction data in case of an illegal instruction exception). How- Imperas (the official reference simulator for compliance testing). ever, it is also legal behavior to simply set MTVAL to zero in case the Finally, we reviewed the still open challenge in compliance testing feature is not supported (for some exception). Conditional behavior of the privileged RISC-V ISA (CSRs in particular) and sketched like this cannot be handled with the current compliance testing possible solutions. approach, since the reference signatures are generated in advance and compared unconditionally to the output signatures. Bits in MIP and REFERENCES MIE can be hardcoded to zero if the respective interrupt source is not available, thus in one case a write succeeds in the other case it [1] A. Waterman and K. Asanovi´c, The RISC-V Instruction Set Manual; Volume I: may be ignored and both are legal behaviors. MSTATUS has flags Unprivileged ISA, SiFive Inc. and CS Division, EECS Department, University which are optional and exhibit different behavior in Machine and [2] of California, Berkeley, 2017. “RISC-V foundation,” https://riscv.org/risc-v-foundation/. Supervisor mode. MSCRATCH can be used by the implementation [3] L. Moore, S. Davidmann, and L. Lapides, “Compliance methodology and ini- at will, and hence can arbitrarily change its value in an architecture tial results for RISC-V ISA implementations,” in Embedded World Conference, specific way. Timing related CSRs such as MCYCLE and MTIME, 2019. as well as the performance counter (which are also optional and can [4] B. Bailey, The Challenge Of RISC-V Compliance, https://semiengineering. be hardwired to zero), yield architecture specific results and thus [5] com/toward-risc-v-compliance/. “RISC-V compliance task group,” https://github.com/riscv/riscv-compliance. should not be compared. Even MINSTRET that simply counts the [6] A. Adir, E. Almog, L. Fournier, E. Marcus, M. Rimon, M. Vinov, and A. Ziv, number of executed instructions cannot be used as signature, because “Genesys-pro: innovations in test program generation for functional processor some platforms use a built-in hardcoded initialization sequence and [7] verification,” D&T, pp. 84–93, 2004. the compliance testing framework uses customized initialization and B. Campbell and I. Stark, “Randomised testing of a microprocessor model shutdown sequences (hence executes different number of instruc- using SMT-solver state generation,” in Formal Methods for Industrial Critical Systems, F. Lang and F. Flammini, Eds., 2014, pp. 185–199. tions). MCAUSE and MEPC are only guaranteed to hold supported [8] Y. Katz, M. Rimon, and A. Ziv, “Generating instruction streams using abstract exception codes and valid virtual addresses, respectively. Invalid CSP,” in DATE, 2012, pp. 15–20. addresses may be freely converted by the implementation before [9] M. Chupilko, A. Kamkin, A. Kotsynyak, and A. Tatarnikov, “MicroTESK: writing them to MEPC. MTVEC can also contain a hardwired read- specification-based tool for constructing test program generators,” in HVC, only value, which again obviously can be architecture specific and [10] 2017. S. Fine and A. Ziv, “Coverage directed test generation for functional verifica- hence need to be considered to be arbitrary. Similarly, PMP registers tion using bayesian networks,” in DAC, 2003, pp. 286–291. (memory protection) are optional. [11] C. Ioannides, G. Barrett, and K. Eder, “Feedback-based coverage directed test Furthermore, Supervisor and User modes are often hardcoded and generation: An industrial evaluation,” in Hardware and Software: Verification cannot be deactivated (which is even the case for Spike). Hence, [12] and Testing, S. Barner, I. Harris, D. Kroening, and O. Raz, Eds., 2011. access to supervisor and user mode CSRs is possible as well, even L. Martignoni, R. Paleari, G. F. Roglia, and D. Bruschi, “Testing CPU emula- tors,” in ISSTA, 2009, pp. 261–272. though the test is runs in Machine mode and is supposed to test [13] “RISC-V torture test generator,” https://github.com/ucb-bar/riscv-torture. Machine mode CSRs. In addition, this has impact on Machine mode [14] “RISCV-DV,” https://github.com/google/riscv-dv. CSRs such as MISA, which has the Supervisor and User bits set too. [15] V. Herdt, D. Große, H. M. Le, and R. Drechsler, “Verifying instruction set In summary the privileged architecture, in particular the CSRs, [16] simulators using coverage-guided fuzzing,” in DATE, 2019, pp. 360–365. “RISC-V formal verification framework,” https://github.com/SymbioticEDA/ expose architecture specific information and provide a large degree riscv-formal. of freedom for implementations. The consequence is that basically [17] “OneSpin 360 DV RISC-V Verification App,” https://www.onespin.com/ all simulators/cores implement a (more or less slightly) different solutions/risc-v. subset/configuration of the privileged architecture, yet all of them [18] V. Herdt, D. Große, and R. Drechsler, “Towards specification and testing of can still be compliant to the RISC-V specification. This signifi- [19] RISC-V ISA compliance,” in DATE, 2020. A. Waterman and K. Asanovi´c, The RISC-V Instruction Set Manual; Volume cantly complicates compliance testing, since it is very difficult to II: Privileged Architecture, SiFive Inc. and CS Division, EECS Department, automatically test for correct behavior. We envision three directions University of California, Berkeley, 2017. for future work: 1) It seems viable to tackle the CSR problem by [20] “libFuzzer - a library for coverage-guided fuzz testing,” https://llvm.org/docs/ writing very fine grained tests6 for each CSR and then select them [21] LibFuzzer.html, 2018. dynamically for each tested platform. This requires the RISC-V V. Herdt, D. Große, P. Pieper, and R. Drechsler, “RISC-V based virtual prototype: An extensible and configurable platform for the system-level,” JSA, privileged architecture specification (which admittedly still officially 2020. is a draft) to list capabilities and interdependencies between CSRs [22] V. Herdt, D. Große, H. M. Le, and R. Drechsler, “Extensible and configurable RISC-V based virtual prototype,” in FDL, 2018, pp. 5–16. 6Including specialized tests that not just compare the whole (CSR) state, e.g. [23] “Spike RISC-V ISA simulator,” https://github.com/riscv/riscv-isa-sim. testing of a performance counter could check that the counter increments when [24] “GRIFT - galois RISC-V ISA formal tools,” https://github.com/GaloisInc/grift. enabled but not care about the exact architecture specific counter value. [25] “Riscv sail model,” https://github.com/rems-project/sail-riscv.