Mutation-based Compliance Testing for RISC-V

                  Vladimir Herdt                                                                              Sören Tempel

Institute of Computer Science, University of Bremen Institute of Computer Science, University of Bremen Cyber-Physical Systems, DFKI GmbH Bremen, Germany Bremen, Germany tempel@uni-bremen.de vherdt@uni-bremen.de Daniel Große Rolf Drechsler Institute for Complex Systems, Johannes Kepler University Institute of Computer Science, University of Bremen Linz, Austria Cyber-Physical Systems, DFKI GmbH Cyber-Physical Systems, DFKI GmbH Bremen, Germany Bremen, Germany drechsler@uni-bremen.de daniel.grosse@jku.at ABSTRACT fixed standard extensions can be added. This includes for instance Compliance testing for RISC-V is very important. Essentially, it integer multiply/divide, single- and double precision floating point, ensures that compatibility is maintained between RISC-V imple- atomic memory operations among many others. Finally, custom mentations and the ever growing RISC-V ecosystem. Therefore, instructions can be added to create application specific solutions. an official Compliance Test-suite (CT) is being actively developed. However, there is a strong risk when allowing such an enormous However, it is very difficult to achieve that all relevant functional flexibility: fragmentation of the RISC-V ecosystem. Simply speak- behavior is comprehensively tested. ing, if designers customize their RISC-V core too much, the benefits In this paper, we propose a mutation-based approach to boost of the common ecosystem are lost as (a) significant adoptions in RISC-V compliance testing by providing more comprehensive test- the tools become necessary and (b) sharing of implementations, ing results. Therefore, we define mutation classes tailored for SDKs, etc. is hindered. This very important problem is addressed RISC-V to access the quality of the CT and provide a symbolic execu- with compliance testing. In contrast to verification, which attempts tion framework to generate new test-cases that kill the undetected to prove that an implementation is correct, compliance testing at- mutants. Our experimental results demonstrate the effectiveness tempts to show that an implementation meets the standard and thus of our approach. We identified several serious gaps in the CT and ensures compatibility with the RISC-V ecosystem. More precisely, generated new tests to close these gaps. compliance testing checks whether registers are missing, modes are not there, instructions are absent, corner-case scenarios are work- KEYWORDS ing as specified, and is performing basic functional sanity-checks RISC-V, Compliance Testing, Mutation, Instruction Set Simulation, for each instruction. Symbolic Execution The importance of compliance testing has been recognized very early by the RISC-V foundation and as a consequence the compli- ACM Reference Format: ance task group has been formed [7]. Intensive ongoing discussions Vladimir Herdt, Sören Tempel, Daniel Große, and Rolf Drechsler. 2021. are taking place on how to proceed in order to develop suitable Mutation-based Compliance Testing for RISC-V. In 26th Asia and South tools, models, and methodologies to solve the RISC-V compliance Pacific Design Automation Conference (ASPDAC ’21), January 18–21, 2021, testing problem [2]. The official approach pursued by the task group Tokyo, Japan. ACM, New York, NY, USA, 6 pages. https://doi.org/10.1145/ is a Compliance Test-suite (CT) [7]. The quality of the test-suite is 3394885.3431584 monitored by leveraging functional coverage metrics (and they 1 INTRODUCTION guide further development of the test-suite) and recent reports in- RISC-V [32, 33] is an open and free Instruction Set Architecture (ISA) dicate that a very high quality of the base RV32I test-suite has been and as such has evolved from academic research into mainstream reached [3]. However, it is important to cross-validate these results adoption. RISC-V is very modular by defining 32, 64 and 128 bit using different methods because it is very challenging to ensure integer ISAs in the base specification. Moreover, to the base ISA that all relevant functional behavior is comprehensively tested. In this paper, we propose a mutation-based approach to boost Permission to make digital or hard copies of all or part of this work for personal or RISC-V compliance testing by providing more comprehensive test- classroom use is granted without fee provided that copies are not made or distributed ing results1. Therefore, we define mutation classes tailored for for profit or commercial advantage and that copies bear this notice and the full citation RISC-V to access the quality of the CT. In particular, we consider on the first page. Copyrights for components of this work owned by others than ACM mutations in the execution unit that mask register and immediate must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a values, modify constants, replace the accessed registers, immediates fee. Request permissions from permissions@acm.org. and operators as well as check for exception side-effects. Based on ASPDAC ’21, January 18–21, 2021, Tokyo, Japan © 2021 Association for Computing Machinery. 1Visit http://www.systemc-verification.org/risc-v for our most recent RISC-V related ACM ISBN 978-1-4503-7999-1/21/01. . . $15.00 https://doi.org/10.1145/3394885.3431584 approaches.

these mutation classes we show that several serious gaps in the found their way into commercial tools, like Certitude from Synop- official CT do exist, and the most recent specification-based compli- sys. This goes back to [17] and is referred as functional qualification. ance testing approach [20] is unable to detect the critical mutants. The principles have been further advanced in [28]. The method Therefore, we provide a symbolic execution framework to generate generates high coverage input vectors for RTL designs by recording new test-cases that kill the remaining mutants. Our experimental branch coverage controlled via mutated guards during symbolic results demonstrate the effectiveness of our approach. Our final simulation. Enhancements for guiding the stimuli generation pro- extended CT closes all gaps and detects bugs in RISC-V simulators. cess with mutation analysis have been presented in [34]. However, while all these works identify the weaknesses of the test-stimuli 2 RELATED WORK or even improve them, they cannot directly be used to generate compliance tests. Compliance testing for RISC-V is an emerging research area. Even the first steps towards the official CT started only in 2018, and 3 BACKGROUND ON RISC-V hence only recently gained significant momentum. Therefore the In this work we consider the RISC-V base RV32I ISA. It defines a 32 number of research papers covering this topic is still limited. We are bit core without any extensions. It has 32 general purpose registers only aware of [18, 20] that specifically target the compliance testing x0 to x31 (with x0 being hardwired to zero) each 32 bit width. problem. [18] leverages coverage-guided fuzzing to generate the CT. Instructions are grouped into different classes (i.e. computational, It primarily focuses on negative testing, i.e. illegal instructions and load/store, branch/jump). They access registers (source: RS1 and exceptions, and thus complements our mutation-based approach. RS2, destination: RD) and immediates to perform their operation. [20] defines a test-suite specification mechanism and leverages Immediates are available in different sizes and signed/unsigned constraint solving techniques to generate a CT according to the interpretation. For example, I_imm is a signed 12 bit immediate, specification rules. It focuses on positive testing aspects and the thus has a value range of [-2048,...,2047]. This allows to define specification mechanism is well suited to reason about different instructions such as ADDI x1, x2, 128 which adds the value of x2 register and immediate values. In contrast to our mutation-based (RS1) with 128 (I_imm) and stores the result in x1 (RD). Format approach, which considers the problem from the angle of inserting and semantics (for the base ISA and extensions) are defined in the bugs, [20] provides a value-driven specification, i.e. for instance unprivileged ISA specification [32]. what value ranges are expected for a specific instruction, and thus In addition, the privileged (architecture) specification [33] covers test-cases are generated with another objective. further important functionality required for environment interac- For the purpose of verification, a set of test generation ap- tion and operating system execution (for example virtual memory proaches specifically targeting RISC-V have also emerged re- support and interrupt handling). In particular, it defines CSRs (Con- cently [1, 10, 22]. The Scala-based Torture Test generator [1] gen- trol and Status Register) and special instructions to access them. erates tests by integrating pre-defined randomized test-sequences. RISCV-DV [10] by Google leverages SystemVerilog in combination 4 with UVM (Universal Verification Methodology) to generate RISC-V MUTATION-BASED COMPLIANCE TESTING instruction streams based on constrained-random descriptions. It This section presents our mutation-based approach to boost RISC-V requires a commercial RTL simulator providing SystemVerilog and compliance testing. We start with an overview (Section 4.1) and then UVM support. Finally, [22] utilizes fuzzing techniques to generate present our mutation classes tailored for RISC-V (Section 4.2). Next, randomized instruction streams as platform dependent binary files we provide more details on how to kill mutants based on symbolic (ELFs). These approaches are designed to continuously generate execution (Section 4.3) and generate compliance test-cases based (randomized) test-cases for verification purposes. Furthermore, they on the solutions (Section 4.4). Our approach thus complements the do not support the compliance testing format. existing CT infrastructure. Beside test-generation methods, there are also a few formal veri- fication approaches for RISC-V. Notable approaches that leverage 4.1 Overview model checking are riscv-formal [8] and the OneSpin 360 DV RISC-V Fig. 1 shows an overview on our approach. Starting point is a set of verification app [6]. However, both approaches clearly target the mutation classes and a reference Instruction Set Simulator (ISS). Each verification of an implementation. mutation class describes how to generate a set of mutants. Each Looking beyond RISC-V, several approaches for test-program mutant represents a mutation in the reference ISS, i.e. a mutant is a generation have been proposed for the purpose of verification. For mutated ISS. In this work we focus on mutations in the execution example, they integrate model-based techniques with constraint unit of the ISS (we provide more information on the mutation solving [13–15, 27] or leverage coverage-guided test generation classes in Section 4.2). based on Bayesian networks [16] and other machine learning tech- In the first step, the set of mutation classes is processed in com- niques [25] as well as fuzzing [29]. bination with the reference ISS to generate a set of mutants. Then, Mutation testing has been intensively investigated over several each mutant is checked against the CT. Killed mutants are consid- decades and has its roots in the software domain as a fault-based ered uninteresting and thus filtered out. A mutant is killed, if it software testing technique (a survey can be found in [26]). In the is detected by at least one test-case of CT (i.e. the mutant and the context of hardware, approaches based on injecting faults into RTL reference produce different results on this test-case). designs to determine the quality of the test-cases have been pro- The (still) alive mutants are passed to the mutation solver (Step 3). posed, for instance [30]. Mutation-based testing and analysis also It leverages a symbolic execution framework to generate specific

              Mutation Class    Compliance                                                             Symbolic                      Solution
                              Testsuite (CT)                                                       Execution Engine
                                                                                               use for
                                                                                               symbolic
                                                                                               reasoning
   Reference      1: Mutation  2: Mutation                                                           3: Mutation                     4. Testcase
   ISS      Generator             Filter                                                                Solver       mutation        Generator
                                                                                           no solution             killed
                                                                                                                                         CT extension

   Mutants are
   mutations in  Original                                   Alive                                        Rest                         RISC-V
   reference ISS  Mutant          Mutant                                                                Mutant                       Testcase

       Figure 1: Overview: mutation-based approach for RISC-V compliance testing

inputs to kill the mutants. Such an input is called a solution. It 1 switch ( op ) { // execute instruction in ISS is possible, that no solution exists, because the mutation has no 23 // ...case BEQ : // Branch EQual instruction influence on the result (e.g. replacing an integer X with X+0). 4 if ( mutation_begin () ) { // mutated paths A solution is essentially a snapshot of the relevant execution 56 if pc( regs [ instr . rs1 () ]= == regs [ instr . rs2 () ]) last_pc - instr . B_imm () ; state and a single instruction that will show a difference in the 7 } else { // unmutated paths output behavior between the reference ISS and mutated ISS when 8 if ( regs [ instr . rs1 () ] == regs [ instr . rs2 () ]) 9 pc = last_pc + instr . B_imm () ; executed from that state. 10 } In the last step the solution is transformed into a RISC-V com- 11 mutation_end () ; 12 break; pliance test-case. The generated test-suite complements the CT in 13 // ... providing stronger test coverage. Furthermore, it allows to access 14 } the quality of the existing CT and reveal coverage holes. Figure 2: Example mutation for the BEQ instruction. 4.2 Mutation Classes We consider mutations in the execution unit of the reference ISS. Each mutation class defines a set of mutations. Only a single In total, we define nine mutation classes with rules tailored for ISS mutation is selected and applied at a time. mutations in the context of RISC-V: M1 Modify the value of a source register by applying a mask 4.3 Killing Mutations via Symbolic Execution that sets a single bit to zero. This ensures that all register Killing a mutation is considered on a per instruction basis. Fig. 2 bits are comprehensively tested. illustrates the basic idea using the BEQ instruction as an example. M2 Similar to M1 but applied on immediate values. The range The normal instruction execution code (Line 8-9) is duplicated and of the mask is set to cover the whole immediate range. the mutation is applied (Line 5-6). In this example the binary oper- M3 Replace a constant with another one by adding or removing ator plus (Line 9) is replaced by minus (Line 6). The resulting code a bit from the left or right side. This ensures that similar block is wrapped with calls to the artificial mutation_begin and constants are used. mutation_end functions (Line 4 and Line 11, respectively). These M4 Replace a load instruction with another one (e.g. load word functions mark the mutation area and are recognized by the sym- with load byte) or replace a store instruction, respectively. bolic execution framework. M5 Replace any of the RS1, RS2 or RD register with each other, To find a solution that kills the mutation, we provide a single e.g. ADD x1, x2, x3 could be mutated to ADD x2, x2, x3 symbolic instruction to the ISS and make the ISS register file un- by replacing RD=x1 with RS1=x2. Each register can also be constrained symbolic. Additionally, we overwrite ISS functions replaced with the hardwired zero register. which are used to access memory, thereby making load/store in- M6 Similar to M5 but applied to immediates, e.g. replace I_imm struction operate on symbolic values. The program counter is set with S_imm, etc. to a fixed concrete value. Based on the symbolic instruction, a path M7 Replace unary operations {!,-,∼} with each other or remove to mutation_begin is searched. At this point, we distinguish two the operation. Another mutation is to replace the new and the sets of paths trough the marked code block: The set of mutated current PC, i.e. which will cause being of by one instruction paths and the set of unmutated paths based on the return value in jumps and return address computations. of mutation_begin (which is controlled by the symbolic execution M8 Replace binary operations with each other. Distinguish be- engine). Then, we calculate the Cartesian product of these two sets, tween computational operations and relational operations. thereby enumerating all possible combinations of unmutated and M9 Move a trap check to the end of the instruction execution, mutated paths. For each resulting combination, we add additional i.e. this will cause side effects (writing a register) to apply solver constraints and search for a solution that kills the mutation. before taking the trap. Through these additional constraints we ensure that only aligned

1 /* [ BEQ template ]** / 15 / [ LW template ]*** */ 5 EXPERIMENTS 2 // .. init relevant regs .. 16 LA RS1 , data_middle 3 J to_beq 17 LW RD , ( I_imm ) RS1 We have implemented our proposed approach for mutation-based 4 ADDI x1 , x1 , 1 18 halt : compliance testing using the ISS of the open source RISC-V VP [19, 56 // .. cover branch range .. 19 // .. exit sequence .. 21, 23] as reference ISS. We focus on the base RV32I ISA, use the ADDI x1 , x1 , 1 20 data_begin : 7 J halt 21 . byte b0 mutation classes described in Section 4.2 as foundation for the 89 to_beq : 22 // ... testing process and leverage angr [4, 31] as symbolic execution back- 10 BEQ RS1 , RS2 , B_imm 23 . byte b2047 end. Beside the official CT [7], we also consider the specification- ADDI x2 , x2 , 1 24 data_middle : 11 // .. cover branch range .. 25 . byte b2048 based CT from [20] in this evaluation. All experiments have been 12 ADDI x2 , x2 , 1 26 // ... performed on a Linux system with an Intel i5-7200U processor. We 13 halt : 27 . byte b4095 14 // .. exit sequence .. 28 data_end : start with a results overview and then present results on the official, Figure 3: Example template for the BEQ and LW instruction. specification-based and our mutation-based CT. Result Overview. Table 1 shows the results. The first two columns show the mutation class and the number of mutants in this class memory addresses are considered and non-terminating self loops (column: #mutants). The remaining columns report results on: 1) are avoided. A mutation is killed, if a difference in the output be- the official CT, 2) the specification-based CT and 3) our symbolic havior is observed. For this reason, we compare the register values, execution framework, to kill the mutants. These three steps are addresses used to access memory, values used to read (or write) applied one after another and only mutants that are still alive (i.e. memory and the PC. If a solution has been found, concrete values not killed) are passed to the next step. The columns #killed and for the register file and memory addresses as well as used memory #alive report the number of killed and still alive mutants after each values are provided (if any). step. In addition, the runtime in seconds is reported for each step. 4.4 Test-case Generation Official CT. It can be observed, that the official CT already pro- Test-case generation transforms a solution into a test-case. We vides strong results in killing the mutants. An average of around use a custom template for the RISC-V compliance test format. The 92% of mutants across all nine mutation classes is killed. It takes template contains case distinctions for different instruction types. around four hours to process all mutants with CT. This corresponds For a computational instruction, we simply generated instruc- to around 6 seconds per mutation. Most of the time is spend in tions to initialize all registers according to the solution and then put executing the 48 tests one after another, which involves loading the the generated instruction into the (RISC-V assembly) test-case. The ELF test files as well as writing and comparing signature results files difference between a mutated and unmutated ISS will be observed and glue code written in Python. Though, performance optimiza- in the resulting register file. tions would be possible in this area (for example by pre-loading For branches and jumps we generate code which ensures that ELFs and signature files) when they become necessary. a difference in the register outputs will be observed in case the While only around 8% of the mutants are not killed, careful anal- resulting PC differs. Fig. 3 shows the test code template. First, the ysis of these alive mutants revealed several interesting error classes registers are initialized based on the solution. Then, a jump to the that are not detected by CT. We discuss them in the following: instruction (BEQ in this case) is performed. Before and after the (1) RISC-V provides three register-based shift operations SLL generated instruction, we add enough enough ADDI instructions to (Shift Left Logical), SRL (Shift Right Logical) and SRA (Shift cover the maximum possible range of the 12-Bit branch immediate. Right Arithmetic). They shift the value of register RS1 by Different destination registers are used in the forward and back- the value of register RS2 and store the result in register RD. ward ADDI to ensure that a different result is always produced (for According to the RISC-V specification, only the lower 5 bit negative/positive immediates). Since difference in output behaviour of the RS2 register should be used for shifting, i.e. the RS2 of branch instructions can only be observed in the PC, we know register is masked by 0b11111. However, changing the mask that the PC of a mutated/unmutated ISS will differ if a solution was to also use upper bits of RS2 for shifting is not detected by found by our symbolic execution engine. CT. We observed similar problems on the immediate-based For load and store instructions the generated solution contains shift instructions. a concrete memory base address (as stored in the RS1 register), the (2) RISC-V provides six branch instructions that perform a condi- relative immediate offset and used memory values. Fig. 3 illustrates tional relative jump. The BEQ (Branch if registers are EQual) the test generation for the LW (Load Word) instruction. First, the instruction is not sufficiently tested by CT. Most of the muta- fixed RS1 base address is replaced by a label (Line 16, LA = Load tions that mask the branch immediate have not been found. Address), to keep the test platform independent (though it should Even using a mask of 0b1111 on the branch immediate is not resolve to the same address on the reference ISS). Then, the LW detected. That means backward jumps are not tested (since instruction is executed (Line 17). Before and after the label, enough the above mutation cuts away the sign bit) and only very data is placed to cover the whole immediate offset range. The mem- small forward jumps (up to 2 instructions) are tested. ory is initialized with the concrete memory values provided by (3) Another interesting class of undetected mutations is mov- the solution. Thus, a difference will be observed in the RD register ing the PC alignment check to the end of the jump instruc- between the mutated and unmutated ISS (for a store instruction, tion. The JAL (offset-based relative jump) and JALR (register- the difference would be observed in the memory). based absolute jump) instructions store the return address

                                            Table 1: Experiment results

 Mutation Class                #mutants     #killedCT: Official                   CT: Spec-based      Symbolic Execution
                                                              runtime   #alive       #killed  runtime  #alive     #killed  runtime
 M1: Mask Register                 1395  1249 [89.5%]           8700s        146  79 [54.1%]   70216s      67   67 [100%]    5954s
 M2: Mask Immediate                 374   343 [91.7%]           2375s         31   4 [12.9%]   12536s      27   27 [100%]    2080s
 M3: Replace Constant                27    23 [85.2%]            141s          4    4 [100%]    1926s       0           /        /
 M4: Replace Load / Store            26     26 [100%]            136s          0           /        /       0           /        /
 M5: Replace Register               243   238 [97.9%]           1344s          5   3 [60.0%]    2609s       2    2 [100%]     149s
 M6: Replace Immediate               96    95 [99.0%]            518s          1      0 [0%]     568s       1    1 [100%]     105s
 M7: Replace Unary Operation         21    19 [90.5%]            169s          2   1 [50.0%]    1114s       1    1 [100%]      92s
 M8: Replace Binary Operation       266   265 [99.6%]           1497s          1      0 [0%]     557s       1    1 [100%]      92s
 M9: Move Trap Check                  7     5 [71.4%]             36s          2      0 [0%]    1114s       2    2 [100%]     160s
 Total                             2455  2263 [92.2%]          14916s        192  91 [47.4%]   90640s     101  101 [100%]    8632s

         into the RD register when performing the jump. However, in               taken (gap 4). Furthermore, we spotted a potential overshift
       case the jump address is misaligned, a trap is triggered and               condition in VP, which is based on undefined behavior in
      no side effects should occur, i.e. RD should not be modified.               C++ (gap 1).
This common class of errors is not detected by CT.
     (4) One more interesting undetected error is storing any wrong
return address for the JAL and JALR instruction.                        6
         The remaining alive mutants correspond to special computa-      DISCUSSION AND FUTURE WORK
tional cases such as masking specific bits from a register or imme-     Our mutation-based approach has been very effective in finding
  diate. It certainly makes sense to strengthen the CT to kill them     several serious gaps in the official CT which can lead to common
  as well, but they are less important compared to the above cases,     implementation bugs that remain undetected by CT. We evaluated
    which have a much higher potential to find implementation bugs.     our approach on a reference ISS and considered mutations in the

Specification-based CT. The specification-based CT contains ad- ISS execution unit for the base RV32I ISA. We envision several ditional 8900 tests that cover a large set of different immediate and directions for future work to extend, complement and further boost register values as well as register access combinations for each in- our approach. We discuss them in the following. struction. Due to the large number of tests (and complex execution One of the first steps would be to consider CSRs and additional infrastructure, which requires around 10 minutes to execute the RISC-V extensions. By leveraging our existing mutation classes test-suite once), the processing time is very high with around 23 and framework, it should be straightforward to integrate additional hours. This CT kills around half of the remaining 192 mutants. It RISC-V extensions with our approach. An interesting point in this partly closes the gaps (1) and (2), but the gaps (3) and (4) as well as direction would also be to produce a minimized test-suite such that the immediate-based shift problem from gap (1) still remain. all mutants are still killed. This would be particularly helpful with additional RISC-V extensions, because the number of test-cases can Mutation-based CT. Using our symbolic execution framework, grow significantly with each extension (since all mutation classes we are able to kill all remaining 101 mutants and close the gaps. are applied in combination with the new instructions). These mutants are not easy to kill, as they have remained undetected Another direction is to devise and evaluate the impact of even by both CT test-suites. It takes around 2.5 hours in total (85 seconds stronger mutation classes, for example by considering multiple in- on average per mutant). Thus, we obtained a mutation-based CT stead of single mutations, on the obtained coverage with respect to with 101 focused test-cases to further complement the existing CT the CT and potential bugs found in RISC-V simulators. In this direc- infrastructure. We cross-validated it on our reference ISS to ensure tion it would also be very helpful to optimize the symbolic encoding that the 101 mutants are indeed killed. and integration with the symbolic execution engine (angr here) as Evaluation. Finally, we evaluated the complete CT infrastructure well as the mutation generation and CT execution to facilitate fast on five RISC-V simulators: riscvOVPsim [7], SPIKE [12], VP [9], exploratory experiments with different mutation classes. GRIFT [5] and SAIL [11]: Some mutants cannot be killed in a platform independent way because they for example rely on a very specific (hardcoded) mem- • No mismatches were detected with the official CT. ory address or PC value in order to trigger the mutation and the • Using the specification-based CT, we detected a configura- compliance testing setup allows each RISC-V simulator to define its tion error in SAIL which causes it to execute RV32I compli- own memory layout by providing a custom linker script (though ance tests with the C extension enabled. the existing set of supported simulators mostly use the same mem- • With our mutation-based CT we additionally found a bug in ory layout). For example a mutation in the memory access unit GRIFT, where the JAL instruction has a side effect in updating might only trigger if the access address is below 0x1000. In case the the RD register even though an unaligned instruction trap is data memory is placed above this address, the mutation cannot be

triggered and thus cannot be killed. One way to tackle this prob- REFERENCES lem in a platform independent way would be to leverage virtual [1] 2017. RISC-V Torture Test Generator. https://github.com/ucb-bar/riscv-torture. memory and thus setup (platform independent) virtual code and [2] 2019. The Challenge Of RISC-V Compliance. https://semiengineering.com/ data memory sections (per test-case) that re-map the (platform toward-risc-v-compliance/. [3] 2019. Imperas delivers highest quality RISC-V RV32I compliance test suites dependent) physical memory sections as necessary. However, be- to implementers and adopters of RISC-V. https://riscv.org/2019/11/imperas- side being more complex, this approach would require a simulator delivers-highest-quality-risc-v-rv32i-compliance-test-suites-to-implementers- with support for virtual memory (which is an advanced feature in and-adopters-of-risc-v/. [4] 2020. angr. https://angr.io/. RISC-V and typically not available in base configurations). [5] 2020. GRIFT - Galois RISC-V ISA Formal Tools. https://github.com/GaloisInc/ In this work we focus on mutations in the execution unit. How- grift. ever, conceptually our approach can also supports different error [6] 2020. OneSpin 360 DV RISC-V Verification App. https://www.onespin.com/ solutions/risc-v. categories. One very interesting part would be to test the virtual [7] 2020. RISC-V Compliance Task Group. https://github.com/riscv/riscv- memory implementation (typically done by an MMU), since it re- compliance. [8] 2020. RISC-V Formal Verification Framework. https://github.com/SymbioticEDA/ quires a significant amount of complex initializations. Besides set- riscv-formal. ting up the CSRs to activate virtual memory support, it is necessary [9] 2020. RISC-V Virtual Prototype. https://github.com/agra-uni-bremen/riscv-vp. to setup appropriate page tables in memory. In addition, the page [10] 2020. RISCV-DV. https://github.com/google/riscv-dv. [11] 2020. RISCV Sail Model. https://github.com/rems-project/sail-riscv. tables need to be setup in a way to reach a very specific muta- [12] 2020. Spike RISC-V ISA Simulator. https://github.com/riscv/riscv-isa-sim. tion which makes it much more complex. Hence, it would be very [13] A. Adir, E. Almog, L. Fournier, E. Marcus, M. Rimon, M. Vinov, and A. Ziv. 2004. interesting to consider extensions in this direction to facilitate com- Genesys-Pro: innovations in test program generation for functional processor verification. D&T (2004), 84–93. prehensive and automated MMU testing. [14] Brian Campbell and Ian Stark. 2014. Randomised Testing of a Microprocessor An orthogonal direction would be to evaluate our mutation- Model Using SMT-Solver State Generation. In Formal Methods for Industrial Critical Systems, Frédéric Lang and Francesco Flammini (Eds.). 185–199. based approach on different reference simulators to access the [15] Mikhail Chupilko, Alexander Kamkin, Artem Kotsynyak, and Andrei Tatarnikov. impact on the generated test-suite. Going further in this direction, 2017. MicroTESK: Specification-Based Tool for Constructing Test Program Gen- the next step would be to evaluate the generated test-suite on erators. In HVC. [16] S. Fine and A. Ziv. 2003. Coverage directed test generation for functional verifi- RISC-V RTL cores. It would be interesting to see what kind of cation using Bayesian networks. In DAC. 286–291. bugs are detected at RTL and measure the obtained coverage. In a [17] Mark Hampton and Stephane Petithomme. 2007. Leveraging a Commercial final step, the mutation-based approach could be applied at RTL to Mutation Analysis Tool For Research. In MUTATION. 203–209. [18] Vladimir Herdt, Daniel Große, and Rolf Drechsler. 2020. Closing the RISC-V generate test-cases specifically tailored for an RTL core and evaluate Compliance Gap: Looking from the Negative Testing Side. In DAC. the results on different RTL cores. [19] Vladimir Herdt, Daniel Große, and Rolf Drechsler. 2020. Enhanced Virtual Proto- typing: Featuring RISC-V Case Studies. Springer. Finally, it would also be very interesting to leverage a sym- [20] Vladimir Herdt, Daniel Große, and Rolf Drechsler. 2020. Towards Specification bolic execution framework, e.g. [24], to perform a (more general) and Testing of RISC-V ISA Compliance. In DATE. 995–998. difference-based testing. The idea is to find test-cases that show [21] Vladimir Herdt, Daniel Große, Hoang M. Le, and Rolf Drechsler. 2018. Extensible and Configurable RISC-V based Virtual Prototype. In FDL. 5–16. differences in behavior between different simulators. Such test- [22] Vladimir Herdt, Daniel Große, Hoang M. Le, and Rolf Drechsler. 2019. Verifying cases are certainly interesting, since they pinpoint the RISC-V ISA Instruction Set Simulators using Coverage-guided Fuzzing. In DATE. 360–365. specification parts that are complex or unclearly formulated. The [23] Vladimir Herdt, Daniel Große, Pascal Pieper, and Rolf Drechsler. 2020. RISC- V based Virtual Prototype: An Extensible and Configurable Platform for the reason is that two simulators implemented a feature differently and System-level. JSA (2020). hence disagree on the understanding of the specification in that [24] Vladimir Herdt, Hoang M. Le, Daniel Große, and Rolf Drechsler. 2019. Verify- ing SystemC using Intermediate Verification Language and Stateful Symbolic point (which should be highlighted by CT). Another argumenta- Simulation. TCAD 38, 7 (2019), 1359–1372. tively similar direction is to perform a difference-based testing on [25] Charalambos Ioannides, Geoff Barrett, and Kerstin Eder. 2011. Feedback-Based the same simulator but with different ISA configurations (since it Coverage Directed Test Generation: An Industrial Evaluation. In Hardware and Software: Verification and Testing, Sharon Barner, Ian Harris, Daniel Kroening, can pinpoint changes between two configurations which may be a and Orna Raz (Eds.). source for common bugs due to the high configurability of RISC-V). [26] Yue Jia and Mark Harman. 2011. An Analysis and Survey of the Development of Mutation Testing. IEEE Trans. Softw. Eng. 37, 5 (Sept. 2011), 649âĂŞ678. 7 CONCLUSION [27] Y. Katz, M. Rimon, and A. Ziv. 2012. Generating instruction streams using abstract CSP. In DATE. 15–20. We proposed a mutation-based approach to boost RISC-V com- [28] Lingyi Liu and Shobha Vasudevan. 2011. Efficient validation input generation in RTL by hybridized source code analysis. In DATE. 1596–1601. pliance testing and demonstrated its effectiveness. Based on our [29] Lorenzo Martignoni, Roberto Paleari, Giampaolo Fresi Roglia, and Danilo Bruschi. mutation classes, we identified several serious gaps in the Compli- 2009. Testing CPU Emulators. In ISSTA. 261–272. ance Test-suite (CT) and generated new tests to strengthen the CT [30] Youssef Serrestou, Vincent Beroulle, and Chantal Robach. 2007. Functional Verification of RTL Designs Driven by Mutation Testing Metrics. In DSD. 222– by closing these gaps. Our approach has also been effective in find- 227. ing bugs in RISC-V simulators. Finally, we provided an extensive [31] Yan Shoshitaishvili, Ruoyu Wang, Christopher Salls, Nick Stephens, Mario Polino, Audrey Dutcher, John Grosen, Siji Feng, Christophe Hauser, Christopher Kruegel, discussion that sketched promising directions for future work. and Giovanni Vigna. 2016. SoK: (State of) The Art of War: Offensive Techniques in Binary Analysis. (2016). ACKNOWLEDGMENTS [32] Andrew Waterman and Krste Asanović. 2019. The RISC-V Instruction Set Manual; Volume I: Unprivileged ISA. SiFive Inc. and CS Division, EECS Department, This work was supported in part by the German Federal Ministry University of California, Berkeley. of Education and Research (BMBF) within the project VerSys under [33] Andrew Waterman and Krste Asanović. 2019. The RISC-V Instruction Set Manual; Volume II: Privileged Architecture. SiFive Inc. and CS Division, EECS Department, contract no. 01IW19001 and within the project Scale4Edge under University of California, Berkeley. contract no. 16ME0127. [34] Tao Xie, Wolfgang Mueller, and Florian Letombe. 2012. Mutation-analysis driven functional verification of a soft microprocessor. In SoC. 283–288.