Sail-RISC-V and Spike for RISC-V Vector:
   Toward Consistent Golden Reference Behavior
                                                                                          Manfred Schlägl, Katharina Ruep and Daniel Große

                                                                              Institute for Complex Systems, Johannes Kepler University Linz, Austria
                                                                                manfred.schlaegl@jku.at, katharina.ruep@jku.at, daniel.grosse@jku.at

                                                                                                              Abstract
    In recent years, the executable specification generated from Sail-RISC-V has increasingly been considered as a
                                                         successor to the widely used Spike ISA Simulator as golden reference for RISC-V, including the complex and highly
                                                               configurable RISC-V Vector Extension (RVV). In this paper, we compare the RVV behavior of Sail-RISC-V
                                                             against Spike using the automated testing framework RVVTS . While Sail-RISC-V largely matches Spike under
                                                         positive testing (0.23% deviations), negative testing reveals substantially more deviations (3.73%), highlighting
   remaining issues in Sail-RISC-V ’s RVV instruction validity checking under dynamic configurations.

               Introduction                                  dynamic configurations still requires further attention
Since the early days of RISC-V, the Spike ISA Sim-           in Sail-RISC-V . Additionally our results demonstrate
ulator [1] has served as the de-facto golden reference       the effectiveness of RVVTS for systematically detect-
model and has been widely used in verification frame-        ing, minimizing, and analyzing deviations in RVV im-
works and architectural validation flows [2, 3, 4]. In re-   plementations. To support the alignment and improve-
cent years, the executable specification generated from      ment of reference-model behavior, we provide RVVTS
the Sail-RISC-V model [5] has increasingly emerged           with Sail-RISC-V         support, the generated RVV test
as a candidate for the next-generation golden refer-         sets, the generated test reports, and the minimized
ence due to its formally grounded specification ap-          test cases as open source on GitHub1.
proach. Both models include the ratified and highly           Spike vs.     Sail-RISC-V
relevant RISC-V Vector Extension (RVV) 1.0, which
introduces powerful data-level parallelism for RISC-V.       First, we use the coverage-guided generator in RVVTS
However, creating and verifying RVV implementations          to produce four compact, high-coverage test sets for
is challenging not only because of the large number of       RV32 and RV64, targeting both positive and negative
specified instructions, but also because their behavior      testing of RVV (with VLEN = 128 bit). Table 1 sum-
depends heavily on dynamic architectural parameters          marizes their configuration, the number of test cases
such as vector length, element width, masking, and           and RVV instructions, and the achieved functional
register grouping.      Consequently, the effective state    coverage. The Invalid+Valid Sequences         (IVS) sets
and parameter space of RVV instructions is orders            contain trap-triggering code sequences for negative

of magnitude larger than that of scalar instructions. testing, whereas Valid Sequences (VS) sets contain Beyond conventional positive testing with valid in- only non-trapping sequences for positive testing. Each struction sequences, robust verification of RVV must set comprises tens of thousands of test cases, hundreds therefore also include negative testing [6] to ensure cor- of thousands of RVV instructions, and achieves >94% rect handling of invalid instructions and configurations. functional coverage. In RVV, such invalid cases arise for example when a Next, we use RVVTS to run the generated test sets vector instruction uses an illegal register group or an on Sail-RISC-V by executing each case on Spike and unsupported element type under the current configu- Sail-RISC-V and comparing the resulting Machine ration. The recently presented open-source RVVTS States (registers, CSRs, #traps, etc.). Any deviation framework addresses these challenges by combining of the Machine States is flagged as a potential failure, coverage-guided test generation targeting positive and for which RVVTS automatically isolates the triggering negative testing with automated test execution, failure instruction and produces a minimized test. The far- code minimization and single-instruction isolation [7]. right of Table 1 summarizes the detected deviations. Using RVVTS , extended with support for The deviation rates for RV32 and RV64 are of the same Sail-RISC-V as a Design Under Test (DUT) (not cov- order of magnitude and are therefore consolidated in ered in this paper), we compare the behavior of RVV the last column. Although Spike and Sail-RISC-V in Sail-RISC-V against the former golden reference behave very similarly in positive testing (VS), with Spike. Negative testing with invalid instruction se- deviations of 0.23%, they differ notably in negative quences reveals 3.73% deviations, showing that the testing (IVS), with deviations of 3.73%. validation of RVV instruction legality under certain 1https://github.com/ics-jku/RVVTS_SailRV_Spike

RISC-V Summit Europe, Bologna, 8-12th June 2026                                                                     1

         Table 1:     Test Sets pre-generated with RVVTS and applied on Sail-RISC-V (git hash a33475aeb8)
                                     RISC-V     #Test    #RVV             Functional Coverage
Test Set                             Config     Cases   Instr.              (riscvOVPsim RVV)           Deteted Deviations
                                                                            Points  Percent             (% w.r.t. #Test Cases)
Invalid+Valid Sequences              RV32      36,016  289,512         30,606 / 31,894  95.96           1,232 (3.42%)   2,868
(IVS)                                RV64      40,939  329,145         31,765 / 33,076  96.04           1,636 (4.00%)  (3.73%)
Valid Sequences                      RV32      34,578  253,309         30,126 / 31,894  94.46             95 (0.27%)     194
(VS)                                 RV64      50,760  372,041         31,179 / 33,076  94.26             99 (0.20%)   (0.23%)

        Table 2:  Deviation Categories on Sail-RISC-V                  are mostly detected by VS.          For all            16 + 191 = 207
                                             Detected Deviations       cases, RVVTS automatically isolates four instructions,
Deviation Category                            IVS         VS
                                               (% w.r.t. SUM)          namely     vf(w)redusum.vs            and         vssubu.vv|vx.
 1   Invalid Accept                          1,803        0            For the former, the deviation can be explained by
     (Missing trap)                          62.87%     0.00%          different order of floating-point element processing
 2   Assertion: Valid Reject                 1,038        0
     (Sail-RISC-V     terminated)            36.19%     0.00%          and from the fact that floating-point arithmetic is
 3   Deviation in Results                      16        191           not distributive.                 For the latter, a closer inspection
     (e.g. registers, CSRs, . . . )          0.56%      98.45%         shows that Sail-RISC-V                    does not set the saturation
 4   Assertion: Invalid Reject                 11         3            bit                      vcsr.vxsat for subtractions where the second
     (Sail-RISC-V     terminated)            0.38%      1.55%          operand is greater than the first. After investigating
SUM (100.00%)                                 2,868      194           the Sail-RISC-V code, we can confirm this as a bug.
                 Using the comprehensive reports automatically gen-                              Conclusions
erated by RVVTS , we analyze the detected deviations                   Overall, our results show that RVV in Sail-RISC-V
in detail. Table 2 lists four main categories revealed                 mostly corresponds to the former golden reference
by IVS and VS, ordered by frequency. For didactic                      Spike under positive testing with valid code sequences

clarity, we discuss 1 , 2 , and 4 first, followed by 3 . (VS), with only 0.23% deviations. However, nega- 1 covers deviations where Spike rejects an in- tive testing with invalid code sequences (IVS) reveals struction (invalid-instruction trap), while the instruc- substantial deviations of 3.73% relative to Spike. In tion is executed by Sail-RISC-V (no trap). By con- particular, the validation of RVV instruction legality struction, these deviations are detected only by IVS, under specific configurations (e.g., element type, group- which is reflected in the results (no cases for VS). ing, etc.) in Sail-RISC-V requires further attention. Given that Spike includes an older and therefore more These findings also underscore the value of RVVTS – mature RVV implementation, we hypothesize that its comprehensive support for positive and negative Sail-RISC-V does not yet fully model certain prohib- RVV testing and its automated code minimization ited RVV instruction cases under specific configura- with instruction isolation. tions (e.g., element type, grouping, etc.). RVVTS with support for Sail-RISC-V , the gener- 2 covers deviations where Spike rejects an instruc- ated RVV test sets, the generated test reports, and tion, whereas the Sail-RISC-V model terminates on the minimized test cases are available as open source an assertion. These deviations are also only detected on GitHub. by IVS. Although Sail-RISC-V detects these invalid Acknowledgments cases, they are not rejected properly within the model This work has partially been supported by the LIT Secure and via a trap. Instead, the cases are caught deep in the Correct Systems Lab funded by the State of Upper Austria. implementation by assertions, leading to a termina- tion, which is clearly incorrect behavior. This confirms References the hypothesis above that Sail-RISC-V has not yet [1] Spike RISC-V ISA Simulator. https://github.com/riscv/ properly covered certain prohibited cases. riscv-isa-sim. 2026. 4 covers deviations where Spike accepts an instruc- [2] RISC-V Vector Tests Generator. https : / / github . com / tion, whereas the Sail-RISC-V model terminates on chipsalliance/riscv-vector-tests. 2026. an assertion. In contrast to 2 , these deviations indi- [3] RISCV-DV. https://github.com/google/riscv-dv. 2026. cate cases in which assertions in Sail-RISC-V are too [4] Alexandre Joannou et al. “Randomized Testing of RISC-V CPUs Using Direct Instruction Injection”. In: IEEE Design strict. For all 11 + 3 = 15 cases, RVVTS automatically and Test 41.1 (2024), pp. 40–49. doi: 10.1109/MDAT.2023. isolates the instruction vrgatherei16.vv. All as- [5] 3262741. sertions relate to the use of supposedly invalid vector Sail RISC-V: Formal Specification of the RISC-V ISA. https://github.com/riscv/sail-riscv. 2026. register groups. After studying the RVV specification, [6] Vladimir Herdt, Daniel Große, and Rolf Drechsler. “Closing the we can confirm that Spike is correct and that these RISC-V Compliance Gap: Looking from the Negative Testing are indeed bugs in Sail-RISC-V . Side”. In: DAC. 2020, pp. 1–6. doi: 10.1109/DAC18072.2020. 9218629. 3 includes all deviations where only results (e.g., [7] Manfred Schlägl and Daniel Große. “Single Instruction Isolation register and CSR values) differ. These deviations for RISC-V Vector Test Failures”. In: ICCAD. 2024, 156:1– 156:9. doi: 10.1145/3676536.3676755.

2                                                                                            RISC-V Summit Europe, Bologna, 8-12th June 2026