Open-Source RISC-V Vector Test Suites: A
                                                                                                             Comparative Analysis

                                                                                       Fatima Saleem1, Umer Imran1, Quswar Abid1 and Dr. Kamran Malik1

                                                                                                           110xEngineers, Pakistan

                                                                                                                   Abstract
                                                           Architecture-level verification of RISC-V Vector (RVV) cores presents a complex challenge due to the diverse behavior of
                                                         vector instructions under various instruction parameters and core configurations. The vector instruction test suite must be
                                                        capable of addressing all legitimate combinations of vector instructions under all the different configurations to accomplish
                                                          comprehensive verification targets. In this paper, we present a comparative analysis of seven open-source test suites for
                                                                  vector ISA: Imperas, RIOS Labs, RISCV-Torture, RISCV-DV, FORCE-RISCV, Yang’s Generator and Tenstorrent. We
                                                          provide an objective evaluation of the attributes and coverage of these test suites using the RVV Specification v1.0 as a
                                                           reference. Our study reveals that a combination of these test generators can provide a fairly comprehensive verification
                                                            strategy for vector cores. On a stand alone basis, FORCE-RISCV covers the complete RVV v1.0 specification upto a VLEN
of 4096. Similarly, Imperas is fairly exhaustive but limited to VLEN of 256.

    Introduction                                           (riscvOVPsim) and a version of its Vector Test Suite
                                                           (VTS Community Edition).           The VTS Community
RISC-V is an open-source Instruction Set Architec-         Edition offers a variety of pre-built test cases for the
ture (ISA) that offers a wide range of instructions to     RISC-V Vector Extension. Imperas has only released
enable the efficient processing of data. Among these       its 32-bit vector test suite, which supports VLEN 128b
instructions are vector instructions, which are used       and 256b only, resulting in reduced coverage of some
to process multiple streams of data in parallel. The       of the vector instruction categories.
RISC-V Vector [1] Architecture Compliance Testing                  The RIOS Lab Vector Test Generator [4] is an
(VACT) is crucial in ensuring that a specific vector       RVV automated test generator created by the RIOS
hardware implementation conforms to the RVV spec-          Lab in collaboration with the RISC-V Foundation for
ification and correctly, as well as completely, imple-     the RISC-V community. The RVV Sail Model, con-
ments the vector instructions. This typically involves     tributed by RIOS Lab, serves as the standard golden
the use of test suites, which are collections of test      model for executing RVV instructions. By utilizing
cases designed to exercise different aspects of the vec-   the configuration features of the RIOS Labs Vector
tor ISA. The aim is to develop comprehensive RVV           Test Generator, users are able to select specific instruc-
architecture tests that cover not only the full vector     tions and set configuration parameters to generate
instructions but also the various configuration pa-        specific tests. This generator is fully compatible with
rameters defined in RVV like vector register length        the latest RISCOF infrastructure, which is widely
(VLEN), register multiplier (LMUL), element width          used in RISC-V testing.
(SEW), etc.                                                  RISC-V Torture [5] is a Scala-based test generator
   It is important to note that the vector test suite is   that supports RVV v0.9 specification and generates
significantly more extensive than the RISC-V scalar        tests by integrating predefined randomized test se-
test suite.     The vector test suite consists of numer-   quences. The tests are randomly generated, and at
ous instructions which can be divided into seven           the end of the test program execution, verification
categories defined in RVV Spec as listed in Table 1        occurs by comparing the register log with the output

(Column 1). To overcome the challenge [2] of ver- from the Spike ISA simulator. Antmicro used RISC-V ifying a RISC-V based vector core, comprehensive Torture Tests to assess Renode’s RISC-V implementa- RVV architecture test suites are being developed by tion. the open-source community. In this paper, we do a RISC-V DV [6] s an open-source UVM-based ran- comparative analysis of all seven open-source RVV dom instruction generator that generates tests compli- test suites to identify the capabilities strengths and ant with RVV v0.9. It provides excellent randomness weaknesses of each test suite. and performance. Andes and Google collaborated Imperas Vector Test Suite[3] is a very compre- to support RVV v0.9 in RISC-V DV. Their upcoming hensive verification tool for RVV processors. Im- plans involve including support for RVV v1.0 and peras has made available its RISC-V Reference Model adding hooks to collect vector extension test cover-

Open-Source RISC-V Vector Test Suites: A Comparative Analysis

                                      Table 1: RVV v1.0 instruction coverage by various tests suites

                                      Tests Suites
RVV Categories (Total     Imperas     RIOS                        RISCV-  RISCV-  FORCE-   Yang’s  Tenstorrent
Instructions)                         Labs                       Torture    DV    RISCV  Generator
Configuration (3)             3        3                            2       2       3        3   3
Loads and Stores (310)        45      141                          202     258     310       46         25
Integer Arithmetic (139)      137     133                          139     136     139      139         60
Fixed-Point Arithmetic (32)   32       32                           32      32      32       32  0
Floating-Point (91)           91       66                           89      75      91       88  0
Reduction Operations (16)     16       15                           16      16      16       16  0
Mask Operations (15)          15       12                           15      15      15       13  5
Permutation (21)              21       17                           20      18      21       19  5

                                        Table 2: Comparative Analysis of different RVV test suites

                                      Test Suite Attributes
Test Suite        Spec        Suite   Checking                  Coverage    Supported    Integrated
                Version       Type    Method                      Tool         ISS            Core
Imperas           1.0         Tests   Signature based         riscvOVPsim  riscvOVPsim  NSITEXE DFP
RIOS Labs         1.0         ATG               Self-Checking  RISCV-ISAC      Sail           -NA-
RISCV-Torture     0.9         RTG     Signature based             -NA-        Spike        Renode
RISCV-DV          0.9         RTG     Signature based             -NA-        Spike     Andes NX27V
FORCE-RISCV       1.0         RTG     Signature based         riscvOVPsim    Handcar          -NA-
Yang’s Generator  1.0         ATG               Self-Checking     -NA-        Spike            ARA
Tenstorrent       1.0         Tests             Self-Checking     -NA-       Whisper       Ocelot

age.FORCE-RISCV [7] is an Instruction Sequence Gen- References

erator (ISG) designed to generate tests for the de- [1] “RISCV Vector Spec v1.0”. In: (Sep 2021). url: sign verification of RISCV processors. It utilizes https://github.com/riscv/riscv-v-spec. randomization to select instructions, registers, ad- [2] The challenge of RISC-V Compliance. url: https: dresses, and data for the tests and can produce valid / / semiengineering . com / toward - risc - v - test sequences with minimal user input. Addition- compliance/. ally, FORCE-RISCV includes a set of advanced APIs, giving the user a high level of control over the in- [3] “Imperas Tests”. In: (Feb 2023). url: https:// struction generation process. Although it supports github . com / riscv - ovpsim / imperas - riscv - the complete RVV v1.0 specification, there is no use tests. case available demonstrating its integration with any [4] “RIOS Lab ATG”. In: (Dec 2022). url: https: RVV implementation to ensure its correctness. //github.com/xiwang-x/rvv-atg. Yang’s Vector Test Generator [8] is an Automatic [5] “RISCV-Torture”. In: (Dec 2020). url: https:// Test Generator (ATG) developed by Yang Liu, a de- github.com/Lampro-Mellon/riscv-torture. veloper from PLCT lab. It verifies the RISC-V Vector implementation. The test generator is written in [6] “RISCV-DV”. In: (Feb 2021). url: https : / / GoLang and employs GNU Make to automate the github.com/chipsalliance/riscv-dv. test generation flow. The generated tests are fully [7] “FORCE-RISCV”. In: (Jan 2023). url: https:// self-checking, and any verification environment for a github.com/openhwgroup/force-riscv. core utilizing the riscv-tests can easily use them. [8] “Yang’s Generator”. In: (Mar 2023). url: https: Tenstorrent Vector Test Suite [9] comprises a col- //github.com/ksco/riscv-vector-tests. lection of binary test files that were produced by their internal test generator. This generator has the [9] “Tenstorrent Tests”. In: (Feb 2023). url: https: ability to create a comprehensive range of tests with / / github . com / tenstorrent / riscv _ arch _ numerous repetitions, effectively covering the entire tests. architectural verification space for vector instructions.

2