From Generation to Failure Categorization: An Open-Source
                               automated RTL Verification Framework for RVV
                 Manfred Schlägl             Jonas Reichhardt                      Daniel Grosse
            Johannes Kepler University  Johannes Kepler University       Johannes Kepler University Linz &
                  Linz, Austria               Linz, Austria                         DFKI Bremen
             manfred.schlaegl@jku.at     jonas.reichhardt@jku.at                   Linz, Austria
                                                                                daniel.grosse@jku.at

Abstract SIMD by supporting dynamic vector lengths (vl) and types (vtype), We present a highly automated, open-source RTL verification frame- configured at runtime with dedicated instructions. work for the RISC-V Vector Extension (RVV) 1.0 that extends the Among RVV implementations, Ara stands out as the first fully open-source RVVTS framework by adding RTL support and a novel open-source processor compliant with the frozen RVV 1.0 speci- Automated Failure Categorization (AFC) stage for scalable result fication (see e.g. [27, 29]). In the most recent version, it delivers analysis. Using the most recent RTL of the silicon-proven Ara vec- very good performance on compute-intensive workloads, sustain- tor processor as DUT, we generate RVV test sets that encompass ing high utilization and scaling efficiently with core and lane counts. both positive and negative testing, and achieve over 96% functional Moreover, Ara achieves state-of-the-art energy efficiency (37.8 DP- coverage – significantly exceeding our evaluated baseline, which GFLOPS/W at 0.8 V; 1.35 GHz in 22 nm) [27]. achieves only 11.04%. Our framework detects more than 82k devi- However, the very features that make Ara compelling – RVV ations, automatically minimizes about 97% of them, and clusters 1.0’s dynamic vector lengths and data types, flexible register group- observed failures into 16 distinct categories. ing, masking, and scalable lane-based microarchitectures – sub- stantially enlarge the state space, rendering ad hoc testbenches CCS Concepts and simple instruction generation insufficient. We substantiate this • Hardware → Equivalence checking; Simulation and emula- argument with the following considerations: For a scalar RISC-V tion; • Computer systems organization → Multiple instruc- instruction (e.g. classical integer add), the architectural behavior is tion, multiple data; Reduced instruction set computing. essentially configuration-independent (integer operation) or varies only by the floating-point rounding mode (≤ 5 possibilities via Keywords the respective frm Control and Status Register (CSR)). Hence, the per-instruction configuration space is constant. In contrast, the RISC-V, RISC-V Vector Extension (RVV), RTL Verification, Processor behavior of an RVV instruction is parameterized by the dynamic Verification CSRs vtype (SEW, LMUL, VTA, VMA), vl and vstart, and optional ACM Reference Format: predication (e.g. masking). For fixed (SEW, LMUL), the RVV spec- Manfred Schlägl, Jonas Reichhardt, and Daniel Grosse. 2026. From Genera- ification permits vl ∈ {0, . . . , VLMAX } and vstart ∈ {0, . . . , vl} tion to Failure Categorization: An Open-Source automated RTL Verification withÍVLMAX = ⌊LMUL × VLEN /SEW ⌋, yielding the triangular Framework for RVV. In Great Lakes Symposium on VLSI 2026 (GLSVLSI ’26), sum VLMAXvl=0 (vl + 1) = ((VLMAX + 1) × (VLMAX + 2))/2 distinct June 22–24, 2026, Canandaigua, NY, USA. ACM, New York, NY, USA, 8 pages. (vl, vstart) pairs – so at least a quadratic growth in configurations https://doi.org/10.1145/3787109.3815255 per instruction even before accounting for masks or other mode bits.1 Thus, the per-instruction configuration space for RVV 1 Introduction is orders of magnitude larger than that in the scalar case. Exploiting Data-Level Parallelism (DLP) can significantly acceler- Recently, the open-source framework RVVTS [30] has been pro- ate algorithms in domains such as modern machine learning and posed to address this challenge. It combines coverage-guided test multimedia. A common approach to leverage DLP is through Single generation with a single-instruction isolation technique to (a) gen- Instruction, Multiple Data (SIMD) extensions, which apply the same erate high-quality RVV tests and (b) automatically isolate failing operation to multiple data elements simultaneously, i.e., a vector. instructions and provide minimized test cases, thereby drastically Classical SIMD architectures, such as Intel SSE, AVX, or ARM Neon, reducing the manual analysis effort. The RVVTS-generated RVV use fixed-size vector registers, which limits flexibility and requires test sets achieve a functional coverage of >90% and employ both pos- recompilation when the vector length changes. In contrast, vec- itive and negative testing, i.e., it generates tests that are expected to tor architectures such as RISC-V Vector Extension (RVV) generalize pass, verifying compliant behavior, as well as tests expected to fail, ensuring correct handling of illegal or exceptional behavior [21]. However, in general two major limitations remain: (i) the verifi- This work is licensed under a Creative Commons Attribution cation capabilities of RVVTS have so far only been demonstrated on International 4.0 License.

GLSVLSI ’26, Canandaigua, NY, USA 1 Putting in concrete numbers for Ara with VLEN = 4096 bits, vtype set to SEW = © 2026 Copyright held by the owner/author(s). 8 bit, LMUL = 1 (no grouping), we get VLMAX = ⌊1 × 4096/8⌋ = 512, leading to ACM ISBN 979-8-4007-2431-2/26/06 (512 + 1) × (512 + 2)/2 = 131, 841 distinct pairs; and this is only for a single vtype https://doi.org/10.1145/3787109.3815255 configuration.

GLSVLSI ’26, June 22–24, 2026, Canandaigua, NY, USA Manfred Schlägl, Jonas Reichhardt, and Daniel Grosse

high-level models, including Virtual Prototypes (VPs), Spike (the ref- Specifically for RISC-V, several instruction stream generators erence model from RISC-V International [11]), and QEMU , because and test frameworks have been developed. One of the earliest exam- RVVTS currently lacks support for Register Transfer Level (RTL) ples is RISC-V Torture Test Generator [6], a random instruction gen- designs; and (ii) the only other available open-source alternative, erator well-suited for scalar RISC-V instructions, but not supporting RISC-V Vector Tests Generator from CHIPS Alliance fails to reach RVV version 1.0. The official Architecture Test Special Interest Group adequate functional coverage. provides architectural tests [5], including the RISC-V Compliance Contributions. This paper makes the following three main contri- Test Generator, which generates tests for most RISC-V extensions. butions: However, it does not cover RVV. Also approaches leveraging sym- (C1) Evidence of insufficient functional coverage in exist- bolic execution for test generation have been proposed [15, 22], ing open-source RVV test generator: We present an in- but they only perform positive testing and do not support RVV. depth evaluation of the RISC-V Vector Tests Generator from RISCV-DV [8], originally developed by Google and now maintained CHIPS Alliance, the default framework for RVV testing in by the CHIPS Alliance, supports RVV but only up to version 0.93. Sail-RISC-V [10], and demonstrate that it achieves only low FORCE-RISCV [3], maintained by the OpenHW Group, supports functional coverage and does not target negative testing. RVV 1.0. However, it requires highly invasive modifications to the (C2) Extension of RVVTS to RTL verification: We extend the DUT, produces only raw traces, and does not include any form RVVTS framework with full RTL support, enabling its applica- of result analysis as shown in [33]. Also provided by the CHIPS tion to RVV implementations as Design Under Tests (DUTs). Alliance is the RISC-V Vector Tests Generator [7], which supports (C3) Automated categorization of RVV failure modes: Due to RVV 1.0 and serves as the default framework for RVV testing in the high coverage of the RVVTS test sets and the complexity Sail-RISC-V [10]. However, it lacks negative testing and automated of advanced RVV RTL implementations, we expect a large result analysis, and achieves low functional coverage as demon- number of potential failures during co-simulation. Hence, we strated later in Section 5.1. introduce a novel rule-based Automated Failure Categoriza- There are commercial solutions for RVV verification. For exam- tion (AFC) stage that automatically minimizes and clusters ple, ImperasDV [4, 35], integrated with major Electronic Design Au- detected failures into failure modes, i.e., equivalence classes tomation (EDA) simulators, provides a reference-model-based pro- of mismatching tests sharing the instruction context and a cessor DV solution with the RVVI interface and the riscvISACOV minimal predicate over RVV parameters. This significantly SystemVerilog functional-coverage library, which is delivered as reduces debugging effort and accelerates root-cause analysis. commercial Verification IP. IP vendors such as Andes Technology In our experimental evaluation, we use the extended RVVTS report using RISCV-DV, co-simulation against an Instruction Set framework on Ara as a strong example DUT. We generate and ap- Simulator (ISS), and commercial tools (e.g., VCS/Verdi) to enhance ply RVV test sets to Ara with VLEN = 4096, encompassing positive, verification coverage for their NX27V RVV core [1], but the over- negative, and mixed test scenarios, and achieve a functional cover- all environment and test suites remain closed and highly project- age of over 96%. We then employ AFC to analyze the Ara failures, specific. To summarize, these solutions are proprietary and leverage automatically minimizing 97% of detected failures and associating commercial simulators, reference models and coverage IP. them with 611 instructions across 16 failure mode categories. Due to Regarding PULP Ara, prior verification efforts [25] focused on the large number of detected failures, a detailed per-failure analysis an early implementation of the vector unit with RVV version 0.7.1. is beyond the scope of this paper. These efforts specifically targeted the Ara vector unit only and em- To support future RVV verification research, the RVVTS frame- ployed Universal Verification Methodology (UVM), RISCV-DV , co- work extended with RTL support and AFC (including categorization simulation with Spike, and SystemVerilog assertions. In this work, rules), together with all results for Ara, is available in a reproducible we focus on the Ara vector processor, which supports RVV 1.0 open-source form on GitHub2. and is fully integrated with the CVA6 core as a black-box RTL DUT. The Ara repository currently provides 1028 hand-crafted RVV test 2 Related Work cases, along with a set of RVV benchmarks, offering only basic Automated test generation for processors is a widely researched functional validation [2]. area. Traditional methods typically decouple the architectural de- To the best of our knowledge, the extended RVVTS framework scription from test generation, leveraging techniques such as con- presented in this paper is the only open-source test framework straint solving [13, 16]. To achieve high coverage, [19] proposed a that offers end-to-end automated RTL testing, from high functional method to construct a coverage model using constraints that de- coverage positive/negative instruction generation to result analysis scribe the execution paths of individual instructions. An alternative and failure mode categorization, with full support for RVV 1.0. approach employing Bayesian networks for coverage modeling 3 was introduced in [20]. In addition, machine learning-based meth- Preliminaries ods [14, 24] and fuzzing-based approaches [17, 23, 26, 34] have also This section provides background on the the RVVTS framework been explored. However, these approaches focus exclusively on and the PULP Ara RVV vector processor considered in this work. positive testing, and require significant manual effort to analyze failing tests and identify root causes. Furthermore, none of these 3 approaches directly support RVV. Compared to the draft RVV 0.9 specification, the ratified RVV 1.0 version introduces hundreds of spec changes, including a completely revised family of vector load/store 2https://github.com/ics-jku/RVVTS_RTL_AFC_Ara instructions and a different vtype layout and vector configuration semantics. Conse- quently, a different verification campaign is needed.

From Generation to Failure Categorization: An Open-Source automated RTL Verification Framework for RVV  GLSVLSI ’26, June 22–24, 2026, Canandaigua, NY, USA

3.1  The RVVTS Framework                                                                               D                          )
In this work, we employ the recently introduced open-source RVVTS              TE ne
framework [30] for systematic verification of RVV. We select RVVTS         2                           |      oye AG          |gnJ
due to its demonstrated ability to uncover previously unknown bugs
in widely used simulators, including three bugs in the SystemC-                all                                            INL)
based RISC-V VP++ [31–33] and two in the QEMU emulator [30].
            RVVTS supports both positive and negative testing strategies
and automates the verification process from test generation to fail-       Figure 1: RVVTS RTL DUT: AraRunner and Ara Test Harness
ure analysis. The framework introduces Single Instruction Isolation
and Code Minimization, which reduce the manual effort required
to analyze failing tests by isolating the root-cause instruction and
generating minimized test cases. The automated verification flow           binary) as input, executes it on the target, extracts the resulting
includes grammar-based, coverage-guided test generation, instru-           architectural state and returns it as output. The architectural state
mentation and build, functional coverage measurement, execution            is finally represented by the RVVTS Machine State data structure,
on a reference simulator (Spike) and the DUT, detection of archi-          which includes registers (integer, floating-point, vector, and CSR), a
tectural state deviations (i.e., potential failures), and subsequent       trap counter, the last executed Program Counter (PC) address, and
failure minimization. In addition to automated execution, RVVTS            hashes for memory areas. In this work, we use the Ara RTL model
provides interactive support through Jupyter notebooks to assist           as a strong example DUT to demonstrate the integration of an RTL
users in tracing detected failures.                                        design into the RVVTS verification flow.
                                                                                                  For the Ara RTL model we create the new AraRunner. To ex-
3.2  The Ara RISC-V Vector Processor                                       ecute programs on Ara, the AraRunner utilizes the verilated Ara
Ara is a 64-bit open-source RISC-V vector processing unit, first           RTL model, as described in Section 3.2. In addition, to meet the
introduced in [18] and later refined in [27, 29]. It was developed         requirements of RVVTS, a small test harness for the Ara RTL model
as part of the PULP platform and is seamlessly integrated with the         is required; we build this harness based on the existing Ara Top-
open-source Ariane CVA6 RISC-V RV64CG core [36]. The most                  Level and introduce some modifications. Please note that, although
recent Ara implements RVV in version 1.0 and supports multiple             AraRunner is specific to Ara, it can be easily adapted for other RTL
vector lanes, enabling parallel processing of slices of RVV registers.     models.
The implementation allows for customization of both the number                             Figure 1 illustrates the complete integration stack of Ara as an
of lanes and the vector register length (VLEN), with a default setting     RVVTS DUT. The blue blocks indicate the unmodified components,
of 4 lanes and a VLEN of 4096 bits.                                                      ~~ such as the RVVTS framework and all main parts of the Ara model
        The Ara unit is modeled at the RTL level using SystemVerilog and   except for the test harness. The green blocks denote the newly
can be simulated alongside the CVA6 core using either commercial           introduced components: the AraRunner within RVVTS on the left
simulators (e.g., ModelSim/QuestaSim) or the open-source tool              and the Ara test harness within the Ara model on the right.

Verilator [12], which is utilized in this work. Verilator translates ~~ The text in the green blocks describes the added behavior. We RTL code into C++, which is subsequently compiled into a single now discuss the AraRunner and the Ara test harness in more detail, statically linked binary, referred to as the verilated model. This by explaining the execution process. The process flow begins on approach enables high-performance, cycle-accurate simulation and the left-hand side of Figure 1 within RVVTS, moves to the right to is widely adopted in academia and the open hardware community. the Ara model, and then returns back to RVVTS. Prior verification efforts of Ara focused on an early vector unit For each test to be executed, RVVTS first calls the AraRunner to implementation with RVV version 0.7.1, using UVM, RISCV-DV , run an ELF binary on the DUT and retrieve the resulting Machine co-simulation with Spike, and SystemVerilog assertions [25]. The State. Besides the test code, the ELF binary already includes instru- current Ara repository provides 1028 hand-crafted RVV test cases mentation code generated by RVVTS, which extracts major parts and a set of RVV benchmarks, offering only basic functional vali- of the Machine State from the target and stores it in the target’s dation [2]. memory before the end of execution. The target PC address, which indicates the end of execution and is referred to as the breakpoint, 4 Extending RVVTS is provided through the global RVVTS configuration. In this section, we present the extensions made to the RVVTS frame- Next, the AraRunner starts the Ara model as new process with work. First, in Section 4.1, we discuss the integration of a RVV RTL the ELF binary and the breakpoint as arguments. The Ara model, model as DUT in RVVTS, using Ara as a representative example. shown on the right-hand side of Figure 1, loads the ELF binary Then, in Section 4.2, we introduce our novel AFC stage, which and begins executing the program. During execution, the Ara test analyzes Machine State deviations to cluster failed cases into mean- harness monitors the PC on each clock cycle to check: (i) whether ingful failure mode categories. the PC has reached the breakpoint, and (ii) whether the PC has remained the same for 10k cycles (hang). 4.1 RTL Model as Design Under Test The former breakpoint case indicates that the program execution completed successfully. The latter hang case is introduced to ad- Supporting a new DUT in RVVTS requires implementing a corre- dress an observation in the current implementation of Ara: certain sponding Execution Runner, which takes a compiled program (ELF instruction sequences may cause the CVA6 core to lock up. While

GLSVLSI ’26, June 22–24, 2026, Canandaigua, NY, USA Manfred Schlägl, Jonas Reichhardt, and Daniel Grosse

the issue is reproducible with specific instruction sequences, we are 1 // Input : elf_filename , breakpoint_addr , hang_threshhold (10000) currently unable to identify a systematic pattern in these sequences. 23 int breakpoint_hits = 0; int hang_pcs [ CVA6Cfg . NrCommitPorts ] = { -1 , -1}; Pseudocode of the Ara test harness handling the breakpoint and 4 int hang_cnt [ CVA6Cfg . NrCommitPorts ] = {0 , 0}; hang cases is shown in Figure 2. Initialization takes place in Lines 2- 5 6 zero ( Ara . memory ); // Set all memory to zero 4. During initialization, the test harness sets up variables for break- 7 elfload ( Ara . memory , elf_filename ); // Load ELF binary point and hang detection, initializes the hang tracking structures 8 9 always @( posedge clk_i ) begin for each commit port, clears the Ara memory, and loads the given 10 // Ara has multiple commit ports ( default 2) -> We have to check all ELF binary into memory. 11 for ( int cpidx = 0; cpidx < CVA6Cfg . NrCommitPorts ; cpidx ++) begin 12 commit_port = Ara . commit_ports [ cpidx ]; The breakpoint and hang detection logic is executed on each 1314 // Check for breakpoint hits on commit port positive clock edge of Ara (Line 9). Since Ara inherits the CVA6 15 if ( commit_port . pc == breakpoint_address && microarchitecture, which supports retiring up to two instructions 16 commit_port . ack && ! commit_port . executed ) begin 17 if ( breakpoint_hits == 0) begin per cycle, the design provides multiple commit ports to report 18 // First breakpoint hit -> Print integer registers and continue simultaneous retirements. Therefore, the test harness has to monitor 19 $display( Ara . regfile ) 20 end else if ( breakpoint_hits == 1) begin the PC values of all commit ports (Line 11). 21 // Second breakpoint hit -> Create mem dump and stop ( COMPLETE ) The breakpoint detection and handling logic, executed for each 22 write_memdump ( Ara . memory , " mem_dump . bin "); 23 $finish() ; commit port on every positive clock edge, is shown in Line 14-26 2425 end breakpoint_hits = breakpoint_hits + 1; of Figure 2. First, the harness checks whether the commit port re- 26 end ports a PC equal to the configured breakpoint address and whether 27 28 // Check for hang on commit port the corresponding instruction has been acknowledged but not yet 29 if ( commit_port . pc == hang_pcs [ cpidx ]) begin executed. Due to the instrumentation scheme used by RVVTS for 30 // PC not changed -> keep track of cycles 31 hang_cnt [ cpidx ] = hang_cnt [ cpidx ] + 1; architectural state extraction, the same breakpoint must be reached 32 if( hang_cnt [ idx ] >= hang_threshhold ) begin twice during execution. On the first occurrence (Lines 17-19), the 33 // Threshhold reached -> Print " Hang " and stop ( HANG ) 34 $display(" HANG "); harness extracts the integer register file, which is still unmodified 3536 end$finish() ; by the RVVTS instrumentation code, and prints it to the standard 37 end else begin output while allowing execution to continue. During the subse- 38 hang_cnt [ cpidx ] = 0; // PC changed -> reset counter 39 end quent execution phase, the inserted RVVTS instrumentation writes 40 hang_pcs [ cpidx ] = commit_port . pc ; // Keep track of PC the remaining architectural state to memory. When the breakpoint 41 end 42 end is reached a second time (Lines 20-23), the harness writes the Ara memory to a dump file and terminates the simulation, making the Figure 2: RVVTS RTL DUT: Ara Test Harness (Pseudocode) stored state available for RVVTS. The hang detection and handling logic is shown in Line 28-39 of Figure 2. For each commit port, the harness checks whether the While the described integration is implemented for the Ara RTL PC remains unchanged compared to the previous clock cycle. If model, the overall approach is not specific to Ara. The concept of the PC does not change, a corresponding hang counter is incre- combining a lightweight RTL test harness with an RVVTS Execution mented. Once the hang counter exceeds the configured threshold Runner can be applied to other RTL-based DUTs with only minor (10k cycles), a hang condition is assumed. In this case, the Ara test adaptations. In particular, the required functionality—program load- harness prints a corresponding message to the standard output and ing, execution monitoring, and architectural state extraction—can terminates the simulation. typically be implemented by extending the existing Top-Level of After termination of the Ara model, either on breakpoint or hang, the respective RTL design. the process flow is back at the AraRunner in RVVTS, left in Figure 1. At this stage, the AraRunner first checks the output of the run for a 4.2 Automated Failure Categorization (AFC) hang message. If a hang is detected, the AraRunner creates a clean As highlighted in the introduction, the combinatorial growth of the new Machine State with the last executed PC set to an invalid value RVV configuration space presents significant challenges for testing. to indicate a failed execution. This Machine State is then returned Although the Single Instruction Isolation method implemented in back to RVVTS. Using this approach, instead of directly reporting RVVTS already categorizes failures based on the isolated instruction, an execution error, enables the RVVTS framework to handle hang the vast state space still leads to a diverse range of failure modes. cases in a manner similar to other Machine State deviations, thereby We define a failure mode as an equivalence class of mismatching allowing the automated result analysis methods of RVVTS to be tests that share the same instruction context, a minimal predicate applied without modification, as demonstrated later in Section 5.3. over RVV parameters (e.g., SEW, LMUL, vl, vstart), and exhibit the If, on the other hand, the execution completes successfully (w/o same architectural symptom. hang), the AraRunner extracts the state of the integer registers from Concrete examples of such failure modes include: (i) Instruction the run’s output and utilizes an existing RVVTS helper component Validity Errors: Instructions executed in unsupported configura- to extract the remaining state from the generated memory dump tions, e.g., reserved vtype encodings or misaligned RVV registers file. Finally, the AraRunner merges these two results to construct a (leading to traps); (ii) Vector Length and Type Handling: Incor- complete Machine State, which is returned to RVVTS, marking the rect handling of vl, register grouping, or element types, including end of the process flow. incorrect vtype updates; (iii) Register Value Deviations: Dif- ferences in RVV register contents after instruction execution; (iv)

From Generation to Failure Categorization: An Open-Source automated RTL Verification Framework for RVV GLSVLSI ’26, June 22–24, 2026, Canandaigua, NY, USA

Control and Status Register Deviations: Unexpected updates Table 1: Test Set generated with RISC-V Vector Tests Generator to CSRs such as vs, fs, or vtype bits; (v) Deadlocks or Hangs: (git hash 841141b775) Execution stalls that prevent test completion, detectable through invalid PC or other architectural state inconsistencies; (vi) Cross- Test Case Million Million Functional Coverage (riscvOVPsim RVV) Instruction Propagation: Systemic issues, e.g., in RVV register Programs Instructions RVV Instr. Points Percent commit or unit communication, which cause multiple instructions 2,391 1,703.06 5.49 (0.32%) 3,653 / 33,076 11.04 to fail in similar ways. These failure modes – distinct from the underlying RTL root causes – must therefore be identified and significantly simplifying the analysis of large failure sets. In Sec- analyzed systematically. tion 5.3, we demonstrate the effectiveness of this approach through Our novel AFC stage addresses this challenge by analyzing the an extensive evaluation using the Ara vector processor as a repre- extracted Machine State from the reference simulator and the DUT, sentative RTL DUT. along with their deviations, to cluster failing cases into meaningful failure mode categories. Within each category, failure cases are 5 still sub-categorized based on the isolated instruction. For each Experimental Evaluation failed test identified by RVVTS, AFC analyzes the corresponding In this section, we demonstrate the effectiveness of the extended Machine States using predefined categorization rules and assigns RVVTS framework and our novel AFC stage by applying them to the failure to a specific failure mode category. The categorization the Ara RTL model. First, in Section 5.1, we evaluate the test set operates as follows: (i) The predefined category rules are evaluated generated by the open-source CHIPS Alliance RISC-V Vector Tests sequentially. (ii) The first rule that matches determines the failure Generator, the default framework for RVV testing in Sail-RISC-V , mode category for the case, ensuring that each failure is assigned as a baseline. Then, in Section 5.2, we introduce the two new test to exactly one category. (iii) If no rule matches (fall-through), the sets generated by RVVTS for Ara, along with the high-level results case is categorized as UNKNOWN . obtained for Ara. After this, in Section 5.3, we use AFC to provide In total, we define rules for 29 failure mode categories, including a more detailed analysis of the results. the fallback category UNKNOWN . These categories are derived systematically from the previously introduced failure modes and 5.1 RISC-V Vector Tests Generator Test Set capture distinct classes of architectural deviations encountered In this section we evaluate the test set generated by the open- during testing. Due to space limitations, it is not feasible to describe source RISC-V Vector Tests Generator (CARVVTG) [7] as baseline all categories in detail within this paper. Instead, we present selected for comparison with the RVVTS test sets. From configurations for representative examples in the following. each RVV instruction, the CARVVTG framework generates a series Two representative rules for Instruction Validity Errors are of independent, self-checking tests as executable programs to be those defining the failure mode categories EXC_INVALID_ACCEPT run on the DUT. Targeting Ara, we configure the framework to and EXC_INVALID_REJECT . As described earlier, the behavior of generate a test set for RV64GV with a Vector Register Length (VLEN) RVV instructions, including their validity, is highly dependent on of 4096 bits. Furthermore, to enable the application of the same the dynamic configuration. The rule for EXC_INVALID_ACCEPT functional coverage metric used by RVVTS (riscvOVPsim [9]), we checks whether the number of traps observed during execution on modify the generator to ensure that no parts of the test programs the reference simulator is higher than on the DUT, i.e., the DUT has are skipped following a detected failure. This guarantees that all accepted instructions that are forbidden according to the reference. instructions included in the test cases are executed and accounted Conversely, the rule for EXC_INVALID_REJECT covers cases where for by the coverage metric. the number of observed traps on the reference simulator is lower CARVVTG generates multiple test programs for each instruction, than on the DUT, i.e., the DUT has rejected instructions that are with each program containing tens of thousands of test cases follow- allowed according to the reference. Both rules ignore other Machine ing the same pattern but with varying values and configurations. State deviations, since such deviations would be a consequence of Table 1 summarizes the major characteristics of the generated test the trapping behavior: an executed instruction may have side effects set, including the functional coverage achieved. Overall, CARVVTG (e.g., modifying register values), whereas a trapping instruction generates 2,391 test programs with a total size of 43 GiB. What would not. stands out immediately is that, despite the test set including over A representative rule for Register Value Deviations is the 1 .7 billion instructions, only 0.32% are RVV instructions. This is one defining the IREG_ONLY category. This rule checks whether explained by the substantial overhead caused by the self-checking all Machine State deviations are exclusively related to the values mechanism for each test case. However, the absolute number of 5.49 of the 32 integer registers (including the zero register). Cases in million RVV instructions is still substantial. What is a major weak- the IREG_ONLY category may, for example, indicate issues in in- ness though is the rather low functional coverage of 11.04%. This struction implementations (e.g., incorrect calculations) or high- is explained by the fact, that CARVVTG uses mostly the same code light instructions that modify integer registers even though they patterns and register selection sets for all test cases. Examples for are not specified to do so. Similar rules exist for exclusive devia- this can be seen in Figure 3. The listings (left and right) show all con- tions in floating-point (FREG_ONLY ) registers and RVV registers crete generated instructions for the vector widening add vwadd.vv (VREG_ONLY ). (left) and the vector widening sum reduction vwredsum.vs (right) Together, the 29 defined rules allow the AFC stage to systemati- included in the test set (excluding vector mask variants). We ob- cally classify failing tests into meaningful failure mode categories, serve that, in both cases, the concrete values for the destination

GLSVLSI ’26, June 22–24, 2026, Canandaigua, NY, USA Manfred Schlägl, Jonas Reichhardt, and Daniel Grosse

                  Table 2: Test Sets generated with CovGuidedTestsetGenerator and applied on Ara (git hash a6436df6ad)

                              Million              Million            Functional Coverage                                               Isolated Failing

Test set Instructions RVV Instr. (riscvOVPsim RVV) Detected Minimized Cases Instructions Points Percent Fails #Cases Percent Overall Exclusive Valid Sequences (VS) 2.17 0.95 (43.78 %) 31,403 / 33,076 94.94 48,973 46,601 95.16 584 13 Invalid+Valid Sequences (IVS) 2.04 1.00 (49.02 %) 31,950 / 33,076 94.60 33,630 33,391 99.29 598 27 Merged Sequences (MS) 4.21 1.95 (46.32 %) 31,951 / 33,076 96.60 82,603 79,992 96.84 611 N/A (VS + IVS)

// Vector Widening Int Add // Vector Widening Int Sum Reduction 5.2 RVVTS Test Sets and Application on Ara vd vs2 vs1 vd vs2 vs1 vwadd.vv v2 , v5 , v4 vwredsum.vs v1 , v5 , v26 RVVTS already comes with pre-generated, high-functional-coverage vwadd.vv v4 , v10 , v8 vwredsum.vs v4 , v12 , v28 (>94%) test sets for positive and negative testing of RV32 and vwadd.vv v8 , v20 , v16 vwredsum.vs v6 , v14 , v4 vwadd.vv v27 , v1 , v14 vwredsum.vs v24 , v8 , v16 RV64 RVV implementations with a VLEN configuration of 512 vdvs2:: {v2 , v4 , v8 , v27 } vd: {v1 , v4 , v6 , v24 } bit [30]. However, Ara is by default configured with a VLEN of {v4 , v8 , v14 , v16 } vs2: {v5 , v8 , v12 , v14 } vs1: {v1 , v5 , v10 , v20 } vs1: {v4 , v16 , v26 , v28 } 4096 bit which prevents application of these pre-generated test sets. Although Ara can be reconfigured to a VLEN of 512 bits, we Figure 3: CARVVTG Instruction Patterns keep the default configuration, as it appears to be the most thor- oughly tested, given that it was used in the tape-out of the Yun System-On-Chip (SoC) [28]. Since the existing pre-generated RVVTS test sets are not com- patible with Ara, we generate new test sets: We configure RVVTS for RV64 and a VLEN of 4096, and generate two test sets. For the and source registers are selected from sets containing only four first test set, RVVTS is configured to generate only code sequences elements (sets in lower part of both listings in Figure 3). Since the that don’t trigger traps when executed on the reference simulator. coverage metric awards points for all possible register combinations, This results in a test set that contains only Valid Sequences (VS), large parts of the coverage points are never hit. targeting pure positive testing. For the second test set, the generator Another weakness of CARVVTG related to the restricted regis- is configured to allow code sequences that trigger traps, resulting ter selection sets, is the lack of support for testing register group in a test set that contains Invalid+Valid Sequences (IVS), targeting overlaps, which is also mentioned in the framework’s documen- positive/negative testing. tation. RVV has very specific rules, when it comes to overlapping The left-hand side of Table 2 summarizes the major characteris- of source and destination registers. For example, overlap is ex- tics of the test sets VS and IVS, as well as the merged test set MS plicitly allowed by RVV for the vector widening sum reduction (i.e., VS + IVS). Both VS and IVS include more than 2 million instruc- vwredsum.vs v0, v0, v1. In contrast, the vector widening add tions. Compared to the CARVVTG test set evaluated in Section 5.1, vwadd.vv v0, v0, v1 is not allowed (trap), because writing dou- which contains 1.7 billion instructions (Table 1), the total number ble width target elements in v0 can interfere with the subsequent of instructions in VS and IVS is substantially smaller. However, the single width source elements in the same register. Both types of fraction of RVV instructions in the RVVTS test sets exceeds 43%, instruction patterns, as exemplified in Figure 3 for vwadd.vv and in contrast to only 0.32% in CARVVTG. As a result, each test set vwredsum.vs, are explicitly excluded by the CARVVTG generator. contains approximately one million RVV instructions, compared As a result, any related bugs remain undetectable. to 5.49 million in the CARVVTG test set. Although the VS and IVS The final major weakness of CARVVTG that we address is its test sets contain fewer RVV instructions, each achieves a functional complete lack of support for negative testing, the significance of coverage of over 94%, which is substantially higher than the 11.04% which will be demonstrated later in Section 5.2 and Section 5.3. In obtained with CARVVTG. Additionally, it is important to note that negative testing, we feed a known invalid instruction to the DUT only the VS test set is directly comparable to CARVVTG, since and verify whether it is correctly rejected with a trap. One exam- CARVVTG does not generate invalid sequences and therefore does ple is the previously mentioned invalid vwadd.vv v0, v0, v1 to not support negative testing. In summary, the test sets generated test whether group overlap is implemented correctly and does not by RVVTS achieve significantly higher coverage with far fewer allow more than specified. While the test programs generated by instructions and, unlike CARVVTG, also include negative testing, CARVVTG include trap handlers, they automatically treat traps as which as we will see shortly, is essential. failures. The inverse case, where a trap is expected and the absence The right-hand side of Table 2 presents the results obtained when of a trap is considered a failure, is not accounted for in the test applying the test sets to Ara via RVVTS. For VS, IVS and MS, the programs. table shows, from left to right, the number of detected failure cases, For CARVVTG, we can therefore conclude that (i) functional cov- the fractions of these cases where RVVTS is able to minimize them, erage is very low at 11.04% due to the reasons outlined above, and and the number of unique instructions that are isolated by RVVTS (ii) the framework lacks support for negative testing. In the follow- as cause the failures. We see, that RVVTS is able to minimize the ing sections, we will demonstrate that extended RVVTS provides majority (>95%) of all these fails. The right-most sub-column of significant improvements in these aspects. Table 2 presents the number of isolated unique instructions detected

From Generation to Failure Categorization: An Open-Source automated RTL Verification Framework for RVV GLSVLSI ’26, June 22–24, 2026, Canandaigua, NY, USA

Table 3: Top 6 Failure Categories on Ara (git hash a6436df6ad)

Failure Category VS #Detected Fails Isolated Failing Instructions in MS (out of 29 categories) IVS MS #Minimized Cases in MS #Instructions RVV Instruction Classes (% w.r.t. all failed cases) (% w.r.t fails in category) Class #Cases #Instr integer 10,405 132 mask 10,290 13 VREG_ONLY 34,315 3,654 37,969 37,108 fixed-point 6,324 32 70.07% 10.87% 45.97% 97.73% 448 floating-point 4,135 89 permutation 2,478 15 reduction 1,774 16 load 1,702 151 VTYPE_VILL_SET_ERROR 90 19,858 19,948 19,948 1 config 19,948 1 0.18% 59.05% 24.15% 100.00% permutation 3,650 10 store 1,404 104 8,384 1,606 9,990 8,521 load 1,349 ARA_HANG 291 fixed-point 1,204 17.12% 4.78% 12.09% 85.30% mask 514 1346 integer 298 floating-point 57 2229 reduction 45 4 MSTATUS_EXT_DUT 6 4,303 4,309 4,297 floating-point 3,277 91 0.01% 12.80% 5.22% 99.72% 101 reduction 589 64 permutation 431 reduction 1,043 16 mask 825 10 EXC_INVALID_REJECT 1,174 1,242 2,416 2,415 integer 511 27 2.40% 3.69% 2.92% 99.96% 65 floating-point 13 8 permutation 11 2 store 75 11 load floating-point 750 91 load 367 107 integer 332 92 EXC_INVALID_ACCEPT 0.01%3 2,309 2,312 2,310 240 32 6.87% 2.80% 99.91% 413 fixed-point permutation 199 15 store 164 63 reduction 148 85 mask 110 OTHERS store 2,076 Zicsr config 1,714 permutation 1,104 12422 10 categories w/ cases 5,001 658 5,659 5,393 155 mask 382 13 categories w/o cases 10.21% 1.96% 6.85% 95.30% config 55 floating-point 46 226 0 cases in UNKNOWN fixed-point 9 7 load 52 52 reduction

exclusively by the given test set, which is calculated e.g. for IVS by rather by a pure positive (VS) or by a positive/negative (IVS) test- subtracting the number of isolated instructions from VS from MS, ing strategy. The following columns present the numbers of cases i.e., 611−584 = 27. These values highlight the importance of positive that are minimized by RVVTS and the number of isolated unique and negative testing: Since instruction sequences containing illegal instructions. The final column presents the isolated instructions, parameter combinations or configurations are, by definition, not grouped into instruction categories based on the RVV spec., along included in the VS test set, bugs related to such invalid sequences with number of cases and number isolated instructions, ranked by can only be detected through negative testing and, therefore, with number of cases. We will now briefly discuss the 6 top failure mode the IVS test set. Conversely, since VS, which targets pure positive categories presented in Table 3. testing, is roughly the same size as IVS but does not contain invalid The top-most category presented in Table 3 is VREG_ONLY . This sequences, it includes a larger number of valid sequences and can, category contains all cases with deviations exclusively in the values therefore, detect cases not covered by IVS. In the next section, we within RVV registers (i.e. there must be no deviations in other reg- will utilize AFC to conduct a more detailed analysis of the results. isters, CSRs, etc.). Cases in this category may, for example, indicate issues in instruction implementations (e.g., incorrect calculations) 5.3 Automated Failure Categorization on Ara or in RVV register handling. As shown in Table 3, such cases are As shown in the previous section, RVVTS is able to trace the majority predominantly identified through positive testing on Ara (~70% for of failures down to individual instructions. However, analyzing the VS compared to ~11% for IVS). root causes of a large number of failures per instruction still requires The second category, VTYPE_VILL_SET_ERROR, contains cases significant manual effort. To tackle this challenge, we next apply related to the vill bit in the vtype CSR, which indicates that the our AFC stage, presented in Section 4.2, to provide a more detailed RVV configuration is illegal, causing subsequent vector instructions analysis of the results. to trap. More specifically, this category includes cases where the The results generated by the applied AFC, are shown in Table 3. vill bit is set on the reference simulator, but not on DUT. For The table rows list the top 6 failure mode categories, ranked by the Ara, all 19, 948 detected cases in this category are related to the number of failures and accounting for more than 93% of all failed exact same issue, namely that the vill bit is not writable using cases. The final row, Others, summarizes the failure mode categories the vsetvl configuration instruction. As shown in Table 3, cases not included in these top 6. The second column presents the number in this category are primarily identified via negative testing (IVS). of detected fails for the VS, IVS and the merged MS test set. From the The third category, ARA_HANG, consists of all cases where a numbers for VS and IVS we can observe whether failures are found hang is detected in Ara, preventing the execution of a test case from

GLSVLSI ’26, June 22–24, 2026, Canandaigua, NY, USA Manfred Schlägl, Jonas Reichhardt, and Daniel Grosse

completing, as described in Section 4.1. The cases are identified by [6] 2026. RISC-V Torture Test Generator. https://github.com/ucb-bar/riscv-torture. checking for an invalid value for the last executed PC in the DUT [7] 2026. RISC-V Vector Tests Generator. https://github.com/chipsalliance/riscv- Machine State. Interestingly, cases in this category are primarily vector-tests. identified via positive testing (VS) and the majority of cases can [8] 2026. RISCV-DV. https://github.com/google/riscv-dv. [9] 2026. riscvOVPsim Imperas RISC-V Instruction Set Simulator (ISS). https://www. be minimized. This indicates issues in instruction processing and imperas.com/riscvovpsim-free-imperas-risc-v-instruction-set-simulator. execution, rather than in instruction validity checks. [10] 2026. Sail RISC-V: Formal Specification of the RISC-V ISA. https://github.com/ riscv/sail-riscv. The fourth category, MSTATUS_EXT_REF, contains cases related [11] 2026. Spike RISC-V ISA Simulator. https://github.com/riscv/riscv-isa-sim. to the vs and fs bits in the mstatus CSR, which represent the en- [12] 2026. Verilator - A fast Verilog/SystemVerilog simulator. https://www.verilator. able and dirty bits for the RVV and floating-point extensions. More org. [13] Allon Adir, Eli Almog, Laurent Fournier, Eitan Marcus, Michal Rimon, Michael specifically, this category includes cases where the number of set Vinov, and Avi Ziv. 2004. Genesys-Pro: innovations in test program generation bits after execution on the DUT is higher than on the reference. for functional processor verification. DTC (2004), 84–93. Cases in this category may hint to instructions that invalidly enable [14] Niklas Bruns, Daniel Große, and Rolf Drechsler. 2020. Early Verification of ISA Extension Specifications Using Deep Reinforcement Learning. In GLSVLSI. extensions. As shown in Table 3, cases in this category are primar- 297–302. ily identified via negative testing (IVS). With MSTATUS_EXT_REF, [15] Niklas Bruns, Vladimir Herdt, and Rolf Drechsler. 2023. Processor Verification using Symbolic Execution: A RISC-V Case-Study. In DATE. 1–6. there is also a direct counterpart to this category, for which, how- [16] Brian Campbell and Ian Stark. 2014. Randomised Testing of a Microprocessor ever, no fail cases were found on Ara. Model Using SMT-Solver State Generation. In Formal Methods for Industrial The two last categories, EXC_INVALID_REJECT/ACCEPT , are Critical Systems. 185–199. [17] Sadullah Canakci, Chathura Rajapaksha, Leila Delshadtehrani, Anoop Nataraja, also counterparts to each other. In both cases, the numbers of traps Michael Bedford Taylor, Manuel Egele, and Ajay Joshi. 2023. ProcessorFuzz: triggered during execution are compared. EXC_INVALID_REJECT Processor Fuzzing with Control and Status Registers Guidance. In International contains the cases where the number of traps was higher on the Symposium on Hardware Oriented Security and Trust. 1–12. [18] Matheus Cavalcante, Fabian Schuiki, Florian Zaruba, Michael Schaffner, and DUT, which indicates that the DUT rejected instructions consid- Luca Benini. 2020. Ara: A 1-GHz+ Scalable and Energy-Efficient RISC-V Vector ered valid by the reference. In contrast, EXC_INVALID_ACCEPT Processor With Multiprecision Floating-Point Support in 22-nm FD-SOI. TVLSI 28, 2 (2020), 530–543. contains the cases where the number of traps was higher on the [19] Mikhail Chupilko, Alexander Kamkin, Artem Kotsynyak, and Andrei Tatarnikov. reference, indicating that the DUT accepted instructions considered 2017. MicroTESK: Specification-Based Tool for Constructing Test Program Gen- invalid by the reference. As shown in Table 3, the latter category, erators. In Haifa Verification Conference. [20] Shai Fine and Avi Ziv. 2003. Coverage directed test generation for functional in particular, is primarily identified via negative testing (IVS). verification using Bayesian networks. In DAC. 286–291. Reaching the final row, OTHERS, we observe that AFC success- [21] Vladimir Herdt, Daniel Große, and Rolf Drechsler. 2020. Closing the RISC-V fully categorized all detected Ara failures into 16 (6 top categories Compliance Gap: Looking from the Negative Testing Side. In DAC. 1–6. [22] Vladimir Herdt, Sören Tempel, Daniel Große, and Rolf Drechsler. 2021. Mutation-

• 10 with cases) out of a total of 29 categories. Notably, all cases are based Compliance Testing for RISC-V. In ASP-DAC. 55–60. successfully categorized, as none fall into the UNKNOWN category. [23] Jaewon Hur, Suhwan Song, Dongup Kwon, Eunjin Baek, Jangwoo Kim, and Byoungyoung Lee. 2021. DifuzzRTL: Differential Fuzz Testing to Find CPU Bugs. In Symposium on Security and Privacy. 1286–1303.

6 Conclusions [24] Charalambos Ioannides, Geoff Barrett, and Kerstin Eder. 2011. Feedback-Based Coverage Directed Test Generation: An Industrial Evaluation. 112–128. We introduced an extended, open-source RVVTS-based framework [25] Victor Jimenez, Mario Rodriguez, Marc Dominguez, Josep Sans, Ivan Diaz, Luca for RTL verification of RVV 1.0, featuring a novel Automated Failure Valente, Vito Luca Guglielmi, Josue V. Quiroga, R. Ignacio Genovese, Nehir Categorization (AFC) stage for scalable failure analysis. Using the Sonmez, Oscar Palomar, and Miquel Moreto. 2023. Functional Verification of a RISC-V Vector Accelerator. IEEE Design & Test 40, 3 (2023), 36–44. Ara vector processor as DUT, we achieved over 96% functional cov- [26] Lorenzo Martignoni, Roberto Paleari, Giampaolo Fresi Roglia, and Danilo Bruschi. erage, significantly outperforming the open-source CHIPS Alliance 2009. Testing CPU Emulators. 261–272. [27] Matteo Perotti, Matheus Cavalcante, Renzo Andri, Lukas Cavigelli, and Luca RISC-V Vector Tests Generator, which serves as the default frame- Benini. 2024. Ara2: Exploring Single- and Multi-Core Vector Processing With an work for RVV testing in Sail-RISC-V , used here as a baseline. Our Efficient RVV 1.0 Compliant Open-Source Processor. TC 73, 7 (2024), 1822–1836. framework detected over 82k deviations, minimized 97% of them, [28] Matteo Perotti, Matheus Cavalcante, Alessandro Ottaviano, Jiantao Liu, and Luca Benini. 2023. Yun: An Open-Source, 64-Bit RISC-V-Based Vector Processor and clustered failures into 16 categories, streamlining debugging With Multi-Precision Integer and Floating-Point Support in 65-nm CMOS. IEEE and root-cause analysis. The extended RVVTS framework and all Transactions on Circuits and Systems II: Express Briefs 70, 10 (2023), 3732–3736. Ara experimental results are publicly available on GitHub. [29] Matteo Perotti, Matheus Cavalcante, Nils Wistoff, Renzo Andri, Lukas Cavigelli, and Luca Benini. 2022. A “New Ara” for Vector Computing: An Open Source Highly Efficient RISC-V V 1.0 Vector Processor Design. In International Conference Acknowledgments on Application-specific Systems, Architectures and Processors. 43–51. [30] Manfred Schlägl and Daniel Große. 2024. Single Instruction Isolation for RISC-V This work has partially been supported by the LIT Secure and Vector Test Failures. In ICCAD. 156:1–156:9. Correct Systems Lab funded by the State of Upper Austria. [31] Manfred Schlägl and Daniel Große. 2025. Fast Interpreter-Based Instruction Set Simulation for Virtual Prototypes. In DATE. 1–7. [32] Manfred Schlägl, Christoph Hazott, and Daniel Große. 2024. RISC-V VP++: Next References Generation Open-Source Virtual Prototype. In OSDA. [33] Manfred Schlägl, Moritz Stockinger, and Daniel Große. 2024. A RISC-V “V” VP: [1] 2026. AndesCore NX27V Processor – 64-bit CPU with RISC-V Vector Exten- Unlocking Vector Processing for Evaluation at the System Level. In DATE. 1–6. sion. https://www.andestech.com/en/products-solutions/andescore-processors/ [34] Flavien Solt, Katharina Ceesay-Seitz, and Kaveh Razavi. 2024. Cascade: CPU riscv-nx27v. fuzzing via intricate program generation. In USENIX Conference on Security [2] 2026. Ara Vector Processor. https://github.com/pulp-platform/ara. Symposium. USA. [3] 2026. FORCE-RISCV RISC-V Instruction Sequence Generator (ISG). https:// [35] Chloe Tain, Savita Patil, and Hussain Al-Asaad. 2025. Survey of Verification of github.com/openhwgroup/force-riscv. RISC-V Processors. JETTA 41, 2 (May 2025), 111–138. [4] 2026. ImperasDV – RISC-V Processor Verification Made Easy. https://www. [36] Florian Zaruba and Luca Benini. 2019. The Cost of Application-Class Processing: synopsys.com/verification/imperasdv.html. Energy and Performance Analysis of a Linux-Ready 1.7-GHz 64-Bit RISC-V Core [5] 2026. RISC-V Architecture Test SIG. https://github.com/riscv-non-isa/riscv-arch- in 22-nm FDSOI Technology. TVLSI 27, 11 (Nov 2019), 2629–2640. test.