Skip to content
STIMSMITH

Automated Failure Categorization

Concept WIKI v1 · 9/5/2026

Automated Failure Categorization (AFC) is a rule-based analysis stage for RVV RTL verification that clusters failing tests by architectural symptoms, instruction context, and RVV-parameter predicates to make large failure sets easier to debug.

Overview

Automated Failure Categorization (AFC) is a stage introduced in the open-source RVVTS-based RTL verification framework for the RISC-V Vector Extension (RVV) 1.0. It is intended to make large sets of RTL verification failures more scalable to analyze by clustering failing cases into meaningful failure-mode categories rather than leaving engineers to inspect each failing test manually.

AFC addresses a specific problem in RVV verification: RVV instructions are parameterized by runtime configuration state such as vtype, vl, vstart, element width, register grouping, and masking. This creates a large configuration space, so even after failures are isolated to individual instructions, many distinct architectural symptoms can remain.

Failure-mode model

The framework defines a failure mode as an equivalence class of mismatching tests that share:

  • the same instruction context,
  • a minimal predicate over RVV parameters such as SEW, LMUL, vl, and vstart, and
  • the same architectural symptom.

The paper distinguishes these failure modes from the underlying RTL root causes: a failure mode is an observable architectural pattern, not necessarily the final bug diagnosis.

Examples of failure modes described for AFC include:

  • instruction validity errors, such as executing unsupported configurations,
  • vector length and type handling errors,
  • RVV register value deviations,
  • control and status register deviations,
  • deadlocks or hangs, and
  • cross-instruction propagation effects.

Inputs and operation

AFC operates on the extracted Machine State from both the reference simulator and the RTL device under test, together with the deviations between those states. For each failed test identified by RVVTS, AFC evaluates predefined categorization rules sequentially. The first matching rule determines the failure-mode category, ensuring that each failure is assigned to exactly one category. If no rule matches, the failure falls through to the UNKNOWN category.

The paper reports 29 defined failure-mode categories in total, including the fallback UNKNOWN category. Within each category, cases are still sub-categorized by the isolated instruction, so AFC complements rather than replaces single-instruction isolation.

Relationship to single-instruction isolation

RVVTS already includes a single-instruction isolation method that traces many failures down to individual instructions. AFC builds on that result: it further groups failures by architectural symptom and RVV-parameter context. This is useful because a single isolated instruction may still fail in many different ways across the RVV configuration space.

Handling hangs

The Ara RTL integration represents detected hangs in a way that AFC and the rest of RVVTS can analyze uniformly. If a hang is detected, the runner creates a clean Machine State whose last executed program counter is set to an invalid value. This allows hang cases to be handled like other Machine State deviations instead of as a separate execution error path.

In the Ara evaluation, one AFC category, ARA_HANG, consisted of cases where a hang prevented a test from completing. These cases were identified by checking for an invalid value for the last executed PC in the DUT Machine State.

Evaluation on Ara

In the reported evaluation on the Ara RVV processor, the extended RVVTS framework with AFC detected more than 82,000 deviations, automatically minimized about 97% of them, and clustered failures into 16 categories. Another passage reports that AFC associated minimized Ara failures with 611 instructions across 16 failure-mode categories.

The six largest failure-mode categories accounted for more than 93% of all failed cases in the Ara study. Discussed examples include:

  • VREG_ONLY: deviations exclusively in RVV register values, which may indicate instruction implementation or RVV register-handling issues;
  • VTYPE_VILL_SET_ERROR: cases involving the vill bit in the vtype CSR, especially cases where the reference set vill but the DUT did not;
  • ARA_HANG: cases where Ara failed to complete execution;
  • MSTATUS_EXT_REF: cases related to vs and fs bits in the mstatus CSR;
  • EXC_INVALID_REJECT; and
  • EXC_INVALID_ACCEPT.

Significance

AFC is presented as the scalable result-analysis part of the broader RVVTS-based framework. Its role is to reduce debugging effort by turning large numbers of raw mismatches into categorized architectural symptoms, while preserving links to isolated instructions for more focused root-cause analysis.

LINKED ENTITIES

4 links
From Generation to Failure Categorization: An Open-Source automated RTL Verification Framework for RVV INTRODUCES
Machine State USES
failure mode category INTRODUCES
single-instruction isolation USES

CITATIONS

11 sources
11 citations
[1] AFC is introduced as a novel stage in an open-source RVVTS-based RTL verification framework for RVV 1.0. From Generation to Failure Categorization: An Open-Source automated RTL Verification Framework for RVV
[2] AFC is motivated by the large RVV configuration space and the need to analyze diverse failure modes even after single-instruction isolation. From Generation to Failure Categorization: An Open-Source automated RTL Verification Framework for RVV
[3] The framework defines a failure mode as an equivalence class of mismatching tests sharing instruction context, a minimal predicate over RVV parameters, and the same architectural symptom. From Generation to Failure Categorization: An Open-Source automated RTL Verification Framework for RVV
[4] AFC analyzes Machine States from the reference simulator and DUT, evaluates predefined rules sequentially, assigns the first matching failure-mode category, and uses UNKNOWN as a fallback. From Generation to Failure Categorization: An Open-Source automated RTL Verification Framework for RVV
[5] The paper defines 29 failure-mode categories for AFC, including the fallback UNKNOWN category. From Generation to Failure Categorization: An Open-Source automated RTL Verification Framework for RVV
[6] AFC examples include instruction validity errors, vector length and type handling errors, register value deviations, CSR deviations, deadlocks or hangs, and cross-instruction propagation. From Generation to Failure Categorization: An Open-Source automated RTL Verification Framework for RVV
[7] For hang handling, the AraRunner creates a Machine State with an invalid last executed PC so RVVTS can process hang cases like other Machine State deviations. From Generation to Failure Categorization: An Open-Source automated RTL Verification Framework for RVV
[8] In the Ara evaluation, AFC detected hang-category cases by checking for an invalid last executed PC in the DUT Machine State. From Generation to Failure Categorization: An Open-Source automated RTL Verification Framework for RVV
[9] In the Ara evaluation, the framework detected more than 82,000 deviations, minimized about 97%, and clustered failures into 16 categories. From Generation to Failure Categorization: An Open-Source automated RTL Verification Framework for RVV
[10] The Ara study reports AFC associating minimized failures with 611 instructions across 16 failure-mode categories. From Generation to Failure Categorization: An Open-Source automated RTL Verification Framework for RVV
[11] The six largest Ara failure-mode categories account for more than 93% of all failed cases and include VREG_ONLY, VTYPE_VILL_SET_ERROR, ARA_HANG, MSTATUS_EXT_REF, EXC_INVALID_REJECT, and EXC_INVALID_ACCEPT. From Generation to Failure Categorization: An Open-Source automated RTL Verification Framework for RVV