Architectural Testing

Definition

Architectural (Compatibility) Testing (AT) ensures that an implementation of an ISA meets all the requirements specified by the architecture under all conditions. For RISC-V, which is micro-architecture agnostic and extremely configurable, it is impossible to create an exhaustive test suite covering all implementations; instead, a suite is built that targets limited areas of the ISA [8bde013b].

Relationship to Design Verification

AT is a subset of Design Verification and is not, by itself, sufficient to guarantee a functionally correct implementation. It only addresses verification of the details specified in the ISA [8bde013b].

Objectives

• Confirm ISA-level conformance under all specified conditions.
• Operate within the practical limits imposed by RISC-V's configurability.
• Provide targeted coverage of architectural features rather than exhaustive verification.

Challenges

1. Standardised test format that maintains uniformity and enables maintenance.
   • The RISC-V Architectural Test SIG has released a Test Format Specification that provides standard macros, keeping tests immune to implementation-specific choices and behaviour [8bde013b].
   • The specification mandates that tests be signature-based: signatures produced by the implementation are compared against a "golden" model signature [8bde013b].
2. A standard ISA coverage specification format to indicate the quality of tests and identify holes or gaps [8bde013b].
3. A tool to measure coverage as per the ISA coverage specification [8bde013b].
4. A tool to generate efficient, directed tests as the ISA specification and testing scopes grow [8bde013b].

Contributions of the Modern Toolchain (RISCV-ISAC / RISCV-CTG)

The paper Automating Generation and Maintenance of a High-Quality Architectural Test Suite for RISC-V (CARRV @ ISCA 2022) presents a toolchain built around the principles of architectural testing with the following contributions [8bde013b]:

1. Low barriers to entry
   • Simple ISA coverage specification format.
   • Flexible and free; independent of any specific simulator or implementation.
   • Requires only knowledge of RISC-V and basic Python.
2. Generated tests are controlled and directed
   • Generated from coverpoints, achieving full coverage with minimal testing.
   • Runnable on any implementation or model.
   • No negative testing — tests only features/behaviour that are implemented.
3. Data propagation analysis ensures that generated signatures are influenced by the tests.
4. Open source tools for coverage measurement and test generation.

Coverpoints in the ISAC Model

A coverpoint specifies a boolean expression over the fields of an architectural element (instruction or state) that must be covered during execution [7408edcd].

• In classical Design Verification, coverpoints are defined in SV-UVM against the RTL. This has a high entry barrier because it is expensive and requires SystemVerilog knowledge, and it is extremely difficult to implement all possible architectural options in a single RTL [7408edcd].
• In ISAC, coverpoints are expressed as pure Python expressions.
• Coverpoints are categorised by the variables/fields they exercise:
  • Instruction Mnemonics (Opcode)
  • Register Operands
  • Operand Combinations
  • Value Combinations
  • CSR Value Combinations
  • Cross Combinations (coverage across multiple instructions)
• Custom Python functions (abstract_comb) are resolved into standard coverpoints during normalization.

Coverage Generation Flow (Tool Flow)

Coverage is computed from an instruction trace produced by the implementation/model. The trace contains [7408edcd]:

• Instruction address
• Instruction encoding
• Architectural state changes (CSR and regfile updates, if any)

The trace parser and decoder are implemented as plugins to support customisation. Architectural states are maintained internally, and the tool supports filtering instructions that influence coverage. Coverage collection can be constrained to specific covergroups for a run, and custom boundaries for the signature region can be specified.

See Also

• RISC-V — the target ISA for the architectural test suite.
• Design Verification — broader verification discipline of which architectural testing is a subset.
• Automating Generation and Maintenance of a High-Quality Architectural Test Suite for RISC-V — the CARRV @ ISCA 2022 paper that motivates and operationalises much of the toolchain.