Source 9bacc3bf... — STIMSMITH

SOURCE ARCHIVE

SHA256: 9bacc3bf3002c64ba46cacd9422c41909964ee61f08b8255f383e77a169886ce

URL: https://carrv.github.io/2022/slides/CARRV2022_slides_2_Kumar.pdf

TYPE: application/pdf

SIZE: 132.7 KB

FETCHED: 7/3/2026, 10:01:09 AM

EXTRACTOR: liteparse

CHARS: 15,069

EXTRACTED CONTENT

15,069 chars

Automating Generation and Maintenance of a High-Quality Architectural Test Suite for RISC-V CARRV at ISCA 2022

S Pawan Kumar 1 Shrreya Singh 2 Neel Gala 1 Allen Baum 3

1InCore Semiconductors

2IIT Gandhinagar

3Esperanto Technologies

19 June 2022

                                                            .   .  .  .  .  . .

Outline

1 Architectural Testing

2 RISCV-ISAC

3 RISCV-CTG

4 Related Work

                                                             .   .  .  .  .  . .

Architectural Testing

1 Architectural Testing

2 RISCV-ISAC

3 RISCV-CTG

4 Related Work

                                                             .   .  .  .  .  . .

Architectural Testing

Objectives

• Architectural (Compatibility) Testing involves ensuring that an implementation of the ISA meets all the requirements under all conditions. • RISC-V is micro-architecture agnostic and Extremely Configurable; Its impossible to create an exhaustive test suite for all implementations. • However, it is possible to build a suite which test limited areas of the ISA. • AT is a subset of design verification and is not sufficient by itself to DV ensure a functionally correct implementation. It only deals with the AT verification of details specified in the ISA.

                                                               .   .  .  .  .  . .
.                                                              .   .  .  .  .  .
©
 2022 InCore Semiconductors Pvt. Ltd.    CARRV @ ISCA, 2022              4/18

Architectural Testing Challenges

1 Standardised format for tests which maintains uniformity and enables maintenance. • RISC-V Architectural Test SIG has released the Test Format Specification1 which provides standard macros for the tests. This keeps the tests immune to implementation specific choices and behaviour. • This spec mandates that the tests be signature based, where the signatures are compared with the signature from a ”golden” model. 2 A standard way of indicating the quality of tests and identify possible holes or gaps, an ISA coverage specification format. 3 A tool to measure coverage as per 2. 4 A tool to generate efficient, directed tests as the ISA specification and testing scopes grow.

Architectural Testing Contributions

1 Low barriers to entry • Simple ISA Coverage specification format • Flexible & Free: Independent of simulator/implementation • Need knowledge about RISC-V and basic python to get started 2 Generated tests are controlled and directed. • Generated from coverpoints, ensuring that full coverage is achieved with minimal testing. • The tests can be run on any implementation/model • No negative testing; Test only for features/behaviour implemented. 3 Data propagation analysis ensures that the signatures generated are influenced by the tests. 4 Open source tools for coverage measurement and test generation

                                                             .   .  .  .  .  . .
     .                                                       .   .  .  .  .  .

RISCV-ISAC

1 Architectural Testing

2 RISCV-ISAC

3 RISCV-CTG

4 Related Work

                                                             .   .  .  .  .  . .

         RISCV-ISAC     CGF

Coverage Definition

• A coverpoint specifies a boolean expression over the fields of an architectural element(instruction/state) that is required to be covered during execution • In classical DV: Define coverpoints in SV-UVM for the RTL • High entry barrier: Expensive and requires SV knowledge • Extremely difficult to implement all possible architectural options in a single RTL • In ISAC, coverpoints are expressed as pure python expressions. • The coverpoints are categorised based on the variables/fields which can be tested. • Instruction Mnemonics(Opcode) • Register Operands • Operand Combinations • Value Combinations • CSR Value Combinations • Cross Combinations (To define coverage across multiple instructions) • Custom functions in python(abstract_comb) which are resolved into the standard coverpoints during normalization. . . . . . . . . . . . . . . © 2022 InCore Semiconductors Pvt. Ltd. CARRV @ ISCA, 2022 8/18

                                        RISCV-ISAC CGF

CGF Description

add_cov : c o n f i g : [ check ISA:= r e g e x ( . ∗ I . ∗ ) ] opcode : {add : 0} r s 1 : { x1 : 0 , x3 : 0} r s 2 : { x2 : 0 , x4 : 0} op_comb : { ' r s 1 == r s 2 != rd ' : 0} val_comb : ' r s 1 _ v a l > 0 and r s 2 _ v a l > 0 ' : 0 ' r s 1 _ v a l > 0 and r s 2 _ v a l < 0 ' : 0 abstract_comb : ' walking_ones ( ” r s 1 _ v a l ” , 6 4 ) ' : 0 ' w a l k i n g _ z e r o s ( ” r s 1 _ v a l ” , 6 4 ) ' : 0 csr_comb : { ' mtval == 0 xdeadbeef ' : 0} c r o s s _ c o v e r a g e : # <> i m p l i e s a l i s t , ? i m p l i e s don ' t c a r e # <i n s t −opcode> : : <var−a s s i g n > : : <val −r u l e s > ' [ ( add ) : ( sw ) ] : : [ a=rd : ? ] : : [ ? : r s 2==a o r r s 1==a ] ' : 0 . . . . . . . . . . . . . . © 2022 InCore Semiconductors Pvt. Ltd. CARRV @ ISCA, 2022 9/18

                                RISCV-ISAC     Implementation Details

Tool Flow

• Coverage is computed from the instruction trace(generated by the een RISCVHSAC TT P implementation/model) containing the pi | following information:• H race parser] Decoder coverage Instruction Address •• Instruction encoding E- Architectural state changes like csr and Noel regfile updates(if any) • Trace parser and Decoder are implemented as plugins to support customisation. • Architectural states are maintained internally. • Supports filtering instructions which influence coverage. • Constrain coverage collection to only specific covergroups for a run. • Custom boundaries for signature region can be specified. • Generates YAML and HTML reports for coverage statistics and data propagation.

                                                                                                    .   .  .  .  .  . .
                                .                                                                   .   .  .  .  .  .

RISCV-ISAC Implementation Details

Data Propagation Reporting

• It is essential for signature based tests(as employed by RISC-V AT) to propagate the results of relevant instructions to the signature region. • ISAC tracks the movement of data into the signature region. The following statistics are reported for each test: • number of instructions that hit unique coverpoints • number of coverpoints hit by multiple instructions • number of signature/memory region overwrites • number of updates to multiple memory regions with the result of a single instruction • number of coverpoint hits without update to signature/memory region

                                                            .   .  .  .  .  . .
    .                                                       .   .  .  .  .  .

RISCV-CTG

1 Architectural Testing

2 RISCV-ISAC

3 RISCV-CTG

4 Related Work

                                                             .   .  .  .  .  . .

          RISCV-CTG

Generating Solutions for Coverpoints

• Has an internal database with all relevant information for an [stro instruction(like fields and their Normatzed domains). Instruction • Uses the CGF format defined by ISAC Ex) (Solutions solutions Register J | to generate tests from coverpoints. • The coverpoints are modeled as a classical Constraint Satisfaction Problem. • A CSP solver2 is then employed to find solutions. The op_comb and val_comb coverpoints are solved independently. Supports two types of solvers • Min-Conflict • Backtracking • Support for skipping the solver when coverpoints are suffixed with #nosat. This is useful when the coverpoint defines all the fields of an instruction definitely. Ex - rs1_val == 2andrs2_val == 5 2Constraint Satisfaction Problem resolver for Python. url: https://github.com/python-constraint/python-constraint. . . . . . . . . . . . . . . © 2022 InCore Semiconductors Pvt. Ltd. CARRV @ ISCA, 2022 13/18

                                      RISCV-CTG

Test generation

• Solutions from op_comb and val_comb are interleaved and any additional fields(such as register for signature pointer) etc are filled in. • Filter the generated test cases using the coverpoints given to ensure that there are none which does not hit at least 1 unique coverpoint. • Replace relevant values in the assembly macro template for the instruction and generate assembly files.

  #d e f i n e RR_OP( op , rd , r s 1 , r s 2 , v1 , v2 , ptr , o f f s e t ) \
  l i r s 1 , v1 ; \
  l i r s 2 , v2 ; \
  op rd , r s 1 , r s 2 ; \
  sw rd , o f f s e t ( p t r ) ;
  // I n s t a n c e o f th e macro i n an add t e s t
  RR_OP( add , x2 , x1 , x2 , 0 x03 , 0 x2 , x3 , 0 )

                                                                               .   .  .  .  .  . .
                                          .                                    .   .  .  .  .  .

                                     RISCV-CTG

Experimental results

• cfg1 - Using Min-Conflict Solver on 8 threads • cfg2 - Using Backtracking Solver on 8 threads

  Suite  Generation                  Average        Test           Coverage
                                     time per test(s)  generated   Time (s)
  (ISA)  Time(s)                        Generation       Cases   Collection
      Cfg1 Cfg2                      Cfg1   Cfg2 Cfg1      Cfg2  Cfg1 Cfg2
 RV32I 293.56 87.87 39.32 10.36 12638 13150 189.07 227.43
 RV32M 136.24 35.536 93.96 24.99 5165 5386 59.21 65.90
  RV32C  80.13  23.80                21.14    4.36 4633    4822    49.07  99.43
 RV64I 864.29 215.88 66.60 19.67 17825 19113 348.82 292.70
 RV64M 575.89 128.55 240.10 63.25 9571 10393 121.23 112.92
 RV64C 319.96 84.24 71.27 10.58 7843 8409 116.42 136.44

                                                                             .   .  .  .  .  . .
                                                                          .  .   .  .  .  .  .

Related Work

1 Architectural Testing

2 RISCV-ISAC

3 RISCV-CTG

4 Related Work

                                                             .   .  .  .  .  . .

    Related Work

Related Work

• Herdt et. al3 use a similar approach(SMT solver) to generate tests. But the coverpoints are specified in a complex format and coverage specification is dependent on external tools. • A Mutation4 based approach relies on an external simulator to produce tests. The same results can be achieved with CTG in lesser time. • Negative testing5 is not feasible for AT due to the permissiveness of the ISA. Any unimplemented functionality can result in unpredictable behaviour of the implementation. Furthermore its difficult to recover from such scenarios or replicate it on the golden model. • No attempt has been made to track data propagation to the signature region in memory.

 3Herdt, Große, and Drechsler, “Towards Specification and Testing of RISC-V ISA Compliance*”.
 4Herdt et al., “Mutation-based Compliance Testing for RISC-V”.
 5Herdt et al., “Closing the RISC-V Compliance Gap”.                                         .   .  .  .  .  . .
     .                                                                                       .   .  .  .  .  .
©
 2022 InCore Semiconductors Pvt. Ltd.    CARRV @ ISCA, 2022                                         17/18

Future Work

• Adding support for new and upcoming extensions like bit manipulation, packed SIMD and floating point. • Testing of privilege architecture specification • Improve coverage collection time by considering coverpoints as hit/miss instead of counting number of hits • Support for testing different hazards.

 . . . . . . .