Automated Coverage Instrumentation

Overview

Automated coverage instrumentation is the step, introduced in ZP Cosim, that automatically identifies and tags the design elements whose states need to be monitored during simulation or emulation in order to infer the coverage effected by test programs. It is presented as Section 3.1, "Automated Coverage Instrumentation," of the ZP Cosim chapter, with subsections covering Implementation (3.1.1), Surelog (3.1.2), Universal Hardware Data Model (3.1.3), Coverage Walker (3.1.4), Implementation of Mux Toggle Coverage Metric (3.1.5), and subsequent coverspace-processing subsections (3.1.6–3.1.11).

Coverage is described as an important aspect of hardware verification, and ZP Cosim introduces a new automated coverage collection implementation built on top of existing open-source offerings. The fundamental requirement of coverage instrumentation is identifying key design elements whose states need to be monitored during simulation or emulation for inferring the coverage effected by test programs.

Why Automate Instrumentation

Depending on the coverage metric in question, the design elements that must be monitored can be:

• Structural elements in the code, such as expressions in assignment statements, case statements, assertions, and similar constructs.
• Functional elements in the design, such as complex composite expressions corresponding to specific functionalities of the hardware.

Automating the identification and tagging of these elements removes the manual burden of marking coverage points in a design and provides a uniform way to collect coverage data from the Processor-Under-Test during both simulation and FPGA emulation.

Coverage Metrics Implemented

ZP Cosim demonstrates and implements automated Mux Toggle Coverage, popularized by Kevin Laeufer et al. [81], together with a proof-of-concept case-statement coverage.

Mux Toggle Coverage

Mux Toggle Coverage is similar in spirit to branch coverage in software testing. The idea is that muxes are the sources of multiplicity in hardware states — i.e., being the fundamental functional switches (or decision statements), the cumulative combination of toggles in mux-selects across all the muxes amounts to all the different possible control paths in the hardware. The cumulative combinations are usually identified by the cross-product of the toggles on all the (interacting) muxes, and this cross-product is a truer indication of actual coverage compared to individual toggles. Later works [63] explore other coverage metrics for efficiency, applicability, and performance reasons.

Case-Statement Coverage

Case-statement coverage is a structural / code coverage metric that assesses the completeness of a test vector in exercising all the case-statement control paths. It is not a standalone metric; it simply provides another perspective on one frontier of verification progress.

Implementation

In order to identify muxes to monitor for coverage collection, ZP Cosim packages a parameterized, modifiable walker that parses the hardware design description and outputs hierarchically referenced full-pathnames of mux select-signals in the design. There are three parts to the coverage metric implementation:

1. Surelog [37] — an open-source SystemVerilog parser and compiler that takes in a SystemVerilog hardware description of the design and generates a walkable parse tree-like object.
2. Universal Hardware Data Model (UHDM) [38] — the data object produced by Surelog that can be thought of as an elaborated intermediate representation of the hardware design description.
3. A customizable walker that can operate on the UHDM object to produce the hierarchical instance names of the muxes, or coverpoints, for monitoring coverage.

A coverpoint is a boolean or multi-bit expression (wire or register) in the Hardware Description Language that is assessed, usually for toggles, for determining the coverage of a test case on the hardware in question.

The coverpoints generated thereof need to be assembled into the top-level wrapper of the hardware design. This can be done in two ways:

• For simulations: SystemVerilog assertions that can be assessed through debug prints and debugged via waveform dumps.
• For emulations and post-processing capabilities: Wired into dedicated coverage collection modules with memory modules for storage of coverage information during emulation, and read interfaces for retrieval from the test environment post emulation.

ZP Cosim provides GNU Make routines to automatically package the identified hierarchically referenced coverpoints by grouping and wiring them into coverage modules in the top-level wrapper of the design under consideration.

Surelog

Surelog [37] is a SystemVerilog pre-processor, parser, elaborator, and UHDM compiler. It adheres to the SystemVerilog 2017 standard [13] and also provides IEEE Design/TB C/C++ Verilog Procedural Interface (VPI) [40] and Python AST Application Programming Interfaces (APIs). In ZP Cosim, Surelog functions as a parser, elaborator, and compiler that takes as input, a file list of the distributed hardware design description in SystemVerilog, among other crucial parameters, and generates a walkable UHDM object. Important parameters include the specification of the top-module, SystemVerilog library extensions, compilation ordering, parameter value overrides, and a switch to request full or folded UHDM elaboration.

Role Within the ZP Cosim Workflow

Automated coverage instrumentation sits in the FPGA-host/host-control loop of ZP Cosim (Figure 3.1, "Reference Block Diagram for self-contained, iterative verification"):

• The Programmable Logic (PL) is the FPGA on-board, which has a synthesized FPGA-shell enveloping the Processor-Under-Test (PUT).
• The Processing Subsystem (PS), a host processor for the FPGA in close proximity, receives the bitstream, flashes it on the FPGA, and launches the control program which:
  • Orchestrates loading of test programs on both the PUT on the Programmable Logic and an instance of the Dromajo golden-reference ISA simulator, iteratively stepping through the PUT's execution trace, cosimulating it with Dromajo, and analysis, and
  • Performs coverage extraction from the PUT upon program termination, processing of the coverage, derivation of mutations (which has not been implemented on ZP Cosim), and program-generation or benchmark step-through.

In case of a mismatch, the PS saves the failing test program for later offline debug through software RTL simulations for greater visibility into the microarchitecture.

Within this loop, automated coverage instrumentation produces the coverpoints that feed into the post-processing and coverspace-reduction stages, which reduce the coverpoint state-space before downstream mutation, encountered-coverage analysis, and benchmark step-through.

Acknowledgments

The ZP Cosim work, including its automated coverage instrumentation, was developed by Anoop Mysore Nataraja as part of a University of Washington dissertation (©Copyright 2023 Anoop Mysore Nataraja), and builds on the existing Zynq-Parrot infrastructure [9] that enables rapid FPGA prototyping for the BlackParrot processor on Zynq 7000 and Ultrascale FPGA devices.