Reinforcement Learning

Reinforcement learning (RL) is a machine learning technique that, in the design verification (DV) context, is used to steer constrained-random stimulus generation toward hard-to-hit coverage points in complex integrated circuit designs. It is deployed alongside supervised learning to enhance existing constrained-random DV environments rather than replace them.

Motivation

As integrated circuits have grown more complex, purely random stimulus struggles to exercise all possible combinations in a timely fashion, while constrained-random approaches depend on extensive human expertise to guide the environment. The guidance burden becomes progressively more challenging and time consuming, and verification time to hit all possible design coverage points often becomes the dominant schedule limitation.

Approach in Design Verification

The technique leverages existing constrained-random DV environment tools and further enhances them using supervised learning and reinforcement learning. This provides better-than-random results in a highly automated fashion, helping achieve full design coverage objectives on an accelerated timescale and with fewer human resources.

Demonstrated Examples

Two hardware verification examples illustrate the technique:

• A Cache Controller design.
• The open-source RISC-V Ariane design paired with Google's RISC-V Random Instruction Generator.

In both cases the machine-learning-based approach performed significantly better on functional coverage and reaching complex hard-to-hit states than a purely random or constrained-random approach.

Relation to Other Techniques

In this application area, reinforcement learning is used in combination with supervised learning to provide automated guidance for constrained-random stimulus, reducing reliance on manual tuning of constraints by human verification engineers.