Skip to content
STIMSMITH

Adaptive Coverage Reward Mechanism

Concept WIKI v1 · 7/10/2026

Adaptive Coverage Reward Mechanism (ACRM) is the module-aware extrinsic reward component of HiFuzz, a hierarchical reinforcement-learning framework for CPU fuzzing. It treats module focus as a multi-armed bandit problem and uses a UCB-style dynamic weighting scheme to shift reward toward under-covered hardware modules.

Overview

The Adaptive Coverage Reward Mechanism (ACRM) is a reward-shaping mechanism used in HiFuzz for hardware fuzzing. HiFuzz combines a macro-level Program Agent and a micro-level Basic Block Agent; ACRM provides balanced module-level coverage feedback as part of that framework. Its purpose is to reduce the weakness of optimizing a single global coverage score, where easy-to-cover modules can dominate the reward and hide under-covered components.

Mechanism

ACRM tracks coverage at the module level rather than only as an aggregate global score. At each training step, it maintains per-module verification status and computes a dynamic module weight. The paper formulates this module selection problem as a Multi-Armed Bandit problem and uses a UCB-based weighting approach.

For module i at step t, ACRM computes an unnormalized weight:

w_tilde_i(t) = mu_i(t) + beta(t) * v_i(t)

where:

  • mu_i(t) is the historical mean coverage gain for the module, representing exploitation.
  • v_i(t) = 1 - current_cov_i(t) / max_cov_i is the module saturation term, representing how under-covered the module remains.
  • beta(t) grows as training progresses, shifting attention from historically high-yield modules toward under-verified modules.

The weights are normalized with a softmax temperature tau:

w_i(t) = exp(w_tilde_i(t) / tau) / sum_j exp(w_tilde_j(t) / tau)

The resulting extrinsic reward passed to the Program Agent is the weighted sum of module coverage gains:

R_ext(t) = sum_i w_i(t) * DeltaCov_i(t)

This design makes the reward adaptive: modules that are saturated receive lower emphasis, while modules with remaining uncovered space receive higher emphasis.

Role in HiFuzz

In HiFuzz, ACRM is one of two reward-related mechanisms used to address reward sparsity and coverage imbalance. The Semantic-Aware Basic Block Encoder supplies dense intrinsic feedback for instruction-level decisions, while ACRM redirects extrinsic reward toward under-covered modules. The final HiFuzz system uses these mechanisms together within the hierarchical generator.

The HiFuzz implementation passes ACRM-derived extrinsic reward into the Program Agent. The Program Agent is trained with Rainbow DQN, while the Basic Block Agent is trained with PPO; the implementation is built in Python on Cocotb, Verilator, and Spike and evaluated on Rocket, BOOM, and CVA6 RISC-V cores.

Reported effects

The paper reports that the largest gains from ACRM-style module weighting appear in hard-to-reach modules such as the FPU. As easier modules saturate, the UCB term lowers their reward weight and shifts attention toward under-covered modules, reducing the masking effect caused by direct optimization of aggregate coverage.

In the ablation study on Rocket, the authors compare against a DQN+PPO baseline that preserves the Program Agent / Basic Block Agent split but removes both semantic intrinsic feedback and adaptive module weighting. Adding ACRM contributes a reported +44.15% improvement over that baseline, and the full HiFuzz system achieves +51.87% over the same baseline. The authors interpret this as evidence that ACRM and the Basic Block Encoder address different parts of the reward problem.

Limitations and interpretation

The ablation isolates ACRM within the same hierarchical generator, not against a separate single-agent instruction-level RL system. The authors note that a true single-agent instruction-level baseline would require a different action interface and validity mechanism, making it less controlled as a component ablation. They also note that reported 24-hour multi-fuzzer coverage curves were not averaged over many random seeds, and that broader multi-seed reporting with means and standard deviations would be useful for quantifying run-to-run variance.

LINKED ENTITIES

4 links
Multi-Armed Bandit USES
HiFuzz PART_OF
Module-Level Coverage Weighting PART_OF
Upper Confidence Bound USES

CITATIONS

9 sources
9 citations
[1] ACRM is used in HiFuzz to provide balanced module-level coverage feedback. HiFuzz: Hierarchical Reinforcement Learning for Semantic-Aware and Adaptive CPU Fuzzing
[2] ACRM formulates module selection as a Multi-Armed Bandit problem and uses a UCB-based dynamic weighting approach. HiFuzz: Hierarchical Reinforcement Learning for Semantic-Aware and Adaptive CPU Fuzzing
[3] ACRM computes module weights from historical mean coverage gain, module saturation, a dynamic beta parameter, and softmax temperature. HiFuzz: Hierarchical Reinforcement Learning for Semantic-Aware and Adaptive CPU Fuzzing
[4] The ACRM extrinsic reward is the weighted sum of coverage gains across modules and is passed to the Program Agent. HiFuzz: Hierarchical Reinforcement Learning for Semantic-Aware and Adaptive CPU Fuzzing
[5] The UCB term shifts attention from saturated modules toward under-covered modules and reduces masking from aggregate coverage. HiFuzz: Hierarchical Reinforcement Learning for Semantic-Aware and Adaptive CPU Fuzzing
[6] In the Rocket ablation, ACRM contributes +44.15% over the DQN+PPO baseline, while the full HiFuzz system reaches +51.87%. HiFuzz: Hierarchical Reinforcement Learning for Semantic-Aware and Adaptive CPU Fuzzing
[7] The DQN+PPO ablation baseline preserves the hierarchical Program Agent / Basic Block Agent split while removing semantic intrinsic feedback and adaptive module weighting. HiFuzz: Hierarchical Reinforcement Learning for Semantic-Aware and Adaptive CPU Fuzzing
[8] HiFuzz is implemented in Python on Cocotb, Verilator, and Spike, with a Rainbow DQN Program Agent and PPO Basic Block Agent, and is evaluated on Rocket, BOOM, and CVA6. HiFuzz: Hierarchical Reinforcement Learning for Semantic-Aware and Adaptive CPU Fuzzing
[9] The paper notes that its 24-hour multi-fuzzer coverage curves were not averaged over many random seeds and identifies broader multi-seed reporting as useful future work. HiFuzz: Hierarchical Reinforcement Learning for Semantic-Aware and Adaptive CPU Fuzzing