Hardware-Guided Reinforcement Learning

Overview

Hardware-Guided Reinforcement Learning (HGRL) is a reinforcement-learning technique that drives the generation of hardware test cases by treating the device under test (DUT) itself as the reward signal. In the GenHuzz white-box hardware fuzzing framework, HGRL dynamically optimizes the token-by-token policy of a language-model-based fuzzer that emits RISC-V assembly instructions, with the objective of maximizing hardware coverage and exposing deeper hardware bugs and vulnerabilities. The technique is designed to learn and exploit complex inter-instruction semantics that static training corpora cannot capture.

Context: Role Within GenHuzz

GenHuzz reframes hardware fuzzing as an optimization problem. Instead of mutating fixed seed corpora, a language model generates RISC-V assembly instructions whose generation policy is optimized online. Hardware-Guided Reinforcement Learning is the component that performs this online optimization: it closes the loop between the fuzzer (the policy / agent) and the hardware (the environment), feeding real-time execution feedback back into the model's policy updates.

Inputs and Data Representation

• Test-case initialization. The fuzzer begins with a set of N assembly instructions I = I_1, I_2, ..., I_N. Assembly language is chosen to strengthen semantic relationships between instructions so that meaningful combinations can reveal subtle hardware vulnerabilities.
• Sequence construction. Instructions are concatenated into a single sequence D using a separator token SEP: D = I_1 SEP I_2 SEP I_3 SEP ... I_N. The SEP operator flags each instruction's beginning and end, helping the fuzzer understand intra-instruction semantics.
• Tokenization. Each instruction I_i is tokenized into T_i = (t_{i,1}, t_{i,2}, ..., t_{i,k_i}), where k_i is the number of tokens in instruction I_i. The combined tokenized dataset is T = T_1 T_SEP T_2 T_SEP ... T_SEP T_N.

Action Space

The fuzzer acts by predicting the next token based on previously generated tokens. In the reinforcement-learning formulation, each predicted token is an action. Because actions are elements of an instruction (e.g., operands) rather than complete instructions, reward assignment is non-trivial.

State Transition and Reward Function

Section 3.2 of the GenHuzz paper formalizes the state-transition and reward-assignment problem. Assigning a reward to each token-level action is more challenging than in common reinforcement-learning tasks because:

1. Delayed rewards. Only a complete instruction can be meaningfully rewarded; the reward for any individual token action is therefore delayed until the full instruction is formed.
2. Unexecuted instructions. Not all instructions will be executed. For example, a branch instruction can skip multiple subsequent instructions. These skipped instructions do not directly contribute to hardware coverage, but they are still essential context for generating the instructions that follow.
3. Exceptions on the DUT. Even syntactically correct instructions can raise exceptions when executed on the DUT. Zero or negative rewards for these instructions are inappropriate because they still contribute to the fuzzing process.

To assign proper rewards to each action, GenHuzz first classifies each instruction's status on the DUT. Possible instruction statuses include:

• valid & executed — the instruction was well-formed and executed by the DUT.
• valid & failed — the instruction was well-formed but raised an exception during execution.
• valid & unexecuted — the instruction was well-formed but skipped (e.g., due to a branch).
• invalid & unexecuted — the instruction was malformed and skipped.

This classification forms the basis for the per-action reward used by Hardware-Guided Reinforcement Learning.

Algorithm: State Transition and Identification

The HGRL loop is summarized by Algorithm 1 ("State Transition and Identification"), with inputs env (the hardware DUT environment), fuzzer (the language-model policy), and GRM (the Golden Reference Model):

Require: env, fuzzer, GRM
 1: obs ← reset(env)
 2: while not done do
 3:   action ← fuzzer(obs)
 4:   next_obs, done ← step(env, action)

At each step the fuzzer selects a token action conditioned on the current observation; the environment advances by executing the corresponding instruction on the DUT, and the new observation is fed back into the policy.

What Hardware-Guided Reinforcement Learning Optimizes For

By incorporating real-time feedback from the hardware, HGRL steers the language-model fuzzer's policy toward test cases that:

• Achieve significantly higher hardware coverage with fewer test cases than state-of-the-art fuzzers (evaluated on three RISC-V cores).
• Expose all known bugs reported in prior studies with fewer test cases.
• Uncover new vulnerabilities: GenHuzz uncovered 10 new vulnerabilities, 5 of which are the most severe hardware vulnerabilities ever detected by a hardware fuzzer targeting the same cores, with CVSS v3 severity scores exceeding 7.3 out of 10.

Relationship to Hardware Coverage

Hardware-Guided Reinforcement Learning uses hardware coverage as its primary optimization signal. Because the reward is derived from the execution status of generated instructions on the DUT, coverage of hardware structures such as registers, functional units, and state machines directly drives the policy gradient. Higher coverage corresponds to deeper exploration of the DUT and a higher likelihood of triggering subtle hardware bugs.

Limitations

Because the RISC-V Golden Reference Model used by GenHuzz does not support speculative or out-of-order processing, GenHuzz (and therefore Hardware-Guided Reinforcement Learning as instantiated here) can only detect static bugs, consistent with prior hardware-fuzzer literature.

Citation

Wu, L., Rostami, M., Li, H., Rajendran, J., & Sadeghi, A.-R. (2025). GenHuzz: An Efficient Generative Hardware Fuzzer. Proceedings of the 34th USENIX Security Symposium, pp. 1787–1805. USENIX Association.