Skip to content
STIMSMITH

Ant Colony Optimization (ACO)

Technique WIKI v1 · 5/24/2026

**Ant Colony Optimization (ACO)** is referenced in the provided evidence as the optimization basis for **INSTILLER**, an RTL fuzzing system for CPU hardware bug detection.[^1] In this context, ACO is used through a variant called **VACO** to distill fuzzing inputs so that generated RTL instruction sequences remain shorter and more efficient.[^1]

Ant Colony Optimization (ACO)

Ant Colony Optimization (ACO) is referenced in the provided evidence as the optimization basis for INSTILLER, an RTL fuzzing system for CPU hardware bug detection.[1] In this context, ACO is used through a variant called VACO to distill fuzzing inputs so that generated RTL instruction sequences remain shorter and more efficient.[1]

Overview

In the cited work, ACO is applied to the problem of register-transfer level (RTL) fuzzing, where inputs are instruction sequences used to test CPU cores before deployment.[1] The paper argues that prior CPU fuzzing approaches can suffer from growing RTL input instruction lengths, and that longer inputs can become ineffective for fuzzing.[1]

INSTILLER addresses this by using a variant of Ant Colony Optimization, called VACO, to reduce or “distill” input instruction sequences while preserving fuzzing effectiveness.[1]

Use in RTL fuzzing

INSTILLER is described as an RTL fuzzer based on ACO.[1] Its ACO-related component is used specifically for input distillation:

  • It keeps instruction inputs short during fuzzing.[1]
  • It uses VACO to distill input instructions.[1]
  • It aims to improve fuzzing efficiency by avoiding unnecessarily long instruction sequences.[1]

The system also includes mechanisms beyond ACO, including support for generating realistic interruptions and exceptions, as well as hardware-based seed selection and mutation strategies.[1]

VACO: ACO variant for instruction distillation

The cited paper introduces VACO, a variant of ACO, as part of INSTILLER.[1] VACO is used to distill RTL input instructions, reducing their length while maintaining or improving fuzzing effectiveness.[1]

Reported experimental results attribute substantial input reduction to VACO: INSTILLER generated 79.3% shorter input instructions than DiFuzzRTL.[1] The paper further reports that this distillation increased execution speed by 6.7% on average.[1]

Reported performance

INSTILLER was evaluated against state-of-the-art fuzzing work on real-world target CPU cores.[1] Compared with DiFuzzRTL, the cited results report:

Metric Reported result
Coverage improvement 29.4% more coverage
Mismatch detection 17.0% more mismatches detected
Input length reduction 79.3% shorter input instructions
Execution speed 6.7% average increase

These results are presented by the paper as evidence that the ACO-derived VACO component is effective for instruction distillation in RTL fuzzing.[1]

Technical significance

Within the provided evidence, ACO is significant as an optimization strategy for improving hardware fuzzing workflows.[1] The key technical contribution associated with ACO is the use of VACO to reduce instruction-sequence length, which can improve fuzzing efficiency and execution speed.[1]

References

[1]: “INSTILLER: Towards Efficient and Realistic RTL Fuzzing,” arXiv:2401.15967, DOI: 10.48550/arXiv.2401.15967.