Definition

Black-box fuzzing is a dynamic software testing technique in which a Program Under Test (PUT) is executed recurrently with generated inputs (seeds) in order to trigger software bugs, without taking into account the internal behavior of the program when constructing those inputs. It is described in the fuzzing literature as "the simplest variant of fuzzing." A complementary definition states that, in black-box fuzzing, the fuzzer regards the software under test as a "black box" and is unaware of its internal workings; the fuzzer therefore cannot access source code or binary internals to direct input generation. ^[1]

Fuzzing Toolchain and Lifecycle

Fuzzing broadly encompasses three major steps — input generation, input modification (mutation), and program execution — and a fuzzer automates these steps on a PUT under a defined security policy. A fuzz campaign is a specific execution of a fuzzer on a PUT with a specific security policy, and fuzz configurations control the parameters of the fuzz algorithm, which vary by fuzzer type. During input generation, the fuzzer produces inputs at random or based on a predetermined set of test cases; during input mutation, the fuzzer alters the generated inputs by randomly changing values, adding errors, or changing structure; during program execution, the updated inputs are fed to the PUT and its behavior is scrutinized. A bug is flagged when the application fails or behaves strangely (e.g., crashes, hangs, or other policy violations). ^[1]

Three common input-generation families cited in the fuzzing literature include:

• Random fuzzing — the fuzzer produces entirely random inputs without any prior knowledge of the program's input domain.
• Mutation-based fuzzing — the fuzzer starts with an initial input corpus and changes it by adding, removing, or altering values of each seed.
• Feedback-based fuzzing — the fuzzer uses heuristics such as coverage or distance-based metrics to direct the input generation process in response to the program's output. Black-box fuzzing falls outside the feedback-based family because no internal-observability signal is available. ^[1]

Fuzzing has proven effective for finding vulnerabilities (e.g., buffer overflows, format-string flaws, race conditions) that are difficult to detect by other testing methods, across software systems including network protocols, operating systems, and web applications. ^[1]

Characteristics

The defining properties of black-box fuzzing are:

• Input generation independent of internal program behavior. A black-box fuzzer selects and mutates inputs without consulting runtime data flow, control flow, or memory access information from the PUT.
• High compatibility with complex software projects. Because it requires no instrumentation or introspection of the target, black-box fuzzing can be applied to systems where source code, debug symbols, or coverage hooks are unavailable.
• Ease of use. Black-box fuzzers are generally straightforward to deploy, requiring little setup beyond providing a target binary and an initial seed corpus.
• Limited bug-finding capability. The lack of internal program information restricts the guidance available during input generation, which limits effectiveness compared to techniques that exploit runtime feedback.

Black-Box Mode in Established Greybox Fuzzers

Coverage-guided fuzzers such as American fuzzy lop (AFL) explicitly include a blackbox fallback for non-instrumented targets: "To fuzz non-instrumented targets, AFL will fall back onto a blackbox mode, and rely on crash and hang detection (timeout) for feedback." AFL additionally provides a QEMU mode to fuzz non-native ISA targets, with an approximate runtime overhead between a factor of 2x and 5x. AFL++ is a community fork of AFL with a newer codebase and features, and exposes a custom mutator API to enable researchers to implement plugins that combine new ideas with existing fuzzing technologies. LibFuzzer, related to the LLVM project, is integrated into the target binary: the user provides an entry point to the target, from which LibFuzzer spawns parallel threads to run with varying inputs; as limitations, the target may not modify global state or provide its own reset. ^[2]

A practical caveat noted in the fuzzing literature is that some greybox/whitebox techniques used by AFL++ and LibFuzzer employ compiler transformations (e.g., inserting special tracing instructions or splitting complex CF edges across multiple basic blocks to expose more partial conditions) that alter the application's runtime behavior and may therefore be inappropriate for real-time IoT targets. ^[2]

Relationship to Other Fuzzing Paradigms

To address the limitations of black-box fuzzing, researchers have proposed Greybox Fuzzing (GF), which collects internal program behavior — such as data flow, control flow, and memory accesses — at runtime to guide input generation. Widely used GF tools such as AFL and LibFuzzer utilize code coverage (e.g., basic blocks, edges, or lines executed) as feedback. If a generated input increases coverage (an "interesting" input), the fuzzer mutates it to derive new candidates, with the intuition that coverage-increasing inputs are more likely to explore new code and expose bugs. ^[2]

Coverage approaches used by such fuzzers include:

• Basic-block coverage — tracking which basic block (BB) has been executed, optionally with per-BB execution counts visualized as a histogram.
• Control-flow edge coverage — tracking the actually taken CF edges.
• Hash digests identifying entire CF paths — guiding towards high path coverage, which can be reached by mutating just one loop limit but may miss other relevant CF edges. ^[2]

Two prominent variants of Greybox Fuzzing are:

• Coverage-based Greybox Fuzzing (CGF): maximizes code coverage of the program to discover bugs anywhere in the program, particularly effective on substantial software changes with long fuzzing runs.
• Directed Greybox Fuzzing (DGF): steers test generation toward specific target sites (e.g., code lines modified by a recent commit) rather than unrelated program components, and is well suited to small-size commits under limited fuzzing budgets.

Comparison with Coverage-Guided Hardware Fuzzing

Black-box fuzzing is conceptually contrasted with coverage-guided hardware fuzzing, which instruments a hardware design to obtain structural coverage signals (such as basic blocks, edges, functional cover-points, or covergroups) and uses these signals to guide mutation of test programs or instruction sequences. Coverage-guided hardware fuzzers rely on instrumentation or on-chip coverage-collection engines (e.g., Coverage Collection Engines, or CCEs) integrated into the SoC to track which parts of the design have been exercised, and they steer test generation toward increasing that coverage. By contrast, black-box fuzzing has no such feedback channel: it cannot instrument the target and must therefore rely on externally observable signals.

Empirical Comparison with Structural Coverage-Based Fuzzing

Structural coverage-based fuzzing instruments the target to track code-coverage signals and steers mutations toward inputs that exercise new edges/basic blocks. Bölcskei et al. (cited in the RISCover work) report that structural-coverage fuzzers can be slower than black-box fuzzers in bug discovery on certain hardware targets. This empirical finding motivates continued exploration of black-box post-silicon fuzzers, especially in architectural vulnerability discovery on RISC-V, where few black-box approaches exist and most prior work is coverage-based. ^[3]

Applications and Use Cases

Database Management Systems

Black-box fuzzing has been applied in industry for decades to improve the correctness of Database Management Systems (DBMSs). Industrial tools such as SQLsmith and SQLancer are representative examples of DBMS-focused black-box fuzzers. In practice, coverage-guided fuzzers have demonstrated substantially higher basic-block coverage than these industrial black-box fuzzers — for example, the coverage-guided fuzzer Ratel covered 38.38% more basic blocks than the best of SQLsmith/SQLancer on GaussDB, 106.14% more on PostgreSQL, and 583.05% more on Comdb2.

Industrial Control Systems and Network-Only Targets

For proprietary industrial devices such as Programmable Logic Controllers (PLCs), network-based black-box fuzzing is often the only available way to test the system, because the device firmware, control applications, and runtime environments are closed source and lack execution-tracing support. The FieldFuzz framework targets the Codesys runtime (used by over 400 devices from 80 industrial PLC vendors) by combining (i) reverse-engineering-enabled remote control of control applications and runtime components, (ii) automated command discovery and status code extraction via network traffic, and (iii) on-system tracing and coverage computation to compensate for the inefficiency of plain network-based fuzzing. FieldFuzz's fuzzing campaigns uncovered multiple vulnerabilities, leading to three reported CVE IDs, and reproduced findings across a diverse set of ICS devices.

IoT Firmware Fuzzing

For Internet of Things (IoT) devices, where obtaining and emulating firmware is difficult, black-box fuzzing of the device has become a viable option. A key challenge is that existing black-box IoT fuzzers cannot form effective mutation optimization mechanisms because of the lack of feedback, and it is difficult or impossible to apply existing grammar-based fuzzing strategies, motivating approaches with syntax inference. [https://arxiv.org/abs/2105.05445v2]

The Snipuzz approach addresses this by running as a client that communicates with the device and infers message snippets from response behavior to drive mutation, narrowing the search space without requiring reverse engineering of companion apps. Snipuzz inherits the advantages of app-based fuzzing (e.g., IoTFuzzer) while utilizing communication responses to perform efficient mutation, and is lightweight in that it does not rely on prerequisite operations such as reverse engineering of apps. Snipuzz was compared against IoTFuzzer, BooFuzz, Doona, and Nemesys, and identified 5 zero-day vulnerabilities across 20 real-world IoT devices, 3 of which were exposed only by Snipuzz (all confirmed by the respective vendors). [https://arxiv.org/abs/2105.05445v2]

Hardware/software co-designed approaches have also been proposed to bring coverage guidance to IoT firmware. TaPaFuzz is an FPGA-accelerated framework for RISC-V IoT fuzzing in which a hardware component executes the IoT firmware program and captures the execution's edge coverage map (returning it, together with the target's return value, back to the host software), while the host iteratively creates inputs, launches runs on the FPGA accelerator, and evaluates execution to drive the next iteration. Seamless hardware/software integration is enabled by the TaPaSCo FPGA abstraction framework, which allows composing SoCs consisting of heterogeneous processing elements. ^[2]

REST API Fuzzing

Black-box fuzzing has been adapted for REST API testing, where conventional black-box fuzzers generate random data without judging input quality. A coverage-level-guided approach uses a Test Coverage Level derived from HTTP response observations (e.g., status codes) as a lightweight proxy for feedback, addressing the blindness of random mutations while remaining a black-box technique. Applied to two large open-source projects, this approach reported and confirmed 89 bugs, and found 351 bugs across 64 remote API services indexed by APIs.guru. [https://arxiv.org/abs/2112.15485v1]

Hardware and CPU Fuzzing (RISC-V)

Black-box fuzzing has also been applied to closed-source hardware targets. RISC-V is an emerging open-source processor architecture (Reduced Instruction Set Computing Five) that is royalty-free, modular, and customizable, making it a popular choice for both industry and research. The RISCover framework is a differential black-box CPU fuzzer that operates from user space to discover architectural security vulnerabilities in closed-source RISC-V CPUs across vendors. Because RTL-level (coverage-guided) fuzzers typically run on minimal SoCs without DRAM, miss system-level interactions, and have long analysis times, and because Bölcskei et al. show that structural-coverage fuzzers can be slower than black-box fuzzers in bug discovery, RISCover instead uses a server–client architecture in which a powerful x86 server orchestrates test-case generation and multiple resource-constrained RISC-V clients execute instruction sequences and report results. RISCover was shown to rediscover 22 of 23 bugs that the state-of-the-art RTL fuzzer Cascade found in soft-cores, and additionally discovered many other architectural bugs — including undocumented instructions, address-handling bugs, decoder bugs, ISA incompatibilities, fault-reporting issues, and QEMU segmentation faults — often within seconds of fuzzing. ^{[1] [3]}

In RISCover, black-box fuzzing of a CPU entails three central challenges:

• C1: Sequence Generation. Without coverage feedback, the fuzzer cannot use instrumentation-based metrics to assess the quality of instruction sequences. RISCover therefore uses number-of-bugs and time-to-bug as evaluation metrics, and employs a bottom-up approach that gradually increases the search space using instruction types inferred from encodings, combined with a weighted random selection of instructions based on the instruction frequency of real-world code.
• C2: Non-deterministic Effects. Because the black-box fuzzer cannot modify, reset, or fully control the CPU's microarchitectural state, architectural effects can depend on factors outside the fuzzer's control (e.g., performance-counter values, prior instruction history, the running OS's memory mappings, or values read from addresses such as the Linux vDSO).
• C3 (orchestration). A centralized design — one powerful server orchestrating multiple resource-constrained RISC-V clients that report register/memory state back — shifts as much work as possible off the under-powered target and onto a powerful x86 server that can keep up with the search space. ^[3]

Differential Testing

Black-box fuzzing has also been used to drive differential testing of emulators versus real hardware, where the goal is to identify discrepancies between an emulation environment and the physical device it mimics. Differential fuzzing compares multiple implementations of the same specification, flagging divergences as potential bugs without requiring any golden model. Prior differential-testing works in this area typically employ black-box input generation, in contrast to coverage-guided approaches that use runtime feedback. Differential testing more broadly has been applied to Web application firewalls, SSL/TLS libraries, compilers, cryptocurrency protocols, deep learning systems, Java Virtual Machines, PDF viewers, mobile applications, file systems, and Java programs. ^[3]

Challenges and Mitigations

The central challenge of black-box fuzzing is the absence of feedback about internal program state:

• In IoT settings, the lack of feedback prevents effective mutation optimization, and grammar-based strategies are often inapplicable, motivating syntax-inference approaches such as Snipuzz; hardware-accelerated FPGA frameworks such as TaPaFuzz reintroduce coverage guidance by capturing edge coverage in dedicated hardware attached to the firmware execution unit. [https://arxiv.org/abs/2105.05445v2] ^[2]
• In REST API settings, blind mutations produce many low-quality inputs, motivating proxy signals such as Test Coverage Level derived from response codes. [https://arxiv.org/abs/2112.15485v1]
• In hardware/closed-source CPU settings, no instrumentation is possible, and feedback must come from observable architectural effects (e.g., register/memory state) reported back to a coordinating server. RISCover formalizes this as the non-deterministic effects problem, where CPU state may depend on factors (performance counters, prior history, OS memory layout, Linux vDSO reads) that the fuzzer cannot control. ^[3]
• In industrial control settings, the proprietary runtime and lack of execution tracing make pure network-based fuzzing inefficient; FieldFuzz augments network fuzzing with reverse-engineering-enabled remote control, automated command discovery, and on-system tracing to recover the feedback that black-box fuzzing lacks.
• In greybox fallback modes (e.g., AFL's blackbox mode), the only signal available is crash/hang (timeout) detection, which is sparse and easy to miss; this is precisely the gap that motivates dedicated blackbox-aware frameworks such as RISCover, FieldFuzz, and Snipuzz. ^[2] [https://arxiv.org/abs/2105.05445v2] [https://arxiv.org/abs/2112.15485v1] ^[3]

Trade-offs and Practical Considerations

The choice between black-box and greybox (or structural coverage-based) fuzzing is governed by a trade-off between deployability and effectiveness. Black-box fuzzing remains attractive for proprietary targets (e.g., closed-source ICS devices, enterprise DBMSs, closed-source RISC-V CPUs, IoT devices) where instrumentation is infeasible, and for large enterprise systems whose distributed nature complicates coverage collection. However, coverage-guided techniques typically achieve higher coverage and bug-finding rates on instrumentable targets (e.g., Ratel vs. SQLsmith/SQLancer on GaussDB, PostgreSQL, Comdb2). Notably, on certain hardware targets, structural-coverage fuzzers have been measured to be slower than black-box fuzzers in bug discovery (Bölcskei et al., as cited in RISCover), complicating the universal preference for coverage guidance. Where real-time constraints apply, compiler transformations used by greybox fuzzers (AFL++, LibFuzzer) may themselves be inappropriate, pushing designers back toward black-box or hardware-assisted approaches. ^{[2] [3]}

Comparison Table: Black-Box vs. Coverage-Guided Hardware Fuzzing