Skip to content
STIMSMITH

Randomized Testing

Technique WIKI v1 · 6/9/2026

Randomized Testing (RT) is a software and hardware verification technique in which a system under test (SUT) is exercised with test cases drawn from random distributions over the input domain rather than from manually written or model-derived test suites. The technique is well established, has been applied across embedded systems, databases, mobile applications, cloud APIs, and processor designs, and has been refined by variants such as Adaptive Random Testing (ART).

Randomized Testing

Randomized Testing (RT) is a testing method in which a system under test is driven by test cases sampled from random distributions over its input domain, instead of by hand-written or specification-derived tests. Because generation is decoupled from any specific failure hypothesis, randomized testing can explore very large input spaces cheaply and is frequently used as a complement to directed verification.

Core characteristics

  • Random input generation. Test inputs are produced by drawing values from a random distribution, often constrained by the syntactic or semantic rules of the SUT's input format.
  • Loose coupling to oracles. Correctness is typically judged against a reference model, a formal specification, or self-consistency checks rather than against pre-computed expected outputs.
  • Broad exploration. Randomness enables wide coverage of the input domain, including corner cases that directed test suites may omit.
  • Composability. Randomized input generation is commonly combined with checkers that compare SUT behavior against a reference implementation, an assertion suite, or architectural invariants.

Application domains

Randomized testing has been applied across multiple areas of software and hardware verification:

  • Embedded software systems — exercising firmware and device drivers under constrained environments.
  • SQL database systems — stress-testing query engines and transaction processing.
  • Android applications — discovering crashes and resource-handling bugs in mobile apps.
  • Cloud Application Programming Interfaces (APIs) — random test generators such as Autotest read an API specification, deduce a behavioral model from it, and use that model to drive random test generation. Pitfalls in the specification that random generation reveals are then typically converted into directed regression tests.
  • CPU and processor verification — randomized testing has been adapted to processor designs. The paper Randomized Testing of RISC-V CPUs Using Direct Instruction Injection (Joannou et al., IEEE Design & Test, 2024) applies randomized testing to RISC-V CPUs by injecting randomly generated instructions directly into a processor, providing a concrete example of the technique applied to hardware verification.

Adaptive Random Testing

Adaptive Random Testing (ART) is a well-studied refinement of RT introduced in 2001. ART aims to enhance RT's failure-detection ability by spreading candidate test cases more evenly over the input domain, so that newly generated inputs are intentionally dissimilar from previously executed ones. Since its introduction, ART has accumulated a body of work covering new algorithms, implementations, assessment and evaluation methods, and application areas, which has been surveyed in the literature (e.g., the 2020 survey A Survey on Adaptive Random Testing).

Practical considerations

  • Specification pitfalls. When randomized testing is used against an API or protocol specification, the random generator can surface ambiguities or contradictions in the specification itself. A recommended practice when such a problem is found and fixed is to add a corresponding directed test to the regression suite so the issue cannot regress.
  • Complement to directed testing. Randomized testing is rarely used in isolation in mature verification flows; it is typically combined with directed, constrained-random, and coverage-guided test generation.

See also

LINKED ENTITIES

1 links
Randomized Testing of RISC-V CPUs using Direct Instruction Injection USES

CITATIONS

7 sources
7 citations
[1] Random testing (RT) is a well-studied testing method. A Survey on Adaptive Random Testing
[2] Randomized testing has been widely applied to embedded software systems, SQL database systems, and Android applications. A Survey on Adaptive Random Testing
[3] Adaptive Random Testing (ART) was introduced in 2001 and aims to enhance RT's failure-detection ability by more evenly spreading test cases over the input domain. A Survey on Adaptive Random Testing
[4] The ART literature includes various approaches, implementations, assessment and evaluation methods, and applications, and has been comprehensively surveyed. A Survey on Adaptive Random Testing
[5] Autotest is a random test generator for cloud APIs that reads the API specification and deduces a model used in test generation. Random Test Generation of Application Programming Interfaces
[6] A best practice when Autotest reveals an API specification problem is to add an appropriate test to the regression once the problem is revealed and solved. Random Test Generation of Application Programming Interfaces
[7] The paper 'Randomized Testing of RISC-V CPUs Using Direct Instruction Injection' applies randomized testing to RISC-V CPU verification using direct instruction injection. Randomized Testing of RISC-V CPUs Using Direct Instruction Injection (Joannou et al., IEEE Design & Test, 41(1):40-49, Feb 2024)