Combinatorial Testing

Concept

Combinatorial Testing (CT) is a black-box software testing technique that systematically exercises interactions among a small number of configuration options (parameters) to detect elusive failures, while avoiding the combinatorial explosion of exhaustive testing. Core CT constructs include covering arrays parameterized by interaction strength t and per-parameter value counts; practical systems often require Constrained Combinatorial Testing (CCT) to exclude or penalize invalid tuples via hard and soft constraints, with solvers ranging from greedy and metaheuristic algorithms to SAT, PB, CSP, and Answer Set Programming (ASP) encodings.

First seen 7/7/2026

Last seen 7/7/2026

Evidence 2 chunks

Wiki v1

WIKI

Combinatorial Testing

Definition and Motivation

Combinatorial Testing (CT) is a black-box testing technique aimed at detecting elusive failures in software systems whose configuration spaces are too large to exhaustively test. CT rests on the empirical observation that most faults are triggered by interactions among a small number of configuration options (parameters). Rather than enumerate every possible combination, CT systematically exercises t-way interactions among parameters, producing test suites that are much smaller than exhaustive ones yet more effective than random testing ^[1].

READ FULL ARTICLE →

NEIGHBORHOOD

No graph connections found for this entity yet. It may appear in future ingestion runs.

explore full graph →

RELATIONSHIPS

1 connections

Constrained Combinatorial Testing ← extends 100% 1e

CCT is defined as an extension of CT with constraint handling.

LINKED ENTITIES

1 links

Constrained Combinatorial Testing EXTENDS Extracted graph relationship

CITATIONS

8 sources

8 citations — click to expand

[1] CT is a black-box testing technique that detects elusive failures by systematically testing t-tuples of configuration options, producing much smaller test suites than exhaustive testing and more effective ones than random testing. catnap: Generating Test Suites of Constrained Combinatorial Testing with Answer Set Programming

[2] Constrained Combinatorial Testing (CCT) extends CT with hard and soft constraints on configuration options and is one of the most widely studied CT techniques. catnap: Generating Test Suites of Constrained Combinatorial Testing with Answer Set Programming

[3] Generating a CCT test suite is to find a Constrained Mixed-level Covering Array (CMCA), an extension of a covering array with constraints expressed as a Boolean formula in CNF. catnap: Generating Test Suites of Constrained Combinatorial Testing with Answer Set Programming

[4] CCT solving techniques include greedy algorithms, metaheuristics-based algorithms, SAT-based complete methods, and off-the-shelf constraint solvers (SAT, PB, CSP) used as back-ends. catnap: Generating Test Suites of Constrained Combinatorial Testing with Answer Set Programming

[5] catnap encodes CCT instances as first-order ASP and solves them with off-the-shelf ASP systems such as clingo, providing compact encodings and easy extension with limiting resources and soft constraints. catnap: Generating Test Suites of Constrained Combinatorial Testing with Answer Set Programming

[6] Developing a universal CCT solver that efficiently generates optimal and suboptimal suites under hard and soft constraints with limited resources remains challenging. catnap: Generating Test Suites of Constrained Combinatorial Testing with Answer Set Programming

[7] Adapting CT to deep learning systems proposes coverage criteria and CT-coverage-guided test generation as a promising approach for testing DL robustness, despite the huge neuron-level state space. Combinatorial Testing for Deep Learning Systems

[8] A comprehensive survey of constrained combinatorial testing covers 129 papers from 1987 to 2018, categorizing constraint handling, identification, and maintenance techniques for constrained problem domains. A Survey of Constrained Combinatorial Testing