Definition

In the cited microprocessor conformance case study, black-box testing is described as a scenario where an abstract description of the system under test (SUT) is used as the basis for test generation [aecb3458]. The goal is to check that the implementation behaves as described by the abstract model, independently of internal implementation details [aecb3458]. The concept is listed as an explicit keyword of the case study alongside White-Box Testing [11aee6c9, f8849d84].

Role in Model-Based Test Generation

The case study uses an assembler-level model that is more abstract than the processor model. This abstraction makes implementation-level details transparent, including interrupt handling, virtual memory and caching, pipelining, and instruction reordering [aecb3458]. In the black-box scenario, abstract test cases are extracted from the processor model to check whether the processor conforms to the assembler model [aecb3458].

Observable Behavior and Conformance

The evidence distinguishes testing scenarios by what is controlled and observed [aecb3458]:

• Sequence testing: The result is inferred from a sequence of system inputs and observed outputs; only initialization of the internal state is necessary, and sometimes a reference to the final state is used.
• Unit testing: A special one-step form of sequence testing in which the output state is more or less completely accessible.

For conformance in the case study, test specifications compare the state controlled according to the model with the state produced by the SUT using a conformance relation _ =k _ that compares register content and just the top k memory cells (instead of infinite memory) [aecb3458]. A test case is composed of an instruction, an initial configuration, and the resulting configuration after executing the instruction [aecb3458].

Application to Processors Without Implementation Access

The case study positions test program generation as useful for validating processor designs at the instruction-set or assembly level [f8849d84, aecb3458]. This is especially suitable for commercial off-the-shelf (COTS) processors, where implementation details are usually unavailable [f8849d84]. In that setting, black-box testing aligns with generating and running low-level test programs against the processor and checking that the observed behavior matches the specified instruction set [f8849d84].

Tooling

The case study uses HOL-TestGen, a model-based testing environment that is an extension of Isabelle/HOL, to synthesize test programs that were run against real hardware in the loop [11aee6c9, f8849d84].

Related Testing Scenarios in the Evidence

The same study also discusses model-based unit testing and sequence testing [aecb3458]:

• Unit testing uses pre- and post-conditions on inputs and results and assumes control over the initial state plus access to internal states after the test.
• Sequence testing uses sequences of inputs and observed outputs, and may compare the final state.

The authors applied these scenarios to instruction categories including memory-related load/store operations, arithmetic operations, logic operations, and control-flow operations [aecb3458].