Test Program

Definition

A test program is a program used to verify or characterize the behavior of a system under test. In the software-engineering literature on compiler, virtual-machine, microprocessor, runtime, and JIT-compiler testing, a test program is the concrete, executable input that exercises the target and whose execution behavior is observed to detect functional bugs, performance regressions, or to localize known defects.

Roles in Testing

The evidence describes several distinct roles that test programs play:

• Functional verification of compilers, virtual machines, and microprocessors. Testing with the use of test programs is applied to ensure the quality of these targets; parameterized descriptions of test programs (test templates) are used to reduce the resources spent on test creation. Test templates testing the same functional properties can be reused across architectures by expressing architecture-dependent constructs as an interface, then customizing them for each target. [b1316731]
• Functional processor verification. In the Genesys-Pro generator, a test program is the concrete output of a test-generation engine that consumes a test template plus a declarative model of the architecture and produces a program that complies with the scenario requested by the template. [068daf2c]
• Performance testing of WebAssembly (Wasm) runtimes. WarpGen generates test programs specifically to expose abnormal performance of Wasm runtimes, with the quality of the generated programs measured by a metric called distinguishability. [arxiv:2412.20100]
• Bug localization in JIT compilers. Test programs are the inputs whose execution behaviors are analyzed by a JIT-compiler bug localizer; the characteristics of these test inputs (passing vs. failing) significantly affect localization accuracy. [arxiv:2307.08885]

Generation

From Test Templates

Test programs are commonly derived from test templates — parameterized descriptions of test programs. The relationship between a template and a test program is generative: rewriting or customizing the template yields concrete test programs for a target architecture. [b1316731, 068daf2c]

In Genesys-Pro, generation occurs on two levels [068daf2c]:

• Stream-level generation determines which instructions appear in the test and in what order.
• Instruction-level generation creates specific instruction instances (e.g., concrete operands, addresses).

The engine uses pseudorandom generation together with generic biasing constraints to vary parameters that are not critical to the requested scenario, so that different engine invocations exercise the same template scenario in different and interesting contexts (for example, varying a Load's source address so that it may trigger a cache or translation event).

From Seed Programs and Operator Synthesis (WarpGen)

For WebAssembly runtime performance testing, WarpGen generates test programs by [arxiv:2412.20100]:

1. Extracting code snippets from historical issue-triggering test programs as initial operators.
2. Inserting an operator into a seed program to synthesize a new test program.

The quality of these generated programs is measured by distinguishability — the ability of a test program to distinguish abnormal performance of specific Wasm runtimes.

From Seed Mutation for JIT Bug Localization

For JIT-compiler bug localization, test programs are generated so that [arxiv:2307.08885]:

• Passing inputs (which do not trigger the bug) are as similar as possible to the initial seed input.
• Failing inputs (which trigger the bug) are as different as possible from the seed.

A structural analysis of the seed program determines which parts of the code should be mutated for each of the passing and failing cases.

Architectural Customization

To support verification of test objects with different architectures, the same functional test template can be reused. Architecture-dependent constructs needed to describe a template are defined as an interface; the template is then customized to the architecture of the object under test by describing implementations of those constructs. Implementations can be described by different sets of architecture-dependent operations (differing in data type and operations used), and different strategies for selecting implementations — full brute force, random selection, and others — can be applied. Rewriting the implementations allows test programs to be derived from a given template for new architectures. The method was successfully applied to check optimizations in a virtual-machine compiler. [b1316731]

Evaluation and Quality Indicators

• Distinguishability. In WarpGen, the indicator called distinguishability refers to the ability of a test program to distinguish abnormal performance of specific Wasm runtimes; it is used to verify the quality of generated programs. WarpGen was applied to four Wasm runtimes and identified seven new performance issues in three of them. [arxiv:2412.20100]
• Passing/Failing mix. In the JIT bug-localization work, the quality of generated test programs is characterized by the presence of both passing inputs (similar to the seed) and failing inputs (different from the seed). Experiments using a prototype implementation indicate that test inputs generated with this approach result in significantly improved bug localization compared with existing approaches. [arxiv:2307.08885]

Continuous Test-Program Verification (PySpike)

In a RISC-V SoC co-verification context, the PySpike tool provides a reference device-under-test (DUT) used as a continuous test-program verifier. A DUT arbitrator coordinates a PySpike target with an external tool such as OpenOCD over a remote bitbang connection, allowing test programs to be checked against a reference model as part of a continuous verification flow. [f3e6748a]