Automatic System-Level Test Generators

Automatic system-level test generators are tools used in the development of Transmeta's Crusoe processor and its Code Morphing Software (CMS) to produce automated tests at both the instruction level and the system level.

Scope and Usage

These generators are employed for both VLIW and x86 testing, in addition to traditional hand-written tests and real workloads. VLIW tests are all written in-house, while x86 tests include both in-house and external sources such as industry-standard tests, applications, operating systems used as tests, and acceptance suites from individual makers.
Automated tests are used for both hardware development and software development, and serve both purposes of finding failures and diagnosing failures.
Tests include instruction-level tests and system-level tests, including performance tests.

Purpose and Advantages

Automated test generators produce tests that are complicated enough to find failures not found by simple tests.
Failures produced by automated tests are often simpler to diagnose than failures in real workloads.
Once bugs are discovered, the test generator can be constrained to exclude or emphasize certain features in order to help find small and easy-to-debug test cases. This "narrowing" can be automated to a degree, and can proceed in parallel with diagnosis of test cases discovered so far.
In practice, automated tests find both hardware and software bugs, including tricky corner cases missed by hand-written tests.

Characteristics of Generator Output

Debugging short instruction sequences from test generators is typically straightforward.
However, easy bugs are often already found with hand-written directed tests, leaving difficult-to-debug conditions such as race conditions to be discovered by automated generators.
Longer-running system tests less often find races and more often find state bugs that span multiple translations. Such state bugs can sometimes be reproduced in simulation, making them easier to reproduce.
System tests can optionally pause occasionally and "settle" to help isolate failures, and can record their progress to simplify reconstruction of the state leading up to failure.

Generation Techniques

A traditional problem with hand-written tests is that the author knows certain cases are unimportant and skips them, thereby missing bugs.
A traditional problem with pseudorandom test generators is that they generate many "boring" cases for each interesting case.
A technique used with good success is biased random number generation. For example, rather than testing a case with two arbitrary inputs, tests use inputs whose bit pattern is usually mostly ones or mostly zeros. This allows the test generator author to provide guidance towards cases which are likely to be problems, but because the inputs are only guided rather than firmly constrained, the random machinery still tests some cases the author considers "boring".

Relation to the Automated Test Farm

The Crusoe in-house farm hardware and software allow automated initialization of CMS, the BIOS, and the hard disk drive. This in turn allows any machine to be used for any purpose. A machine can be allocated, configured, and used for remote interactive debugging, and then, without human intervention, reallocated and reconfigured for another use such as batch testing, with any desired versions of CMS, BIOS, and disk image. Such an automated test farm consumes the tests produced by automatic system-level test generators.