Overview

Automatic instruction-level test generators are tools used to automatically produce tests for both VLIW and x86 testing environments. They complement traditional hand-written tests and real workloads, and are employed for both hardware and software development, as well as for finding and diagnosing failures.

Characteristics and Benefits

Automated test generators produce tests that are complicated enough to find failures not found by simple tests. These generated tests are often simpler to diagnose than failures found in real workloads. In practice, automated tests find both hardware and software bugs, including tricky corner cases that are missed by hand-written tests.

Once bugs are discovered, the test generator can be constrained to exclude or emphasize certain features in order to help find small and easy-to-debug test cases. This process of "narrowing" down test cases can be automated to a degree, and can proceed in parallel with the diagnosis of test cases found so far.

Challenges and Techniques

A traditional problem with hand-written tests is that the author "knows" certain cases are unimportant and so skips them, potentially missing bugs. Conversely, a traditional problem with pseudo-random test generators is that they generate many "boring" cases for each interesting case.

A technique used with good success to mitigate these issues is biased random number generation. For example, rather than testing a case with two arbitrary inputs, the generator tests with inputs whose bit pattern is usually mostly ones or mostly zeros. This allows the test generator author to provide guidance towards cases which are likely to be problems, while still allowing the random machinery to test some cases the author might think are "boring".

Debugging Considerations

Debugging short instruction sequences from test generators is expected to be straightforward. However, often "easy" bugs have already been found with hand-written directed tests, leaving difficult-to-debug race conditions and the like. Longer-running system tests, by contrast, less often find races but more often find state bugs that span multiple translations.

Usage in Development

In the context of Transmeta Crusoe development, automatic instruction-level and system-level test generators were used alongside a highly automated test farm. These tools were part of the suite used to build, run, and debug the Crusoe processor and its code morphing software (CMS).