Automated test farm

The automated test farm is one of the key tools used during the development of the Transmeta Crusoe processor. It is listed alongside reference x86 simulators, corner-case tests, a fast simulator, fault-isolating debuggers, and test generators as a tool used to "build, run, and debug Crusoe."

Purpose and capabilities

The farm consists of in-house farm hardware and software. Its primary capability is to allow automated initialization of CMS (the Code Morphing Software), the BIOS, and the hard disk drive. This in turn allows any machine in the farm to be used for any purpose:

• A machine can be allocated, configured, and used for remote interactive debugging.
• Then, no matter what its state, it can — without human intervention — be reallocated and reconfigured for another use, such as batch testing, with any desired versions of CMS, BIOS, and disk image.

Because hardware was always in high demand and limited supply (even with processors in volume production, in-house debug systems were in limited supply and often needed rework), this flexibility was essential.

Interaction with debugging and checkpoints

The farm is tightly integrated with the simulator and debugger infrastructure. Automated farm testing:

• Executes long-running tests to find bugs.
• Leaves checkpoints so humans can quickly get to the point of failure found by automation.

These checkpoints may be loaded by different debugger sessions, and also serve as an easy way to hand off debugging sessions between developers.

Tests and generators exercised by the farm

The farm runs tests from a broad range of sources, drawing on both VLIW and x86 test suites. Tests include:

• Instruction-level tests and system-level tests, including performance tests.
• VLIW test suites — all written in-house.
• x86 test suites — both in-house and external, where external tests include tests bought from other companies, industry-standard tests, applications and operating systems used as tests, and acceptance suites used by individual makers.
• Application- and OS-based tests — including real workloads.
• Performance tests.
• Benchmarks used as tests — noting that "test" implies an acceptance or rejection criterion; some benchmarks have a checkable results file, but many do not, so "crashes" and "fails to complete" are also treated as checkable failures.
• Corner-case tests.

The farm also uses several automatic instruction-level and system-level test generators, in addition to traditional hand-written tests and real workloads. These automatic generators produce tests complicated enough to find failures not found by simple tests, and the tests they produce are often simpler to diagnose than failures in real workloads. Once bugs are discovered, the generators can be constrained to exclude or emphasize certain features to help find small and easy-to-debug test cases; although "narrowing" is often time-consuming, it can be automated to a degree and can proceed in parallel with diagnosis.

Complementary automation techniques

In practice, automated tests find both hardware and software bugs, including tricky corner cases missed by hand-written tests. A traditional problem with hand-written tests is that authors "know" certain cases are unimportant and skip them, missing bugs; a traditional problem with pseudo-random test generators is that they generate many "boring" cases for each interesting case. To address this, the project used biased random number generation — for example, using inputs whose bit pattern is usually mostly ones or mostly zeros — to provide guidance toward cases likely to be problems while still letting the random machinery test some cases the author thinks are "boring."

Manual testing alongside the farm

Despite the high degree of automation, candidates are still subjected to human testing, and some difficult or risky changes are also human-tested. The fast VLIW simulator allowed human testing to begin long before hardware was ready, increasing maturity by the time hardware became available.