Application and OS-based tests

Application and OS-based tests refer to the use of real-world applications and complete operating systems as test stimuli within Transmeta's Crusoe validation program. Rather than relying solely on hand-written or automatically generated instruction- and system-level tests, the Crusoe team treated commercially available applications and OSes themselves as tests, alongside industry-standard tests and acceptance suites supplied by individual makers (e.g., OEMs or OS vendors).

Role in the test strategy

The Crusoe test strategy combined several test sources:

• In-house VLIW tests — all written internally.
• In-house x86 tests — developed internally.
• External x86 tests, which comprised:
  • Tests purchased from other companies.
  • Industry-standard tests.
  • Applications and OSes used as tests.
  • Acceptance suites provided by individual makers.

The unifying notion is that "test" implies an explicit acceptance or rejection criterion. Some benchmarks ship with a checkable results file, but many do not. Even so, outcomes such as a program "crashing" or "failing to complete" were treated as clearly checkable failures, justifying the inclusion of these workloads in the validation flow. Although benchmarks are not designed for fault isolation, some failures are easy to diagnose and can expose bugs missed by any other test; at minimum, these failures are triaged.

Relationship to other tooling

• The Crusoe testing effort runs on an Automated test farm that allocates, configures, initializes, and reconfigures machines automatically — allowing the same hardware to be repurposed between interactive debugging, batch testing, and reruns of application- or OS-based workloads.
• Automated test generators complement (but do not replace) application and OS-based tests by producing difficult cases that are easier to diagnose, while applications and OSes stress realistic code paths.
• A small remote debugger stub built into CMS enables post-mortem triage of failures found by application and OS-based runs, including work on machines that have crashed or become unresponsive.

Practical notes

• Failures detected via applications and OSes are logged through the same bug-tracking workflow (Gnats, plus customer-facing workflow tools) as other test failures.
• Video-output matching tools were used to detect regressions in graphical applications, with frames preserved for human review when automatic matching produced ambiguous or false results.
• Hardware scarcity made the automated, remotely-reconfigurable farm essential for squeezing maximum coverage out of each board, including for lengthy application- or OS-level soak runs.

See also

• Automated test farm — infrastructure used to execute application- and OS-based tests at scale.