Benchmarks used as tests
In the Transmeta Crusoe hardware, software, and development workflow, externally sourced benchmarks form one category of test inputs alongside hand-written tests and automatically generated instruction- and system-level tests. Tests are organized into two broad suites: a VLIW test suite written entirely in-house, and an x86 test suite that combines in-house tests with externally supplied tests.
Sources of external tests
External x86 tests come from several origins:
- Tests bought from other companies.
- Industry-standard tests.
- Applications and operating systems used as tests.
- Acceptance suites used by individual makers.
These externally sourced items are collectively referred to as "benchmarks used as tests" within the broader test mix.
Distinguishing benchmarks from other tests
Not every benchmark is a complete test in the strict sense. "Test" implies an acceptance or rejection criterion, and while some benchmarks include a checkable results file, many do not. The Crusoe team nonetheless treats several easily recognized outcomes as checkable failures:
- Crashes.
- Failure to complete.
Benchmarks are not designed for fault isolation, but some failures they produce are easy to diagnose and can expose bugs that no other test finds, so failures from benchmarks are at least triaged rather than discarded.
Relationship to other testing practices
Benchmarks sit alongside, not in place of, other testing mechanisms in the workflow:
- Hand-written tests and real workloads.
- Automated instruction-level and system-level test generators used for both VLIW and x86.
- Human testing, which begins early because the fast VLIW simulator lets it start before hardware is ready.
Benchmark-derived failures therefore feed into the same triage, diagnosis, and bug-tracking (Gnats) processes as failures from hand-written and generated tests.
Operative environment
Benchmark execution depends on the surrounding test infrastructure. The automated test farm initializes CMS, the BIOS, and the hard disk drive so any machine can be reallocated and reconfigured for purposes such as batch testing, with any desired versions of CMS, BIOS, and disk image. Video-output pattern-matching tools were sometimes used to evaluate benchmark runs, though they produced false failures and frames were therefore retained for human review.