Overview
Acceptance test suites are external test collections used to validate the Transmeta Crusoe processor's x86 implementations. They are drawn from "other companies," industry-standard tests, applications and operating systems used as tests, and "acceptance suites used by individual makers," serving as input to the Crusoe hardware/software validation flow described in the 2009 Crusoe retrospective paper.
Role in the Crusoe testing program
The Crusoe validation effort distinguished between in-house and external test sources:
- In-house tests: All VLIW tests were written in-house, while some x86 tests were also written in-house.
- External tests: x86 tests were augmented with tests bought from other companies, industry-standard tests, applications and OSes used as tests, and acceptance suites used by individual makers.
Acceptance suites are explicitly part of this external x86 test portfolio and contribute to the overall assertion that "test" implies an acceptance-or-rejection criterion for the system under test.
Acceptance vs. rejection criteria
In the Crusoe methodology, "test" is defined by an acceptance or rejection outcome. Where formal criteria were available, benchmarks and tests used checkable results files; where they were not, crashes and "fails to complete" were still treated as recognizable checkable failures. Acceptance suites therefore participate in a broader triage strategy that classifies failures as checkable (results files, crashes, hangs) or uncheckable (silently wrong answers), with the checkable subset being the primary signal for acceptance.
Relationship to fault isolation and triage
The paper notes that benchmarks "are not designed for fault isolation," but that "some failures are easy to diagnose and can expose bugs not found by any other test," making it worthwhile "to at least triage these failures." Acceptance test suite outcomes are part of this triage workflow: their results are collected, characterized, and either flagged as accepted or routed for further diagnosis.
Interaction with the automated test farm
Acceptance test suites are executed on machines managed by the in-house automated test farm, which:
- Performs automated initialization of CMS, the BIOS, and the hard disk drive, so any machine can be used for any purpose.
- Allocates and configures machines for purposes such as remote interactive debugging or batch testing, and can reallocate and reconfigure them without human intervention.
- Supports running different versions of CMS, BIOS, and disk images as needed for a given test run.
This farm infrastructure is what makes routine, unattended execution of acceptance test suites tractable alongside the hand-written tests, automatic instruction-level and system-level test generators, and real workloads also used in validation.
Generation, narrowing, and complementarity
Although acceptance test suites themselves are externally authored, they sit within a larger pipeline that also includes automatic instruction-level and system-level test generators. These generators can be constrained to exclude or emphasize certain features to produce small, easy-to-debug test cases after a failure is discovered, and narrowing can proceed in parallel with diagnosis. Acceptance suites therefore complement, rather than replace, the generated and hand-written test sets: generated tests are "complicated enough to find failures not found by simple tests" and "often simpler to diagnose than failures in real workloads," while acceptance suites contribute real-world workload coverage and maker-specific acceptance criteria.
Debugger and hardware integration
A small remote debugger stub is built into CMS and, on boot, queries a special debug connector; if "live," it downloads a larger debug stub. The debugger's user interface is kept consistent between simulation and hardware so developer experience is nearly the same in both contexts. Failures originating from acceptance test suite runs on hardware can therefore be investigated using the same toolchain as simulation failures.
Known limitations in the test pipeline
The paper notes that some failure modalities were imperfectly captured:
- For graphical output checks, automated video-output pattern matching "sometimes yielded false failures," so video frames were retained for human review.
- Hardware was always in high demand and limited supply, and in-house debug systems often needed rework even after volume production.
These constraints affect how thoroughly acceptance test suite failures can be reproduced and diagnosed, particularly those that manifest in peripherals or hardware-specific paths.