46.4

Threadmill: A Post-Silicon Exerciser for Multi-Threaded Processors

  Allon Adir, Maxim Golubev, Shimon Landa, Amir Nahir,
           Gil Shurek, Vitali Sokhin, Avi Ziv
              IBM Research - Haifa, Israel

{adir, maximg, shimonl, nahir, shurek, vitali, aziv}@il.ibm.com

ABSTRACT Statistics show that close to 50% of chips require additional Post-silicon validation poses unique challenges that bring-up unplanned tape-outs because of functional bugs [1]. Given tools must face, such as the lack of observability into the de- this and facing condensing schedules, it is not unusual for sign, the typical instability of silicon bring-up platforms and project schedules to specify several planned tape-outs at in- the absence of supporting software (like an OS or debug- termediate stages of the project before the final release of gers). These challenges and the need to reach an optimal the system. As a result, an implementation of the system utilization of the expensive but very fast silicon platforms on silicon running at real-time speed is available to serve the lead to unique design considerations - like the need to keep development stage known as post-silicon validation. Tradi- the tool simple and to perform most of its operation on plat- tionally, this stage is used to provide a stamp of approval form without interaction with the environment. for the design, aiming to expose any remaining electrical de- In this paper we describe a variety of novel techniques sign flaws, catch the few functional bugs that escape into optimized for the unique characteristics of the silicon plat- silicon, validate the system’s performance, enable closure of form. These techniques are implemented in Threadmill – firmware and system-software, and help characterize produc- a bare-metal exerciser targeting multi-threaded processors. tion issues. These days however, intermediate silicon proto- Threadmill was used in the verification of the POWER7 types need to play a more significant role in design lifecycle, processor with encouraging results. and also serve as the next-level vehicles for functional veri- fication of the system. Categories and Subject Descriptors Post-silicon is getting a lot of attention in the hardware verification community in recent years [12, 13], but this at- B.6.3 [Logic Design]: Design Aids—Verification tention is focused on the on-line checking and debugging ca- pabilities of the silicon platforms (e.g., [2, 8, 11, 17, 9]). Very General Terms little has been published on post–silicon validation method- Verification ologies (e.g., [14, 3]) and other aspects of post-silicon vali- dation, such as stimuli generation [3] and coverage [4]. Post–silicon validation is carried out by highly experi- Keywords enced teams of designers and domain experts, deploying a Post-Silicon Validation, Functional Verification, Stimuli Gen- wide range of testing methods: from starter sets of simple eration, Multi-Threading tests, through massively pre-generated buckets of tests [14], specialized test-applications, exercisers [15], and layers of

1. INTRODUCTION the system’s software stack (i.e., OS, applications). Post-silicon validation uses a variety of platforms, ranging

The Holy Grail of a single tape-out product seems to be from the first manufactured wafers to a series of experimen- drifting away for today’s state-of-the-art processor and mul- tally constructed configurations of the system. These plat- tiprocessor hardware systems. The functional verification of forms are characterized by high manufacturing costs and these systems involves hundreds of person years and requires relatively low availability. Also, when compared to simu- the compute power of thousands of workstations [16]. Yet, lation platforms, they offer real-time execution speed, but the intricacy of modern microarchitectures and the complex- low dynamic observability into the system’s state. A related ity of the system topology makes it virtually impossible to limitation is the high overhead for loading and offloading eliminate all functional bugs in the design before tape-out. memory and state information, in particular for a bare-metal system, where no OS services are available. Finally, some post-silicon platforms (e.g., on wafer) provide very limited Permission to make digital or hard copies of all or part of this work for memory space for programs and data. personal or classroom use is granted without fee provided that copies are The characteristics of the post-silicon platforms create not made or distributed for profit or commercial advantage and that challenges and impose tradeoffs that shape the way these copies bear this notice and the full citation on the first page. To copy platforms are used for functional verification. While post- otherwise, to republish, to post on servers or to redistribute to lists, silicon platforms offer a huge number of execution cycles, requires prior specific permission and/or a fee. their low availability and high cost calls for high utilization DAC'11, June 5-10, 2011, San Diego, California, USA in terms of maximizing the time spent in executing test- Copyright © 2011 ACM 978-1-4503-0636-2/11/06...$10.00

Permission to make digital or hard copies of all or part of this work for 860
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. To copy

46.4

cases and minimizing overhead. The overhead associated preliminary test execution step, when the exact state of the with loading a test image onto the platform may become required resources can be determined. a bottleneck, in particular for bare-metal platforms. Pack- Some generation processes are too complex to run on the aging a large number of pre-generated tests together would tested platform. One example is the generation of data for reduce the number of required loads and thus mitigate the floating-point operands. Here, large buckets of operand-sets overhead. However, this method requires sophisticated in- are generated off-platform, based on user directions, and are frastructure to enable efficient synchronization and commu- embedded in the exerciser’s image. These sets of operands nication between a large test generation farm and the tested are later used by Threadmill while generating on-platform. system. Alternatively, tests could be generated on the tested Threadmill is a distributed application, generating and system by a post-silicon exerciser [15], thus eliminate en- checking distributively on each processor and thread. This tirely the need to be loaded on the platform. An exerciser scheme makes it simpler to achieve scalability on massively is an application that once loaded to the system, continu- parallel systems, but makes the orchestration of correlated ously generates test-cases, executes them, and checks their inter-thread, inter-processor, and system scenarios far more results. The on-platform test-generation process can signif- challenging. Implementing the generator engine as a dis- icantly reduce the number of cycles left for the execution tributed application means that multi-threaded interacting of test-cases, thus threatening to eliminate the advantage test-cases need to be created in a distributed way. Planning of the silicon prototype as a verification platform. Conse- for interaction require synchronization or coordinated deci- quently, an on-platform test generation engine must be fast sions between the generator’s threads. An efficient way to and light, and therefore simple, compared to technologies achieve coordination over generation decisions, which should used for pre-silicon verification. be random in nature, is to use a shared seed for the dis- The requirement for simplicity goes beyond the utiliza- tributed pseudo-random decision procedures. Yet, certain tion issue. The low observability makes hardware failures user directions require that strong synchronization points, extremely hard to debug and therefore simple software must such as barriers and critical sections, are inserted to lock be used to ease the effort. In addition, we’d like to deploy step the distributed generation processes. However, such the tool in the very early stages of the post-silicon valida- procedures significantly reduce the efficiency of the scheme, tion effort when the OS cannot yet be run on the system and and therefore user directions are analyzed to plan and min- “complex” operations such as reading files from an I/O de- imize the use of generation-time synchronization points. vice are not supported. The limited observability, the need Threadmill uses multi-pass consistency checking [15], i.e., to maximize utilization, and the simplicity requirement sug- each test-case is executed multiple times (’passes’) and the gest that on-platform, simple checking methods would be end-of-test values of some system resources (e.g., memory, preferred. Reference models that are commonly used in pre- registers) are compared for consistency. This method proved silicon verification are less appealing here: off-platform us- to cope well with the majority of bugs that escape into sili- age would require off-loading test execution results, while con. To support it, the generator must guarantee that gen- on-line usage would introduce complexity and significantly erated scenarios lead to predictable results, at least for the reduce the platform’s utilization. checked resources. The effectiveness of this method greatly In [3] we presented a unified methodology for pre- and depends on existing and injected variability between passes post-silicon verification and described Threadmill, an ex- (e.g., in thread scheduling or in operation modes of the sys- erciser designed to support the post-silicon aspects of this tem). Consistency checking is one way to cope with the chal- methodology. Threadmill is a bare-metal exerciser, with a lenge of checking in the absence of a reference model. The simple and fast pseudo random generation engine. Thread- method is also readily amenable to Threadmill’s distributed mill is directable, that is it enables fine user direction and test-cycle control scheme. In addition to consistency check- control over the generated tests. It supports various check- ing, Threadmill supports user specified self-checking proce- ing methods, and provides mechanisms to assist in analysis dures. These are typically designed to verify predictable of failures. aspects of an abstract scenario outlined by the user. The description of Threadmill in [3] focuses on its architec- We demonstrate the effectiveness of two of Threadmill’s ture, and the methodology it supports, as well as some of the generation techniques (using off platform floating-point data design decisions made to support the unified methodology. generation and muti-threaded memory collision generation) We now describe in detail some of techniques and mecha- through experiments. nisms used to overcome the technical challenges induced by The rest of the paper is organized as follows: Section 2 the platform and by Threadmill’s design decisions. gives an overview of Threadmill. Sections 3, 4, and 5 de- Threadmill’s design point induces several challenges. For scribe how Threadmill overcomes specific challenges related efficiency and simplicity reasons, Threadmill avoids using to test generation on a post-silicon platform. Sections 6 and techniques that are widely deployed by pre-silicon tools and 7 describe Threadmill’s checking and debugging techniques. are considered essential to enable fine user direction. It does Finally, we give our conclusions in Section 8. not use constraint satisfaction techniques and is not sup- ported by a reference model, neither to assist in test gen- 2. THREADMILL OVERVIEW eration nor to support on-platform checking. The task of Threadmill is a bare-metal, user-directable exerciser, with planning (or even predicting) the program’s control flow be- a simple and fast pseudo-random generation engine. The comes a challenge when a reference model is not available to high-level tool architecture of Threadmill is depicted in Fig- support the generation process. For instance, we use a dual ure 1. Threadmill was developed to support a unified pre- pass technique to generate a branch instruction. The gen- and post-silicon verification methodology [3]. Its goal is to erator plans the intended target address of the branch but support a verification process guided by a verification plan the construction of the instruction itself is finalized during a by enabling the validation engineers to guide the exerciser

861

46.4

                           Fe                                   description as simple data structures; and fixed (test-template
       (=)                                                      independent) code that is responsible for the exercising. The
                          [me]                                  image created by the builder is then loaded onto the sili-
                                                                con/accelerator platform where the exerciser indefinitely re-
                                                                peats the process of 1) generating a random test-case based
   = Comer1             (Geren)                                 on the test-template, the configuration and the architectural
                                                                model, 2) executing it, and 3) checking its results.
       P=                                                              We designed Threadmill with the intention of focusing

                                                                its name).   It does this by generating multi-threaded test-
   Architectural Model |  Nyy)                                  on the validation of multi-threaded designs (as implied by
                                                                cases. Since Threadmill runs on-platform, this implies that
                                                                Threadmill itself is a distributed, multi-threaded, tool. More-
Figure 1: Threadmill architecture                               over, to maximize platform utilization and increase Thread-
                                                                mill’s scalability, we implemented Threadmill’s test gener-
                                                                ation component as a concurrent program, that is, every
                                                                thread generates its own part of the test-case.

through test-templates. 3. STATIC BRANCH GENERATION Threadmill was designed to address the unique character- istics of the post-silicon platform. To minimize interactions We designed Threadmill’s test generation component to with the environment we chose to develop Threadmill as be simple and fast. Therefore, we opted for a static test an on-platform exerciser. Thus once Threadmill is loaded generator, i.e., one that does not make use of a reference on silicon it will generate, execute, and check tests “indefi- model. Test generators that use a reference model (such nitely”. As described above, the lack of debugging capabili- as Genesys-Pro [6]) can leverage it to provide information ties coupled with the typical instability of the bring-up plat- about the expected state of the processor before and after form induces the requirement for a simple design. Therefore, the generation of each instruction. This information can Threadmill does not use sophisticated generation techniques be used to create more interesting events. For example, the like CSP [7], and does not rely on a reference model. generator can decide to select two registers that are expected Threadmill was developed to be simple enough to also to hold large numbers in order to trigger an overflow event operate on hardware-acceleration platforms. Accelerators when generating a multiply instruction. The problem then are hardware platforms dedicated to fast RTL simulation. is how to generate valid and interesting test-cases without The unique characteristics of the acceleration platform, as the ability to track the processor’s state. A partial replace- well as its implication on the verification methodology, are ment to the reference model could be to explicitly reload described in [3, 4] and are beyond the scope of this paper. resources, such as registers, with desired values right before The main input to Threadmill is a test-template that spec- their use within the test-case. However, this solution might ifies the desired scenarios from a verification plan. The test- interfere with the generation of the requested scenarios. For template language of Threadmill is very similar to the lan- data-oriented events, such as divide-by-zero, a simple yet guage of Genesys-Pro [6], IBM’s primary test-generator for effective solution is to reserve registers to hold interesting pre-silicon simulation-based core verification. To adhere to values. Of course, the generator has to ensure that the re- the requirements for simplicity and generation speed, several served registers are not modified during the test-case. constructs that require long generation time, such as events, Threadmill generates instructions in the order in which are not included in Threadmill’s language. The language en- they will be executed. Therefore, it needs to know the out- ables verification engineers to write the test-templates as an come of a conditional branch to know at which address to activity separated from the generator’s development activ- generate the next instruction. This is hard to do without a ity. The language consists of four types of statements: basic reference model because it is impossible to evaluate the con- instruction statements, sequencing-control statements, stan- dition without knowing the expected values of the resources dard programming constructs, and constraint statements. involved in the condition. One way to overcome this issue Users combine these statements to compose complex test- is to create the known machine state by inserting a well- templates that capture the essence of the targeted scenarios, known instruction sequence into the test so the outcome of leaving out unnecessary details. This allows them to direct the conditional branch is known at generation time. But the generator to a specific area. Other inputs to Thread- this approach misses many dependency scenarios between mill are the architectural model, testing knowledge, and the the condition register setting and the branch. system topology. To allow generation of an arbitrary instruction stream The Threadmill execution process starts with a builder ap- with conditional branches Threadmill completes the con- plication that runs off-line to create an executable exerciser struction of the branches in two stages. First, as part of image. The role of the builder is to convert the data incorpo- the general test generation flow, Threadmill decides on the rated in the test-template and the architectural model into desired condition result and target address of the branch. data structures; these structures are then embedded into the ~~ However, instead of placing the branch instruction into the exerciser image. This scheme eliminates the need to access test stream, it writes an illegal instruction form. This way, files or databases while the exerciser is running. when the test-case is executed, the illegal instruction causes The exerciser image is composed of three major compo- an illegal instruction interrupt. The handler of this inter- nents: a thin OS-like layer of basic services required for rupt calls a generation function dedicated to the completion Threadmill’s bare-metal execution; a representation of the of the branch instruction generation. This function is in- test-template, architectural model and system configuration voked during the actual run of the test-case, and can there-

   862

46.4

fore observe the current state of the machine. This observed malized, -infinity, etc.). We defined a coverage model that state is used to construct a condition expression that will includes the occurrence of all the legal number forms in the evaluate to the desired result (as determined in the first input and outputs operands of the 10 fp instructions. This stage). For example, suppose that at the first stage the model includes 280 different legal events. We ran Thread- generator decides that branch A should be taken. At run mill in three modes. In the first mode we used random time, the illegal instruction interrupt is taken and the han- values for the inputs of the fp instructions; in the second dler observes that register R4 contains a positive value. The mode we used only data collected from tables constructed handler now constructs a condition that would cause branch with FPgen off-line as described above; in the third mode A to be taken if R4 is positive. The handler then replaces we used pre-generated data in 70% of the cases, while in the the illegal instruction with the generated branch and the ex- remainder random values are used. When only random in- ecution of the test-case resumes. As described in Section 6, puts were used, 35.71% of legal events were covered. When Threadmill executes each test-case multiple times for check- only pre-generated data was used, 92.86% of legal events ing purposes. In the first execution (which serves as a kind were covered. In the intermediate mode, where only 70% of of reference-model to the following executions) the branch instructions used pre-generated data, 97.5% of legal events only runs after the illegal instruction interrupt and the han- were covered. While this may seem surprising, the reason dler. In subsequent executions of the same test, the branch for this is that FPgen focuses on generating legal interesting instruction is not delayed and runs right after its preced- operand values, and therefore does not cover the illegal forms ing instructions, without the handler. This approach, which well (specifically, out-of-range numbers for single precision defers some of the generation to the test-case execution, is instructions). Therefore, combining pre-generated data with very expensive performance-wise and is used sparingly, only pure random values gives the best coverage results. in the cases where a reference model is crucial. The approach also requires execution of the same test-cases multiple times, but this multi-pass execution is a worthy cost because of its 5. CONCURRENT GENERATION OF other benefits for checking purposes. CONCURRENT TEST-CASES The decision to make Threadmill’s generation component 4. GENERATING FLOATING-POINT a concurrent program (as described in Section 2) poses a INSTRUCTIONS significant challenge: Threadmill’s generation component is required to create multi-threaded test-cases in a distributed Verification of the floating point unit (FPU) hardware im- way. Multi-threaded test-cases typically include some form plementation is known as an intricate problem. The numer- of data sharing among the threads, for example, to bet- ous corner cases of the vast test space, coupled with the com- ter stimulate the hardware’s memory coherence mechanisms. plexity of the implementation of floating point operations, Therefore, the concurrently generating threads must come turn the FPU verification effort into a unique challenge in up with a shared set of addresses to be used in the respective the field of processor verification. thread parts of the test-case. Thus, some form of synchro- FPgen [5] is a test generation framework targeted toward nization between the generating threads is needed. How- the verification of the floating-point (fp) datapath, through ever, executing synchronizing sequences, such as a barrier the generation of data operands for fp instructions. The or a critical section, can be very costly performance-wise. generated data can be directed to trigger a large variety To address this challenge we developed a mechanism that of events related to the floating point computation, includ- enables the threads to make joint random decisions with no ing intermediate and final data characteristics. It does this synchronization. By “joint” we mean that all threads make by leveraging various sophisticated stochastic and analyti- the same decision, while keeping the decision random. This cal techniques. Such techniques would be too complex for a is achieved by a shared random seed. All the generating computation by our on-platform test generator exerciser. threads are provided with a common random number, cre- A simple approach could be to generate random float- ated by the Threadmill builder. Every generating thread ing point values for the instructions’ inputs. However this then makes all joint random generation decisions based on technique has very limited coverage of the more intricate a random number generator initialized with this joint seed. events related to fp instructions. Our approach uses FP- The joint random decision mechanism is heavily used in gen “off-line” (i.e., not during the on-platform generation) the generation of the addresses to be used by the threads to construct a large table of interesting data operands for when constructing their load/store instructions. In addi- the various fp instructions. Since this data generation is tion to jointly choosing the shared addresses, the gener- done off-line, we are less constrained by performance and ating threads must also jointly agree on the collision type complexity considerations. As part of the image building (i.e., write-write, write-read, true- and false-sharing colli- process, we select a random choice of data from the table sions, etc). Our approach is to decide on the type of collision of fp operands. This is then used to create a smaller table, allowed for every memory location. Thus the whole memory which we incorporate into the exerciser image. The exerciser is mapped to the selected random collision types by using a can now randomly select data for the generated fp instruc- joint random hash function, which is created using the joint tions to trigger the interesting events targeted by FPgen. random seed. In order to demonstrate the effectiveness of this method, Without making these decisions jointly, the probability of we conducted the following experiment: we used Threadmill generating colliding accesses among the threads is negligible to generate tests with 10 different types of fp instructions (assuming the shared memory space is large). To demon- (fadd, fsub, fmul, fdiv, fsqrt, in both single and double vari- strate the effectiveness of the joint random collision gen- ations). We distinguished 22 types of fp number forms - 10 eration scheme, we ran Threadmill on three configurations single forms and 12 double forms (such as normalized, denor- with 2GB of memory - two, four and eight threads. We

 863

                                                                     46.4

Threads2 Read-Read Write-Write Read-Write Total the same overriding instruction as above is injected into the 4 67 64.4 129 260.4 test-case. 200.4 228.6 387.6 816.6 It is beneficial to apply the multi-pass consistency check- 8 476.8 527.6 833.2 1837.6 ing technique with some variability between the different passes. Some variability is created automatically, for exam- Table 1: Collision experiment results ple due to page tables filling up and different cache states. Between passes Threadmill also modifies certain things that have no impact on the architectural behavior of the test- used a test template specifying 50 load/store instructions case, such as thread priority. It is important to note that per thread. We measured the number of generated collision the consistency checking technique targets bugs that can be events, where a collision event is defined as a pair of accesses detected due to different operation timing or order in the by different threads that target (possibly different) locations various passes. Specifically, the instructions of the different in the same cache line (of size 128 bytes in our design). For threads executing their parts of the test in parallel can be ex- example, suppose thread 0 loads twice from some cache line pected to be interleaved differently in different passes. Many and thread 1 stores three times to the same cache line. This multi-threading related bugs require a fine relative timing of would be counted as 6 read-write collision events. The re- the threads’ operations. If the problematic timing occurs in sults of the experiment are presented in Table 1 where every one of the passes then the bug could be observed by the row gives the average of 5 different runs. consistency check. In addition, the ordering of instructions executing in the pipeline of the same thread can also be dif- 6. CHECKING TECHNIQUES ferent. However, some type of bugs cannot be detected. For The goal of checking is to detect failures as soon as pos- example, a bug in the processor’s datapath causing 1+1=3 sible and report as many details as possible to facilitate will go undetected. However, we do not expect such bugs an effective debug process towards discovering the failure’s to reach the post-silicon stage at all, but rather be detected root cause. In pre-silicon verification, the tests run on a at a much earlier stage. Finally, this checking technique fits testbench. The testbench incorporates checkers that detect Threadmill very well. Threadmill generates multi-threaded most of the failures. These checkers, however, are not avail- test-cases by having each thread generate and execute its able when running on the post-silicon platform. Specifically, own part in the test-case. The consistency checking is done a reference model is not available to us due to its high com- as follows. At the beginning of a pass, each thread initial- plexity. We therefore have to devise and apply alternative izes its registers, as well as the checked memory areas to checking methods when running Threadmill. While there which it is allowed to write. The thread then samples them are some checkers in the design itself that are monitored by for checking at the end of the pass. As a result, no extra the environment (e.g., machine went into an error state), our synchronization barriers are needed, which is significant for focus here is on checking done by the exerciser itself while achieving good platform utilization. it is running on the platform. Self-checking tests is another checking technique used by Threadmill employs a technique called multi-pass consis- Threadmill. In this case, the test scenario has some as- tency checking. In this technique we execute each generated sertions embedded in it, and any violation of an assertion test-case several times (passes). After each pass we check triggers a failure. The following example is based on a set that certain resources, including registers and memory areas, of tests for shared memory multiprocessor by Collier [10]. are consistent and have the same end-of-test values. In this The system is partitioned into n writer threads, numbered way, we have the first pass essentially serve as a reference 0 through n-1, and remaining reader threads. Each writer model. This consistency is guaranteed by restricting the thread i makes a series of atomic writes to a shared memory test-case generation as follows. First, we do not randomly location, with an ascending series of values starting from i, generate any instruction whose behavior is not fully pre- i+n, i+2n, ..., and so forth. Each reader thread repeatedly dictable. This includes, for instance, the Load-and-reserve makes an atomic read from the shared memory location, and instruction, which does a load and additionally attempts to figures out from which writer thread it originated from by establish a reservation in memory. This attempt may suc- using simple modular arithmetic. The reader thread then ceed or fail depending on run-time conditions that involve asserts that each series of read values originating from the other threads in the system, and cannot, in general, be pre- same writer thread is monotonically non-decreasing. dicted during test-case generation. Second, we forbid any 7. DEBUGGING FAILING TESTS write-write collisions on checked memory areas since we can- not predict the order between the colliding write accesses. Debugging failures on a post-silicon platform is a notori- Third, we must ensure that any data written to a checked ously difficult problem. Many of the debugging aids that are resource is deterministic, in the sense that it has the same available on simulation platforms cannot be supported on sil- value in every pass. This is achieved either by having an- icon because of the very limited observability into the state other checked resource serve as the source of the data or of the design. Furthermore, despite the fact that Thread- by simply using a constant. There is one exception to the mill itself is a piece of software, standard software debug- latter restriction. A thread doing a load from a checked gers cannot be used to debug it because they are typically memory area that is written by another thread could be too complex for the fragile state that the design is in dur- reading unpredictable data. In this case, Threadmill will ing bring-up. The design’s instability during bring-up can inject into the test-case an additional instruction that over- also cause unexpected behavior of Threadmill’s core soft- rides the target register with some constant, if that target ware components. It is thus often difficult to determine if a register is checked. It is important to note that we do allow failure is due to a bug in the design or in Threadmill. loads from unchecked memory areas as well, in which case An important technique for debugging failures found by

                                               864

46.4

Threadmill is to recreate the failing execution on a plat- verification and post-silicon validation. In Proceedings form that does provide observability into the design - such of the 2011 Design, Automation and Test in Europe as simulation or hardware acceleration. It is not enough just Conference, pages 1590–1595, 2011. to re-execute the failing test-case since the bug could also [4] A. Adir, A. Nahir, A. Ziv, C. Meissner, and have occurred during the test generation (which typically J. Schumann. Reaching coverage closure in post-silicon takes much longer than the test execution). In addition, a validation. In Proceedings of the 6rd Haifa Verification Threadmill image generates, executes, and checks a great Conference, pages 60–75, 2010. number of test-cases, and a bug found through one test-case [5] M. Aharoni, S. Asaf, L. Fournier, A. Koyfman, and could be the result of a fault that occurred during the han- R. Nagel. FPgen - a deep-knowledge test generator for dling of a previous test-case. Restarting the image from the floating point verification. In Proceedings of the 8th start on the simulation/acceleration platform is typically not High-Level Design Validation and Test Workshop, possible because it could take days to simulate all the cycles pages 17–22, 2003. executed on silicon before the bug occurs. Our approach is to [6] M. L. Behm, J. M. Ludden, Y. Lichtenstein, restart the failing exerciser image on simulation/acceleration M. Rimon, and M. Vinov. Industrial experience with a few test-cases before the failure. We do this by reusing the test generation languages for processor verification. In corresponding random seeds that were used by the generat- Proceedings of the 41st Design Automation ing threads (as saved in the limited memory trace). Conference, pages 36–40, 2004. Another approach for test-case recreation is to take the [7] E. Bin, R. Emek, G. Shurek, and A. Ziv. Using a test-template of the Threadmill image that caught the bug constraint satisfaction formulation and solution and use it with Genesys-Pro, the pre-silicon test genera- techniques for random test program generation. IBM tor counterpart of Threadmill. Genesys-Pro uses a test- Systems Jouranl, 41(3):386–402, 2002. template language similar to Threadmill’s and can be used [8] K.-h. Chang, I. L. Markov, and V. Bertacco. to generate test-cases for simulation with the hope that the Automating post-silicon debugging and repair. In bug re-occurs in one of them. This approach does not guar- Proceedings of the 2007 international conference on antee a precise recreation, but our experience shows that Computer-aided design, pages 91–98, November 2007. using the same test-template can get the design into the gen- eral area of the bug and improve the chances of re-triggering [9] K. Chen, S. Malik, and P. Patra. Runtime validation the fault under the same or similar circumstances. of memory ordering using constraint graph checking. In Proceedings of the 14th International Symposium on 8. CONCLUSION AND FUTURE WORK High-Performance Computer Architecture, pages The growing importance of post-silicon validation to the 415–426, 2008. functional verification process increases the need to utilize [10] W. W. Collier. Reasoning About Parallel silicon prototypes as a vehicle for functional verification. Architectures. Prentice Hall, 1992. We presented Threadmill, a bare-metal exerciser targeting [11] F. M. De Paula, M. Gort, A. J. Hu, S. J. E. Wilton, multi-threaded processors. Threadmill generates test-cases and J. Yang. Backspace: formal analysis for on platform, employing a variety of novel techniques opti- post-silicon debug. In Proceedings of the 2008 mized for the unique characteristics of the silicon platform. International Conference on Formal Methods in Threadmill is an important enabler of the unified pre- Computer-Aided Design, pages 1–10, November 2008. and post-silicon verification methodology described in [3]. [12] S. Mitra, S. A. Seshia, and N. Nicolici. Post-silicon This methodology relies on a shared test plan and similar validation opportunities, challenges and recent test template languages for the pre- and post-silicon test advances. In Proceedings of the 47th Design generators. Threadmill was used in the verification of the Automation Conference, pages 12–17, 2010. POWER7 processor chip. Results of this experience con- [13] A. Nahir, A. Ziv, R. Galivanche, A. J. Hu, firm our beliefs about the benefits of the various techniques M. Abramovici, A. Camilleri, B. Bentley, H. Foster, described in this paper. V. Bertacco, and S. Kapoor. Bridging pre-silicon We are exploring further opportunities to leverage smart verification and post-silicon validation. In Proceedings off-line pre-generation to improve the test quality – focusing of the 47th Design Automation Conference, pages on creating interesting address translation paths. This will 94–95, 2010. enable Threadmill to become a more important player in [14] H. G. Rotithor. Postsilicon validation methodology for areas such as caches and memory and I/O subsystems. microprocessors. IEEE Design & Test of Computers, 17(4):77–88, 2000. 9. REFERENCES [15] J. Storm. Random test generators for microprocessor [1] International technology roadmap for semiconductors design validation, 2006. 2009 edition - design. http://www.inf.ufrgs.br/emicro. http:/www.itrs.net/Links/2009ITRS/2009Chapters [16] D. W. Victor et al. Functional verification of the 2009Tables/2009 Design.pdf. POWER5 microprocessor and POWER5 [2] M. Abramovici, P. Bradley, K. Dwarakanath, P. Levin, multiprocessor systems. IBM Journal of Research and G. Memmi, and D. Miller. A reconfigurable Development, 49(4):541–554, 2005. design-for-debug infrastructure for socs. In Proceedings [17] I. Wagner and V. Bertacco. Reversi: Post-silicon of the 43rd Design Automation Conference, pages validation system for modern microprocessors. In 7–12, July 2006. Proceedings of the IEEE International Conference on [3] A. Adir, S. Copty, S. Landa, A. Nahir, G. Shurek, and Computer Design, pages 307–314, 2008. A. Ziv. A unified methodology for pre-silicon

 865