Functional Verification Methodology for Microprocessors Using the Genesys Test-Program Generator Application to the x86 Microprocessors Family

Laurent Fournier Yaron Arbetman Moshe Levinger IBM Haifa Research Lab IBM Haifa Research Lab IBM Haifa Research Lab laurent@vnet.ibm.com yaronar@vnet.ibm.com mosh@vnet.ibm.com

Abstract hard-to-reach parts of the design. Secondly, the simulation includes running existing sets of tests (legacy tests or commer- Even though the importance of microprocessor design cial test suites) [10]. These tests typically have claimed cover- verification is widely acknowledged, no rigorous method- age, and, most importantly, have already been successfully ology is being commonly followed for its realization. This employed in the verification of similar designs. Obviously, this paper attempts to delineate such a methodology, and component is available only if the targeted architecture is an shows how it is promoted by Genesys, an automatic established one. For the x86 architecture, such sets of tests are pseudo-random test-program generator. The methodol- generally available and are being updated along with each ogy relies on a verification plan which induces smart sets architecture upgrade. Thirdly, after the hardware is ready for of tests that carry out the verification tasks. The paper use, some extensive applications (e.g., operating systems) are reports on an application of this methodology, using run. Finally, automatic test generation, usually mainly random Genesys, to verify an x86 design and describes, in partic- and of restricted scope, might be performed. The overall proc- ular, how this methodology could have helped to avoid ess should be interleaved with some means of coverage. Cover- known escape bugs, such as the recent two infamous age is a major constituent of the verification process and Pentium Floating Point bugs. therefore deserves further attention. This however is beyond the scope of this paper. In addition, designers should routinely keep in mind the verification implications of their development or modifications, thereby collaborating toward a verification- 1.0 Introduction aware design process [5]. It will be shown, in this paper, how It is widely recognized that functional verification emerges the suggested methodology, and Genesys in particular, pro- as the bottleneck of the design development cycle. This is due motes such a process. to a combination of several correlated factors: exponential Genesys, a follow-on of the Model-Based-Test-Generator increase in design complexity, tighter time-to-market require- [1,4], enables the combination of randomness and control, thus ments, and higher quality expectations. In parallel, verification generating a virtually infinite number of high quality tests. It means are not evolving at a matching pace. The cost of the late was primarily developed to minimize the effort to apply it to discovery of the recently found Pentium FDIV flaw (around any architecture, and to allow the usual architectural changes $475,000,000) demonstrates the implications of having a and upgrades to be easily implemented within the tool. Moreo- design that does not totally conform to its architectural specifi- ver, its most powerful property is its ability to be externally and cation [2]. It is therefore not surprising that, for a typical micro- incrementally enriched by Testing Knowledge (TK) in order to processor design project, up to half of the overall resources influence the quality of the generated tests [4]. In this way, cor- spent, are devoted to its verification [3, 4]. This paper suggests ner cases can be assigned a suitable probability to occur, an overall methodology for the functional verification of micro- whereas their chance of appearing randomly would be practi- processors, and explains how this methodology is promoted by cally nonexistent. In fact, Genesys and, in particular, the incre- Genesys, a pseudo-random test-program generator developed at mental TK paradigm, is especially well suited to the the IBM Haifa Research Lab [4]. In addition, it reports some of methodology described in this paper. the insights gained through the application of this methodology The importance of the x86 architecture cannot be over- to x86 microprocessors. stated, and it seems that this importance will not decrease in the Common verification practices include tests obtained from foreseen future. It is therefore of interest to study the results of several sources [6,7]. Firstly, a small fraction of the tests are applying the proposed methodology to the verification of an usually devised manually to target corner cases, or otherwise x86 design.

Functional Verification Methodology 17 December 1998 1

The remainder of the paper is organized as follows: Section Since it is inefficient to run the same regression every day, 2 suggests a functional verification methodology. Section 3 tests run on successive regressions should be different, yet be presents the Genesys system, and Section 4 describes how it able to fulfil the same underlying general purposes. Section 4.1 copes with and suits the implementation of the described meth- describes how this goal is easily achieved using a random test- odology. A case study, done by analyzing the implementation program generator, such as Genesys. of the above methodology on an x86 microprocessor, is reported in Section 5. Section 6 concludes the paper. 2.2.2 Specific Tests Generally many tests produced with significant effort are 2.0 General Methodology worthwhile keeping for periodic reruns. These tests might be written by the designer himself (sometimes manually) to pin- The overall strategy relies on the early composition of a point some hard-to-reach corner cases. It would be inefficient Verification Plan. The Verification Plan will ultimately induce not to exploit these tests and lose the Testing Knowledge they sets of tests that realize the verification tasks. This section sug- include. Section 4.2 shows how Genesys promotes the accumu- gests a framework for composing and implementing a Verifica- lation and further usage of such Testing Knowledge. Addition- tion Plan, independently of the tools available for realizing the ally, tests that have uncovered bugs in the design should be plan. Section 2.1 describes the Verification Plan, whereas Sec- preserved. However, the discovery of a bug should induce a tions 2.2 - 2.4 survey the different means available for its imple- broader set of actions, as described in Section 2.4. mentation. 2.2.3 Coverage Subsets 2.1 Verification Plan To complete the periodic regression set, a library of tests It is obvious that any verification activity should be pre- with some known coverage properties, should be available. Dif- ceded by the composition of a Verification Plan that describes ferent coverage models are possible, the important point being the overall methodology. This includes the description of high that the successful running of those tests should provide a rela- level verification goals leading to detailed verification tasks. tively high degree of confidence. They may be run each time a The plan should be composed from a deep understanding of the major design change is performed. They may be partitioned architecture and the main properties of the micro-architecture. into sets, each having a specific subgoal. In this context, a par- Available verification means (e.g., existing tests, test genera- ticular set might be run any time its corresponding part in the tors, etc.) should not be taken into account at this point, since design is suspected or undergoes a major change. For example, they could bias or limit the scope of the Verification Plan. As a set responsible for the Floating Point Unit might be simulated the design evolves, new tasks are added to cover emerging each time this unit is significantly modified. It is also advisable implementation details. The plan is therefore a living document to run those sets periodically (e.g., each weekend) and not only reflecting the present state of the design, and can also be incre- as a result of external events. However, these tests can not be mented upon discovery of a bug. run too often, since such sets of tests, in order to maintain the coverage level, are inherently huge and time-consuming to run. 2.2 Test Repertory In any event, to maintain efficiency, the use of such coverage The following subsections define the different types of tests subsets should be delayed until the design has reached a rela- required during the verification process. A summary appears in tively stable state. Table 1. It is common practice to purchase existing suites of tests claiming coverage, especially for established architecture such as x86. These suites can also be built using a random test gener- 2.2.1 Periodic Regression Sets (PRS) ator, such as Genesys (see Section 4.1). However, one should Starting early in the design, a test regression should be run keep in mind that there are always bugs which cannot be found frequently in order to yield a reasonable confidence in the over- by any existing static set of tests. all design status. It functions as a kind of “barometer” follow- ing the design evolution from start to end. Consequently, it 2.2.4 Full-Load Testing should be relatively small so that it can run fast, yet be compre- Many intricate design mechanisms are difficult to fully hensive enough to provide useful feedback. These goals are exercise using standard tests. For example, causing a complex conflicting, but can be simultaneously met by directing a PRS pipeline to reach full capacity depends on timing dependencies to all the different capabilities of the design, while providing among several internal signals and is not usually accessible by only loose coverage. regular tests. It is therefore common to directly trigger these The contents of this set should be adapted to the state of the mechanisms by artificially causing them to be in a threshold design, starting from very simple tests and evolving as new state. Standard tests can then be exercised in such a context. capabilities are added. Its running frequency is dependent on the number of problems uncovered, and on the time needed to correct them. Typically, such a regression should be routinely run on a daily basis.

Functional Verification Methodology 17 December 1998 2

2.2.5 Random Testing tional testing and as a reliable measure for deciding when suffi- When the design reaches high stability and bugs become cient confidence has been acquired. relatively difficult to find, the testing described in the previous subsections should be mixed with massive unconstrained ran- 2.4 Bug Driven Activity dom testing. A random test generator is clearly needed for this The discovery of a functional design bug is an important purpose. Randomness is a fundamental property since bug loca- event. There are a few lessons to be gained from it. These stem tions are usually unpredictable. However, one should be aware primarily from the two main properties of design bugs (and of the limitations of random testing. Indeed, pure random test- bugs in general): temporality and locality. This points out that, ing is inefficient since the space domain is enormous and the on the one hand, the test which revealed the bug should be probability of interesting cases is infinitely small. Section 4.3 retained in order to make sure that the bug will not re-occur describes how Genesys avoids the pitfall of pure randomness. (temporality) and, on the other hand, additional tests should be built for adjacent design areas (locality). For the latter task, the 2.2.6 Hardware Testing core of the bug should be understood and serve as a basis for Once the hardware is up and running, some testing can be additional tests which would also tackle related areas. resumed directly on the silicon. Beyond the fact that it is impor- At this point, it is important to point out that the legality of a tant to check the final product itself, the main advantage of run- test often expires. In other words, a test that is perfectly legal at ning tests directly on hardware (as opposed to a simulation some point in time, may become invalid after changes have environment) is that it is much faster. But, since debugging on been made to the design. Since changes are often made during hardware is typically very inconvenient, testing should also the design cycle, the lifetime of a test is typically limited. This continue in the simulation environment. fact points out that important tests, such as bug finders, might Large applications, such as operating systems, are usually need to be “adjusted”. This can be a very time-consuming task. run. Although they are important confidence builders, one Section 4.2. shows how this problem can be overcome using should keep in mind the limited value of such applications in Genesys. terms of verification. They tend to repeatedly exercise a very Finding a bug can also serve as a feedback mechanism for small portion of the design, therefore it would be premature to the Verification Plan. Had the bug been found by some random reach conclusions from running them successfully. testing, it might point out that there is a hole in the Verification In any event, it is profitable at this point to run very long Plan. It should have induced tasks, and thus tests, covering the tests on the hardware. These tests can also originate from a test discovered failure. This could also lead to the addition of new generator. Again, as for massive random generation, the wealth tasks for other related potential failures. of the default Genesys Testing Knowledge (see Section 3.3.3) Section 4.2 shows how Genesys, by generalizing bug data will cause those long tests to have a high verification value. and driving more tests, naturally lends itself to bug driven activ- ity in general, and to the locality property, in particular. 2.3 Verification Process At the beginning of the design process, bugs are easy to 3.0 Genesys find. There is therefore no point in bombarding the design with a large number of tests. This would only overwhelm the verifi- The main objective of this section is to present the basic cation team with a huge number of failures. Similarly, unsys- concepts and properties of the Genesys test-program generator. tematic testing, i.e. running tests without specific properties, is The general structure of the Genesys system is described in equally inefficient. It induces a non homogenous debugging Section 3.1, while Section 3.2 points out its main properties. process, in which some parts of the design arbitrarily receive more attention than others. 3.1 General Structure Testing should be done incrementally to avoid a non homogenous debugging process, which would complicate the debugging. The initial PRS should be very simple, small and systematic. For example, it is recommended to start with tests Architecture Model that are aimed at single instructions and limited to the simplest and Architecture processor mode, and then have the test complexity evolves at a Testing Knowledge Simulator pace which matches the design status. Once relative stability has been reached, coverage subsets can be run. Later, as bugs become rare, massive random testing User Test-Program should begin. Then, when the hardware is running, it will be Interface Generator used for further testing (i.e., long tests, huge applications such Test Programs as Operating Systems). The overall process should be interleaved with some means of coverage. It should serve as a feedback mechanism for addi- FIGURE 1. System components and interactions

Functional Verification Methodology 17 December 1998 3

Genesys is a model-based test-program generator which probability. In Genesys, TK is in the form of Generation and

dynamically generates tests using a generation-simulation cycle Validation functions (written in C) and is held in the external for each instruction. It has been widely used during the last specification model. The most noteworthy property of Genesys seven years at many IBM sites all over the USA. It is also used is that this TK can be incrementally added by the user itself to as a major verification means by several other companies [3]. bias the creation of any test. The TK can be linked to the data, The system has already been employed to model several proc- length or address of any resource and it can influence the gener- essor architectures and to verify their implementation. ation of any event. In such a way, the scope and sharpness of The system consists of three basic interacting components: Genesys’ test-programs are truly unlimited: they are deter- a generic, architecture-independent test generator which is the mined by the investment made by its users. Depending on how engine of the system, an external specification (the model) it is defined in the specification model, TK can be taken into which holds a formal description of the targeted architecture, account during random generation (then called default TK) or and a behavioral simulator which is used to predict the results only when specifically requested (called specific TK). The Test- of instruction execution (see Figure 1 in which the User Inter- ing Knowledge paradigm is reported at length in [1]. face is depicted as the fourth block). The user can control test generation by specifying desired biasing towards special 3.2.3 Model-Based events. This biasing can also be saved to a user directives file. Genesys relies on a formal and declarative model of the The external specification model also allows incorporation architecture captured separately in its architecture model com- of testing knowledge. It is employed in order to generate prob- ponent. This structure allows to conveniently integrate architec- ing test cases and allows expert users to add knowledge to the ture changes, which are so common during design system in a local and relatively simple manner. development. More generally, it enables to adapt Genesys to a 3.2 Major Properties wide range of different architectures, with minimal effort. The latest trend of designing a hybrid microprocessor (e.g., Merced The following subsections describe the main properties of which mixes x86 with VLIW) emphasizes the importance of Genesys, especially stressing those which will be of primary having a tool which is easily customized to any architecture. importance for implementing the verification methodology. 3.2.1 Control 3.2.4 Short Tests Contrary to typical test cases which assume initiation from Genesys enables the creation of programs ranging from the Reset state of the design, Genesys allows you to start a test completely deterministic to totally random programs. By the from any (legal) state. It enables you to bring the design means of a directives file, control is given to guide the genera- directly into a state which otherwise would have been relatively tion to the desired extent, while any parameter not explicitly hard to reach. The sole output of the generator is a test file constrained is randomly set to any consistent value. The ability which consists of a sequence of instructions starting from a to target the whole range of test randomness is of key impor- given initial state, and a section of expected results describing tance for implementing the proposed methodology. Namely, the the expected values of the various processor resources. Both methodology induces the creation of fully random tests, spe- properties, initial state and expected results, contribute to the cific scenarios, and test templates with varying levels of ran- production of short, easy-to-debug and incisive tests. domness (see Section 4).

3.2.2 Testing Knowledge 4.0 Implementing the Methodology Using Another major feature of Genesys is its ability to easily Genesys acquire additional Testing Knowledge (TK). The incorporation The goal of this section is to demonstrate how Genesys inte- of TK into a random generator enables you to adjust the distri- grates and promotes the methodology described in Section 2. bution of the probability of the targeted test space. As a simple Although a large part of this methodology is independent of the example, the result of zero for an ADD instruction is typically particular test generator used, some of the most noteworthy of special importance while its relative probability to occur ran- characteristics of Genesys contribute significantly to the suc- domly is practically inexistent. Informing the test generator that cessful implementation of the proposed methodology. More the result of zero is important, and should thus be generated specifically, even though all the major properties appearing with a reasonable probability, is an example of the process of under Section 3.2 contribute to the overall quality of the verifi- adding TK to the generator. As a more advanced example, cation process, two of them will be shown to be particularly many architectures define instructions with two memory oper- important: first the ability to keep directives files instead of ands (e.g., string instructions in x86). Having the two memory tests, and second the possibility to incrementally and externally locations overlap is usually an exceptional event rarely occur- add testing knowledge to Genesys. Instead of keeping tests, ring randomly, and worthwhile to add as TK to the generator. directives files, which may have varying degrees of control, More generally, adequate weights should be given to corner impart the desired randomness and solve the test legality prob- cases which otherwise would be occurring with negligible lem. More exclusive to Genesys, the testing knowledge frame-

Functional Verification Methodology 17 December 1998 4

work enables to optimize the quality of each of the stages the feature will continue to be tested randomly, within many induced by the methodology. In the following subsections, we different contexts. will follow these stages and explain the role of Genesys in Second, when a bug is found by chance, not as a result of them. the scheduled verification tasks. There is some similarity between new design features and such a bug discovery. In both 4.1 Getting Started cases, a new corner case is identified. Therefore, upon bug dis- Given a deep understanding of the architecture and an over- covery, the TK should be increased as well. In addition, to view of the principal design properties, the first goal is to com- tackle the locality property (Section 2.4), once the essence of pose a Verification Plan. At this point, all of the different the bug is distinctly grasped, it should be captured in a Genesys expected behaviors, including every corner case, are identified directives file, leaving all other biasing directives random. Con- and listed as targets for testing. They should be translated into sequently, this directives file can be seen as a generic engine to TK, including default TK, within Genesys, especially if their target not only the bug, but all the nearby design regions sur- probability to appear at random is relatively low. In such a way, rounding the bug as well. Incidentally, keeping directives files all the different identified behaviors are easy to target, yet will (instead, or in addition to tests) also solves the “test legality appear randomly with a reasonable probability. expiration” problem described in Section 2.4. Then, there is a need for the development of the periodic regression set. As indicated previously, it is desirable for such a 4.3 Massive Testing test set to be different in successive runs. This can be achieved In later stages of the design process, massive random testing by composing the set using Genesys directives files instead of is performed both in the simulation environment and directly on static tests. In such a way, a countless number of different tests hardware (given an appropriate interface). The default TK can be induced. Moreover, randomness will be profitably added present in Genesys allows you to tackle all the different design to the process. behaviors, even though no effort is done to control the genera- It should be observed at this point that having short tests, tion process. Genesys allows the combination of quality and due to the initial state and expected results structure, makes the quantity, thereby achieving a large number of smart cycles. entire debugging process much more efficient. This is true throughout the verification process, but it is of particular impor- tance in the beginning stages. Contrary to standard self-check- 5.0 Case Study: An x86 Design ing tests that start from Reset, simple and short tests should focus on elementary goals. There is no need to get involved in Due to its preponderance and complexity, the x86 architec- complex Reset sequences and redundant self-checking seg- ture is a suitable candidate for corroborating the methodology ments. suggested in Section 2. This methodology was applied during Following periodic regression, coverage subsets should the development of an x86 microprocessor within IBM, and become available. A random test generator, capable of generat- relied on the recent availability of Genesys-x86, a Genesys- ing a countless number of different tests, can assist in the elabo- based pseudo-random test generator for the x86 architecture. ration of such subsets. Given a coverage model, massive Although genericness is one of Genesys’ properties, significant generation can be filtered to keep only those tests involving new effort was needed to support a complex architecture such as the coverage tasks belonging to this coverage model. As an exam- x86. One of the main challenges presented to Genesys by the ple for a coverage model, a subset can be built by using stand- x86 architecture was the presence of instructions with variable- ard software techniques to cover the code of an architecture size and complex addressing modes [8]. Section 5.1 reports the simulator (such as the one present in the Genesys system). principal insights gained. Section 5.2 demonstrates how Testing Knowledge encapsulation could have prevented the recent Pen- 4.2 Testing Knowledge Encapsulation tium bugs. Both the Verification Plan and the Testing Knowledge 5.1 Applying the Methodology to x86 Verification evolve together with the design. They must be upgraded on two main occasions. First, when a certain architecture behaviour is The main insight gained is that confining verification to being implemented in the design in an unpredictable manner. (usually purchased) test suites and long applications such as Nothing in the architecture book hints that this behaviour operating systems, is far from sufficient. The full test repertory requires special attention. This event is of particular importance described in Section 2.2 is needed for thorough verification. as it is a significant source of numerous escape bugs. It is obvi- Figure 2 displays the number of bugs uncovered at one of ously the designer’s duty to convey the information to the veri- IBM sites by Genesys-x86 tests, in comparison to tests origi- fication team. Thus, beyond the elaboration of a few tests by the nating from different test sources but mainly commercial test designer itself, this should cause the Verification Plan to be suites (the upper curve being Genesys). Genesys’ tests come updated, and several tests to be created around the newly identi- from both the periodic regression and the massive random test- fied corner cases. Genesys, in particular the TK paradigm, pro- ing. It should be noted that Genesys-x86 was employed after vides very efficient groundwork for dealing further with such the test suites had been comprehensively exercised on the cases. By directly upgrading Genesys with this additional TK, design simulation environment. This highlights the fact that the

Functional Verification Methodology    17 December 1998    5

bugs found by Genesys-x86 might not have been found other- observation can be easily translated into Testing Knowledge wise, at least not within an appropriate time frame. It is note- within Genesys. Thus, analysis of bugs, in an intra-architecture worthy to imagine the erroneous conclusions (and their manner, leads to the definition of what can be called generic implications) which would have been reached from relying Testing Knowledge. Incidentally, having the long bit sequence only on bug curves other than Genesys’. It was surprising to TK in Genesys-x86 causes the tool to generate tests which witness the simplicity of some of the bugs left by those suites uncover both Pentium bugs with a reasonably high probability. and later discovered by Genesys-x86. For example, a regular Table 2 below shows a range of values which cause the Pentium jump to high memory (beyond 216) revealed a bug showing that II FIST32 bug (‘x’ and ‘y’ can be replaced with ‘0’ or ‘1’) [9]. this simple operation was not exercised by any of the suites. A similar sequence also caused the FIST16 bug. This is partly due to the lack of randomness inherent in any The probability of randomly generating such a mantissa for given static set of tests. Whatever their size and claimed cover- a normal floating point number is 1:232, as you need a 31 ‘0’ age, there are always simple bugs which stay untargeted. Of sequence with a ‘1’ before it (the leading ‘1’ is a must for a nor- course, Genesys-x86 also revealed many bugs resulting from mal FP number). Using the bit sequence pattern, the probability the complex interactions of different instructions and even units to hit this type of mantissa drops to approximately 1:27. For the of the design. As an example of a more subtle bug, Genesys exponent part, using the bit sequence pattern reduces the proba- generated a test where a MTCR0 changing the processor mode (from protected to real) appeared in a non taken leg of a JMP bility from 1:215 to  15  or 1:910 taking into account the sign instruction. This mode switching was not correctly cancelled  12  upon returning to the right instruction stream. bit. Hence, the probability of getting this event using the bit The hardware was functional at first tape-out and success- sequence pattern is around 1:217 (~1:117,000). As a result, one fully ran DOS and other operating systems. As expected, the should expect to find this type of FP bug after generating a few test subsets coming from Genesys kept finding bugs at a high hundred thousand FIST32 instructions when the bit sequence rate after this stage. Again, the reason is that, even though run- pattern is used, whereas, using pure random generation, the ning large applications such as operating systems is undoubt- edly a good confidence builder and a threshold which has to be probability to hit the bug is 1:249 which entails the unrealistic successfully passed, they do not exercise a significant portion need to generate thousands of billions of FIST32 instructions. of the architecture. Many areas are left untouched, leaving potential bugs untargeted. In particular, many different initial 6.0 Conclusions states proposed by Genesys’ tests are never reached. A rigorous methodology for tackling functional microproc- 5.2 Pentium Bugs essor design verification has been described. The principal con- It is interesting to check whether the described methodol- tribution of this paper is to set the general guidelines for ogy and Genesys-x86 would have prevented known escape obtaining a compound functional verification framework, and bugs such as the last 2 Pentium FP bugs (the FDIV bug and the to describe how this can be optimally implemented by a test- FIST bug [9]). Both were implementation dependent bugs program generator such as Genesys. The goal of the paper is to which could not have been suspected from merely reading the assist in the composition of verification processes which typi- Pentium architecture book. Consequently, a mandatory precon- cally include only part of the testing suggested in this frame- dition for uncovering these bugs is that the unsuspected algo- work. Although related and having a recognized importance, rithm implemented by the designer is revealed to the the fields of formal verification and coverage are beyond the verification team, but this is exactly the requirement described scope of this paper. at the beginning of Section 4.2. We see then that such expensive It has been shown how an incomplete methodology applied bugs reflect a hole in the verification process. It could be a to x86 microprocessors, i.e. running only existing suites of tests methodology problem suggesting a lack of coordination and long applications, or missing coordination between design- between the designer and the verification team, or it could stem ers and the verification team, can lead to false conclusions on from a lack of means to sufficiently integrate the newly the design’s correctness. In addition, the two main properties of revealed algorithm within the verification process. In any event, Genesys, namely external Testing Knowledge and comprehen- Section 4.2 explains how the suggested methodology would sive control through directives files, have been shown to natu- have dealt with these problems by introducing the implementa- rally suit the described functional verification methodology. tion-specific information into Genesys’ Testing Knowledge. Nevertheless, there is another less obvious way through which Genesys-x86 might have discovered these bugs. Accu- mulated experience has shown that there are some cases which are typically more bug-prone in design. For example, although in general it is not clear from the architecture definition, long sequences of 0’s or of 1’s in source operands or in intermediate results, are often the source of bugs in today’s FP designs. This

Functional Verification Methodology 17 December 1998 6

                                                                                     Design
Type              Description                                                        Stage               Goal
Periodic          A frequently run test regression. Small but comprehensive          Starting early in   Barometer of the
Regression        enough to provide useful feedback on the design status. Should     the design          design status
Set (PRS)         be adapted to the state of the design.
Specific Tests    Tests which pinpoint some hard-to-reach corner cases.              Small number        Verify corner cases
                                                                                     of bugs in PRS
Coverage Sub-     A library of tests having some known measure of coverage.          Design is rela-     Quality checker.
sets              Should be used each time a major design change is performed. A     tively stable       Assurance vs.
                  subset for a given part (such as FPU) may be run when the part                         changes
                  changes.
Full-Load Test-   Standard tests run after the design was artificially put under a   Unit verification   Verify the unit in a
ing               full-load state, such as a full complex pipeline.                                      full-load state
Random Test-      Massive unconstrained random testing produced by a random          Design is stable    Overall design
ing               test program generator.                                                                check. Final search
                                                                                                         for bugs
Hardware          Running large applications such as operating systems, and mas-     Hardware is         Final verification on
Testing           sive random generation of long tests.                              functional          product itself
TABLE 1. Type of tests and their usage





                         Bugs found by various verification means
90
80                       Genesys
                         Manual Testing
70                       Commercial Test Suites
60                       Applications

50

40

30

20

10

                  00 1 2 3     4     5     6   7                                     8 9                 10
                             Weeks



FIGURE 2. Comparative evolution of bugs found at an IBM site (Genesys-x86 vs. other verification means).

Sign Exponent Mantissa 1 100000000100100 1xxxxx100000000000000000000000000000000yyyyyyyy TABLE 2. Template for an FP number which uncovers the Pentium II FIST32 bug

Functional Verification Methodology    17 December 1998                                                  7

Bugs

References.

[1] Y. Lichtenstein, Y. Malka and A. Aharon, “Model-Based Test Generation For Processor Design Verification”, Innovative Applications of Artificial Intelligence (IAAI), AAAI Press, 1994. [2] H.P. Sharangpani, M.L. Barton, “Statistical Analysis of Floating Point Flaw in the Pentium Processor”, Intel Corporation, 1994. [3] F. Casaubieilh et al., “Functional Verification Methodology of Chameleon Processor”, 33rd Design Automation Conference, Las Vegas, June 1996, pp. 421-426. [4] A. Aharon, D. Goodman, M. Levinger, Y. Lichtenstein, Y. Malka, C. Metzger, M. Molcho, and G. Shurek, “Test Program Generation for Functional Verification of PowerPC Processors in IBM”, 32nd Design Automation Conference, San Francisco, June 1995, pp. 279-285. [5] A. L. Sangiovanni-Vincentelli, P. C. McGeer, A. Saldanha, “Verification of Electronic Systems”, 33rd Design Automation Conference, pp. 106-111. [6] J. Monaco, D. Holloway, R. Raina, “Functional Verification Methodology for the PowerPC 604 Microprocessor”, 33rd. Design Automation Conference, pp. 319-324. [7] M. Kantrowitz, L. M. Noack, “I’m Done Simulating; Now What?”, 33rd Design Automation Conference, pp. 325-330. [8] D. Lewin, L. Fournier, M. Levinger, E. Roytman, G. Shurek, “Constraint Satisfaction for Test Program Generation”, IEEE 14th Phoenix Conference on Computers and Communications, 1995. [9] Intel Pentium II flag erratum, Intel home page at: http:// developer.intel.com/design/news/flag/tech.htm [10] G. Ganapathy, R. Narayan, G. Jorden, D. Fernandez, “Hardware Emulation for Functional Verification of K5”, 33rd Design Automation Conference, pp. 315-318.

Functional Verification Methodology 17 December 1998 8