SiliFuzz: Fuzzing CPUs by proxy

Kostya Serebryany Maxim Lifantsev Konstantin Shtoyk Doug Kwan Google Google Google Google kcc@google.com maxim@google.com kostik@google.com dougkwan@google.com Peter Hochschild Google phoch@google.com

Abstract 1 Introduction

CPUs are becoming more complex with every gen- When discussing misbehaving CPUs we distinguish eration, at both the logical and the physical levels. between logic bugs and electrical defects. A logic This potentially leads to more logic bugs and electri- bug is an invalid CPU behavior inherent to a partic- cal defects in CPUs being overlooked during testing, ular CPU microarchitecture or stepping, e.g. [2, 3, which causes data corruption or other undesirable 4]. An electrical defect is an invalid CPU behavior effects when these CPUs are used in production. that happens only on one or several chips. Often, These ever-present problems may also have simply but not always, an electrical defect affects only one become more evident as more CPUs are operated of the physical cores on the chip. A defect may be and monitored by large cloud providers. present in a CPU from day one (if it was missed by the vendor during testing) or it may be a result of If the RTL (“source code”) of a CPU were avail- physical wear-out (e.g., circuit aging). For the pur- able, we could apply greybox fuzzing to the CPU poses of this paper the distinction between day-one model almost as we do to any other software [1]. defects and wear-out is not important. However our targets are general purpose x86_64 The term “Silent Data Corruption” (SDC) has CPUs produced by third parties, where we do not been coined [5, 6, 7, 8] to represent CPU defects have the RTL design, so in our case CPU imple- (and bugs) that do not cause an immediate observ- mentations are opaque. Moreover, we are more able failure, but instead silently lead to incorrect interested in electrical defects as opposed to logic computation. This phenomenon has been known for bugs. a very long time, and is directly related to concepts We present SiliFuzz, a work-in-progress system like fault-secure operation, data integrity checking, that finds CPU defects by fuzzing software proxies, and silent errors. In many production environments like CPU simulators or disassemblers, and then ex- SDCs are more dangerous than crashes, because ecuting the accumulated test inputs (known as the most server software ecosystems are built with toler- corpus) on actual CPUs on a large scale. The major ance to crashes, but not with tolerance to unobserved difference between this work and traditional soft- data corruption. Not all CPU defects induce SDCs, ware fuzzing is that a software bug fixed once will but many do. be fixed for all installations of the software, while In this paper we present SiliFuzz, a system for CPU defects we have to test every individual for finding CPU defects. It uses coverage-guided core repeatedly over its lifetime due to wear and fuzzing of software CPU simulators and disassem- tear. In this paper we also analyze four groups of blers to generate a representative set of tests for CPU defects that SiliFuzz has uncovered and de- CPUs (known as the corpus). We call this approach scribe patterns shared by other SiliFuzz findings. “fuzzing by proxy.” Our underlying assumption is

    1

that if a test set provides good code coverage for the   ering as many corner cases as possible.

proxy, these tests are also likely to trigger interesting • Execution (§2.5): A system to check every core CPU behavior, and potentially uncover defects. We of every machine in the fleet. treat our test targets, CPUs, as opaque—we do not • Feedback loop (§2.6): A feedback loop between make any assumptions about the underlying struc- known bad machines and SiliFuzz. ture of a CPU, nor do we use any vendor-internal debugging capabilities. Similar approaches have SiliFuzz works completely in user space and thus been used before for other targets. For example, a does not directly test kernel-specific instructions. test set generated by fuzzing the Linux kernel USB subsystem was used to find bugs in the Windows ker- 2.1 Fuzzing and Coverage nel USB subsystem [9]. Our approach is also a kind of differential testing [10], since we cross-validate First, we need to describe the basics of software different CPU cores on many machines. fuzzing. Fuzzing is a technique of testing a target We do not claim novelty in the academic sense— (an application, or an API) with a large number of all components of our system, such as fuzzing hard- test inputs generated on the fly. The goal is to make ware as software and fuzzing an opaque system via these inputs as interesting and as diverse as possi- proxy, have been known before [1, 9]. We also do ble in order to trigger corner cases. In other words, not compare our system to others that are similar, as fuzzing aims to maximize the combined code cov- we do not have access to any such systems. Further- erage. “Code coverage” may have different mean- more, the system we present here is early work in ings, from basic block coverage to bounded path progress and it needs more work to reach maturity. coverage. In this paper we use “coverage” as a syn- Nonetheless, SiliFuzz is already producing results onym of software control flow edge coverage. The and we feel it is important to share our results with two most popular approaches to software fuzzing a wider audience. are generation-based fuzzing and coverage-guided The rest of the paper is organized as follows. We mutation-based fuzzing. start with an overview of SiliFuzz in Section 2, pro- Generation-based fuzzing (e.g. for software: [11, vide a brief evaluation in Section 3, cover the re- 12], or for hardware: [13, 14]) randomly creates lated literature in Section 4, and discuss the future an infinite number of test inputs, typically based direction of work in Section 5 before concluding. on knowledge of the input grammar. Each new Appendices A-D describe four groups of defects input is produced from scratch, not using any kind found in CPUs from two major x86_64 vendors. of feedback from the system under test. This is frequently called blackbox fuzzing. Coverage-guided mutation-based fuzzing starts 2 SiliFuzz Overview from a seed corpus of test inputs and applies small random mutations and crossovers to random ele- In this section we describe the SiliFuzz system in ments of the corpus. The mutants are executed by its current work-in-progress form. It consists of the the target application and the coverage feedback following components: is collected. If a given mutant triggers previously • Preparation: unseen coverage it is added to the corpus. Guided fuzzing is typically performed by general-purpose – Snapshot (§2.2): A format for representing and fuzzing engines not developed for one specific tar- serializing short, self-contained programs and get, such as libFuzzer [15], AFL [16], or Hongg- their reference output(s). fuzz [17]. This is frequently called greybox fuzzing. – Player (§2.3): A program for executing snap- A variation of coverage-guided fuzzing, often shots one-by-one, on a CPU or a simulator. called structure-aware fuzzing, requires that the mu- – Snapshot generation (§2.4): A pipeline for gen- tations are applied to inputs in a way that preserves erating sets of snapshots, with the goal of cov- validity and syntactical correctness. Syzkaller [18]

          2

  is one such fuzzing engine. libFuzzer allows users to    commands and processing execution results. The
define their own mutators, turning it into a structure-    harness performs commands issued by the driver
aware fuzzing engine.                                      including running the actual code. Having two pro-
            Many fuzzing engines, blackbox and greybox,    cesses provides the necessary isolation between the
      use dictionaries—user-defined byte sequences that    driver and potentially crashing code of the snapshot
are likely to trigger interesting behaviors if inserted    being played.
into the test inputs.                                           Consider a simple snapshot consisting of a sin-
                                                           gle NOP instruction placed at the memory address
2.2 Snapshot                                               0x10000000. It will cause the following sequence
    A SiliFuzz snapshot describes instructions and data    of commands to be generated by the driver and exe-
      necessary for executing some relatively short se-    cuted by the harness:
       quence of binary code. Our typical snapshot con-    1. MapMemory          { start = 0x10000000,
      tains less than 100 bytes of code and runs in mi-                             num_bytes = 4096 }
croseconds, but it can be arbitrarily large.      Valid    2. WriteMemory        { start = 0x10000000,
     snapshots do not contain syscalls or other sources                         data = "\x90\xCC\x00. . .",
       of nondeterminism. They are executed in a single                             num_bytes = 4096 }
thread.                                                    3. ProtectMemory      { start = 0x10000000,
 A snapshot consists of the following parts:               4. ExecuteSnapshot   perms = r-x }
    • Initial state of all user readable CPU registers,        { registers = < . . .,
 including the program counter.                                                 rip = 0x10000000 > }
        • Memory mappings with their read/write/execute    5. ChecksumMemory { start = 0x10000000,
          permissions and the corresponding data bytes.                             num_bytes = 4096 }
      Note that snapshots expected to cause page faults        The \xCC byte is an x86 trap instruction marking
 require that particular pages not be mapped.              the end of the snapshot. The rest of the memory

• Expected end state(s) of the execution. If a snap- page at 0x10000000 is filled with zeroes to guaran- shot executes differently on different CPU mi- tee deterministic execution. croarchitectures it will have multiple end states. If the execution causes a CPU exception this is 2.4 Snapshot Generation captured as the corresponding signal number and signal address. SiliFuzz leverages open-source x86_64 CPU em- • Additional metadata describing how this particu- ulators and disassemblers as proxies to generate lar snapshot was produced (e.g. whether it was snapshots with the help of libFuzzer. The intuition captured from a program, produced by combining behind this approach is that if a particular input several snapshots, etc). sequence triggers interesting behavior (new cover- age) in the proxy system then the same input may 2.3 Player also trigger interesting behavior in a real CPU. We also generate random instruction sequences with The player is a program that can replay a snapshot, ifuzz [19]. capture the outcome and determine if the outcome matches one of the expected end states. We pin the 2.4.1 XED execution to a particular core. A mismatch indicates a potential defect in the CPU core that executes the XED [20] is an x86_64 instruction encoder/de- snapshot. The player is separated into two processes coder. It can decode up to 15 bytes of data into one referred to as the driver and the harness. The driver x86_64 instruction at a time, and report the number is a stateful execution orchestrator. It performs op- of decoded bytes. As such, it makes it a good target erations like transforming snapshots into a series of for fuzzing with libFuzzer (see Listing 1).

     3

    Listing 1: XED libFuzzer integration

int LLVMFuzzerTestOneInput(const uint8_t *data, size_t data_len) { xed_decoded_inst_t xedd; xed_tables_init(); xed_decoded_inst_zero(&xedd); xed_decoded_inst_set_mode(&xedd, XED_MACHINE_MODE_LONG_64, XED_ADDRESS_WIDTH_64b); xed_decode(&xedd, data, data_len); return 0; }

The result of fuzzing XED is a corpus of byte se- most of the inputs added to the corpus are those that quences which represent valid and non-valid x86_64 trigger new coverage in Unicorn and are thus mostly instructions. The corpus maximizes the coverage valid instruction sequences. of XED itself, and thus likely triggers interesting behaviors in the CPU instruction decoder or other 2.4.4 Snapshot Fixing and End State Recording components. Appendix B is a demonstration of such interesting behavior. Fuzzing via XED, ifuzz, and Unicorn does not produce snapshots directly—rather, it produces in- 2.4.2 ifuzz struction sequences. These sequences may contain syscalls and other sources of nondeterminism, and The ifuzz library [19] is a part of the Syzkaller perform invalid memory access or contain illegal project. It provides both generation and mutation instructions. They also lack a well-defined end state. facilities for x86_64 instructions based on a model We apply a number of techniques to convert these of the x86_64 ISA derived from XED. When used instruction sequences into proper snapshots. These in generation mode, ifuzz only creates random valid can be broken down into several stages: instruction sequences. Since ifuzz is not guided by 1. Adding and removing memory mappings. We coverage, it essentially produces an infinite number intercept memory page faults and map a limited of outputs with no way to choose a good subset. number of additional pages accessed by a snap- We use a sample of ifuzz-generated instruction se- shot. The mapped pages are currently filled with quences in two ways: first, we create snapshots from zeroes. them to test on real CPUs and second, we use them 2. Nondeterminism elimination. We eliminate as a dictionary for fuzzing Unicorn. all system calls by tracing all snapshots. We 2.4.3 Unicorn then verify that every snapshot successfully and deterministically replays multiple times on many Unicorn [21] is a lightweight multi-platform, multi- machines. Each snapshot is limited to 3 seconds architecture CPU emulator framework based on of CPU time by default. QEMU. We applied libFuzzer to Unicorn in a 3. End state determination. Finally, each snap- setup similar to the fuzz target available in the shot is run on all supported Google produc- Unicorn source tree [22] to generate instruction tion platforms (microarchitectures) and the per- sequences. In essence, this fuzz target maps the platform end states are captured. Currently, we fuzzer-generated data at a fixed address and starts only keep snapshots that have identical end states simulated execution from the first byte. on all microarchitectures as additional protection libFuzzer is oblivious to x86_64 instruction en- against hard-to-detect sources of nondetermin- coding rules and the nature of how the result is ism. processed. However, during the fuzzing process We do not eliminate illegal instructions, primarily

    4

because there is no way to understand whether an          SiliFuzz is chosen, job scheduling software on the
instruction is illegal without executing it on all CPU      machine reserves 4 cores (2 hyperthreaded pairs)
generations. Appendix B demonstrates another rea-        for the test and allows SiliFuzz to run for up to 2
son for not removing illegal instructions.              minutes.     The core count, time limit, and testing
                                                        frequency are chosen for all tests to keep screening
2.5    Execution at Scale                                overhead below a certain percentage of machine uti-
2.5.1   SiliFuzz Checker                                  lization. The job scheduling software also ensures
    the 4-core window eventually covers all cores on the

For running SiliFuzz at scale, we created a program, machine. After SiliFuzz finishes, results are logged SiliFuzz checker, that efficiently runs multiple snap- into a machine database for analysis. shots. This checker is used in background testing of production machines, described below. The checker 2.6 Using Known Bad Machines for Feed- reads snapshots from a corpus residing on a disk, back materializes snapshots in memory and sends them to a pool of snapshot players, one for each core Machines that are believed to be defective are taken being tested. Our current corpus consists of very out of production and set aside for detailed diagnos- short snapshots. Reading a snapshot from a file tics. These machines are put into a quarantine pool system and re-materializing it in memory is there- so that we can run a number of tests on them. Test- fore expensive compared to the execution time of ing usually takes weeks for a machine since some the snapshot. To amortize the cost of reading snap- failures are dependent on frequency, voltage and shots, the checker groups snapshots in batches and temperature and we need to vary these parameters reuses snapshots in a batch multiple times. Multi- to run our tests with different combinations. The ple batches are also executed concurrently to hide quarantine pool is a valuable tool for isolating defec- I/O delays. In our current setting, we batch 50 ran- tive machines in our fleet but it is also an important dom snapshots together and create a random list of tool for SiliFuzz development. It allows us to: 1000 snapshot executions out of the batch using a uniform random distribution. These numbers are • Compare SiliFuzz to other SDC detection tools picked empirically to reduce the reading cost by a and measure its relative performance and ef- factor of 20. Execution of snapshots on the list are fectiveness. The population of the pool is not distributed evenly across all available cores using constant—machines enter and leave the pool over multiple queues. time, so we have to perform the comparison regu- larly. 2.5.2 Background Testing for Production Ma- • Avoid SiliFuzz regressions. In addition to com- chines paring SiliFuzz against other tools, we can also compare different versions of SiliFuzz. This is To efficiently screen the fleet, SiliFuzz needs to especially useful for regression testing. spend a non-trivial amount of time on every core of • Enrich the corpus via known bad machines. Our every machine. Machines in Google’s data centers hypothesis is that a snapshot that uncovers one de- are continuously screened for SDC defects with a fect on one machine has a good chance to uncover collection of tests, including SiliFuzz. This is done some defect on another machine. Our data has an while machines are in use in production. To mini- early indication supporting this hypothesis. So, mize the performance impact of this screening, only we are preserving such snapshots in our corpus, a small number of machines in a cluster are being regardless of their proxy coverage. tested at any given time. On a machine being tested, • Establish defect patterns. The quarantine pool one of the testing tools, for example SiliFuzz, is serves as a source from which defect patterns picked randomly based on a probability distribution can be extracted. Some failures recorded there customized for the machine’s architecture. When are analyzed by a human to see if there is any

           5

commonality. remaining 40% of the machines were detectable The pool includes machines where the Jump Con- by both types of snapshots. This suggests that the ditional Code (JCC) [2] issue can be reproduced. A two types of snapshots are largely complementary. specially crafted snapshot can trigger the problem Our insight is that CPU defects are common enough with high probability allowing us to continuously that any sufficiently non-trivial fuzzing will uncover and reliably test the SiliFuzz infrastructure. some defects when applied at large scale. The twice as many defects found by fuzzing trivial proxies 3 Evaluation compared to using a sophisticated generation-based fuzzer suggests that fuzzing by proxy is worth fur- ther investigation, especially searching for better Our current corpus contains snapshots generated by proxies. fuzzing Unicorn and XED with libFuzzer and a num- Analyzing CPU defects is a labor-intensive task ber of random snapshots produced by ifuzz—about that is often hindered by poor reproducibility and 500,000 in total. The corpus has been deployed to a complicated processes around getting access to spe- large portion of the Google production fleet and de- cific production machines. In Appendices A-D we tected multiple faulty machines. We only consider describe four groups of defects that we analyzed a machine defective if we are able to reproduce the in detail. The first three affect more than one ma- same problem on the same core(s) over multiple chine each. The fourth defect falls into a pattern that days. In about 70% of the cases a single pair of we observe on multiple machines, but the precise logical cores sharing the same physical core is behavior is unique. Defect A was uncovered by a affected. In most other cases only a single logical Unicorn-produced snapshot, Defect B was uncov- core is affected. In just a handful of cases we ob- ered by a XED-produced snapshot, and Defects C serve many different cores failing. Some snapshots & D were independently uncovered by ifuzz- and have detected defects on more than one machine, al- Unicorn-produced snapshots. We cannot guarantee though those defects are not necessarily equivalent. this attribution with certainty, because early in the We were able to group some of the defects detected project we did cross-pollination between the three by different snapshots based on similar behavior corpora. Going forward, we will need to more care- and likely similar causes. fully track attribution of defects in the presence of Some machines flagged by SiliFuzz have been cross-pollination. independently confirmed as defective by other tests. We have detected many other defects that we Others have been implicated in production incidents. did not analyze in detail, however we noticed some Finally, about 45% of SiliFuzz findings are unique common patterns: and have not been previously identified by any other tool or automation available to us. The data we • Wrong results for integer, floating point and vec- collected fleetwide shows a wide distribution of tor instructions. Often the result has just one in- time-to-failure (CPU time consumed by SiliFuzz correct bit. on a given machine before detecting the first defect) • Spurious SIGFPE, SIGSEGV, or SIGILL. with some machines failing almost immediately and • Missing floating point exception. others taking days to detect a single defect. • No SIGILL where one should be raised according We have observed that our approach of fuzzing to the specification. by proxy finds more defects compared to purely • Sticky flags, i.e. certain bits in EFLAGS or similar random snapshots produced by ifuzz. Preliminary status registers are always set despite the lack of data collected from the quarantine pool shows that the underlying condition. from the machines identified by SiliFuzz, about 40% • Undershooting REP(NE)-prefixed string instruc- were uniquely identifiable by snapshots produced tions. from Unicorn and XED. About 20% were similarly • Floating point data pointer register (RDP) value identified only by ifuzz-produced snapshots. The mismatch.

    6

We do not know whether these common patterns Generation-based fuzzing has been applied to are representative of all CPU defects, or are sim- hardware for several decades [14, 13]. However, in ply easy to find using our technique. However we the past, it typically did not utilize coverage feed- observe a correlation between types of defects we back to guide mutations. More recently, Trippel et find most frequently and the parts of Unicorn that al. [1] have translated an RTL design into a software are implemented as C code (floating point, vector model and applied off-the-shelf coverage-guided instructions, etc) as opposed to being just-in-time software fuzzing engines to the model. Our work compiled, and our method provides better coverage uses a real CPU as the fuzz target, for which we feedback for the C code, than for the JIT-compiled do not have the original RTL design. A similar code. approach to ours was adopted by Martignoni et al. [30] but in the opposite direction. They used a real 4 Related Work CPU as the golden reference to perform differential testing to find bugs in CPU emulators. The SDC problem with electrical defects in micro- While pre- and post-silicon manufacturing tests processors was known from at least the 1960s. It are performed by CPU vendors before CPUs are became more widely known at least two decades delivered to customers [31, 32, 13, 14, 33], SiliFuzz ago [23, 5]. Today, for large service providers own- aims to find defects in CPUs already deployed in ing multiple data centers, SDC-inducing defects production systems. Crucially, unlike system-level are a very real problem. Facebook discusses their testing performed by CPU vendors, SiliFuzz has SDC issue in [6] while Google provides a high level to work without the vendors’ expertise, tools, and overview of SDC in [7]. knowledge and instead leverage a large fleet of de- Miller, Fredriksen, and So [24] first introduced ployed CPUs at scale—which often is infeasible in fuzz testing. Since then, there has been much pre-production verification. progress in this area. Readers can refer to [25] for a recent survey of fuzzing techniques. Our work 5 draws on some previous work. In essence, SiliFuzz Future Work is a form of differential testing [10], which uses identical test inputs to discover differences between We are just starting our work on finding CPU defects. test subjects. It also heavily relies on advances in Future work will be concentrated along four axes: coverage-guided mutation-based (or “evolutionary”) scale, speed, automation, and quality, with quality fuzzing [26]. being the most challenging. There has been some previous work in fuzzing CPU hardware. Sandsifter [27] uses depth-first- Scale search-based fuzzing and page fault analysis to exhaustively determine x86 instruction lengths on We need to run on as many cores for as much time CPUs from various vendors. This is done to de- as possible. This is expensive, and we are looking tect undocumented instructions and hardware bugs. for ways to reduce the cost. Sandsifter focuses on instruction encoding as op- posed to validating the functionality. Easdon [28] describes a way of detecting undocumented instruc- Speed tions only accessible to an operating system’s ker- nel. UISFuzz [29] is another hardware instruction The player needs to be further optimized to execute set fuzzer for undocumented instructions similar to more snapshots per second, and reduce syscalls and Sandsifter. It uses instruction format information I/O operations. Slow-executing snapshots need to to speed up search. Like Sandsifter, it focuses on be removed without sacrificing the combined proxy instruction encoding. code coverage.

    7

Automation of software has 100% control flow edge coverage, Detected defects need to be automatically analyzed, it may still have bugs not detectable by the test cor- deduplicated and reported. Defective machines need pus. Complete code coverage of the software proxy to be automatically evicted from the fleet without in our approach guarantees even less when testing false positives. CPU vendors need to be provided the hardware. We may be able to develop or reuse with actionable reports, reproducers, and often the better metrics specifically for fuzzing CPUs. We hardware itself. will continue to rely on software proxies, but ide- ally we would also collect coverage metrics directly Quality from hardware. However, we are likely to remain constrained by the CPUs being mostly opaque to us. Quality of the corpus is a function of two values: Better proxies. XED only allows fuzzing the coverage it provides, and time needed to execute decoder-like functionality of CPUs. Unicorn is a it (or, simply, its size). If the corpus is too large, functional CPU simulator and thus does not give it becomes too expensive to execute all of it on all any coverage feedback for the microarchitectural cores. If the coverage is low, we will not find many behavior. We will need to find ways to fuzz mi- defects. There are many ways we can increase the croarchitectural models, e.g. based on gem5 [35] or coverage while maintaining a tolerable size. Simics [36]. Mutation strategies. We currently mutate the in- Quality of the player. The player currently exe- struction sequences as simple byte sequences. This cutes multiple snapshots in the same process (har- is interesting from the point of view of stressing ness) with non-trivial set up code in between. We the CPU instruction decoders (or maximizing the expect that minimizing this set up code will im- coverage of XED), but produces too many invalid prove our chances to find defects since every snap- instruction sequences. They are mostly not added to shot execution will then inherit random microarchi- the corpus since they generate little new coverage, tectural state from previously executed snapshots. but they slow down the corpus generation due to We can improve further by running multiple non- the high volume of such snapshots under the current overlapping snapshots in parallel threads, such that fuzzing strategies. We will need to add structure- we stress cache coherency and similar subsystems. aware mutations that produce only valid x86_64 Multi-state. Currently we discard a considerable sequences. Specialized snapshot mutation strategies number of snapshots (up to 20%) because they con- are also expected to have a strong effect (see e.g. tain multiple end states. These can be caused by “register scrambling” from [34]). Cross-pollinating nondeterminism in these snapshots or by meaning- snapshots generated by different proxies is likely ful microarchitectural differences. Being able to to have a positive effect, and we already do it in a differentiate the two (e.g. by statically analyzing the limited way via dictionaries. disassembled code of each snapshot) will further Corpus distillation, a process of choosing the improve the quality of our snapshot corpus. smallest corpus subset while maintaining coverage. Bugs vs. defects. SiliFuzz may potentially be used Individual corpora coming from Unicorn and XED for finding CPU bugs. Unlike electrical defects, are already distilled, but their combination may have logic bugs can be detected on a single CPU, or even redundancies. We currently do not distill the ifuzz on a CPU model, before the actual CPU even exists. corpus at all. As we start using more sources of For example, our system can easily reproduce the snapshots (e.g. harvesting snapshots from real pro- JCC bug [2] and thus, theoretically, can find it too. grams) we will need an efficient distillation strategy, However, in order to reliably detect bugs we need via better coverage metrics and better proxies. to allow snapshots with multiple end states and dis- Better coverage metrics. Code coverage, while tinguish valid differences between end states (e.g. the best we have for software testing, is not a very one CPU family not supporting a given instruction) strong metric of corpus quality. Even when a piece from differences caused by a bug.

    8

6     Conclusion                                              [6]   H. D. Dixit, S. Pendharkar, M. Beadon, C. Ma-
                                                                    son, T. Chakravarthy, B. Muthiah, and S. Sankar.
           In this paper we presented our work-in-progress sys-     “Silent Data Corruptions at Scale”. In:      CoRR
             tem for finding CPU defects. We fuzz proxies, such     abs/2102.11245 (2021). arXiv: 2102.11245.

as software CPU simulators and disassemblers, us- [7] P. H. Hochschild, P. Turner, J. C. Mogul, R. Govin- ing traditional software fuzzing techniques, and then daraju, P. Ranganathan, D. E. Culler, and A. Vah- use the accumulated corpus to cross-check millions dat. “Cores that don’t count”. In: HotOS. ACM, of CPU cores. We have detected a large number of 2021. DOI: 10.1145/3458336.3465297. defects, analyzed four of them in detail, and ana- [8] Google. Silent Data Corruption. URL: https : / / lyzed common patterns among the others. support.google.com/cloud/answer/10759085/ We expect this and similar technologies to be silent-data-corruption (visited on 09/15/2021). in widespread use in the coming years since CPU [9] A. Konovalov. Coverage-Guided USB Fuzzing defects are here to stay. with Syzkaller. Talk, OffensiveCon, Berlin. Feb. 2019. URL: https://youtu.be/1MD5JV6LfxA. Acknowledgements [10] W. M. McKeeman. “Differential Testing for Soft- ware”. In: Digit. Tech. J. 10.1 (1998), pp. 100– 107. URL: http://www.hpl.hp.com/hpjournal/dtj/ We thank our colleagues Mike Fuller, Eric Liu, vol10num1/vol10num1art9.pdf. Marco Elver, Dmitry Vyukov, Andrew G. Morgan, [11] X. Yang, Y. Chen, E. Eide, and J. Regehr. “Find- Chris Leary, Mike Gunter, Subhasish Mitra, and ing and understanding bugs in C compilers”. In: Eric A. Schmidt Jr. for the help with preparing this ACM SIGPLAN Conference on Programming Lan- paper. guage Design and Implementation (PLDI). ACM, 2011. DOI: 10.1145/1993498.1993532. References [12] A. Helin. Radamsa. URL: https : / / gitlab. com / akihe/radamsa (visited on 09/15/2021). [1] T. Trippel, K. G. Shin, A. Chernyakhovsky, G. [13] J. Shen and J. A. Abraham. “Native mode func- Kelly, D. Rizzo, and M. Hicks. “Fuzzing Hard- tional test generation for processors with applica- ware Like Software”. In: CoRR abs/2102.02308 tions to self test and design validation”. In: IEEE (2021). arXiv: 2102.02308. International Test Conference (ITC). IEEE Com- puter Society, 1998. DOI: 10.1109/TEST.1998. [2] Intel. Mitigations for Jump Conditional Code Er- 743296. ratum. Nov. 2019. URL: https : / / www . intel . [14] P. Parvathala, K. Maneparambil, and W. Lind- com / content / dam / support / us / en / documents / say. “FRITS - A Microprocessor Functional BIST processors/mitigations-jump-conditional-code- Method”. In: IEEE International Test Conference. erratum.pdf. IEEE Computer Society, 2002. DOI: 10 . 1109 / [3] A. Edelman. “The Mathematics of the Pentium TEST.2002.1041810. Division Bug”. In: SIAM Rev. 39.1 (1997), pp. 54– [15] libFuzzer – a library for coverage-guided fuzz 67. DOI: 10.1137/S0036144595293959. testing. URL: https://llvm.org/docs/LibFuzzer. [4] R. R. Collins. “The Pentium F00F Bug”. In: Dr. html (visited on 09/15/2021). Dobb’s Journal (May 1998). URL: https://www. [16] american fuzzy lop. URL: https : / / lcamtuf . drdobbs.com/embedded- systems/the- pentium- coredump.cx/afl/ (visited on 09/15/2021). f00f-bug/184410555. [17] Honggfuzz. URL: https://honggfuzz.dev (visited [5] H. Nguyen, Y. Yagil, N. Seifert, and M. Reitsma. on 09/15/2021). “Chip-level soft error estimation method”. In: [18] syzkaller project authors. syzkaller kernel fuzzer. IEEE Transactions on Device and Materials Re- liability 5.3 (2005). DOI: 10.1109/TDMR.2005. URL: https://github.com/google/syzkaller (visited 858334. [19] on 09/15/2021). syzkaller project authors. syzkaller – ifuzz. URL: https://github.com/google/syzkaller/tree/master/ pkg/ifuzz (visited on 09/15/2021).

              9

[20] Intel. The X86 Encoder Decoder (XED). URL: [32] A. Klaiber and S. Chau. “Automatic Detection https://intelxed.github.io (visited on 09/15/2021). of Logic Bugs in Hardware Designs”. In: Fourth [21] Unicorn. URL: https://www.unicorn-engine.org International Workshop on Microprocessor Test (visited on 09/15/2021). and Verification, Common Challenges and Solu- [22] Unicorn – fuzz_emu_x86_64.c. URL: https : / / tions (MTV). IEEE Computer Society, 2003. DOI: github . com / unicorn - engine / unicorn / blob / 10.1109/MTV.2003.1250262. 772558119af66269742 / tests / fuzz / fuzz _ emu _ [33] S. Gurumurthy, D. Bertanzetti, P. Jakobsen, and x86_64.c (visited on 09/15/2021). J. Rearick. “Cache-resident self-testing for I/O [23] D. S. Milojicic, A. Messer, J. Shau, G. Fu, and circuitry”. In: IEEE International Test Conference A. Muñoz. “Increasing relevance of memory hard- (ITC). IEEE Computer Society, 2009. DOI: 10 . ware errors: a case for recoverable programming 1109/TEST.2009.5355549. models”. In: Proceedings of the 9th ACM SIGOPS [34] K. Czechowski, V. W. Lee, E. Grochowski, R. European Workshop. ACM, 2000. DOI: 10.1145/ Ronen, R. Singhal, R. W. Vuduc, and P. Dubey. 566726.566749. “Improving the energy efficiency of Big Cores”. [24] B. P. Miller, L. Fredriksen, and B. So. “An Empir- In: ACM/IEEE International Symposium on Com- ical Study of the Reliability of UNIX Utilities”. puter Architecture (ISCA). IEEE Computer Soci- In: Commun. ACM 33.12 (1990). DOI: 10.1145/ ety, 2014. DOI: 10.1109/ISCA.2014.6853219. 96267.96279. [35] The gem5 authors. “The gem5 Simulator: Version [25] V. J. M. Manès, H. Han, C. Han, S. K. Cha, M. 20.0+”. In: CoRR abs/2007.03152 (2020). arXiv: Egele, E. J. Schwartz, and M. Woo. “The Art, 2007.03152. URL: https://gem5.org. Science, and Engineering of Fuzzing: A Survey”. [36] Intel. Simics® Simulator Public Release. May In: IEEE Transactions on Software Engineering 2021. URL: https://software.intel.com/content/ (2019). DOI: 10.1109/TSE.2019.2946563. www/us/en/develop/articles/simics- simulator. [26] american fuzzy lop – Historical notes. URL: https: html (visited on 09/15/2021). //lcamtuf.coredump.cx/afl/historical_notes.txt (visited on 09/15/2021). [27] C. Domas. “Breaking the x86 ISA”. In: Black Hat USA (2017). URL: https : / / github . com / xoreaxeaxeax/sandsifter. [28] C. Easdon. Undocumented CPU Behavior: Ana- lyzing Undocumented Opcodes on Intel x86-64. Talk. June 2018. URL: https://www.cattius.com/ images/undocumented-cpu-behavior.pdf. [29] X. Li, Z. Wu, Q. Wei, and H. Wu. “UISFuzz: An Efficient Fuzzing Method for CPU Undocu- mented Instruction Searching”. In: IEEE Access 7 (2019), pp. 149224–149236. DOI: 10 . 1109 / ACCESS.2019.2946444. [30] L. Martignoni, R. Paleari, G. F. Roglia, and D. Bruschi. “Testing CPU emulators”. In: Proceed- ings of the Eighteenth International Symposium on Software Testing and Analysis (ISSTA). ACM, 2009. DOI: 10.1145/1572272.1572303. [31] A. Adir, S. Copty, S. Landa, A. Nahir, G. Shurek, A. Ziv, C. Meissner, and J. Schumann. “A unified methodology for pre-silicon verification and post- silicon validation”. In: Design, Automation and Test in Europe (DATE). IEEE, 2011. DOI: 10.1109/ DATE.2011.5763252.

           10

Appendices                                               gal instructions. While this is technically a defect, as
                                                         the CPUs involved did not behave according to spec-
A  F2XM1 Defect                                          ifications, the problem does not affect normal oper-
    ation of our fleet because typical production code

The first defect we found with SiliFuzz is related to does not contain illegal instructions. It is still possi- the F2XM1 x87 instruction, which computes 2x − 1 ble that the same defect has another, more harmful, for a given value of x. We observed on several CPUs manifestation that we did not observe. that this instruction sometimes returns incorrect re- sults. Listing 2 shows a code snippet from one of C FCOS Miscomputes the snapshots detecting this issue. Listing 2: F2XM1 Defect The third defect we found involves FCOS and other fldln2 // load natural log(2) on transcendental functions. Listing 3 shows an instruc- // stack top. tion sequence that sometimes produces incorrect re- f2xm1 // compute 2^x - 1 using sults on one of the machines screened by SiliFuzz. // stack top value. The sequence comes from a hand written reproducer Upon investigation, it turned out that only the inspired by failing snapshots. F2XM1 instruction matters. It does not matter if the Listing 3: FCOS Miscomputes input comes from an FLDLN2 instruction or loaded // Prologue from memory. On a given machine, we observed fldln2 that failures only appear occasionally on the defec- fldln2 tive core. The incorrect results seem to be constant fprem1 for a given physical core but vary between affected // Payload fld1 cores. The precise underlying failing mechanism fcos is still unknown to us. We have also confirmed the fault with a vendor-supplied diagnostic tool, The expected value of the above should be which reports F2XM1-related issues on those cores 0.5403023059 (or 0x3fe14a280fb5068c in binary we found. representation) but the actual value is 0.5403023043 This defect looks similar to the one described (binary 0x3fe14a280edd9be7). The difference is by Dixit et al. [6], but we do not know if it is the less than 0.0000003%. The erroneous value is con- same. sistent for both logical cores sharing the physical core. B Overshoot of an Illegal Instruc- One interesting aspect of this defect is that there seems to be some hidden state or memory. The tion prologue part of the instruction sequence seems to put the core in a broken state and then the payload The second defect found by SiliFuzz is a class of starts miscomputing. The prologue can be run in issues related to illegal instructions. x86_64 sup- the same process as the payload or in a different ports numerous instruction prefixes that can be ap- process. The CPU recovers from the broken state plied only to some instructions, and not to others. itself after an unknown period of time. We have In normal circumstances, a prefix followed by an found ways to trigger miscomputation even without incompatible instruction would always raise an ille- the prologue. Simply running the payload in a loop gal instruction fault (#UD). We observed some ma- with different values on ST(0) starts miscomputing chines sometimes overshoot past these ill-formed after some number of iterations. instructions without raising a #UD fault. An attempt to map the values that miscompute is These snapshots originated from fuzzing the x86 presented in Figure 1. Our current theory is that bit disassembler, which explains why they contain ille- 23 (i.e. 1<<23) in the binary encoding of the input

             11

0.08



 0.06 |
 0.00 | | | Il | | || |

Figure 1: Miscomputing FCOS. AxisX is the argument to COS, Y is X ifmiscomputed, 0 otherwise.

argument to FCOS triggers the problem: when the
bit  is set  the computation works    as expected but
when it is reset the instruction   almost universally
miscomputes on affected cores.

D                     Missing x87 Data Pointer Update

‘We observed a defect in which one of the x87 error
registers    is not consistently updated.  The legacy
x87 floating-point state contains a register for the
last data address used by a non-control x87 instruc-
tion.  In the example sequence seen in Listing 4, the
FISTTPLL instruction should set the      data pointer
to the store address.      On the machine on which we
found this defect, the data pointer is instead some-
times set to zero.   This missing update only happens
for the data pointer but not other error registers and
this       symptom is only found on a particular core
pair of the     machine. We believe this is a genuine
hardware defect.

     Listing 4: Missing x87 Data Pointer Update
cmp             tedx
rer      $0x£3,

£isttpll        (%rdi) # should  set x87  dp
adc      (%rax),










             12