SOURCE ARCHIVE
EXTRACTED CONTENT
141,623 chars RemembERR: Leveraging Microprocessor Errata for Design Testing and Validation
Flavien Solt, Patrick Jattke and Kaveh Razavi ® ®
Computer Security Group, ETH Z¨urich
Abstract—Microprocessors are constantly increasing in com- [25]. Finally, many bugs can only be found in post-silicon
plexity, but to remain competitive, their design and testing cycles testing under real-world conditions [26]–[31]. These design
must be kept as short as possible. This trend inevitably leads to testing and validation methods, unfortunately, do not scale to
design errors that eventually make their way into commercial
products. Major microprocessor vendors such as Intel and AMD the complexity of today’s microprocessor designs [16]. In the
regularly publish and update errata documents describing these testing steps, the lingering question is whether the test cases are
errata after their microprocessors are launched. The abundance providing a sufficient coverage [32]–[34]. Similarly, expensive
of errata suggests the presence of significant gaps in the design verification efforts should be targeted to those parts of the
testing of modern microprocessors. design where critical bugs are likely to lurk.
We argue that while a specific erratum provides information about only a single issue, the aggregated information from the body Microprocessor errata. In response to the discovery of bugs of existing errata can shed light on existing design testing gaps. after production, microprocessors vendors regularly publish Unfortunately, errata documents are not systematically structured. errata documents: human-readable documents containing a list We formalize that each erratum describes, in human language, a set of triggers that, when applied in specific contexts, cause of errata [15], [35]–[38]. The goal of publishing this list of certain observations that pertain to a particular bug. We present defects is to document known bugs and to provide system RemembERR, the first large-scale database of microprocessor designers with workaround guidance where appropriate. The errata collected among all Intel Core and AMD microprocessors organization of errata differs across vendors, but the structure since 2008, comprising 2,563 individual errata. Each RemembERR of each erratum entry remains similar. Each erratum, from entry is annotated with triggers, contexts, and observations, extracted from the original erratum. To generalize these properties, both Intel and AMD, includes a description with information we classify them on multiple levels of abstraction that describe about the conditions under which the bug occurs and a brief the underlying causes and effects. discussion of its implications once triggered. Furthermore, each We then leverage RemembERR to study gaps in design testing entry includes information about the proposed workarounds by making the key observation that triggers are conjunctive, while and whether or not the bug has been fixed. While the individual observations are disjunctive: to detect a bug, it is necessary to apply all triggers and sufficient to observe only a single deviation. erratum is useful for keeping track of a bug and informing Based on this insight, one can rely on partial information about users about it, we argue that grouping them reveals precious triggers across the entire corpus to draw consistent conclusions information that can guide future design testing and validation. about the best design testing and validation strategies to cover the existing gaps. As a concrete example, our study shows that RemembERR. To extract the relevant information from the we need testing tools that exert power level transitions under errata, we created RemembERR, a comprehensive, annotated MSR-determined configurations while operating custom features. database of all errata of Intel Core and AMD microprocessor families since 2008, with a total of 2,563 entries. Creating I. INTRODUCTION this database itself presented challenges since the errata are not machine-readable: (a) they lack an identical structure What are the bugs that we could not discover before between documents, (b) they contain a significant number of we sent the microprocessor design for fabrication? This is errors such as duplicate entries in the same document, reused probably the most important question that design test engineers errata numbers, and erroneous Model Specific Register (MSR) repeatedly ask themselves. The question is not getting any numbers, and (c) a lack of classification and consistency in easier to answer with the ever-increasing complexity of modern notations. To facilitate guiding testing and validation, we create microprocessors [1]. Despite advances in design testing tools a new classification of errata for RemembERR. We manually and techniques [2]–[23], we still see plenty of post-production annotate each RemembERR entry with its necessary triggers, bugs after new microprocessors are released, indicating that the contexts to which the bug applies, and the observations there exist gaps in design testing and validation. In this paper, that can be made once the bug is triggered. We call this we identify these gaps and propose concrete actions to cover level the concrete level of our classification. Although useful, them by leveraging a new classification based on errata reported the concrete level can sometimes be too erratum-specific to by Intel and AMD. generalize. For example, a particular offset inside a certain Design testing and validation. Before a microprocessor is machine-specific register must be written to trigger a bug. To shipped to customers, it goes through a variety of testing study causes and effects in an aggregate manner, we further and validation steps. In the early stages, a design simulation classify and annotate RemembERR entries at two higher levels using random or human-driven inputs may reveal bugs [2], of abstraction, which we call the abstract and class levels. [16]. Once the design matures, formal verification techniques Equipped with RemembERR, we then study trends to identify ensure the correctness of selected design parts [17]–[19], [24], design testing gaps. We make a key observation that in almost
every erratum, trigger conditions are conjunctive, while contexts Testing with simulation has two major shortcomings. First, and observations are disjunctive. This means that to discover a simulation-based testing is extremely slow. Therefore, it can bug, all triggers must be activated (e.g., a misaligned load that process only a few inputs in a reasonable time, making it causes a page fault), in any of the applicable contexts (e.g., challenging to reach all possible system states. For example, in user mode), and observing any behavior deviating from the open-source CVA-6 64-bit RISC-V core requires four days the expected behavior (e.g., a machine check exception) is to boot Linux in simulation [43]. We expect more complex sufficient to detect the bug. This powerful insight allows us to CPUs, tailored towards high performance, to be even more extract valuable information from aggregated errata, regardless complex by several orders of magnitude. Simulation becomes of how vague each individual erratum may be on its triggers mostly ineffective for complex modern microprocessors without and/or observations (the contexts are usually clear). Importantly, limiting the test cases to those effective in triggering bugs. this information about triggers, contexts, and observations is Second, simulation cannot expose issues related to the physical necessary for directing design testing and validation campaigns design, such as timing violations, data loss after power to discover bugs that are not currently missed by the existing gating [44], or interaction with real-world peripherals and tools and techniques. memories. Emulation, in spite of being significantly faster, Our study shows that more than 40% of bugs are uncovered suffers from the latter shortcoming as well. only when two distinct trigger types are combined. Moreover, Formal methods. Unlike simulation, which may suffer from most triggers do not interact with each other, while others limited input coverage, formal verification methods aim to seem to be closely related and together, they bring up new prove that certain properties always hold given some allowed bugs. Exploiting these interactions is crucial to boost future inputs. This approach makes it possible to prove correctness for design testing and validation of microprocessors and to keep all expected inputs — achieving completeness. However, these up with their increasing complexity. powerful formal methods have three weaknesses. First, they Contributions. We make the following contributions: typically do not scale to complex designs with many stateful • We propose a new classification of design flaws based on elements [16], [45]–[47]. Therefore, a typical approach is to necessary triggers, and sufficient contexts and observa- verify only selected design parts while modeling the rest [48]– tions. [50]. Second, properties may be difficult to express formally, • We create RemembERR, a comprehensive database created and there can exist many properties for complex designs [51], from 2,563 public errata across all 12 first generations of [52]. Third, properties related to power management or other Intel Core and 13 current AMD microprocessor families. physical effects may be difficult to reason about [44], [53]– • Using RemembERR, we study trends in post-production [56]. As each property is proven exhaustively, quickly rendering microprocessor bugs and develop testing and validation verification time infeasible, the test and validation engineers guidelines that relate triggers, contexts, and observations. must carefully decide which properties to prove. Open sourcing. We make the entire RemembERR database, in- Silicon testing. Complex bugs often escape traditional pre- cluding our annotations, publicly available1 so that researchers silicon testing and validation [26]–[31], [57]. Therefore, silicon and design test engineers can draw conclusions specific to their testing remains a crucial part of design validation, and takes goals and automate their tools. up to 50% of the testing cost for commercial designs [58]. In contrast to simulation, silicon testing achieves far higher II. BACKGROUND throughput, but it does not reach the completeness offered This section provides some brief background on existing by formal methods. Furthermore, silicon testing makes the hardware bug detection techniques (Section II-A) and errata design’s internals inaccessible. documents (Section II-B). B. Errata A. Bug detection methods For each design generation (Intel) or family (AMD) of We provide background on the three commonly used microprocessors, vendors typically provide a specification techniques for detecting hardware bugs [16]: simulation, formal update document, also known as errata, for listing known bugs methods, and silicon testing. after a product has been shipped. When a customer observes Simulation. Design simulation is a traditional testing tech- that a microprocessor deviates from its original specification, nique that is already used early in the design’s development they can look through the errata documents to verify whether cycle [16]. During a simulation, the design is given sequences it is a known bug. The errata also provide information on of inputs. The outputs and the resulting state of the design how to avoid triggering unwanted behavior. Notably, the bugs are then compared against a golden model [12] or inspected described in errata documents can no longer be fixed and manually [16], [39]. Modern simulators provide a rich set of remain for the lifetime of the affected microprocessors. features, such as undefined values (i.e., don’t care values), Organization. Following their intended purpose, errata docu- signal injection, and various coverage metrics [40]–[42]. ments produced by Intel and AMD are human-readable PDF documents listing the individual bugs. Each erratum has a 1https://github.com/comsec-group/rememberr title, a description, implications, workarounds, and a status
Table I. An erratum for Intel Core 12th generation. closed-source counterparts [59]–[61]. By aggregating errata
ID: ADL001 and building an annotated database, we intend to make this Title: X87 FDP Value May be Saved Incorrectly information more accessible than it currently is. Description: Execution of the FSAVE, FNSAVE, FSTENV, or FNSTENV instructions in real-address mode or virtual-8086 mode may save an incorrect Structure. The way errata are structured is suitable for reading value for the x87 FDP (FPU data pointer). This erratum does not apply if the by an experienced human but is not optimized for automated last non-control x87 instruction had an unmasked exception. data mining. A clear specification of what each field contains Implications: Software operating in real-address mode or virtual-8086 mode or implies is missing. The useful information is often spread that depends on the FDP value for non-control x87 instructions without unmasked exceptions may not operate properly. Intel has not observed this across the title, description, and implication (and sometimes erratum in any commercially available software. workaround) fields, with a high degree of redundancy. This Workaround: None identified. Software should use the FDP value saved by observation calls for creating and maintaining an improved the listed instructions only when the most recent non-control x87 instruction erratum structure, scheme and tooling support that would be incurred an unmasked exception. more adapted for data mining and to rule out redundancy while Status: For the steppings affected, refer to the Summary Table of Changes. remaining human-readable. Table II. An erratum for AMD Zen 3 family. Guiding design testing and validation. In complex CPU ID: 1361 designs, all testing and validation methods must be directed. Title: Processor May Hang When Switching Between Instruction Cache and Formal methods require knowing the bug type to target and Op Cache. prioritizing the parts of the design that are most susceptible. Description: Under a highly specific and detailed set of internal timing Furthermore, formal properties must be local and specific to conditions, running a program with a code footprint exceeding 32 KB may minimize the impact of state explosions. For dynamic methods cause the processor to hang while switching between code regions that consistently miss the instruction cache and code regions contained within such as simulation and silicon testing, it is crucial to know the Op Cache. which input signals to provide in which context and what Implications: System may hang or reset. effects to expect if a bug is triggered [39], [62]. In Section VI, Workaround: System software may contain the workaround for this erratum. we provide an in-depth discussion on how the annotated errata Status: No fix planned. information can enhance existing validation methodologies. For example, errata reveal that specific bugs require ongoing indicating whether a fix is available for current or future releases PCIe communication. Is connecting a PCIe device enough of the same CPU generation or family. Intel released separate to discover all PCIe-related bugs? Looking at all the errata, erratum documents for the Mobile and Desktop version of we observe that some PCIe-related bugs require triggering its Core microprocessors until generation 5. After that, they a reset signal. Furthermore, how can we efficiently observe released only one document per generation. AMD uses a single whether a bug was triggered? This knowledge of the interaction document per CPU family (i.e., per CPU microarchitecture). between different input types, contexts, and effects is crucial for maximizing a testing campaign’s efficiency and efficacy. Errata examples. We provide two recent errata examples. In Table I, we show the first erratum for Intel Core 12th generation IV. REMEMBERR CPUs, and in Table II, the most recent erratum for AMD Zen In this section, we introduce RemembERR, an annotated 3 family CPUs. database of 2,563 errata from AMD and Intel microprocessors. III. MOTIVATION: LEARN FROM THE PAST In Section IV-A, we first describe the scope and our method- ology. Based on this (yet unannotated) database, we present The number of published errata has not significantly de- essential observations about the current state of microprocessor creased over time, as we show in Section IV. Strikingly, we will errata in Section IV-B. show that some bugs require years to be reported, while similar bugs were already found in previous designs. These trends A. Methodology point to gaps in existing design testing and validation tools Figure 1 presents an overview of our methodology. Our and techniques. A data-driven approach using the information approach can be summarized into four steps: 1a First, we contained in the errata can shed light on these gaps and provide acquired the latest errata documents from Intel and AMD, directions for covering them. and 1b analyzed duplicate errata. This already allows us Accessibility. Each erratum is specific to one bug in a to make general observations about errata’s current state particular design, complying with a certain Instruction Set (Section IV-B). 2 We then generalized the triggers, contexts, Architecture (ISA). This makes deriving any valuable insights and observable effects to derive a universal classification from a series of individual errata difficult. Further, it does not scheme for errata (Section V-A). 3 Using automation, we incite communities that build and verify other microprocessors classified a portion of the errata, and for the rest, we used to read and learn from known pitfalls and spots that require four-eyes manual classification. The result is the annotated special testing focus. This is becoming increasingly more RemembERR database. 4 Lastly, we leveraged RemembERR important as the complexity of community-driven micropro- to derive novel insights for filling the gaps in existing design cessors is progressively catching up with their proprietary and testing and validation (Section V-B).
1a responsible for non-systematically (redundantly) distributing
A information across errata fields.
1b B RemembERR Duplicates. As we will show, it is common that two (or
(annotated) multiple) designs from the same vendor with different release
? dates are affected by the same erratum. RemembERR contains
= 4 all the duplicates as often as they appear across documents.
3 This provides useful information about bugs shared among
Trg_MBR Ctx_PRV Eff_HNG generations or families. However, to allow filtering for unique
_cbr _boo _unp entries, RemembERR features a keying mechanism that assigns
_pbr _vmg _hng
... ...
2 Trg_MOP Ctx_FEA ... a unique identifier to each cluster of identical errata.
... ... Eff_FLT
RemembERR ... AMD identifies errata across microprocessor families using
(non-annotated) Classification Scheme a unique numeric identifier: two families are affected by the
Fig. 1. Overview of our methodology. same erratum if both have an erratum with the same number in
their corresponding errata document. This mechanism protects
Table III. Inspected errata documents. Left: Intel Core CPUs, against intra-document duplicates. Besides different errata
right: AMD CPUs. (M): Mobile, (D): Desktop. numbers, some cases are indistinguishable given the limited
Intel AMD information in the errata’s fields. For example, errata no. 1327
and no. 1329 only differ in their suggested workaround but
Gen. Reference Fam. Models Reference may originate from distinct root causes. In total, we collected
1 (D) 320836-037US 10h 00-0F 41322-3.84 506 errata from AMD, of which 385 are unique.
1 (M) 322814-024US 11h 00-0F 41788-3.00 Intel errata documents do not provide a simple way to
2 (D) 324643-037US 12h 00-0F 44739-3.10 identify duplicates across generations. Instead, we base our
2 (M) 324827-034US 14h 00-0F 47534-3.18 duplicate detection on the errata titles. As a first step, we
3 (D) 326766-022US 15h 00-0F 48063-3.24 marked all errata with the same title as duplicates. Extensive
3 (M) 326770-022US 15h 10-1F 48931-3.08 manual inspection of all the candidate duplicates shows that
4 (D) 328899-039US 15h 30-3F 51603-1.06 when the titles are (nearly) identical, all other fields are
4 (M) 328903-038US 15h 70-7F 55370-3.00 identical as well. Except for minor phrasing variations or
5 (D) 332381-023US 16h 00-0F 51810-3.06
5 (M) 330836-031US 17h 00-0F 55449-1.12 slightly different levels of detail. As a second step, we manually
6 332689-028US 17h 30-3F 56323-0.78 analyzed remaining errata that have not been marked yet as
7/8 334663-013US 19h 00-0F 56683-1.04 duplicates, sorted by decreasing title similarity, given that title
8/9 337346-002US similarity is a strong indicator of potential duplicates. We could
10 615213-010US manually identify 29 pairs as duplicates. In total, we collected
11 634808-008US 2,057 errata for Intel, of which 743 are unique.
12 682436-004US Because Intel and AMD use different identifiers for their
Examined documents. We comprehensively examined all the errata, it is difficult to determine whether a bug is common
errata documents listed in Table III. Vendors usually withdraw between products of these two vendors; at least, we could
errata documents once the processor line is not supported not find any occurrence giving strong evidence. Arguably,
anymore, which makes finding the errata documents not always Intel and AMD designs are proprietary; hence they might not
straightforward. We scraped the web thoroughly and took the share hardware blocks. It is, hence, unlikely for identical bug
most recent findable document for each generation (Intel) or instances to occur across vendors.
family (AMD). We examined all the errata from the Intel Core B. Observations
series and all the errata from AMD CPUs since 2008. The data gathered in RemembERR allows us to make several
Errata in errata. Errata documents contain many errors novel observations about the current state of errata. themselves. Examples are two revisions pretending to have 1) Timeline: We first analyze the number of reported bugs added the same erratum (affects 8 errata across 3 documents), accumulated over time. Unfortunately, bug discoveries are not some errata are never mentioned in the revision notes (affects timestamped; hence, we approximate the timestamp of each 12 errata across 2 documents), the same name refers to two erratum by identifying in which revision of the errata document different errata (affects an erratum named AAJ143), there are it first appeared. We then use the errata document’s release or missing or duplicate fields in errata (affects 7 errata across 4 update date to approximate the timestamp. documents), or there are errors in the MSR numbers (affects In some cases, the revision summary does not indicate 3 errata across 3 documents). In rare cases, errata may be in which revision a certain erratum was added. Fortunately, repeated inside the same errata document (affects 11 errata errata are sequentially numbered. Hence, we can approximate pairs across 6 documents). These errors are a clear indicator the date by assuming that the subsequent erratum was added that the writing of errata is a manual process. Humans not only simultaneously. In rare cases, we observed contradicting dates: express errata in a human language, but they also seem to be revision logs falsely pretend that the same erratum was added
Number of disclosed errata (cumulated)
200
175
150
125
100
75
50
Core 1 (D) Core 2 (D) Core 3 (D) Core 4 (D) Core 5 (D) Core 6 Core 8-9 Core 11
25 Core 1 (M) Core 2 (M) Core 3 (M) Core 4 (M) Core 5 (M) Core 7-8 Core 10 Core 12
100 10h 12h 15h-0 15h-3 16h-0 17h-0 19h
75 11h 14h 15h-1 15h-7 16h-3 17h-3
50
25
0
2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023
Disclosure date
Fig. 2. Disclosure dates of Intel Core errata (top) and AMD CPU errata (bottom). The y-axis represents the cumulative number
of disclosed errata. The data point represents the errata’s release date.
in two consecutive revisions. In this case, we consider the date Core 1 (D) Core 1 (M) Core 2 (D) Core 2 (M) Core 3 (D) Core 3 (M) Core 4 (D) Core 4 (M) Core 5 (D) Core 5 (M) Core 6 Core 7-8 Core 8-9 Core 10 Core 11 Core 12
of the earlier of the two revisions as the correct one.
Figure 2 shows the cumulative growth of errata over time, Core 1 (D) 165 87 49 43 35 35 17 17 13 14 6 6 6 6 0 0
where duplicate entries are counted individually. We observe Core 1 (M) 125 44 37 29 29 18 18 15 15 4 4 4 4 0 0
that Intel updates its errata documents significantly more Core 2 (D) 130 112 75 75 35 35 27 28 11 11 11 11 0 1
frequently than AMD. Desktop and mobile processors released Core 2 (M) 112 61 61 20 20 19 19 7 7 7 7 0 1
at close dates have very similar curves, for example, Intel Core
2, 3, and 4 during the year 2013. This may suggest that the Core 3 (D) 114 114 36 36 27 27 11 10 10 10 0 1
same bugs tend to affect multiple generations. In Section IV-B2, Core 3 (M) 117 36 37 28 28 12 11 10 10 0 1
we study this bug transmission effect across design generations Core 4 (D) 167 165 66 65 29 27 22 26 1 0
in more detail. Core 4 (M) 177 67 67 30 28 22 26 1 0
Figure 2 further demonstrates that vendors keep introducing Core 5 (D) 120 108 34 31 25 29 2 1
new bugs into their products. While the latest microarchitectures Core 5 (M) 134 37 35 29 32 2 1
seem to be less affected, it is likely that many bugs have not
yet been discovered or reported. Core 6 187 138 117 116 7 5
Core 7-8 153 123 126 9 5
(O1) Observation. The number of reported errata does not Core 8-9 133 113 4 4
significantly decrease over time with new designs. Core 10 130 9 5
All cumulative curves tend to be concave. The more time Core 11 30 10
passes, the fewer bugs are found in a given period. In most Core 12 32
older designs, the curve stagnates towards the end, where only Fig. 3. Bug heredity: number of common bugs across Intel
very few new bugs are discovered after many years from the microprocessor generations.
initial release of the CPU, especially for Intel Core designs.
This observation confirms the intuition that finding new bugs
in a design becomes increasingly more difficult or that older this phenomenon to provide an answer to the questions: (a) How
designs are not as rigorously tested anymore compared to newer often are bugs transmitted across generations or families?
designs. (b) Are transmitted bugs rediscovered multiple times?
(O2) Observation. The increase in errata for a given design Transmission. By definition, distinct AMD families have
is usually concave. distinct microarchitectures. Our data corroborates that, as we
find fewer shared errata between AMD families, compared to
2) Heredity: It is known from well-studied bugs such Intel Core generations. Furthermore, AMD provides limited
as Meltdown [63], Foreshadow [64], RIDL [64] and Zom- chronological information, as depicted in Figure 2. Hence, we
bieLoad [65], that different designs may suffer from exactly the focus this part of our study on Intel errata.
same bug. One cause for this phenomenon may be the reuse of Figure 3 shows the number of identical errata between pairs
microarchitectural blocks across design generations. We study of Intel errata documents. We can observe that Desktop and
Num. errata (cumul.)
Number of latent errata
100 100 forward-latent
backward-latent
80 75
50
60
Core 6 Core 8-9 25
40 Core 7-8 Core 10 0
2016 2017 2018 2019 2020 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
Disclosure date
Fig. 4. Disclosure dates of Intel Core errata for bugs that are Fig. 5. Forward-latent and backward-latent errata among Intel
shared by all Intel Core generations from 6 to 10. Core generations.
mobile processors share the vast majority of bugs, matching our was allocated to testing older CPU generations, for example,
observation of similar curves in Section IV-B1. The processors to prepare for the release of the Skylake microarchitecture.
that share a substantial part of their microarchitecture are The increasing forward-latent numbers typically denote cores
salient in this diagram, such as Intel Core generations 6 to 10. sequences with similar microarchitectures, where a bug has
Note that if a security bug is discovered only after multiple not been fixed, although it was known before the official CPU
generations, an attacker could have exploited it for years release. The number of forward-latent errata has always tended
without being uncovered. Therefore, the duration between bug to increase, and this trend has accelerated since 2015. Note
introduction and discovery is not a suitable proxy for estimating that these curves, for the time interval displayed here, may
criticality, especially regarding security. In Figure 3, long non- increase in the future with the rediscovery of more errata in
zero horizontal lines indicate long-lasting bugs. 6 bugs stayed existing or future Intel Core generations.
from Core 1 to Core 10, and one erratum from Core 2 was These results suggest that either the test and validation cycles
still identified 11 generations later, more than 10 years after are very long (in order of many years), making it difficult to
its initial discovery. react to newly discovered bugs during this phase, or these bugs
are difficult to mitigate without fundamentally changing the
(O3) Observation. Bugs are often shared between genera- microarchitecture.
tions of microprocessors. Shared bugs may stay for up to
11 generations. (O4) Observation. Most of the design flaws that are shared
between generations were already known before releasing
Rediscovery. We conducted a further study to answer the the subsequent generation.
question: from errata shared between microprocessors, which
proportion was already reported in an earlier generation at the Observation O4 indicates a correlation between long CPU
time of release? development cycles and the difficulty of finding complex bugs.
Figure 4 shows the reporting date for the 104 bugs shared 3) Workarounds: The vendors propose different workaround
by all Intel Core generations from 6 to 10. This set of bugs types, depending on where the workaround should be applied,
corresponds to a salient region of common bugs in Figure 3. i.e., which actor should (not) perform a specific action to ensure
The first data point corresponds to the release date of each proper functionality. Based on this, we classify the workarounds
generation. Clearly, most of the shared design errors were into five categories: BIOS, software, peripherals, absent, and
known before the release of the subsequent generation, some None. The category absent indicates existing workarounds
even many years before. without any specific information, such as “Contact [...] for
This raises the question of where bugs are first discovered: information on a BIOS update.” Instead of absent, whenever
in older designs and then confirmed on more recent ones possible, we classify the workaround into a specific category
(forward), or are they usually first found on more recent designs even if the exact information is missing. Vendors use an
and then confirmed on older ones (backward)? While Figure 4 additional category documentation fixes to describe originally
provides a qualitative insight, to answer this question, we define intended behavior that was wrongly documented. This category
a forward-latent erratum as an erratum that was reported in one is negligible in size as it represents less than 0.5% of the total
design and (strictly) later reported in a later design. Similarly, number of errata.
we say that an erratum is backward-latent if it was reported in We summarize our results in Figure 6, where identical
a design (strictly) before being reported in an earlier design. errata are merged. The errata that can be mitigated in the Figure 5 shows the forward-latent and backward-latent errata BIOS are arguably the least critical, as long as the mitigation for Intel Core CPUs (again, the AMD errata documents lack does not substantially affect performance or security. Errata sufficient chronological information for such an analysis). requiring conditions in the peripherals or the software are more The salient portion of backward-latent errata around the year challenging to mitigate due to the plethora of legacy hardware 2015 may represent a period at Intel where less resource and software. In total, 28.9% (AMD) and 35.9% (Intel) of all
Number of errata Proportion of errata (%)
Core 1 (D) Core 1 (M) Core 2 (D) Core 2 (M) Core 3 (D) Core 3 (M) Core 4 (D) Core 4 (M) Core 5 (D) Core 5 (M) Core 6 Core 7-8 Core 8-9 Core 10 Core 11 Core 12
Fam. 10h Fam. 11h Fam. 12h Fam. 14h Fam. 15h-0 Fam. 15h-1 Fam. 15h-3 Fam. 15h-7 Fam. 16h-0 Fam. 16h-3 Fam. 17h-0 Fam. 17h-3 Fam. 19h
Intel AMD Section V-A, after which we explain the classification scheme’s
40 36.5 35.9 40 39.8 categories. Using the classified data, we present new insights
23.1 31.6 28.1 about bugs based on our classification results in Section V-B.
20 20
0 0.8 0 0 1.9 0.8 A. Categories
BIOS/Firmware Software Peripherals None Absent BIOS/Firmware Software Peripherals None Absent A crucial part of our classification is the definition of concrete
categories. We first made an exploratory pass over the errata
documents to determine appropriate categories for triggers,
Fig. 6. Suggested workarounds of errata by category. contexts, and effects. To make the classification useful for our
intended purpose, we require our concrete categories to be
200 Intel AMD No fix (a) unambiguous: a category should clearly be distinctive from
other categories to improve our classification’s reliability,
150 Fix (b) usable: categories should be helpful to guide the design
testing process,
100 (c) and self-explanatory: a one-sentence description should
50 be sufficient to understand the category.
0 For instance, requiring a reset signal to observe faulty behavior
is an unambiguous trigger (i.e., unlikely to be misclassified or
misunderstood). It is a usable trigger because it is necessary
to trigger observable behavior, and it is self-explanatory. If
Fig. 7. Proportion of fixed vs. unfixed bugs. reset signals are not needed to find some bugs of interest, we
should apply more relevant, directed test cases to increase
effectiveness and close design testing gaps.
unique errata do not have any suggested workaround at all. 1) Classification methodology: In the following, we describe
(O5) Observation. A substantial number of errata do not our systematic approach for designing our errata classification
have any suggested workaround. scheme. After that, we explain how we efficiently classified
the errata consistently and reliably.
As we discuss soon, this is not because bugs were fixed but Goal. We designed a hierarchical classification scheme that
because most bugs are deeply rooted in the design, limiting allows us to seamlessly switch between different levels of
possible workarounds. abstraction. These abstraction levels are crucial for making the
- Fixes: In some cases, the vendors fix the root cause of necessary observations and recommendations for improving a bug, as indicated explicitly in dedicated parts of the errata design testing and validation. If the recommendations are too documents (in dedicated tables or in a status field). Fixes precise, methods cannot easily generalize the insights when are distinct from workarounds as the former rules out the looking for new bugs. If they are too abstract, however, then bug from the design completely, while the latter dynamically limited guidance will hamper efficiency and coverage. aims at preventing the bug from interfering with proper design Our classification scheme is composed of three levels: the functionality. Fixes may require a re-spin of the processor, concrete level, the abstract level, and the class level. We which is an update of the CPU’s design masks. We could not explain them for the example of triggers. First, the concrete find any requirements driving the decision to fix a bug rather level represents the exact action that is described in the erratum. than proposing a workaround. Most likely, the decision is made For example, “the core resumes from the C6 power state” is based on the bug’s criticality by considering functionality or an action described at the concrete level. Second, the abstract security impact, and also the complexity of fixing the bug. level represents a slightly higher level of abstraction. As an Figure 7 shows the number of bugs that are fixed in different example, a transition between core power states is an action designs. Clearly, the vast majority of bugs are never fixed. For described at the abstract layer. The abstract level is crucial Intel CPUs, there has been a weak trend over the last few since design testing and validation tools must achieve generality generations toward fixing bugs. to maximize coverage. In the example before, considering only transitions from the core C6 power state may not catch
(O6) Observation. Bugs are rarely fixed. unknown bugs that only manifest when transitioning from other
power states. Finally, the class level represents the highest
V. CLASSIFICATION level of abstraction; in our example, power management is the
representation of the action at the class level. This last level of
This section introduces an errata classification scheme based abstraction provides even more generality, contributes to better
on triggers, contexts, and effects. We start by describing the readability and allows us to make more general conclusions
methodology we applied to design our classification scheme in about the bugs triggered by a particular trigger class.
Number of errata
Agreements (%)
Intel AMD Intel AMD
800 400 100
600 300 80
60
400 200 40
200 100 20
0 0 0
0 1 2 3 4 5 6 0 1 2 3 4 5 6 0 1 2 3 4 5 6 0 1 2 3 4 5 6
Step Step Step Step
Fig. 8. Number of errata per errata classification discussion Fig. 9. Percentage of human-classified errata-category pairs step. classified identically by both humans before the discussion.
Methodology. We define the categories for triggers, contexts, in total, we defined 60 categories. First, we merge identical and observable effects in an iterative way. We process all unique unique errata in the decision-making process, resulting in 1,128 errata to extract concrete triggers, contexts, and observable remaining errata. This still amounts to 1128 × 60 = 67,680 effects. For each of them, we check if we already have a classification decisions per human, even without considering corresponding abstract category. If so, we then label the erratum the discussions for mismatches yet. We measured a typical with this abstract category; otherwise, we create a new abstract average duration of 30 seconds per classification decision, category, and we check if we have a corresponding class which amounts to more than 560 hours of high-focus work per category. If a corresponding class category exists, we add the human merely for the individual classification part. new abstract category to the existing class category; otherwise, Fortunately, some classes can be automatically filtered out we create a new class category and attach the new abstract as irrelevant for a given errata, given the text describing it. category to it. We provide a detailed overview of class and Some others can be automatically said to be clearly relevant abstract categories in Tables IV to VI. to an erratum. With conservative filtering based on regular RemembERR is a cross-ISA database as typically, only expressions, we could reduce the number of decisions to 2,064 items (i.e., triggers, contexts, or effects) at the concrete level per human in the individual phase. These remaining decisions may be ISA-specific. Therefore, RemembERR can naturally are difficult to make automatically and reliably. For example, be extended with errata from designs implementing other ISAs if a reset signal is a trigger or an effect in an erratum based on (e.g., POWER, ARM). its description. For guiding the human-based classification, we Four-eyes classification. Some errata contain expressions that designed a syntax highlighting engine with regular expressions are specific enough to be classified automatically using regular to emphasize parts of the errata descriptions relevant to a expressions into some categories, but many errata-category given category. With this tool’s assistance, we could reduce pairs require manual analysis for classification. Besides being the amount of pure classification work and discussion to time-consuming, manually extracting and annotating such approximately 30 hours per human in total. We release all code an immense database of complex items is error-prone. To along with the RemembERR database and envision that such significantly improve the reliability of our results, two of computer-assisted classification tools will encourage further the researchers involved in this work independently classified contributions to errata classification. the errata. After completing the classification, they discussed 2) Triggers: Inputs that cause an exceptional observable and resolved each mismatch individually. To improve the effect are often not clearly stated or unspecified. At first sight, classification and clarify our understanding of the categories, this renders the majority of errata unusable as they cannot easily the discussions were made iteratively in seven successive steps be reproduced. However, we tackle this major challenge by for each design, using the same method but with the next batch designing a trigger classification scheme based on conditions of individually classified errata. Figure 8 shows the cumulative that are necessary to cause an observable effect. Effectively, number of errata in each classification step. Figure 9 shows this means we define the required conditions under which the evolution of the agreement of the decisions of the two suitable inputs can trigger a certain bug. This new classification humans. Note that, since the AMD errata were classified after method comes with several benefits. First, it allows deriving the Intel errata, the data provided in Figure 9 is chronological. valuable insights even if only a limited amount of information There are multiple reasons for mismatches, notably (i) human is available. This makes our scheme especially useful for newer errors as classification is a tedious, long, and difficult process; microprocessors or ISAs where fewer errata are available. (ii) imprecise description of the trigger, contexts, or effects Second, the categories we defined are largely independent in errata, leaving room for interpretation; and (iii) ambiguous and not exclusive, which allows for a simple estimation of a classification categories. We note that the agreement percentage bug’s complexity: the more necessary conditions are involved, is generally above 80%. the more complex the bug is to trigger. Furthermore, this Software-assisted classification. The cumulative number of classification scheme can easily be augmented in the future categories for triggers, contexts, and observable effects is large: with new trigger classes, if needed.
Table IV. Classification of triggers. Table V. Classification of contexts.
Trg_MBR a data operation on a... Ctx_PRV related to privileges
- _cbr cache line boundary. - _boo booting or being in the BIOS.
- _pbr page boundary. - _vmg being a virtual machine guest.
- _mbr memory map boundary such as canonical. - _rea operating in real mode. Trg_MOP a memory operation involving... - _vmh being a hypervisor.
- _mmp an interact. with a memory-mapped element. - _smm being in SMM.
- _atp an atomic/transactional memory operation. Ctx_FEA related to features
- _fen a memory fence or a serializing instruction. - _sec security feature enabled (SGX, SVM, etc.).
- _seg a condition on segment modes. - _sgc running in a single-core configuration.
- _ptw a core page table walk. Ctx_PHY non-digital conditions
- _nst translation on nested page tables. - _pkg package-specific.
- _flc flushing some cache line or TLB. - _tmp temperature-specific.
- _spe a speculative memory operation. - _vol voltage-specific. Trg_FLT related to exceptions and faults
- _ovf a counter overflow. Table VI. Classification of observable effects.
- _tmr a timer event. Eff_HNG related to hangs
- _mca a machine check exception. - _unp an unpredictable behavior.
- _ill an illegal instruction. - _hng a hang of the processor. Trg_PRV related to privilege transitions - _crh a crash of the processor.
- _ret a resume from System Management or OS - _boo a boot failure.
- _vmt mode. Eff_FLT related to faults a transition between hypervisor and guest. - _mca a machine check exception.
Trg_CFG related to dynamic configuration - _unc an uncorrectable error.
- _pag a paging mechanism interaction. - _fsp one or multiple spurious faults.
- _vmc a virtual machine configuration interaction. - _fms one or multiple missing faults.
- _wrg a configuration register interaction. - _fid a wrong fault identifier or order. Trg_POW related to power states Eff_CRP related to corruptions
- _pwc a transition between power states. - _prf a wrong performance counter value.
- _tht a change in thermal or power supply condi- - _reg a wrong MSR value. tions, or throttling. Eff_EXT related to physical outputs
Trg_EXT related to external inputs - _pci issues observable on the PCIe side.
- _rst a (cold or warm) reset. - _usb issues observable on the USB side.
- _pci an interaction with PCIe. - _mmd multimedia issues (e.g., audio, graphics).
- _usb an interaction with USB. - _ram abnormal interaction with DRAM.
- _ram a specific DRAM configuration. - _pow abnormal power consumption.
- _iom an access through the IOMMU. input (e.g., a PCIe device). We write abstract descriptors as the
- _bus system bus (HyperTransport, QPI, etc.). concatenation of two elements as well: (i) a prefix determining Trg_FEA related to features the class where the abstract category belongs to, and (ii) a suffix
- _fpu floating-point instructions. determining the abstract category, given the prefix. For example,
- _dbg debug features such as breakpoints. the abstract category Trg_EXT_rst refers to applying cold
- _cid design identification (CPUID reports). or warm resets.
- _mon monitoring (MONITOR and MWAIT). 3) Contexts: Some bugs can only happen in specific settings,
- _tra tracing features. for example, in a virtual machine guest or during BIOS/UEFI
- _cus other specific features (SSE, MMX, etc.). initialization. Bugs that can be provoked from user mode repre- sent a particular security risk as unprivileged user applications
In Table IV, we show all the categories for trigger that are usually executed in this mode. Contrary to triggers, contexts we defined on the abstract and class levels. We write class are disjunctive: there may exist multiple contexts in which the descriptors as the concatenation of two elements: (i) a prefix same bug can be triggered. That said, for a given erratum, it determining whether it refers to a trigger, context, or effect, and is sufficient to be in any of its contexts to observe the bug. In (ii) a suffix determining the class, given the prefix. For example, Table V, we list all the abstract and class context categories the class Trg_EXT consists of all triggers involving external that we derived from our considered errata.
Affected errata (%)
trg CFG wrg trg POW tht trg POW pwc trg FEA cus trg EXT pci trg FEA dbg trg MOP mmp trg PRV vmt trg FEA tra trg CFG pag trg EXT ram trg FLT mca trg MOP seg trg EXT usb trg MOP atp trg EXT rst trg FEA fpu trg MOP nst trg CFG vmc trg FLT ovf trg EXT iom trg PRV ret trg EXT bus trg MBR mbr trg FLT tmr trg FEA cid trg FEA mon trg FLT swf trg MBR cbr trg MOP spe trg MBR pbr trg MOP fen trg MOP flc trg MOP ptw
trg CFG pag Proportion of errata (%) trg CFG vmc trg CFG wrg trg EXT bus trg EXT iom trg EXT pci trg EXT ram trg EXT rst trg EXT usb trg FEA cid trg FEA cus trg FEA dbg trg FEA fpu trg FEA mon trg FEA tra trg FLT mca trg FLT ovf trg FLT swf trg FLT tmr trg MBR cbr trg MBR mbr trg MBR pbr trg MOP atp trg MOP fen trg MOP flc trg MOP mmp trg MOP nst trg MOP ptw trg MOP seg trg MOP spe trg POW pwc trg POW tht trg PRV ret trg PRV vmt
15 40 Intel
10 30 37.9 31.6 28.4 32.6 AMD
5 20
0 10 14.214.6 3.6 4.7 0.9 1.4 0.5 0.4 0.2 0.4
0
1 2 3 4 5 6 7
Fig. 10. Most frequent triggers of all errata. Fig. 11. Number of involved triggers
Number of errata by the number of triggers.
4) Observable Effects: The main objective of our effect
classification is to find an answer to the question: where to
look at when testing a design against an erratum with multiple 11 11 1 7 1 16 10 2 1 6 1 3 2 2 3 5 1 10 28 4 4 2 2 3 17 trg CFG pag
observable effects? 5 2 10 6 5 1 1 2 1 1 4 11 2 3 1 5 21 trg CFG vmc
In Table VI, we describe all the abstract and class categories 6 6 7 16 97 72 1 21 12 2 2 16 42 9 1 9 2 2 111 4 1 211 7 3 6 3 144 2713 5 122 trg CFG wrg
trg EXT bus
for observable effects that we derived from the errata under 3 3 14 1 3 7 11 25 51 4 9 131 trg EXT iom
trg EXT pci
study. Similar to contexts, an erratum’s observable effects 2 21 2 2 2 34 12 1 109 1916 trg EXT ram
trg EXT rst
are disjunctive. For example, if a corrupted configuration 2 3 1 1 3 1 3 2 1 2 9 2 2 3 2 2 trg EXT usb
1 1 1 2 trg FEA cid
register inevitably leads to an unexpected fault, for instance, 13 8 13 8 3 4 1 11 1 1 8 11 1 12 2 5 2 7 16 trg FEA cus
because its corruption triggers an exception, then this bug 3 16 2 4 1 2 2 2 4 2 7 2 17 4 2 10 16 trg FEA dbg
1 2 1 2 3 2 2 3 9 1 1 1 trg FEA fpu
simultaneously belongs to two effect categories: wrong MSR 2 1 4 9 3 1 trg FEA mon
3 3 3 8 3 7 1 5 7 5 4 12 trg FEA tra
value (Eff_CRP_reg) and spurious faults (Eff_FLT_fsp). 4 1 2 1 1 2 1 14 47 13 2 31 trg FLT mca
trg FLT ovf
There are also cases where an effect is observable in different 132 1 4 1 7 6 3 trg FLT swf
trg FLT tmr
ways. To give an example, an operation bringing the CPU to 1 11 21 21 11 1 8 12 3 trg MBR cbr
trg MBR mbr
some incorrect power state can be observed either by reading 3 1 1 trg MBR pbr
2 6 2 1 1 2 1 1 1 trg MOP atp
a configuration register or by measuring the CPU’s power 1 1 1 trg MOP fen
consumption. 2 1 1 1 1 1 trg MOP flc
8 4 2 4 9 6 7 trg MOP mmp
Only a few bugs can be considered non-critical: criticality 1 3 1 3 13 trg MOP nst
1 2 trg MOP ptw
generally depends on the assumptions made by the software 1 4 12 trg MOP seg
trg MOP spe
running on the faulty CPU. Therefore, it is necessary to be 49 2 2 trg POW pwc
trg POW tht
conservative in this matter. For example, crashes and hangs are 8 trg PRV ret
trg PRV vmt
evidently critical: systems depending on the liveliness of the Fig. 12. Pairwise cross-correlation between distinct abstract
CPU would critically suffer. On the other extreme, seemingly triggers. The values represent the number of errata documents
innocuous wrong values in performance monitors could as that require at least these two triggers.
well have critical consequences [66], not only on performance
due to incorrect monitoring but also on security, since several
recently proposed security defenses depend on the integrity of with other components (DRAM, memory-mapped components,
performance counters [67]–[76]. Wrong performance counter peripherals such as PCIe) seems to be particularly challenging,
values open exploitable breaches in these defense systems. thus resulting in many bugs. Such inputs correspond to stimuli
B. Insights that are difficult to supply to simulation or emulation prototypes
In this section, we leverage RemembERR to present new that solely rely on the logical operation and that rule out power
insights about the most common triggers, contexts, and obser- or peripheral physical layer considerations. Such bugs seem
vations. To avoid any bias in our analysis, we use RemembERR to be mostly discoverable by silicon testing. On the contrary,
with deduplicated (unique) errata. only five errata for AMD and one for Intel mention that the
bug can only be triggered in simulation.
Triggers. Our analysis starts by studying the most frequent
triggers for Intel and AMD designs. We present the re- (O7) Observation. Most errata require specific MSR in-
sults in Figure 10. We can see that the most frequent teraction or configuration combined with throttling, power
triggers are either related to specific configurations set state transitions, or peripheral inputs.
up by writing to model-specific registers (trg_CFG_wrg),
power throttling (trg_POW_tht), or to power state transi- Figure 11 shows how many errata have a certain number of
tions (trg_POW_pwc). More generally, many bugs require triggers. 14.4% of the errata do not specify any clear trigger or
triggers related to power management, virtualization, external refer to trivial triggers such as usual load and store operations
inputs, or features such as debugging or tracing. This suggests or intense workloads, and are therefore excluded from the
that implementing power management or communicating figure. Mixing the errata from the two vendors, in total 49%
Trigger occurrences (%) Relative trigger representation (%)
Core 1 (D)
Core 1 (M)
Core 2 (D)
Core 2 (M)
Core 3 (D)
Core 3 (M)
Core 4 (D)
Core 4 (M)
Core 5 (D)
Core 5 (M)
Core 6
Core 7-8
Core 8-9
Core 10
Core 11
Core 12
Trigger occurrences (%) Trigger occurrences (%)
60 trg CFG trg FEA trg MBR trg POW 3
trg EXT trg FLT trg MOP trg PRV Intel AMD
50 2
40
30 1
20
0
10
0 trg EXT bus trg EXT iom trg EXT pci trg EXT ram trg EXT rst trg EXT usb
Fig. 15. Relative representation of triggers related to external
Fig. 13. Trigger classes over Intel Core generations. stimuli between Intel and AMD.
20 Intel AMD 4 Intel AMD
15 3
10 2
5 1
0 trg MBR trg MOP trg FLT trg PRV trg CFG trg POW trg EXT trg FEA 0 trg FEA cus trg FEA dbg trg FEA fpu trg FEA mon trg FEA tra trg FEA cid
Fig. 14. Relative representation of trigger classes between Intel Fig. 16. Relative representation of triggers related to specific and AMD. features between Intel and AMD.
of the errata require at least two combined triggers to cause external communication have constantly been dominating.
a faulty behavior. However, we cannot derive whether bugs Without resorting to the former, more than 60% of the known
involving multiple triggers are rare or have been tested less. errata cannot be reproduced in the 10th generation. Errata
8.7% of Intel and 20.8% of AMD unique errata mention that triggered by privilege transitions are gaining importance in
a “complex set of conditions” is required to trigger the bug. the last generation. Importantly, all trigger classes are always
We ignored these indications as they are not precise enough to necessary to trigger some bugs, except in the latest two
be exploited reliably. generations. We additionally note that errata increasingly relate
Figure 12 shows pairwise correlations between triggers to specific features (trg_FEA), again, except for the latest
over all examined errata from AMD and Intel. This figure two generations. Arguably, the latter generations may be too
provides two specific insights: the relevant complex triggers recent to draw conclusions, as we expect more errata to be
and their interaction. Triggers typically interacting with other released in the coming months and years.
triggers are visible through highly populated lines. Regarding
concrete interaction, a complex trigger can consist of debug (O9) Observation. It is necessary to apply all trigger
features (trg_FEA_dbg) and virtual machine state transi- classes to trigger all known bugs.
tions (trg_PRV_vmt), as we can see their intersection is
salient. This insight is crucial for an efficient and thorough Figure 14 shows the relative trigger class representation
testing campaign. For example, many bugs involving DDR between Intel and AMD errata. In this figure, for each vendor,
(trg_EXT_vmt) or PCIe (trg_EXT_pci) will never be we counted the total number of triggers for all unique errata and
triggered until power levels change. grouped them by the trigger classes. Overall, the representation
of each trigger class is highly similar between the two vendors,
(O8) Observation. Some abstract triggers tend to correlate which is interesting given that not only the designs are different,
strongly, while most do not. but also the vendors, testing and validation processes certainly
substantially differ. Only the trigger classes related to external
Figure 13 shows how the trigger classes evolved over stimuli and specific features vary significantly between the two
different generations of Intel Core designs. Notably, errata vendors.
triggered at memory boundaries (trg_MBR) are absent in the
two latest Intel Core generations. This could be explained (O10) Observation. The representation of trigger classes
by different reasons: Intel’s testing approach might have over the errata corpora is very similar for Intel and AMD.
become more rigorous in this direction, or this kind of bug
is now more difficult to find, or they have not yet been Figures 15 and 16 show a more specific analysis of the
found and reported. Errata triggered by specific features or two latter trigger classes and clearly indicate the more specific
Affected errata (%) Affected errata (%)
eff CRP reg ctx PRV vmg eff HNG hng eff HNG unp ctx FEA sec eff CRP prf eff FLT mca ctx PHY tmp eff FLT fsp ctx PRV boo eff EXT pci eff FLT fms ctx PRV smm eff EXT usb eff EXT ram ctx PRV vmh eff EXT mmd ctx PHY vol eff FLT fid eff FLT unc ctx CFG sgc eff HNG boo eff HNG crh ctx PHY pkg
Affected errata (%)
MCx STATUS
MCx ADDR
IBS OP DATAx
IBS FETCH CTL
CPUID PWR THERM
PERF LEGACY CTL3
PERF LEGACY CTL0
IBS OP CTL
BR INST RETIRED
RTIT STATUS
MCx STATUS
MCx ADDR
IBS OP DATAx
IBS FETCH CTL
PERF LEGACY CTR
IBS OP CTL
HWCR
MCx MISC
PStateDef
MCx CTL
Intel AMD
7.5 6.7 6.9
5.0 4.6 6
2.5 3.5 3.5 1.8 1.7 4 4.2
0.0 1.2 0.4 0.3 2 1.6 1.2 1.2 0.9 0.8 0.8 0.8 0.6 0.6 2.9 1.91.6 1.3 1.1 1.1 1.1 0.8 0.5
0
Fig. 17. Most frequent contexts of all errata.
20 19.6 17.3 13.7 13.5 Fig. 19. Most frequent MSR containing observable effects for
10 8.7 7.9 7.4 6.7 4.5 4.3 4 3.2 2.5 1.2 1.2 Intel and AMD.
0
(O12) Observation. Corrupted registers and hangs are the
most common observable effect on Intel and AMD designs.
Fig. 18. Most frequent effects for all errata. Model Specific Registers. Based on our previous observation
that corrupted registers are the most common observable
effect, we wanted to know which registers provide information
differences between Intel and AMD errata. Concerning the about unexpected behavior. Figure 19 shows the most frequent triggers related to external stimuli, it is important to note that observable effects in Intel and AMD designs. For both some CPUs offload certain peripheral functionalities to an vendors, the machine check status registers (MCx_STATUS external chipset whose errata are not necessarily included in and MCx_ADDR) witness a bug in most cases (7.1% to 8.5% the documents under study. Concerning the triggers related of all unique errata), followed by Instruction Based Sampling to specific features, we observe a clear overrepresentation of (IBS) registers and performance counters. triggers related to custom features and tracing features in Intel compared to AMD. (O13) Observation. Among MSRs, Machine Check Status Contexts. In the next step of our study, we want to determine Registers most often indicate a bug’s occurrence. the context that most of the bugs require. Similar to the triggers, In summary, we designed a new hierarchical errata clas- this knowledge is crucial for efficiently testing designs as sification scheme that helps underlining new insights about certain bugs may only occur in specific contexts. reported bugs in complex designs. In Section VI, we concretely Figure 17 shows the most frequent contexts among Intel discuss how these insights may lead to improvements in design and AMD errata. Our data shows that running from within validation methodologies and toolchains. a virtual machine (ctx_PRV_vmg) is particularly prone to bugs. An explanation for this could be that today’s hardware VI. APPLICATIONS TO DESIGN TESTING virtualization extensions (e.g., Intel’s VT-x or AMD’s SVM) In this section, we answer two questions. First, why do are complex and deeply rooted in the CPU’s design, making design testing and validation tools and methodologies, widely their rigorous testing more challenging. used in the industry, fail at detecting bugs reported in errata? (O11) Observation. Most errors occur in the context of Second, how would they benefit from RemembERR to detect hardware support for virtual machine guests. past, present, and future bugs? We focus on how Remem- bERR can improve different families of testing and validation methodologies rather than focusing on specific tools, given that Effects. Next, we investigated which effects are the most vendors often make use of in-house tools [17]. valuable indicators for determining whether a bug was triggered. Figure 18 shows the most frequent observable effects in A. Dynamic methods Intel and AMD designs. Most bugs manifest themselves as a Dynamic methods consist of applying inputs to a simulated, corrupted register (eff_CRP_reg), a hang (eff_HNG_hng), emulated, or physical (manufactured) design and verifying or an unpredictable behavior (eff_HNG_unp). While an compliance of signals with a specification or expected values. unpredictable behavior is not clear (vendors do not provide While simulation [39]–[42], [77] and emulation [78], [79] are more information in these cases), the first two cases can easily useful to find simple bugs early in the design process, silicon be observed and may provide useful indicators for discovering testing is necessary to find many complex bugs [26]–[30]. new bugs. Two major challenges when looking for bugs with dynamic
methods are the immense input space and the vast observation prevent timing-sensitive bugs from happening.
space, where state observation may interfere with attempts to Besides, recent fuzzing work would benefit from enhanced
trigger bugs. In both cases, RemembERR offers potential for observation heuristics. RFUZZ [13] is incapable of discovering
improvement. any bug on its own [12] as it does not compare its state with any
Challenge: input space. CPUs usually have many pins and reference. Authors of TheHuzz [12] confirmed that directing
take sequential inputs over many cycles until a bug is eventually observations is one major challenge for a synthesizable porting
triggered; therefore, an exhaustive exploration is infeasible. A of their simulator-based testing system. DifuzzRTL [10] relies
common response to this challenge is Constrained Random on a golden model implemented in software and may benefit
Verification (CRV) [2]. CRV applies a series of input signals from limiting its observation volume as well. Knowledge about
that comply with a set of constraints, such as respecting a bus which design parts to observe, in correlation with the supplied
protocol. However, while this method can find shallow bugs in inputs, has the potential to empower such new fuzzer proposals.
common design paths, the probability of triggering complex Runtime detection. Previous work proposes inserting pervasive
bugs is comparatively small. CPU modifications for online bug detection [14], [15], [36],
Today’s fuzzers have not settled on seed input corpora or [37]. They are all data-driven, and in particular, each work
a way to generate them. For example, RFUZZ [13] requires performed an ad-hoc partial and non-systematic errata study.
that “only some parameters to the fuzzer, such as the mutation RemembERR provides all the necessary data to strengthen
technique and seed inputs to use, need to be specified by the these systems and foster future work in this area without
user.”. While reference [80] pretends to improve over the state requiring researchers to conduct time-consuming errata studies
of the art using an empty seed file, some successful experiments repetitively.
were seeded with some input sequences that stress interesting
design features. DifuzzRTL [81] does not specify how it
chooses its initial seed corpus, while HyperFuzzing [82] only B. Formal methods
vaguely says that it requires “an initial pool of inputs for the
fuzzer seeded with a few interesting behaviors.” TheHuzz [12] Formal methods statically analyze a design along with
takes its configuration instructions statically from existing properties that are usually specified manually [16]. For instance,
codebases, does not target specific functionalities, and samples Formal Property Verification (FPV) [17], [18], [24], [90], [91]
its test instructions uniformly. Therefore, the young movement ensures that a set of assertions is never violated in a given
toward fuzzing hardware designs will seemingly profit from a design. This technique is often used in two design testing
more carefully selected initial input corpus. stages [19]: (a) after thorough CRV to validate corner cases, and
Active research has targeted input generation for the (b) when a silicon bug happens that has not been detected before. pre-silicon phase [3]–[9], [20]–[23], [83]–[89], but it does Another instance of a formal method is Secure Path Verification not extend to emulation or silicon testing. Therefore, input (SPV) [90]–[94], which checks for unexpected information generation for emulation or silicon testing is still an open flows across designs, therefore allowing for design-global problem. RemembERR provides the best possible solution policies that are otherwise difficult to express with classical that does not require modifying the design in silicon. This assertions. Formal methods are subject to state explosion and is a significant advantage as modifying physical design is an require a manual definition of the policies. enormous effort and ends up not testing the original design Challenge: state explosion. A common approach for tackling in all aspects. RemembERR precisely indicates which sets of the state explosion problem [16], [45]–[47] is to treat parts of inputs empirically interact and could trigger bugs, as shown the design as a “black box” and replace them by a model [25]. in Figure 12. This knowledge can then be integrated into However, this limits the validation to properties specific to automatic dynamic testing of an emulated or manufactured those parts that are not black-boxed [16]. An interesting yet design, taking the best of both worlds: targeted inputs and high unaddressed challenge is choosing the subset of the design that execution speed under real-world conditions. can be black-boxed to find a given class of bugs. Challenge: observation space. A too frequent and exhaustive RemembERR provides a large amount of empirical data observation can have detrimental effects. RemembERR pro- for identifying modules that typically interact with each other vides empirical observation points that indicate CPU malfunc- and could cause bugs. Our most immediate observation is that tions and correlates them with the set of input types provided. power management modules seem to have been vastly excluded This enables a much more fine-grained observation strategy, from the parts of the design that are formally verified. We argue where the observation footprint is minimal. that the input correlation knowledge provided by RemembERR In simulation, an excessive observation causes a longer will substantially help to scope black-boxing more suitably. run time. In emulation and silicon testing, the observation Challenge: handwritten policies. Policies and assertions challenge becomes critical because almost all observations must are usually hand-written, for example, as SystemVerilog be performed online, e.g., by reading performance counters. assertions [95] or in Property Specification Language [96]. Excessive observations not only reduce testing performance This process is manual and error-prone, and to the best of our but also hinder triggering bugs because heavy inspection may knowledge, no existing work proposed a way to resolve it.
RemembERR provides the necessary data to foster research Table VII. An erratum in the proposed format.
in automatic data-driven policy generation for formal verifica- ID: [Some unique identifier shared with identical errata in other designs]
tion. Title: x87 FDP Value May be Saved Incorrectly
Triggers:
C. Manual inspection Abstract: Trg_FEA_fpu
Concrete: Execution of FSAVE, FNSAVE, FSTENV, or FNSTENV
Manual inspection is integral to design validation [16] but is Contexts:
challenging for complex designs and bugs of interest. Without Abstract: Ctx_PRV_rea
any knowledge of common errors, manual inspection is deemed Concrete: Operating in real-address mode or virtual-8086 mode
to fail. For example, an engineer responsible for testing a Effects:
memory controller will benefit from the dozens of concrete Abstract: Eff_HNG_unp
Concrete: Incorrect value for the x87 FDP
bug instances that happened in the past and in other designs Comments: This erratum does not apply if the last non-control x87 instruction
to ensure that they do not repeat. However, the current state had an unmasked exception.
of RemembERR is limited by the black-box nature of errata Root cause: [Here, an explanation of the root cause may be provided]
as they are published today. A description of the root cause Workaround: None identified.
associated with each erratum would provide enormous help Status: No fix.
in empirically distinguishing safe from dangerous hardware
design practices. tremendously from the errors of their more mature siblings,
VII. DISCUSSION which will help in preventing similar mistakes.
We discuss selected observations we made while creating Recommendations for errata formatting. Current errata and interpreting RemembERR and provide further information documents are formatted for humans, and given the mistakes for its future users. present in the documents, it is likely that vendors do not have Patchable errors. The errata documents do not show all known any form of a systematic knowledge base for erratum storage errors present in CPUs when they are released. Two classes of and analysis. One vendor has confirmed not having such a bugs are missing. First, bugs that are patchable by microcode database, and that their only source of errata knowledge is the updates. Such updates are usually not documented but can be errata documents and the employees’ experience. reverse-engineered [97]. Second, bugs that are no longer valid, We propose a new format for errata descriptions, as exem- e.g., because a re-spin has been released and the older version plified by the transformation of Table I into Table VII, because is no longer officially supported. Errata of this type (about 2%) the current state of the art for errata description (composed of are listed in the summary of errata documents, but details (e.g., a title, a problem description, implications, workarounds, and the description) remain hidden. a status field) is unsatisfying for systematic analysis. Other microprocessor vendors. Intel and AMD are not the Root cause. The root cause information is currently absent only major vendors of complex microprocessors that provide from almost all errata. One CPU vendor confirmed that triggers errata for their designs. ARM, for example, does so as well. and effects are intentionally left inaccurate to avoid revealing We focused our work on Intel and AMD because they produce design details, therefore there is limited hope for root cause their design entirely from scratch, without relying on other publication but such databases may be maintained internally. vendors of major blocks (e.g., complete cores). Therefore, we With information on root causes, an errata database would go expect these errata to be the most insightful as the vendors one step further than RemembERR by providing empirical data control the whole design process chain. Besides that, Intel and correlated with triggers, contexts and effects for identifying AMD microprocessors have a long history of designs, which root causes, which is known to be a difficult problem [27], [29], gives us access to valuable long-term data. [30], [33], [98]. Root cause information would additionally Learning from the past. The entire corpus of errata that we first underline which design parts are the most difficult to analyzed relies exclusively on bugs that have already been implement correctly and what are the most common mistakes. discovered, albeit some of them more recently. Therefore, VIII. RELATED WORK our analysis cannot tell much about yet undiscovered bug Following, we present existing work that examines pub- classes and trigger-context-effect correlations. However, in lic CPU bug information (Section VIII-A). Afterward, we Section IV we showed that bugs are sometimes rediscovered introduce previous work that improves silicon testing (Sec- years apart, suggesting that industrial testing and validation tion VIII-B). methods can still gain rigor from our analysis. For example, our findings may help direct testing efforts to specific areas A. Errata-based (i.e., contexts) that are known to be most affected by bugs. The Tables VIII and IX summarize work that studied errata efficiency of design testing can be improved by focusing on from open-source and commercial CPUs, respectively. Previous the most common triggers of these areas and the components work provides fragmented information extracted from errata to where they typically have an observable effect. In addition, provide a classification that justifies a specific approach that the growing open-source microprocessor community will learn solves a given problem. Unlike RemembERR, none provides
Table VIII. Summary of work that examined errata in open-source CPUs.
Year Errata Criteria Goal
Constantinides et al. [14] 2008 296 (OpenSPARC) Type Programmable module to react to errata online Miroslav et al. [99] 2003 280 (students) Type Design a testing flow capable of catching errors Van Campenhout et al. [98] 2000 Students & research Type Recommendations for end users
Table IX. Summary of work that examined errata in commercial CPUs.
Year Errata Criteria Goal
RemembERR 2022 2,563 Triggers, contexts, effects Provide support for data-driven design testing Hicks et al. [15] 2015 301 Severity Monitor hardware security invariants Wagner et al. [38] 2008 37 Location Find internal signals corresponding to bugs Narayanasamy et al. [37] 2006 172 Type, severity Programmable module to react to errata online Sarangi et al. [36] 2006 470 Location, severity Programmable module to react to errata online Aviˇzienis et al. [35] 1999 535 Location, severity Taxonomy for dependable systems Wichmann [100] 1993 – – Recommendations for end users
clear insights on representative bug triggers and effects in ACKNOWLEDGMENTS modern CPUs. Insights from RemembERR are exploitable for The authors would like to thank the anonymous reviewers future research in design testing and validation. for their valuable feedback. This work was supported by a B. Directed silicon testing Microsoft Swiss JRC grant, the Swiss State Secretariat for Education, Research and Innovation under contract number There are two directions aiming to improve inputs and MB22.00057 (ERC-StG PROMISE), and the Swiss National observations for silicon testing. Science Foundation under NCCR Automation, grant agreement Eliminating the golden model. Golden models are a bottle- 51NF40 180545. neck for silicon testing. Wagner et al. [32] propose to use X. ARTIFACT APPENDIX complementary pairs of instruction blocks to ensure that the A. Abstract CPU’s architectural state is left untouched after execution if Our artifacts include the annotated RemembERR database no bug has been triggered on the CPU. Foutris et al. [34] along with the source code used to build and annotate it. We identify equivalences between instructions from different ISAs also provide code of all experiments described in this paper, and to compare executions on largely different CPUs. provide a Docker image to make reproducing the results easier. Bug observability. In some cases, it is difficult to triage a Further, we added an example script to encourage readers to bug. Lin et al. [33], [101] propose methods to transform write their own queries. Note that generating RemembERR an instruction sequence to accelerate the observability of a from scratch is a lot of work: parsing and annotating involved triggered bug. Farahmandi et al. [102] propose an observability tens of hours of high-focus work for two humans. Reproducing measure consisting of test sequences. the experiments is quick (<1 h). RemembERR is complementary to and compatible with The Readme.md file in our repository contains detailed these directions by providing insights for more effective input instructions. generation and guidelines about the elements to observe for B. Artifact check-list (meta-information) efficient testing. • Data set: The RemembERR database IX. CONCLUSION •• Run-time environment: Python3 Hardware: Any Linux machine We analyzed 2,563 errata from all Intel Core and AMD • Output: Figures and numbers CPUs since 2008. Individually, these errata provide little insight • Experiments: All data shown in the paper other than a description of a particular bug, but collectively • How much disk space is required? ≈3 GB (including the they provide insightful information about gaps in current • software dependencies) How much time is needed to prepare workflow: <1h design testing and validation practices. To this end, we built • How much time is needed to complete experiments: <1h RemembERR, a large-scale database of annotated errata. We • Publicly available?: Yes solved the challenge of unclear triggers by the observation that • Code licenses (if publicly available)?: GPLv3 triggers are almost always conjunctive. In contrast, contexts and • Data licenses (if publicly available)?: GPLv3 effects are disjunctive: observing the most convenient location • Workflow framework used?: Luigi (Python-based, pip package) is sufficient. In our analysis using RemembERR, we discovered • Archived?: https://doi.org/10.5281/zenodo.7011959 the most common triggers, contexts, and effects in errata, which C. Description we then correlated to provide concrete guidelines for the next 1) How to access: https://github.com/comsec-group/reme generation of design testing and validation tools. mberr
- Hardware dependencies: None. G. Experiment customization
- Software dependencies: We provide instructions for We provide an example custom script that bootstrap the Ubuntu. The apt dependencies are the following: learning process of how to use the database for custom analyses. build-essential Please refer to Readme.md. libpoppler-cpp-dev software-properties-common H. Notes python3.8-dev None. libgl1 libglib2.0-0 I. Methodology software-properties-common Submission, reviewing and badging methodology: git cm-super • https://www.acm.org/publications/policies/artifact- dvipng review-badging texlive-latex-extra • http://cTuning.org/ae/submission-20201122.html texlive-fonts-recommended • http://cTuning.org/ae/reviewing-20201122.html python3.8 REFERENCES python3-pip python3-distutils [1] A. Danowitz, K. Kelley, J. Mao, J. P. Stevenson, and M. Horowitz, python3-apt “Cpu db: Recording microprocessor history: With this open database, you can mine microprocessor trends over the past 40 years.” Queue, vol. 10, no. 4, pp. 10–27, 2012.
The pip dependencies are the following: [2] A. B. Mehta, Constrained Random Verification (CRV). Springer, 2018. camelot-py==0.10.1 [3] A. Adir, E. Almog, L. Fournier, E. Marcus, M. Rimon, M. Vinov, and A. Ziv, “Genesys-pro: innovations in test program generation for colorama==0.4.4 functional processor verification,” IEEE Design & Test of Computers, luigi==3.0.3 [4] vol. 21, no. 2, pp. 84–93, 2004. numpy==1.22.3 A. Ahmed and P. Mishra, “Quebs: Qualifying event based search in concolic testing for validation of rtl models,” in 2017 IEEE International openpyxl==3.0.9 Conference on Computer Design (ICCD), 2017, pp. 185–192. pandas==1.4.2 [5] A. Ahmed, F. Farahmandi, and P. Mishra, “Directed test generation pdftotext==2.2.2 using concolic testing on rtl models,” in 2018 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 2018, pp. pikepdf==5.1.1 [6] 1538–1543. readchar==3.0.5 M. Chen, P. Mishra, and D. Kalita, “Automatic rtl test generation from systemc tlm specifications,” ACM Transactions on Embedded Computing matplotlib==3.5.1 Systems (TECS), vol. 11, no. 2, pp. 1–25, 2012. opencv-python==4.5.5.64 [7] M. Chen, X. Qin, H.-M. Koo, and P. Mishra, System-level validation: high-level modeling and directed test generation techniques. Springer See Readme.md or requirements.txt in our reposi- [8] Science & Business Media, 2012. L. Liu and S. Vasudevan, “Efficient validation input generation in rtl tory for details. by hybridized source code analysis,” in 2011 Design, Automation & 4) Data sets: RemembERR (provided as part of our arti- [9] Test in Europe. IEEE, 2011, pp. 1–6. E. Sadredini, R. Rahimi, P. Foroutan, M. Fathy, and Z. Navabi, facts). “An improved scheme for pre-computed patterns in core-based soc 5) Models: None. architecture,” in 2016 IEEE East-West Design & Test Symposium (EWDTS). IEEE, 2016, pp. 1–6. D. Installation [10] S. Nilizadeh, Y. Noller, and C. S. Pasareanu, “Diffuzz: differential fuzzing for side-channel analysis,” in 2019 IEEE/ACM 41st International Clone the repository and install the apt and Python depen- [11] Conference on Software Engineering (ICSE). IEEE, 2019, pp. 176–187. T. Li, H. Zou, D. Luo, and W. Qu, “Symbolic simulation enhanced dencies. You may use a Python virtual environment: coverage-directed fuzz testing of rtl design,” in 2021 IEEE International python3 -m venv /.venv/rememberr [12] Symposium on Circuits and Systems (ISCAS). IEEE, 2021, pp. 1–5. source /.venv/rememberr/bin/activate A. Tyagi, A. Crump, A.-R. Sadeghi, G. Persyn, J. Rajendran, P. Jauernig, and R. Kande, “Thehuzz: Instruction fuzzing of processors using golden- reference models for finding software-exploitable vulnerabilities,” in Alternatively, you may find all tools preinstalled in the [13] 31st USENIX Security Symposium (USENIX Security 22), 2022. Docker image that we provide. K. Laeufer, J. Koenig, D. Kim, J. Bachrach, and K. Sen, “Rfuzz: Coverage-directed fuzz testing of rtl on fpgas,” in ICCAD). IEEE, 2018, pp. 1–8. E. Experiment workflow [14] K. Constantinides, O. Mutlu, and T. Austin, “Online design bug detection: Rtl analysis, flexible mechanisms, and evaluation,” in 2008 Follow the instructions in Readme.md. 41st IEEE/ACM International Symposium on Microarchitecture. IEEE, 2008, pp. 282–293. F. Evaluation and expected results [15] M. Hicks, C. Sturton, S. T. King, and J. M. Smith, “Specs: A lightweight runtime mechanism for protecting software from security- Numbers will be provided in stdout and figures in the critical processor bugs,” in Proceedings of the Twentieth International directory specified in Readme.md. Conference on Architectural Support for Programming Languages and Operating Systems, 2015, pp. 517–529.
[16] G. Dessouky, D. Gens, P. Haney, G. Persyn, A. Kanuparthi, H. Khattri, [37] S. Narayanasamy, B. Carneal, and B. Calder, “Patching processor design J. M. Fung, A.-R. Sadeghi, and J. Rajendran, “{HardFails}: Insights errors,” in 2006 International Conference on Computer Design. IEEE, into {Software-Exploitable} hardware bugs,” in 28th USENIX Security 2006, pp. 491–498. Symposium (USENIX Security 19), 2019, pp. 213–230. [38] I. Wagner, V. Bertacco, and T. Austin, “Using field-repairable control [17] L. Fix, “Fifteen years of formal property verification in intel,” 25 Years logic to correct design errors in microprocessors,” IEEE Transactions of Model Checking, pp. 139–144, 2008. on computer-aided design of integrated circuits and systems, vol. 27, [18] L. Fix and K. McMillan, “Formal property verification,” in EDA for IC no. 2, pp. 380–393, 2008. System Design, Verification, and Testing. CRC Press, 2018, pp. 20–1. [39] F. Solt, B. Gras, and K. Razavi, “Cellift: Leveraging cells for scalable [19] D. S. Vincenzoni, “Formal property verification: A tale of two methods,” and precise dynamic information flow tracking in rtl,” in 31st USENIX https://www.edn.com/formal-property-verification-a-tale-of-two- Security Symposium (USENIX Security 22), 2022, pp. 2549–2566. [20] methods/, accessed: 2022-06-21. [40] Siemens, “Modelsim,” https://eda.sw.siemens.com/en-US/ic/modelsim/, M. R. Fadiheh, J. Urdahl, S. S. Nuthakki, S. Mitra, C. Barrett, D. Stoffel, accessed: 2022-06-21. and W. Kunz, “Symbolic quick error detection using symbolic initial [41] Synopsys, “Vcs,” https://www.synopsys.com/verification/simulation/vcs state for pre-silicon verification,” in 2018 Design, Automation & Test .html, accessed: 2022-06-21. [21] in Europe Conference & Exhibition (DATE). IEEE, 2018, pp. 55–60. [42] Cadence, “Xcelium logic simulation,” https://www.cadence.com/ko KR/ E. Singh, K. Devarajegowda, S. Simon, R. Schnieder, K. Ganesan, home/tools/system-design-and-verification/simulation-and-testbench- M. Fadiheh, D. Stoffel, W. Kunz, C. Barrett, W. Ecker et al., verification/xcelium-simulator.html, accessed: 2022-06-21. “Symbolic qed pre-silicon verification for automotive microcontroller [43] J. Balkind, K. Lim, F. Gao, J. Tu, D. Wentzlaff, M. Schaffner, F. Zaruba, cores: Industrial case study,” in 2019 Design, Automation & Test in and L. Benini, “Openpiton+ ariane: The first open-source, smp linux- [22] Europe Conference & Exhibition (DATE). IEEE, 2019, pp. 1000–1005. booting risc-v system scaling from one to many cores,” in Workshop on K. Ganesan, F. Lonsing, S. S. Nuthakki, E. Singh, M. R. Fadiheh, Computer Architecture Research with RISC-V (CARRV), 2019, pp. 1–6. W. Kunz, D. Stoffel, C. Barrett, and S. Mitra, “Effective pre-silicon [44] S.-H. Chen and J.-Y. Lin, “Implementation and verification practices verification of processor cores by breaking the bounds of symbolic of dvfs and power gating,” in 2009 International Symposium on VLSI [23] quick error detection,” arXiv preprint arXiv:2106.10392, 2021. Design, Automation and Test. IEEE, 2009, pp. 19–22. V. M. Suryasarman, S. Biswas, and A. Sahu, “Automation of test [45] E. M. Clarke, W. Klieber, M. Nov´aˇcek, and P. Zuliani, “Model checking program synthesis for processor post-silicon validation,” Journal of and the state explosion problem,” in LASER Summer School on Software [24] Electronic Testing, vol. 34, no. 1, pp. 83–103, 2018. Engineering. Springer, 2011, pp. 1–30. Cadence, “Jasper fpv app,” https://www.cadence.com/en US/home/too [46] F. Erata, S. Deng, F. Zaghloul, W. Xiong, O. Demir, and J. Szefer, ls/system-design-and-verification/formal-and-static-verification/jasper- “Survey of approaches for security verification of hardware/software gold-verification-platform/formal-property-verification-app.html, systems,” Cryptology ePrint Archive, 2016. accessed: 2022-06-21. [47] F. Farahmandi, Y. Huang, and P. Mishra, “Formal approaches to [25] O. Demir, W. Xiong, F. Zaghloul, and J. Szefer, “Survey of approaches hardware trust verification,” in The Hardware Trojan War. Springer, for security verification of hardware/software systems.” IACR Cryptol. 2018, pp. 183–202. ePrint Arch., vol. 2016, p. 846, 2016. [48] A. Gupta, M. KiranKumar, and R. Ghughal, “Formally verifying graph- [26] A. Adir, M. Golubev, S. Landa, A. Nahir, G. Shurek, V. Sokhin, ics fpu,” in International Symposium on Formal Methods. Springer, and A. Ziv, “Threadmill: A post-silicon exerciser for multi-threaded 2014, pp. 673–687. processors,” in Proceedings of the 48th Design Automation Conference, 2011, pp. 860–865. [49] T. Schubert, “High-level formal verification of next-generation micro- [27] O. Friedler, W. Kadry, A. Morgenshtein, A. Nahir, and V. Sokhin, processors,” in Proceedings 2003. Design Automation Conference (IEEE “Effective post-silicon failure localization using dynamic program Cat. No. 03CH37451). IEEE, 2003, pp. 1–6. slicing,” in 2014 Design, Automation & Test in Europe Conference & [50] M. M. Wilding, D. A. Greve, R. J. Richards, and D. S. Hardin, “Formal Exhibition (DATE). IEEE, 2014, pp. 1–6. verification of partition management for the aamp7g microprocessor,” in [28] H. D. Foster, “Trends in functional verification: A 2014 industry study,” Design and Verification of Microprocessor Systems for High-Assurance in Proceedings of the 52nd Annual Design Automation Conference, Applications. Springer, 2010, pp. 175–191. 2015, pp. 1–6. [51] A. Armstrong, T. Bauereiss, B. Campbell, A. Reid, K. E. Gray, [29] J. Keshava, N. Hakim, and C. Prudvi, “Post-silicon validation challenges: R. Norton-Wright, P. Mundkur, M. Wassell, J. French, C. Pulte et al., How eda and academia can help,” in Design Automation Conference. “Isa semantics for armv8-a, risc-v, and cheri-mips,” POPL, 2019. IEEE, 2010, pp. 3–7. [52] T. Bourgeat, I. Clester, A. Erbsen, S. Gruetter, A. Wright, and [30] D. Lin, E. Singh, C. Barrett, and S. Mitra, “A structured approach to A. Chlipala, “A multipurpose formal risc-v specification,” arXiv preprint post-silicon validation and debug using symbolic quick error detection,” arXiv:2104.00762, 2021. in 2015 IEEE International Test Conference (ITC). IEEE, 2015, pp. [53] S. Greenberg, J. Rabinowicz, and E. Manor, “Selective state retention 1–10. power gating based on formal verification,” IEEE Transactions on [31] D. Josephson, “The good, the bad, and the ugly of silicon debug,” in Circuits and Systems I: Regular Papers, vol. 62, no. 3, pp. 807–815, Proceedings of the 43rd annual Design Automation Conference, 2006, 2014. pp. 3–6. [54] A. M. Gharehbaghi and M. Fujita, “Specification and formal verification [32] I. Wagner and V. Bertacco, “Reversi: Post-silicon validation system for of power gating in processors,” in Fifteenth International Symposium modern microprocessors,” in 2008 IEEE International Conference on on Quality Electronic Design. IEEE, 2014, pp. 604–610. Computer Design. IEEE, 2008, pp. 307–314. [55] H. Choi, M.-K. Yim, J.-Y. Lee, B.-W. Yun, and Y.-T. Lee, “Formal [33] D. Lin, T. Hong, F. Fallah, N. Hakim, and S. Mitra, “Quick detection verification of an industrial system-on-a-chip,” in Proceedings 2000 of difficult bugs for effective post-silicon validation,” in DAC Design International Conference on Computer Design. IEEE, 2000, pp. 453– Automation Conference 2012. IEEE, 2012, pp. 561–566. 458. [34] N. Foutris, D. Gizopoulos, M. Psarakis, X. Vera, and A. Gonzalez, [56] N. Reddy, S. Menon, and P. D. Joshi, “Validation challenges in recent “Accelerating microprocessor silicon validation by exposing isa trends of power management in microprocessors,” in 2020 IEEE diversity,” in Proceedings of the 44th Annual IEEE/ACM International International Symposium on Defect and Fault Tolerance in VLSI and Symposium on Microarchitecture, ser. MICRO-44. New York, NY, Nanotechnology Systems (DFT). IEEE, 2020, pp. 1–6. USA: Association for Computing Machinery, 2011, p. 386–397. [57] M. Dusanapudi, S. Fields, M. S. Floyd, G. L. Guthrie, R. Kalla, [Online]. Available: https://doi.org/10.1145/2155620.2155666 S. Kapoor, L. Leitner, C. F. Marino, J. McGill, A. Nahir et al., [35] A. Avizienis and Y. He, “Microprocessor entomology: a taxonomy of “Debugging post-silicon fails in the ibm power8 bring-up lab,” IBM design faults in cots microprocessors,” in Dependable Computing for Journal of Research and Development, vol. 59, no. 1, pp. 12–1, 2015. Critical Applications 7, 1999, pp. 3–23. [58] P. Mishra, R. Morad, A. Ziv, and S. Ray, “Post-silicon validation in the [36] S. R. Sarangi, A. Tiwari, and J. Torrellas, “Phoenix: Detecting and soc era: A tutorial introduction,” IEEE Design & Test, vol. 34, no. 3, recovering from permanent processor design bugs with programmable pp. 68–92, 2017. hardware,” in 2006 39th Annual IEEE/ACM International Symposium [59] D. Petrisko, F. Gilani, M. Wyse, D. C. Jung, S. Davidson, P. Gao, on Microarchitecture (MICRO’06). IEEE, 2006, pp. 26–37. C. Zhao, Z. Azad, S. Canakci, B. Veluri et al., “Blackparrot: An agile
open-source risc-v multicore for accelerator socs,” IEEE Micro, vol. 40, Association, Aug. 2022, pp. 3237–3254. [Online]. Available: https:
no. 4, pp. 93–102, 2020. //www.usenix.org/conference/usenixsecurity22/presentation/trippel
[60] F. Zaruba and L. Benini, “The cost of application-class processing: [81] J. Hur, S. Song, D. Kwon, E. Baek, J. Kim, and B. Lee, “Difuzzrtl: Energy and performance analysis of a linux-ready 1.7-ghz 64-bit risc- Differential fuzz testing to find cpu bugs,” in 2021 IEEE Symposium v core in 22-nm fdsoi technology,” IEEE VLSI, vol. 27, no. 11, pp. on Security and Privacy (SP). IEEE, 2021, pp. 1286–1303. [61] 2629–2640, 2019. [82] S. K. Muduli, G. Takhar, and P. Subramanyan, “Hyperfuzzing for soc C. Celio, D. A. Patterson, and K. Asanovic, “The berkeley out-of-order security validation,” in ICCAD, 2020, pp. 1–9. machine (boom): An industry-competitive, synthesizable, parameterized [83] F. Corno, E. Sanchez, M. Reorda, and G. Squillero, “Automatic test risc-v processor,” EECS Department, University of California, Berkeley, program generation: a case study,” IEEE Design & Test of Computers, [62] Tech. Rep. UCB/EECS-2015-167, 2015. vol. 21, no. 2, pp. 102–109, 2004. S. Mitra, S. A. Seshia, and N. Nicolici, “Post-silicon validation [84] G. Squillero, “Microgp—an evolutionary assembly program generator,” opportunities, challenges and recent advances,” in Design Automation Genetic programming and evolvable machines, vol. 6, pp. 247–263, [63] Conference. IEEE, 2010, pp. 12–17. 2005. M. Lipp, M. Schwarz, D. Gruss, T. Prescher, W. Haas, A. Fogh, [85] G. Squillero and G. Squillero, “Evolving assembly programs: how games J. Horn, S. Mangard, P. Kocher, D. Genkin et al., “Meltdown: Reading help microprocessor validation,” IEEE Transactions on Evolutionary kernel memory from user space,” in 27th USENIX Security Symposium Computation, vol. 9, pp. 695–706, 2005. (USENIX Security 18), 2018, pp. 973–990. [64] S. Van Schaik, A. Milburn, S. Osterlund, P. Frigo, G. Maisuradze,¨ [86] P. Bernardi, E. E. S. S´anchez, M. Schillaci, G. Squillero, and M. S. K. Razavi, H. Bos, and C. Giuffrida, “Ridl: Rogue in-flight data load,” Reorda, “An effective technique for the automatic generation of in 2019 IEEE Symposium on Security and Privacy (SP). IEEE, 2019, diagnosis-oriented programs for processor cores,” IEEE Transactions pp. 88–105. on Computer-Aided Design of Integrated Circuits and Systems, vol. 27, [65] M. Schwarz, M. Lipp, D. Moghimi, J. Van Bulck, J. Stecklina, no. 3, pp. 570–574, 2008. T. Prescher, and D. Gruss, “Zombieload: Cross-privilege-boundary data [87] E. S´anchez, M. S. Reorda, and G. Squillero, “Efficient techniques sampling,” in Proceedings of the 2019 ACM SIGSAC Conference on for automatic verification-oriented test set optimization,” International Computer and Communications Security, 2019, pp. 753–768. Journal of Parallel Programming, vol. 34, no. 1, pp. 93–109, 2006. [66] A. Carelli, A. Vallero, and S. Di Carlo, “Performance monitor counters: [88] D. Gizopoulos, M. Psarakis, M. Hatzimihail, M. Maniatakos, interplay between safety and security in complex cyber-physical systems,” A. Paschalis, A. Raghunathan, and S. Ravi, “Systematic software-based IEEE Transactions on Device and Materials Reliability, vol. 19, no. 1, self-test for pipelined processors,” IEEE VLSI, vol. 16, no. 11, pp. pp. 73–83, 2019. 1441–1453, 2008. [67] T. Zhang, Y. Zhang, and R. B. Lee, “Cloudradar: A real-time side- [89] J. Hudec and E. Gramatov´a, “An efficient functional test generation channel attack detection system in clouds,” in International Symposium method for processors using genetic algorithms,” Journal of Electrical on Research in Attacks, Intrusions, and Defenses. Springer, 2016, pp. Engineering, vol. 66, no. 4, p. 185, 2015. 118–140. [90] Synopsys, “Vc formal,” https://www.synopsys.com/verification/static- [68] M. Chiappetta, E. Savas, and C. Yilmaz, “Real time detection of and-formal-verification/vc-formal.html, accessed: 2022-06-21. cache-based side-channel attacks using hardware performance counters,” [91] Siemens, “Questa formal verification apps,” https://eda.sw.siemens.com Applied Soft Computing, vol. 49, pp. 1162–1174, 2016. /en-US/ic/questa/formal-verification/, accessed: 2022-06-21. [69] S. Ferracci, “Detecting cache-based side channel attacks using hardware [92] G. Cabodi, P. Camurati, S. F. Finocchiaro, C. Loiacono, F. Savarese, performance counters,” Ph.D. dissertation, Sapienza, University of Rome, and D. Vendraminetto, “Secure path verification,” in 2016 1st IEEE 2019. International Verification and Security Workshop (IVSW). IEEE, 2016, [70] M. Alam, S. Bhattacharya, D. Mukhopadhyay, and S. Bhattacharya, pp. 1–6. “Performance counters to rescue: A machine learning based safeguard [93] W. Hu, X. Wang, and D. Mu, “Security path verification through joint against micro-architectural side-channel-attacks,” Cryptology ePrint information flow analysis,” in 2018 IEEE Asia Pacific Conference on [71] Archive, 2017. Circuits and Systems (APCCAS). IEEE, 2018, pp. 415–418. X. Wang, C. Konstantinou, M. Maniatakos, and R. Karri, “Confirm: [94] Cadence, “Jasper spv app,” https://www.cadence.com/ko KR/home/too Detecting firmware modifications in embedded systems using hardware ls/system-design-and-verification/formal-and-static-verification/jasper- [72] performance counters,” in ICCAD). IEEE, 2015, pp. 544–551. gold-verification-platform/security-path-verification-app.html, accessed: X. Wang, C. Konstantinou, M. Maniatakos, R. Karri, S. Lee, P. Robison, 2022-06-21. P. Stergiou, and S. Kim, “Malicious firmware detection with hardware [95] E. Cerny, S. Dudani, J. Havlicek, D. Korchemny et al., SVA: the power performance counters,” IEEE Transactions on Multi-Scale Computing of assertions in systemVerilog. Springer, 2015. [73] Systems, vol. 2, no. 3, pp. 160–173, 2016. [96] R. Armoni, L. Fix, A. Flaisher, R. Gerth, B. Ginsburg, T. Kanza, R. Elnaggar, K. Chakrabarty, and M. B. Tahoori, “Run-time hardware A. Landver, S. Mador-Haim, E. Singerman, A. Tiemeyer et al., trojan detection using performance counters,” in 2017 IEEE Interna- “The forspec temporal logic: A new temporal property-specification [74] tional Test Conference (ITC). IEEE, 2017, pp. 1–10. language,” in International Conference on Tools and Algorithms for the Y. Xia, Y. Liu, H. Chen, and B. Zang, “Cfimon: Detecting violation Construction and Analysis of Systems. Springer, 2002, pp. 296–311. of control flow integrity using performance counters,” in IEEE/IFIP [97] P. Koppe, B. Kollenda, M. Fyrbiak, C. Kison, R. Gawlik, C. Paar, International Conference on Dependable Systems and Networks (DSN and T. Holz, “Reverse engineering x86 processor microcode,” in 26th 2012). IEEE, 2012, pp. 1–12. USENIX Security Symposium (USENIX Security 17), 2017, pp. 1163– [75] C. Li and J.-L. Gaudiot, “Online detection of spectre attacks using 1180. microarchitectural traces from performance counters,” in 2018 30th International Symposium on Computer Architecture and High Perfor- [98] D. Van Campenhout, T. Mudge, and J. P. Hayes, “Collection and analysis mance Computing (SBAC-PAD). IEEE, 2018, pp. 25–28. of microprocessor design errors,” IEEE Design & Test of Computers, [76] H. Sayadi, H. Wang, T. Miari, H. M. Makrani, M. Aliasgari, S. Rafatirad, vol. 17, no. 4, pp. 51–60, 2000. and H. Homayoun, “Recent advancements in microarchitectural security: [99] M. N. Velev, “Collection of high-level microprocessor bugs from formal Review of machine learning countermeasures,” in 2020 IEEE 63rd verification of pipelined and superscalar designs,” in International Test International Midwest Symposium on Circuits and Systems (MWSCAS). Conference, 2003. Proceedings. ITC 2003. Citeseer, 2003, pp. 138–138. IEEE, 2020, pp. 949–952. [100] B. A. Wichmann, “Microprocessor design faults,” Microprocessors and [77] Veripool, “Verilator, the fastest verilog/systemverilog simulator,” https: Microsystems, vol. 17, no. 7, pp. 399–401, 1993. //veripool.org/verilator/, accessed: 2022-06-21. [101] D. Lin, T. Hong, Y. Li, S. Eswaran, S. Kumar, F. Fallah, N. Hakim, [78] T. Feist, “Vivado design suite,” White Paper, vol. 5, p. 30, 2012. D. S. Gardner, and S. Mitra, “Effective post-silicon validation of system- [79] I. K. Ganusov, M. A. Iyer, N. Cheng, and A. Meisler, “Agilex™ on-chips using quick error detection,” IEEE Transactions on Computer- generation of intel® fpgas,” in 2020 IEEE Hot Chips 32 Symposium Aided Design of Integrated Circuits and Systems, vol. 33, no. 10, pp. (HCS). IEEE Computer Society, 2020, pp. 1–26. 1573–1590, 2014. Security Symposium (USENIX Security 22). Boston, MA: USENIX [102] F. Farahmandi and P. Mishra, “Observability-aware post-silicon test [80] T. Trippel, K. G. Shin, A. Chernyakhovsky, G. Kelly, D. Rizzo, generation,” in Post-Silicon Validation and Debug. Springer, 2019, pp. and M. Hicks, “Fuzzing hardware like software,” in 31st USENIX 111–123.