SOURCE ARCHIVE
EXTRACTED CONTENT
147,822 charsLinköping University | Department of Electrical Engineering
Master’s thesis, 30 ECTS | Electrical Engineering
2026 | LiTH-ISY-EX--26/5836--SE
Fuzzing IPv4 modules on FPGAs
– Design and comparison of on-chip and host-based fuzzers
Alexander Lindskog
Filip Ripstrand
Supervisor : Petter Källström
Examiner : Kent Palmkvist
External supervisor : Axel Stafström
Linköpings universitet
SE–581 83 Linköping
+46 13 28 10 00 , www.liu.se
Upphovsrätt
Detta dokument hålls tillgängligt på Internet - eller dess framtida ersättare - under 25 år från publicer- ingsdatum under förutsättning att inga extraordinära omständigheter uppstår.
Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner, skriva ut enstaka kopior för enskilt bruk och att använda det oförändrat för ickekommersiell forskning och för undervisning. Över- föring av upphovsrätten vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning av dokumentet kräver upphovsmannens medgivande. För att garantera äktheten, säkerheten och till- gängligheten finns lösningar av teknisk och administrativ art.
Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i den omfattning som god sed kräver vid användning av dokumentet på ovan beskrivna sätt samt skydd mot att dokumentet än- dras eller presenteras i sådan form eller i sådant sammanhang som är kränkande för upphovsmannens litterära eller konstnärliga anseende eller egenart.
För ytterligare information om Linköping University Electronic Press se förlagets hemsida http://www.ep.liu.se/.
Copyright
The publishers will keep this document online on the Internet - or its possible replacement - for a period of 25 years starting from the date of publication barring exceptional circumstances.
The online availability of the document implies permanent permission for anyone to read, to down- load, or to print out single copies for his/hers own use and to use it unchanged for non-commercial research and educational purpose. Subsequent transfers of copyright cannot revoke this permission. All other uses of the document are conditional upon the consent of the copyright owner. The publisher has taken technical and administrative measures to assure authenticity, security and accessibility.
According to intellectual property law the author has the right to be mentioned when his/her work is accessed as described above and to be protected against infringement.
For additional information about the Linköping University Electronic Press and its procedures for publication and for assurance of document integrity, please refer to its www home page: http://www.ep.liu.se/.
© Alexander Lindskog
Filip Ripstrand
Abstract
Fuzz testing has proven highly effective in software verification, yet its application to hard- ware systems remains comparatively underexplored. This thesis investigates the feasibility and effectiveness of FPGA-based fuzzing for IPv4 hardware designs, with a particular fo- cus on methods for fuzzing designs after synthesis and deployment on FPGA hardware. To evaluate this, three fuzzing approaches are implemented and compared: a fully synthe- sized hardware fuzzer capable of autonomous pseudo-random input generation of IPv4- packets directly on an FPGA and two software-driven approaches in which an external computer communicates with the FPGA over Ethernet to fuzz the synthesized design. One software fuzzer follows the same strategy as the hardware-based implementation, while the other integrates the widely used AFL++ fuzzing framework. The implemented ap- proaches are evaluated in terms of throughput, coverage and FPGA resource utilization. Mux-coverage instrumentation is used for measurements and feedback, and is evaluated in terms of extra resource usage and timing effects. Results show that FPGA-hosted fuzzing can achieve substantially higher test throughput than software-based fuzzing approaches, while still enabling meaningful coverage of the target design. The findings demonstrate that synthesized hardware fuzzing is a viable approach for high-speed verification of network-oriented FPGA designs and highlight important trade-offs between performance, flexibility and instrumentation complexity.
Acknowledgments
We would like to thank Sectra for the opportunity and idea for this thesis and a special thanks to our supervisor at Sectra, Axel Stafström, for the valuable guidance and feedback through- out this project. We would also like to thank our examiner Kent Palmkvist and supervisor Petter Källström.
iv
Contents
Abstract iii
Acknowledgments iv
Contents v
List of Figures vii
List of Tables viii
List of Abbreviations ix
1 Introduction 1 1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Aim . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.3 Research questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.4 Delimitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.5 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2 Theory/Related work 3 2.1 Network protocol layers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2.2 Internet Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.3 Ethernet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.4 FPGAs and resource usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.5 Testing and verification of systems . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.6 Foundations of fuzzing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.7 Software fuzzing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.8 RTL-fuzzing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.9 Randomness in hardware designs . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.10 Instrumentation of hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3 Method 18 3.1 Test setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 3.2 Conceptual approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 3.3 Testing and measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
4 Implementation 22 4.1 Implementation of DUT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 4.2 Implementation of Ethernet communication on the FPGA . . . . . . . . . . . . . 23 4.3 Implementation of hardware fuzzer . . . . . . . . . . . . . . . . . . . . . . . . . 24 4.4 Implementation of software fuzzer . . . . . . . . . . . . . . . . . . . . . . . . . . 26
5 Results 30 5.1 Resource usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
v
5.2 Timing with and without instrumentation . . . . . . . . . . . . . . . . . . . . . . 31 5.3 Coverage metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
6 Discussion 38 6.1 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 6.2 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 6.3 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
7 Conclusion 45 7.1 Research questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 7.2 Consequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Bibliography 47
A Measurements 52
B Ethernet subsystem overview for the FPGA 54
vi
List of Figures
2.1 OSI-model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.2 IPv4 Datagram header . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.3 C code for addition of two 16-bit values using 1’complement . . . . . . . . . . . . . 6 2.4 Structure of an Ethernet frame . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.5 Fuzzing algorithm. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.6 Fuzzer setup with software simulation via Verilator. . . . . . . . . . . . . . . . . . . 13 2.7 RTL fuzzing via differential testing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.8 Mux coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
4.1 Breakdown of modules in the DUT . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 4.2 Module schematic of HW fuzzer with internal FSMs . . . . . . . . . . . . . . . . . . 24 4.3 Software fuzzing modules on the FPGA . . . . . . . . . . . . . . . . . . . . . . . . . 26 4.4 AFL++ fuzzing setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
5.1 Resource usage for fuzzer/harness and DUT. . . . . . . . . . . . . . . . . . . . . . . 31 5.2 Total resource usage with module composition. . . . . . . . . . . . . . . . . . . . . . 31 5.3 Coverage over time for HW fuzzer with random configuration 1. . . . . . . . . . . 33 5.4 Coverage over time for HW fuzzer with random configuration 2. . . . . . . . . . . 33 5.5 Coverage over time for HW fuzzer with random configuration 3. . . . . . . . . . . 34 5.6 Coverage over time for SW fuzzer with random configuration 1. . . . . . . . . . . . 34 5.7 Coverage over time for SW fuzzer with random configuration 2. . . . . . . . . . . . 35 5.8 Coverage over time for SW fuzzer with random configuration 3. . . . . . . . . . . . 35 5.9 Coverage over time for SW fuzzer using AFL++. . . . . . . . . . . . . . . . . . . . . 36 5.10 Coverage over time for SW fuzzer using AFL++ with an invalid seed. . . . . . . . . 36 5.11 Coverage over time for SW fuzzer with random configuration 3 at 80 MHz . . . . . 37
B.1 Ethernet subsystem overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
vii
List of Tables
5.1 Total system logic elements after place and route. . . . . . . . . . . . . . . . . . . . 30 5.2 Hierarchical breakdown of resource usage . . . . . . . . . . . . . . . . . . . . . . . . 31 5.3 Timing for fuzzer and DUT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 5.4 Probability configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 5.5 Fuzzing frequency and coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
6.1 Approximate resource usage for other common FPGA boards . . . . . . . . . . . . 40
A.1 Fuzzing frequency and coverage, all measurements . . . . . . . . . . . . . . . . . . 53
viii
List of Abbreviations
ALM Adaptive Logic Module API Application Programming Interface AXI Advanced eXtensible Interface CLB Configurable Logic Block CRC Cyclic Redundancy Check CSR Control and Status Register DFF D Flip-Flop DUT Device Under Test FIFO First In First Out FPGA Field Programmable Gate Array FSM Finite State Machine HW hardware IHL header length IP Internet Protocol IPv4 Internet Protocol version 4 ISA Instruction Set Architecture LE Logic Element LFSR Linear-Feedback Shift Register LUT Lookup Table MAC Media Access Control OS Operating System PUT Program Under Test RTL Register Transfer Level SoC System on Chip SW software TTL Time to Live WSL Windows Subsystem for Linux XLR eXchangeable Logic and Routing
ix
1 Introduction
This chapter presents the motivation and aim, lists the research questions, states the delimi- tations, and outlines the contributions.
1.1 Motivation
Invalid network protocols have been the cause of many different security exploits [1]. Verifi- cation and testing are important tools to catch these exploits before they can cause any harm.
Fuzzing has become a popular method for finding vulnerabilities by automatically gener- ating test cases with unexpected inputs, which can find corner cases the engineers never thought of. Fuzzing techniques can be classified by the transparency of the Device Under Test (DUT) and test case generation strategy [1]. Transparency is divided into black-, white- and grey-box where the visible parts of the system is output, detailed information and in- strumented parts respectively. The generation strategy can be either generation or mutation based. Generation based generation uses the protocol structure to generate interesting test cases whereas mutation based generation can use normal test cases and mutate them.
Hardware designs are becoming larger and more complex, which require more time and effort to verify and test. Recent studies show that 70 % of the design cycle is spent on verifi- cation [2]. This establishes a demand for more efficient and automated testing methods com- pared to traditional formal and dynamic verification. While formal verification can achieve high test exhaustion, dynamic verification is more accessible [3]. In recent years fuzzing has been successfully applied to hardware designs using different approaches. Most hardware fuzzers rely on simulation or existing software fuzzers like AFL to run in software while the DUT is either executed on a Field Programmable Gate Array (FPGA) [3] or in software [4]. Even more recently implementation of the fuzzer completely on FPGA has been explored with promising results [5].
1
1.2 Aim
The primary aim of this thesis is to investigate the fuzz-testing approach on hardware design mainly by developing the fuzzer itself as a hardware design. The purpose of the thesis is to address a gap in research regarding fuzzing of hardware designs. The common approach is to stay at Register Transfer Level (RTL) or convert to a software model for which software- based fuzzers can be applied directly. In contrast, by exploring a synthesizable fuzzer this work targets an area that is to a large degree unexplored. Furthermore, most research focuses on fuzzing processor designs or as in RFuzz [3] and DirectFuzz [6], small board-level com- munication protocols. This thesis will therefore explore RTL Internet Protocol (IP) designs which, to the best of our knowledge, have not been studied before. These topics were cho- sen in collaboration with Sectra, who are interested in efficient ways to find vulnerabilities in hardware designs.
1.3 Research questions
Does the hardware fuzzing logic use a reasonable amount of resources making it a vi- able testing method over using a software solution?
Can instrumentation be added to an RTL design with reasonable resource overhead while preserving timing margins?
How does coverage over time and throughput compare between a hardware fuzzer, a software fuzzer using the same strategy and a more sophisticated software fuzzer?
1.4 Delimitations
This thesis is limited to comparing fuzzing approaches for the Internet Protocol version 4 (IPv4) protocol, with the DUT synthesized and evaluated on an FPGA.
1.5 Contributions
Both authors contributed jointly to the entire report. Responsibilities were more clearly di- vided for the fuzzers: Alexander took primary responsibility for implementing and docu- menting the hardware fuzzer, while Filip implemented and documented the two software fuzzers.
2
2 Theory/Related work
In this chapter the related projects and theory of the work is described.
2.1 Network protocol layers
A network normally consists of multiple layers where each layer consists of one or a few protocols in order to offer a service. These layers are together called a protocol stack. The Internet Protocol stack, which is the main model for layering the internet, consists of five layers. The model is described by Kurose and Ross [7, pp. 78–82].
The uppermost layer, also called the fifth layer, of the Internet Protocol (IP) stack is the Ap- plication layer, which is used by applications to send messages between different clients and servers or similar. This could be requests from websites to its server or email services. The Transport layer, or layer 4, transports application layer messages between two endpoints of- ten by dividing the message up into so called segments. There are two main protocols, UDP and TCP. UDP is fast and simple but offers no guarantees and data can be lost. TCP as- sures delivery of segments and offers congestion control and other features. The third layer is called the Network layer and is responsible for actually moving a segment of data between the two endpoints. In order for this to be done the packets, called a datagram, has to be routed through different networks. This layer includes the IP protocol which is used by end users and routers to move datagrams to the correct destination. The second layer, called the Link layer services the network layer by moving packets called frames between two adjacent nodes in the network. The first layer called the Physical layer is used by the link layer to actu- ally transmit individual bits between the two nodes. This will look very different depending on what type of link it is, for example a cable and a Wi-Fi link will transfer and interpret bits in two different ways.
There is another protocol stack model called the Open Systems Interconnection (OSI)-model, also described by Kurose and Ross [7, pp. 78–82]. All the five layers of the Internet protocol stack are included and work the same, but the OSI model has two more layers in between the application layer and transport layer, see Figure 2.1. These layers are called the Presentation layer and the Session layer. The first four layer from the physical layer to the transport layer
3
is the same. Then comes the session layer, as the fifth layer. It provides synchronization of data exchange, different types of checkpoints and data recovery. The presentation layer is the sixth layer and is used to interpret and present data that is to be or has been exchanged. It handles for example data compression and encryption. In this model the application is instead the seventh layer.
Application Application
Presentation Presentation
Session Session
Transport Transport
Network Network
Link Link
Physical Physical
Figure 2.1: OSI-model
2.2 Internet Protocol
Internet Protocol version 4 (IPv4) is, in the protocol stack, a network layer protocol [7, pp. 357–
361]. It is widely used and has started to partially be replaced, by IPv6, due to its limitation in
address space. An IPv4-packet is often referred to as a datagram and this datagram consists
of a header containing relevant information and a data segment. The header is the relevant
part as the data could consist of anything as long as the number of bytes is correct.
0 4 8 12 16 20 24 28 31
Version IHL Type of Service Total length
Identification Flags Fragment offset
Time to Live Protocol Header Checksum
Source Address
Destination Adress
Figure 2.2: IPv4 Datagram header
As can be seen in Figure 2.2 an IPv4 header typically consists of 20 bytes (5x32 bits) divided
into different fields. These fields are as follows.
• Version, this field consists of 4 bits that tell which IP version is used.
• IHL, this field refers to how long the header length is. Default is 5 for 5x32 bits but
custom options can be added as part of the data that follows.
• Type of Service field is used for distinguishing different types of packets from each other.
This can be used to prioritize packets that are used by real-time systems.
• Total length tells how many bytes the datagram is, header and data combined.
4
• Identification, flags, fragment offset, these fields are used for IP fragmentation which can be used when there are limits on packet size in lower level protocols
• Time to Live (TTL) is a field that is decremented at each router it passes and when it reaches 0 the packet is dropped. This is in order for a packet to disappear from the network instead of ending up in a loop and circulating forever.
• Protocol, this field is used to tell how to interpret the data by the upper layer. For exam- ple if the data consist of a TCP or UDP packet
• Header Checksum, this field is used for detecting bit errors in a received IPv4 datagram
• Source and Destination Address, source address refers to who sent the datagram and destination address refers to where the datagram is going
After the header follows, depending on the header length field, a custom options segment and then the actual data of the datagram.
2.2.1 Validity of an IPv4 packet For an IPv4 datagram to be considered valid at a the network layer level certain fields have to be correct. In a larger context where the packet is received by a higher layer such as the transport layer and the datagram is sent through a network all fields are important. But in a smaller context where only the network layer is considered, and no network of routers/end- points exists many fields can be ignored. The fields that must be correctly set for a standard IPv4 datagram is the version field, header length field, total length field, header checksum field, and possibly the TTL field, if a TTL value of 0 is considered invalid.
Version The version field indicates which network layer protocol is being used. The version field will be considered wrong if it does not give the value 4 for IPv4. If it is anything else the packet will be considered invalid and dropped or will be interpreted as some other protocol like IPv6 by a router which will also cause errors.
TTL The TTL field means Time To Live. The field is decremented at every router the datagram passes by until it reaches 0, then it is dropped. This is for the packet to be dropped and not circulate in the network forever if it would happen to get stuck in a network loop. If many datagrams would end up in a loop and the routers were unable to determine if a datagram should be dropped then network traffic would over time add up by the circulating datagrams causing delays and traffic congestion in the network.
IHL and total length The two length fields, i.e. the header length and total length, must match the actual lengths they describe. The default value for the header length field, also called the IHL field, is 5. If it is less than 5, then the header will be too short to be a valid IPv4 header. However, if the value is so large that it signifies that the header is longer than the total length of the datagram it will result in an early header termination error for the module/system handling the datagram.
The total length field, which is given in bytes, has to match the actual length of the datagram. If this value is larger than the actual number of bytes it will result in an early termination error as the actual last byte is received prior to the expected last byte. If the value is smaller
5
than the actual number of bytes it will also result in an error as the entire expected datagram has been received but data keeps coming.
Header checksum The header checksum is used to detect errors, and does not have a default value since it is calculated based on the other header fields. In order to calculate the checksum every 2 bytes in the header is considered a number and all these numbers are summed together using 1’s complement arithmetic (checksum field is set to 0 during calculation), see Figure 2.3. The total sum is then stored in the checksum field bitwise inverted. Bitwise inversion is performed to make error-checking faster. The receiver calculates the checksum on the header and adds the result to the received checksum. An uncorrupted header will yield ’0xFFFF’, or -0 in 1’s complement arithmetic.
1 uint16_t add1c16b(uint16_t a, uint16_t b){
2 uint32_t t = a + b;
3 uint16_t res = (t & 0xffff) + (t >> 16);
4 return res;
5 }
Figure 2.3: C code for addition of two 16-bit values using 1’complement
The checksum is used by the receiver of the datagram to make assumptions regarding if errors has been introduced during transmission. If the calculated checksum is not the same as the received checksum, i.e. the result is not ’0xFFFF’, a data error is assumed to have occurred and the datagram is dropped. The checksum has to be recomputed at every router that changes any field as the checksum would in that case no longer match the new header data. For example at a router the TTL field is decremented which would change the result of the checksum.
2.3 Ethernet
Ethernet is a link layer protocol used in wired local networks [7, pp. 504–505]. An Ethernet packet is referred to as a frame and its structure can be seen in Figure 2.4. The frame con- sists of a header with 6 bytes of Media Access Control (MAC) destination address, 6 bytes source address and 2 bytes of EtherType. EtherType is used to tell how the payload is to be interpreted or what type of payload the frame encapsulates. For example, if the payload is an IPv4 datagram the EtherType can be set to the hexadecimal value of 0x0800, which is the EtherType code for IPv4.
To start an Ethernet transfer, 8 bytes of a so called Preamble is sent. These 8 bytes are used to wake up the receiving side, synchronize the two devices and initiate the transfer. The first 7 bytes are the same and consists of alternating 1:s and 0:s in order to synchronize. The 8-th byte looks the same except for the last 2 bits, which are the first consecutive 1:s and informs the receiver that the actual data is following. At the end of a frame, a Cyclic Redundancy Check (CRC) checksum can be added and is used to detect errors. This CRC checksum is not strictly necessary but is dependent on the settings of the connected devices.
2.4 FPGAs and resource usage
A Field Programmable Gate Array (FPGA) is a programmable logic device; it is a type of integrated circuit. The challenge that a FPGA solves is how to make hardware programmable. Hardware is static and once on the chip can not be added or removed. The solution an
6
80 00 20 7A 3F 3E 80 00 20 20 3A AE 08 00 IP, ARP, etc 00 20 20 3A Destination MAC Address Source MAC Address EtherType Payload CRC Checksum MAC Header Data (4 bytes) (14 bytes) (46 - 1500 bytes) Ethernet Type ll Frame (64 to 1518 bytes) Figure 2.4: Structure of an Ethernet frame
FPGA uses, is to have hardware elements, such as Lookup Tables (LUTs), whose function is dependent on some configuration [8, pp. 37–38].
In addition to the elements which perform logical operations, there are routing elements. These elements are responsible for routing signals between logic blocks and this wiring is also programmable [8, p. 24]. This results in a hardware design that is static but the configuration is programmable and can realize any design. Despite the hardware being programmable and in that way flexible it is still limited in that the amount of hardware elements/blocks are limited.
Since the FPGA has a limited amount of hardware resources an interesting aspect of an im- plemented design is how much chip area or resources it uses. If the circuit board is to be used for multiple tasks then depending on the size of different systems it is possible to run out of hardware for the design. There are often restraints on designs as well and as more of the chip resources is used routing gets more difficult as well which can cause problems with time constraints etc.
2.4.1 Basic hardware blocks/LEs The basic building block in an FPGA is a Logic Element (LE) and it is programmed to realize a function. Depending on the FPGA and its manufacturer these LEs can look different. Some major manufactures are Altera, American Micro Devices (AMD) and Efinix. Historically an LE consists of a LUT and D Flip-Flop (DFF) or other type of register and it’s common as a comparison metric even when the actual building block has another layout. For example Al- tera uses a building block called Adaptive Logic Module (ALM) that consists of four registers and a few LUTs [9] that can be combined in different ways depending on how the function is best realized. The ALM uses four 3-input LUTs and two 4-input LUTs which can be com- bined and used in different modes depending if the ALM is used as normal LUTs or used to realize arithmetic functions.
AMD uses something called Configurable Logic Block (CLB) [10] which consists of two slices where each slice has access to four 6-input LUTs and eight registers. This building block is difficult to compare to other building blocks so the measurement of classic LEs is used where one 6-input LUT is approximately 1,6 LEs [10, §Device Resources]. Efinix uses the eXchangeable Logic and Routing (XLR) cell [11] as its basic building block. The XLR cell is used both for logic and routing [12] and is comparable to 1,2 classical LEs as it can be configured in more ways than a LE with a LUT and a flip flop, for example as an adder and a flip flop.
2.5 Testing and verification of systems
Testing and verification of systems is extremely important especially if it comes to critical systems or systems with high security. There are different ways to test and verify systems and often multiple approaches are used together.
7
2.5.1 Static verification Static verification is the process of verifying a design by inspecting it, for example by checking that code conventions are followed or calculating that the requirements are met. A type of static verification is called formal verification which is when a design is proved to be correct or proved to act correct with respect to some property or requirement of the system. This is done through formal methods of mathematics.
2.5.2 Dynamic verification Dynamic verification is the process of verifying a design by testing it. This is generally done by having test cases that are used as input to the Device Under Test (DUT) and verifying that the behaviour is as expected. It is also used for stress testing, i.e. repetitively performing an activity to see how much load a DUT can handle.
Fuzzing is a dynamic verification approach that generates test cases automatically. This means that not many predefined tests have to be written which can otherwise be time con- suming and result in a lot of work if the system is large or accepts many different inputs.
2.6 Foundations of fuzzing
Fuzzing has developed into an effective technique for detecting vulnerabilities or bugs, es- pecially in software systems. The first fuzzing tool, and the term "fuzzing", was developed by Miller et al. [13] in order to test tools in the UNIX system after it was noticed that random characters in commands caused programs to crash when receiving these unexpected inputs. This problem led to the development of a tool that would test a utility by feeding it random- ized inputs. The tool was then used on other UNIX utility tools and proven effective by the number of bugs found in these other tools. Since then the technique has developed with different tools and approaches depending on the type of system.
2.6.1 Classification of fuzzing Fuzzing is normally grouped into three types of fuzzing depending on the target and the knowledge of its inner workings. These are black-box fuzzing, white-box fuzzing and grey- box fuzzing [14, §2.2.1]. Black-box fuzzing is when the internal knowledge of the program is not considered. It is assumed to be a black-box where some input is tested and its corre- sponding output is observed. This is at one end of the spectrum of knowledge of the system internals. The other end of this spectrum is white-box fuzzing where all internals of the system are known and can be used by the fuzzer in order to test more obscure cases. The internals could include source code knowledge and runtime data. In between these extremes is grey-box fuzzing where some limited amount of internal knowledge can be obtained and used by the fuzzer. A normal metric for software programs with this type of fuzzing is some sort of code coverage which is a metric of which branches have been taken for a certain input.
2.6.2 Mutation and feedback Fuzzing can also be classified according to its type of input creation strategy, one is generation-based fuzzing and the other is mutation-based fuzzing [14, §2.2.2]. Generation- based fuzzing is a strategy where the input is built from scratch using an input format or specification. If an array of data is interpreted in a certain way by dividing the array into multiple fields of data then generation-based fuzzing could be used to fill in all the fields which results in a complete input. This input format/specification or set of fields can also be called an input grammar. Mutation-based fuzzing works the other way around, it takes
8
Oracle
Initial Preprocess Schedule InputGen InputEval Bugs
Config
Continue ConfUpdate
Figure 2.5: Fuzzing algorithm.
one or more predefined so called input seeds. This seed is then mutated in certain ways, for example by inverting random bytes.
Another way to group is through feedback. This refers to runtime information that is used during fuzzing to guide the next input. A no-feedback approach does not acquire any infor- mation for guiding test case input generation. If a feedback approach is used then informa- tion from the current test case is used to create an input for the next test case [14, §2.2.3]. One approach with this is to use code coverage information, such as coverage of branches seen for the current input, to detect when new coverage is found so that this test case can be further mutated. This would likely mean that same branch is taken again and deeper branches can be taken allowing for more coverage. There are many so called coverage-guided fuzzers that use this approach or a similar approach, one example is AFL/AFL++ [15][16].
2.6.3 The systematic fuzzing approach Fuzzing has historically not been very well defined. In the AFL white-paper [15], a popular fuzzer called AFL is described as not being an implementation of any theory but rather a col- lection of techniques that have been proven to give good results in practice. Since the fuzzing approach can be used for many applications with different strategies, there exist many im- plementations and types of fuzzers. For example fuzzing websites is vastly different than fuzzing an embedded device due to more limited hardware resources for an embedded de- vice and the access to the device through interfaces or Application Programming Interfaces (APIs). In a paper by Chen et al. [14, §2.3], there is a compilation of different fuzzers and what type of targets they are designed for. It lists fuzzers designed for files, protocols, JSON- data, APIs, Operating System (OS) kernels, web, compilers, browsers etc. This shows that the areas of application are scattered widely through various different fuzzers which implement different strategies. In light of this, systemization of fuzzing has been pushed by Manés et al. [17] to provide a unified model and taxonomy around fuzzing.
Fuzzing is defined by Manés et al. [17] as the execution of a Program Under Test (PUT) with inputs that the PUT may not expect. The model provided by Manés et al. is an algorithm which takes a set of test configurations to start from and a time limit and outputs a set of bugs found by the algorithm. In the paper [17], a test configuration is defined in definition 6 as the parameters that control the fuzz algorithm. This algorithm is broken down to six functions where one function does preprocessing and the other five functions are a part of the fuzzing iteration, see Figure 2.5. The functions does the following:
Preprocess The Preprocess function is responsible for potential modifications of the initial fuzz con- figurations and instrumentation of the PUT. The modifications of the configurations can in-
9
volve removing redundant configurations or compressing configurations also called seed trimming [17]. This is necessary for efficient testing in order to consume less memory when the set of possible inputs is very large, e.g. large data fields. The instrumentation can be to insert coverage points with a special compiler or to construct driver applications or a snap- shot of a program which can be loaded to skip long startup time. A model for the input can also be inferred in this stage to be used in the InputGen stage. It should be noted that these actions can either be done manually or automatically.
Schedule The Schedule function takes in the current set of configurations [17], time and the time limit and outputs a chosen test configuration to be used in the current iteration. This choice should either expand the knowledge of the PUT or try to exploit the PUT with the knowledge obtained by previous iterations with the end goal of maximizing the the unique number of bugs or coverage of the PUT. A common attribute of scheduling algorithms is to prioritize faster configurations that cover more of the PUT.
InputGen The InputGen function takes the given test configuration and generates a test case [17]. Since the output is dependent on the given input the generation strategy is responsible for the results produced by the fuzzing algorithm and is therefore one of the most important parts of the algorithm. The test case is either generated based on a defined model or by mutating a given input. One way to define the model is to manually specify a grammar which all input should abide by, this can be a specific language or protocol. Another way is to infer the model automatically which is done in Preprocess or ConfUpdate. A model can in some instances also be a program which is altered to produce partly invalid output. One example of this is when testing a decoder, then the output of the corresponding encoder is used as input to the decoder and the alterations of the encoder will be able to produce slightly invalid test cases. Mutation based input generation is an alternative to model based generation and does not need a predefined model to create new test cases from but instead an initial set of seeds to make small modifications to. Common techniques for modifying the seeds are bit-flipping, arithmetic mutation, block-based mutation and dictionary based mutation. Bit-flipping is an approach where a fixed or random number of sequential bits is inverted. Arithmetic mutation is an operation where a sequence of bytes is considered as an integer and an arithmetic operation is performed on the sequence where the result replaces the original byte sequence. Block-mutation considers a sequence of bytes as a block and randomly inserts, deletes, replaces, appends a block of bytes to the seeds or permutates the order of blocks. Dictionary-based mutation uses a set of predefined values such as -1, 0, 1 or others when mutating integers as these values are commonly used and often have a significant meaning in applications.
When the fuzzer has full access to the PUT’s internals, like with white-box fuzzing, more ways to generate inputs are possible. One of these ways is dynamic symbolic execution where concrete execution is combined with symbolic execution. In symbolic execution each path through the PUT results in an expression. From the expressions the input values to trigger errors or unexpected outputs can be provided by a solver. These generation methods can find much more complex inputs that the methods available in grey- and black-box fuzzers can not.
InputEval The InputEval function takes the configuration, test case and a bug oracle and evaluates against the PUT resulting in execution info and potentially a detected bug. The bug oracle
10
is a program that determines if a test case has broken a correctness policy of the PUT. This policy does not have to be a bug in the program, it can also be e.g. performance problems. Originally the oracle checked the exit code of a given program to determine the success of finding a bug, i.e. the program exited with a fatal signal. This method has the advantage of being able to use the PUT without instrumentation but is limited to exposing major bugs and not minor bugs like stack buffer overflows. To expose these minor bugs transformations of the program can be performed. These transformations may include spatial and temporal memory checks, illegal control flow or undefined behaviours if the program is written in a language like C. To be able to find semantic bugs usually the technique of differential testing is used where the behaviour of the PUT is compared with a reference model, another similar program. When their output differ it is likely to indicate that a bug has been found.
ConfUpdate The ConfUpdate function updates the set of test configurations based on the current fuzzing iteration’s test configuration and execution info. Here the fuzzer can direct the fuzzing and do seed selection and trimming. The ability to update the configuration is dependent on the information received from the InputEval stage and in turn the instrumentation of the PUT. This means grey- and white-box fuzzers are able to do more in this step and many of these fuzzers have focused on methods to improve the configuration updating. As part of the evolutionary algorithm most mutation based fuzzers use, ConfUpdate is responsible for determining which seeds to keep. Here the important decisions are to add new seeds that are promising to improve coverage and remove seeds that do not improve coverage. Seeds whose coverage can more efficiently be provided by other seeds are also removed. During this step a model can be inferred by analysing output to generate state machines or grammar.
Continue The Continue function takes the current set of test configurations and determines if the fuzzing should continue or not. The decision is often made with regards to a time limit, max number of iterations or a goal coverage.
2.6.4 Applying the model Applying this model on the general classifications of fuzzers shows that the visibility of the DUT affects the different functions. For example, black-box fuzzers put more effort in the Preprocess and InputGen functions for the grammar of the input than they do in the ConfUpdate and InputEval functions where they do not have much information to work with due to the black-box DUT. Grey-box fuzzers focus more on the ConfUpdate, InputEval and Schedule functions with input seed selection since they have access to more information of the DUT to make decisions based on whether an input should be kept for further use or not.
2.7 Software fuzzing
Fuzzing as a dynamic verification approach historically comes from the software develop- ment side with Miller and UNIX development [13]. Since then software fuzzing has grown and become more widely used. As mentioned in Section 2.6.3, there are a multitude of software fuzzers available that focus on different APIs, interfaces or systems. A common approach in software fuzzing is grey-box or white-box fuzzing, often with some sort of in- strumentation that provides a coverage metric to guide the fuzzer [17]. One such fuzzer that has become prominent and has been used as a baseline for multiple other fuzzers is AFL [17][15][18].
11
2.7.1 AFL AFL was originally developed at Google by Zalewski [18] and has found popularity with many related projects [19] and forks of the AFL-project that have become independent [17], one such fork is AFL++ [16]. AFL is considered a grey-box fuzzer and it uses coverage as feed- back in order to discover interesting inputs [15]. This feedback is interpreted as branch cover- age which is the coverage metric injected when a software is compiled and instrumented via AFL. Branch coverage measures how many of all the logical branches of some code, have been executed. Whether a particular branch has been executed, is registered in shared memory by some lines of code, injected at all branch points by AFL. This coverage in shared memory can be compared between test inputs to see if changes occurred, if changes did occur the input is considered an interesting input that found a new branch not previously taken.
2.7.2 AFL, strategy In regard to mutation strategy, AFL does not have a fundamental theory behind it, there is no mathematical best strategy and no proof of concept for one strategy [15]. AFL instead uses a combination of techniques and mutation principles that have been tested in practice and that have proven to be effective. These principles and techniques that AFL uses, are agnostic to the format of the input data. It also results in a fuzzer that is deterministic at first before moving on to non-deterministic mutations later on. The deterministic step includes sequential bit flips of varying lengths, additions and subtractions and/or insertions of interesting integers. The non-deterministic stage includes insertions, deletions and splicing of different test cases. Whenever a mutated input is determined to have found new branches or state transitions through the branch coverage metric, that input is added to a First In First Out (FIFO) queue of interesting seeds and can then be used again as a basis for a new test case that is mutated.
The intuition behind this approach, which uses mutation and feedback, is that if new cov- erage points have been found, the corresponding input can be used as a basis for future mutation. If a new coverage point or branch has been found there are likely more coverage points nested inside and protected by an outer comparison. Mutating the newly found input is more likely to find these nested coverage points as the base input already reaches the re- quired depth among the branches where one would like to search for new coverage points. By using the interesting inputs as a basis for mutation the strategy prioritizes a deep search among the branches and a broad search is done at the newly found depth in order to find even deeper branches.
2.8 RTL-fuzzing
When it comes to fuzzing Register Transfer Level (RTL)-designs there have been development since 2018 with RFuzz [3] where new fuzzers have built on it, such as DirectFuzz [6], DiFuz- zRTL [20], TurboFuzz [5], ProcessorFuzz [21] and Profuzz [22], that have all improved and extended RFuzz. The most common approach for fuzzing these RTL-designs is to simulate either directly at RTL or to use a tool like Verilator [23] to simulate at a software level.
2.8.1 Simulated fuzzing RFuzz [3] is one of the first tools designed for fuzzing RTL-designs. Its method is to utilize a software fuzzer like AFL [18][15] to provide input to an RTL-design. AFL, however uses coverage feedback in order to guide its fuzzing. Such coverage is usually acquired through inserting coverage points or instrumentation into the compiled code when fuzzing software programs [18, §3]. This is not possible when fuzzing RTL-designs and hardware. RFuzz solves this problem by using mux-coverage as a replacement for code coverage, see Sec- tion 2.10. This coverage can then be used instead of a more traditional code coverage as
12
feedback to AFL. RFuzz uses two main approaches for fuzzing the DUT. The first approach is by using a synthesized design, see Section 2.8.2. The other approach is via simulation with Verilator [23].
Verilator is an open-source tool that is used for simulating Verilog or SystemVerilog code. The tool compiles the Verilog or SystemVerilog code into an optimized and potentially mul- tithreaded model that is wrapped in a C++/SystemC module. These models execute faster than normal SystemC code and is much faster than traditional simulators that interpret the code during simulation. It also claims to be on par with or have better performance than many closed-source simulators [23]. RFuzz uses this tool to simulate the DUT and it then communicates with the fuzzer via buffers in shared memory [3, §4], see Figure 2.6.
CPU Shared Memory Verilator
Input buffer
Fuzzer DUT
Coverage buffer
Figure 2.6: Fuzzer setup with software simulation via Verilator.
DirectFuzz [6] is another fuzzing tool that built upon RFuzz. DirectFuzz implements directed fuzzing which allows for focusing the fuzzing on certain submodules in a larger system. This is useful in a system containing several subsystems or components, and when new compo- nents are progressively added to the large system. The fuzzer can then save time by directing the focus on the newly added component instead of fuzzing older parts that remained un- changed. DirectFuzz, like RFuzz, uses Verilator to simulate the hardware design.
Saravanan et al. explains in the article on Profuzz [22, §2], that using Verilator results in an inherent mismatch between key intrinsic hardware characteristics and a software level ab- straction that cannot fully capture these characteristics, such as concurrency and signal inter- actions. Profuzz for this reason chooses another approach to DirectFuzz by not using software abstraction simulations through Verilator but instead simulates directly at RTL-level.
The fuzzing tools named previously focus on coverage and trying to maximize it in order to test the entire design and strive to check all functionality. This approach is good for testing the entire design but is limited in detecting semantic bugs. This is a problem that DiFuz- zRTL [20] tackles by using differential testing. DiFuzzRTL is mainly focused on CPU designs where semantic bugs can easily occur by a few instructions run sequentially which causes a different behaviour than expected. As mentioned, DiFuzzRTL uses differential testing by running two simulations and detecting differences between them. It uses an Instruction Set Architecture (ISA) simulation as a golden reference model which refers to a deterministic soft- ware model of the expected or intended behaviour. DiFuzzRTL then compares the final state between the ISA simulation and the RTL simulation and if there is a mismatch it results in a potential bug [20, §4]. DiFuzzRTL uses register coverage, see Section 2.10, instead of using mux coverage. This approach better captures Finite State Machine (FSM) state transitions due to its clock sensitivity and also scales better [20, §3]. DiFuzzRTL adds new interesting test cases based on the coverage from the RTL simulation, see Figure 2.7.
13
Add new seed based on the RTL coverage Simulation
Seed Corpus Mutation engine SimInput Compare Potential
state/trace Bug
ISA Simulation
Figure 2.7: RTL fuzzing via differential testing.
ProcessorFuzz [21] uses the same approach as DiFuzzRTL, namely by using differential fuzzing with an ISA simulation as a golden reference model and compares the result with an RTL simulation. ProcessorFuzz improves on DiFuzzRTL [20] by using Control and Status Registers (CSRs) as coverage, see Section 2.10, instead of DiFuzzRTLs register coverage which can be a bit misleading as a coverage metric [21, §2]. ProcessorFuzz adds new interesting test cases based on the coverage from the ISA simulation, see Figure 2.7. In order to optimize runtime, ProcessorFuzz executes the test case in the ISA simulation which is generally faster than RTL simulation and then determines if the input is interesting through that simulation. The RTL simulation only runs the test cases that were determined to be interesting through added coverage.
2.8.2 FPGA accelerated fuzzing As can be seen in section 2.8.1, there are many fuzzer variants that use simulations for their fuzzing strategy. However, research on fuzzing synthesized designs is limited. RFuzz [3], as mentioned in section 2.8.1, uses two approaches. One approach involves simulation with Verilator and the other one is FPGA accelerated. The FPGA accelerated approach synthesizes the DUT on the FPGA. The fuzzer writes test cases and reads results to/from buffers in shared memory the same way as it does in the simulation approach meaning that fuzzer itself is agnostic to how the rest of the system is set up. The communication and access between the fuzzer/shared memory and the synthesized DUT happens via a high bandwidth direct memory access channel [3, §4]. This requires a System on Chip (SoC) with an FPGA and CPU on the same board.
Another fuzzer that synthesizes onto hardware is TurboFuzz [5]. It runs the entire fuzzer on the FPGA including the DUT. The fuzzer also has different modes for test case generation strategy, random and mutation. The generated test case is then executed on the DUT and also in an ISA software emulator running on its SoC board’s CPU. This is the only part of the system together with DDR4 memories that is not synthesized onto the FPGA. The ISA emulator on the CPU acts as a reference model and the results of the ISA emulator and the DUT are compared. For debugging potential bugs found through this differential testing, TurboFuzz takes advantage of its SoC´s hardware snapshot feature which can capture the entire state of the FPGA and transfer the snapshot to an external simulator for analysis [5, §3].
2.9 Randomness in hardware designs
Many fuzzers use some element of randomness in their strategy for creating new test cases. It can be via generation where the fields are filled with random values or the mutation is done
14
on random bits in the seed as described in Section 2.6.2. Generally speaking there are two overarching approaches for generating random values in hardware which can be grouped into Pseudo Random Number Generators (PRNG)s and True Random Number Generators (TRNG)s.
2.9.1 Pseudo-randomness, PRNG PRNGs can be used to generate numbers that appear random but in reality are not. PRNGs usually work based on some function that starts with a seed value and creates a new value [24]. This process is deterministic and all the values can be calculated if the seed and function is known but the pattern of numbers can appear random in that a value and the next value seem disconnected from each other.
One common approach for generating pseudo random numbers is via a Linear-Feedback Shift Register (LFSR). An LFSR [25] is a shift register that advances the bits towards the most significant bit. The least significant bit is decided and set via a feedback mechanism that usually consists of a combination of xor-gates where the inputs are a fixed number of bits in the register. An LFSR generates a number pattern based on the seed value or starting value in the register and the feedback function, this number pattern is however limited in length before the pattern starts repeating. The longest pattern an LFSR can produce before repeating is 2n ´ 1 where n is the number of elements in the shift register, this is called the maximal length [26]. If the register is large enough and the seed/feedback function is well chosen so that the repeating sequence is long then the numbers generated can appear random [26]. If the number of bits in the shift register is at least 24 it is likely but not guaranteed that the pattern is long enough so that it can be considered random [27].
2.9.2 True randomness, TRNG TRNGs are called so because they generate truly random numbers, they do so based on some physical entity that behave in a random way such as electrical noise or environmental sen- sors [24, §4]. Since these approaches are truly random they are more appropriate to use in applications where random numbers are used as a security feature.
2.10 Instrumentation of hardware
When fuzzing software the binaries are usually instrumented with some coverage metric such as branch coverage. This is done through a compiler that injects a few lines of code at all the coverage points in order to keep track of the coverage points [15]. However, this is not as straight forward when it comes to hardware, especially synthesized hardware. In order to use certain mutation techniques or fuzzers a coverage metric is needed. In software it is straight forward since the code is run sequentially and if for example a certain branch is taken the injected code is executed and sets variables that inform the fuzzer about the branch. In hardware there is no code being executed but the design instead consists of hardware modules that exist in parallel and it is only a signal or wire that connects multiple modules.
2.10.1 Mux coverage RFuzz [3] implemented a solution to the hardware coverage problem. This solution makes use of the signals controlling the if- and case-statements and that these constructs result in multiplexers in synthesis. It uses 2:1 mux select signals as coverage points, and multiplexers with more inputs can be converted to a series of chained 2:1 multiplexers. The select signals, that control the multiplexers, are also routed out of the module as an output vector. This can be realized with minimal extra hardware, and the output is then a bit-vector of all the mux
15
signals from the coverage points [3, §3.5], see Figure 2.8. RFuzz implements this with custom transforms in compiler passes that is plugged into an existing compiler where the resulting output is an RTL-code file (Verilog) that has injected mux coverage into the original design [3, §4.1] and this design can then be synthesized.
These select signals that can now be accessed outside of the module can be used as coverage of the hardware. Full coverage has been achieved if all select signals have been toggled. A mux coverage point is considered toggled if it evaluates to true and false in a single test. If this has happened, it would be interpreted as as both branches having been taken and that logic being tested. In order to monitor these signals, extra hardware is needed to keep track of which signals has been toggled and if a signal has changed value.
Figure 2.8: Coverage points using 2:1 mux select signals resulting in a coverage vector.
2.10.2 Register coverage Jaewon et al. mentions in one of their papers [20] two main limitations of the RFuzz approach. The first is that the mux coverage approach is clock insensitive. In a scenario where a test case is executed during multiple clock cycles it will in the end reach a certain state. Another test case, executed in the same way, can reach another state. In both cases, the two bit-vectors of mux coverage can look the same but the bits have been toggled in a different order. This is because the mux coverage can not capture the nuances of which clock cycle a toggle occurs in which is highly relevant for larger FSMs where the order can determine the state. The mux coverage poorly captures these state transitions because it merges multiple states into one state. The other limitation is that the required resources for the instrumentation increases quadratically [20, §3] with the number of muxes, due to logic needed for monitoring all the signals. This limits scalability and can limit runtime performance.
Instead, DiFuzzRTL [20] uses register coverage. It is based on control registers instead of the mux control signals which makes it more scalable [20, §3]. For this register coverage, DiFuz- zRTL injects three new registers into each RTL module which are called regstate, covmap and covsum. All values in the control registers are hashed into the regstate register and then the logic tries to write a 1 to the covmap register in a slot determined by the regstate register. The 1 marks that the corresponding state has been investigated. If the slot being written to con- tained a 0 i.e. the state had not already been explored then the covsum value is incremented by 1. Since the regstate, covmap and covsum registers are updated every clock cycle this ap-
16
proach is clock sensitive. The covsum registers from all the RTL modules are then summed up resulting in the final coverage value [20, §4.C].
2.10.3 CSR transition coverage Canakci et al. demonstrates in their paper on ProcessorFuzz [21] a scenario where the register coverage used by DiFuzzRTL can be misleading. The scenario put forward is a remainder register in a division module, which is 130-bits wide and controls multiple mux selection signals and therefore considered a control register. It is shown that in this case most of the coverage increase comes from this wide remainder register which in reality does not contribute to real coverage increases. The reason for this misleading case is that the remainder register is mostly concerned about the data-path and does not control the FSM [21, §2.C].
In order to tackle this problem ProcessorFuzz [21] uses the following approach with Control and Status Registers (CSRs). CSRs are the system registers in an ISA specification and these registers are the registers which directly keep track of the current state of the processor. A value change in one of these registers most likely means a more significant change in the architectural state of the processor. ProcessorFuzz compares the CSR values of the previous instruction and the current instruction in an ISA simulation, if they differ in a new way not previously seen then the current input is considered interesting [21, §3]. The reason laid out for using the transitions instead of the values is that it is possible to consider the previous state as well as the current state. A bug in one state may only occur if the state was reached via a certain other state.
17
3 Method
This chapter presents the methods used to evaluate the fuzzing approaches. Section 3.1 de- tails the hardware setup and the Device Under Test (DUT), used to benchmark fuzzer ef- fectiveness. Section 3.2 presents the fuzzing approaches at a conceptual level. Section 3.3 describes the measurement procedures and evaluation metrics used to obtain the results.
3.1 Test setup
This section describes the available hardware and the DUT.
3.1.1 Hardware, FPGA The available hardware is a Field Programmable Gate Array (FPGA) System on Chip (SoC) board by Arrow called the Everest DEV Board [28]. It has a Microchip MPF300T FPGA [29][30] that comes with 300 k Logic Elements (LEs). The board comes with several ports such as three RJ45 connectors for 10/100/1000 Mbps Ethernet and UART via USB. It also has a total of 3 GB of DDR3 memory divided over three memories. The board features a VSC8575 chip [28][31] for handling the physical connection with the RJ45 Ethernet connectors at the physical layer on the SoC board capable of 1 Gb of data throughput.
3.1.2 DUT For the DUT, modules from the Git-repo verilog-ethernet [32] will be used. This repo contains multiple network modules written in verilog which is synthesizable for layer 2, 3 and 4 that can be combined or used separate. This gives some flexibility in having a DUT that only contains the features that are of interest in fuzzing.
3.1.3 DUT instrumentation Since the DUT needs to be instrumented in order to have access to some sort of coverage met- ric for both feedback to fuzzers and for measurement purposes, the mux-coverage approach used by RFuzz [3] and explained in Section 2.10.1 will be used. The reason for this is that the modules available in the Git repository use a limited amount of control registers unlike a
18
processor design. The modules that will be used has multiple registers in the data-path but not many for control and state transitions and therefore the theory is that the mux-coverage approach is the more suitable of the approaches available and explained in Section 2.10.
3.2 Conceptual approach
In this section the conceptual approach of the hardware fuzzer synthesized on the FPGA together with the DUT is described. In order to have something to compare and measure against two software approaches will be used as well. One software approach will implement the same strategy as the hardware fuzzer but execute on an external computer and send the test inputs to the FPGA. The other approach is to adapt a more widely used software fuzzer that also runs on an external computer and sends the test inputs to the FPGA.
3.2.1 Hardware black-box When fuzzing the DUT with a synthesized fuzzer the DUT will be considered to be a black- box from the fuzzer’s perspective. Since the DUT will be considered as a black-box, no cover- age information will be used by the fuzzer but mux-coverage will still be included and used for measurements and gaining insights. The synthesized fuzzer will use a simple random generation strategy that is based on the grammar of an Internet Protocol version 4 (IPv4) datagram with its different fields. Since this implementation doesn’t have any requirements for true randomness and only needs values that can be considered random, an LFSR with enough bits will be used to generate these values. Once a test case is generated it will be used as input in the DUT and since the DUT is considered as a black-box the only information used to evaluate the input is whether or not the DUT accepts the input as valid, drops the input or signal an error.
3.2.2 Software black-box The first software approach will use the same strategy as the hardware fuzzer, i.e. a random generation strategy based on the grammar of an IPv4 datagram. This is in order to make a direct comparison between a synthesized fuzzer and a non-synthesized fuzzer. For this approach the DUT will be regarded as a black-box in the same way as the hardware fuzzer, however there will still be coverage information available for measurement purposes but that information will not be used by the fuzzer.
3.2.3 Software grey-box, AFL++ The second software approach will use a more sophisticated and widely used fuzzer, in this case AFL++ [33]. AFL++ is a coverage guided fuzzer, this means that the DUT will be con- sidered a grey box and the coverage information previously used only for measurements will also be used by the fuzzer. AFL++ uses the same techniques described in Section 2.7.2.
3.3 Testing and measurements
Synthesis and building the FPGA setup is done with Microchip’s Libero SoC Design Suite.
3.3.1 Hardware fuzzing Measurement of the fuzzer is done via Wireshark [34], which is a network analyser. With Wireshark, the reports can be captured as they are sent from the fuzzer and then analysed. A good measurement requires no interference on the channel by the Operating System (OS) sending packets on the channel or different devices on the board still performing setup
19
steps.In order to achieve this, the fuzzer waits until it receives a certain start frame mean- ing it can be controlled by the user more than just through a reset button. This is needed because if the fuzzer starts running as soon as the FPGA device is turned on, the OS may detect a new channel and start sending packets for establishing a connection which can in- terfere with the capturing of reports from the fuzzer. Another possibility is that the Ethernet subsystem is not fully finished in programming on-board chips external to the FPGA chip causing delays because the system is not ready.
Once the start frame is sent the fuzzer is active and starts reporting packets. The fuzzer sends a report for every input that causes incorrect behaviour, every time the coverage changes and every N-th packet where N is a parameter set at compile time. All these reports are captured with Wireshark and some meta information is provided such as time stamps. The coverage can be parsed out from the captured packets and time measurements can be calculated from the time stamps associated with a packet.
3.3.2 Software fuzzing The same approach as in section 3.3.1 is used for measuring with both the similar software fuzzer and AFL++. Wireshark is used for capturing the report packets received from the FPGA and can then be analysed in the same way since the report packets look essentially the same as the hardware fuzzer. With software approaches the sent test cases are sent via the same link and in order to easily distinguish the sent frames from the received frames the Ethernet headers are different for sent and received packets which can then be used as a filter for one or the other via Wireshark.
For the software approaches no start frame is needed since the fuzzing is controlled manually by starting and executing a program. Since the coverage reported is a total coverage and not per test case this has to be reset between fuzzing runs which is done manually via a push button on the FPGA. Due to the significant startup time of the Ethernet subsystem, a manual reset for everything but that subsystem was provided, in addition to a total reset. By resetting the system, the coverage is also reset for a new fuzzing run. This approach is also used for the hardware fuzzer but is done implicitly since the hardware fuzzer has to be reset to run again.
Compatibility between operating systems AFL++, which is used as a software fuzzer, is implemented for Linux and uses Linux system calls. This is incompatible with the available computer devices which runs a Windows OS. To solve this problem Windows Subsystem for Linux (WSL) [35] was be used.
3.3.3 Resource usage Resource usage for the whole system is taken after the Place and route step where we focus on the LEs. The Lookup Tables (LUTs) and D Flip-Flops (DFFs) for the different modules are measured after the Synthesis step where a hierarchical breakdown is provided. This does not correspond perfectly with LEs since Place and route has not been executed yet for these resource usage numbers. After synthesis the numbers given say how many resource elements are needed for the main design, not how many are actually used which is what place and route does.
3.3.4 Frequency Timing is provided by Libero SoC and the max frequency is determined by choosing a high frequency for the fuzzer including the DUT which cannot be obtained. When the timing
20
verification fails the tool reports the worst slack path obtained and that is used as the basis for the maximum frequency which the design can run at.
21
4 Implementation
This chapter describes how the different fuzzers and the surrounding systems are imple- mented and how the different systems work in a more detailed manner.
4.1 Implementation of DUT
The Device Under Test (DUT) is implemented as a combination of modules from the verilog- ethernet [32] Git repository and a few additional modules. The first module from verilog- ethernet is ip_eth_rx.v [36] and the reason for the choice of this particular module is that it is an Internet Protocol version 4 (IPv4) module that parses all the fields in the IPv4 header. It also calculates and compares the checksum to determine if it is correct meaning that this module captures the scope of the project in regard to IPv4. The module takes an Ethernet frame as input where the Ethernet header has already been parsed and the encapsulated packet is received without any prior parsing via an 8-bit Advanced eXtensible Interface (AXI)- stream [37]. The output of the module is all fields of a default IPv4 header individually in parallel and the data of the datagram in serial via an AXI-stream.
The second module from verilog-ethernet used is the ip_eth_tx.v module which is inverted in regards to inputs and outputs compared to ip_eth_rx.v and is its corresponding counterpart. In between is a buffer module that reads the entire frame/datagram into a buffer. If any errors are reported by ip_eth_rx.v the packet is discarded but if not, the packet is forwarded to the outputs of the DUT via ip_eth_tx.v, see Figure 4.1.
4.1.1 Instrumentation To instrument the DUT we use the method implemented by RFuzz [3], and also described in section 2.10.1, using the tool Yosys [38] to convert Verilog and VHDL designs into an interme- diate representation called FIRRTL [39] which is the assumed input to the instrumentation.
The instrumentation is made through compiler passes incorporated into the FIRRTL com- piler. There the control signals are wired to the top module into a vector with all the control signals throughout the whole hierarchy. This boolean vector, covcurr, is used to accumulate
22
DUT
Instrumented part
ip_eth_rx.v Buffer ip_eth_tx.v
module
Buffer
Figure 4.1: Breakdown of modules in the DUT.
the coverage of the DUT according to the following equation,
covtot[i] := covtot[i] + (covprev[i] ‘ covcurr[i]) (4.1)
which is used for detecting when a control signal has toggled. The coverage is later given by řiN=´01 covtot[i] (4.2) N where N is the number of control signals. This is added to the top module of the provided design and the resulting design is a Verilog module.
The instrumented modules are the ip_eth_rx.v, buffer and ip_eth_tx.v modules, see Figure 4.1. Because the buffer itself is only an array for storing data it is not included in the instrumenta- tion of the DUT. It is excluded because of its simple nature of storing data in registers instead of a dedicated memory block causing the instrumentation to include a multitude of coverage signals that are not of interest, one signal for each register.
4.2 Implementation of Ethernet communication on the FPGA
In order to communicate with the fuzzer from an external computer, Ethernet is used. The implementation follows two user guides [40][41] with some differences due to a different Field Programmable Gate Array (FPGA) board. The Ethernet subsystem uses mostly Mi- crochip vendor IP (Intellectual Property) blocks for its implementation. On the System on Chip (SoC) board there are three RJ45 Ethernet connectors and these are connected to a quad- port VSC8575 chip, which is a Gigabit Ethernet transceiver also located on the SoC [31][41]. There is also a ZL30722 chip [42], which is a clock generation chip that provides reference clocks for the VSC8575 chip and Ethernet modules on the FPGA. These chips need to be con- figured which is done via a Mi-V soft processor running at 80 MHz that interfaces to the external chips and executes firmware to correctly setup and communicate with these devices.
The main module is the CoreTSE Microchip vendor IP module [43] configured with an AXI stream interface with a 32-bit wide data bus for input and output frames. It also inserts, or removes depending on direction, a Cyclic Redundancy Check (CRC) field and padding if needed since a frame has to be at least 60 bytes excluding CRC, see Figure 2.4. By configuring it with an AXI interface and generating a corresponding clock to use for AXI, it makes the
23
usage and interfacing with other modules in the setup simpler considering that AXI is a common occurrence in the verilog-ethernet [32] repo.
The CoreTSE module provides AXI clock inputs for receiving and transmitting. In order to avoid different clock domains between the fuzzer logic and the Ethernet subsystem AXI in- terface these systems are part of the same clock domain. The clock is generated inside the Ethernet subsystem by a phase-locked loop (PLL) which is already used to provide another system clock to the soft processor etc. The generated clock is routed out of the Ethernet sub- system to the fuzzer and DUT where it drives the input clock of those modules resulting in a fuzzer clock domain separated from most of the Ethernet subsystem and its vendor IP mod- ules. The clock used in the AXI interface and fuzzing logic runs at 200 MHz meaning that is the clock frequency of the hardware fuzzer/software harness and DUT.
See Appendix B for a full Ethernet subsystem view and the user guides [40][41] for a more detailed explanation of the Ethernet subsystem.
4.3 Implementation of hardware fuzzer
The hardware fuzzer framework is implemented using the principles described in Sec- tion 2.6.3 with modules for scheduling, input generation and input evaluation. The modules can be seen in Figure 4.2. The preprocessing is done manually and defines a grammar of an IPv4 datagram to be used for generation of test inputs and serve as the framework for a test case. In this grammar framework different, fields will be populated with random values or a predefined good value based on a generic threshold value that determines the probability of selecting the random or good value.
Fetch Generate Test Start Test Header Fetch Sync Send Config Test Start Header Case Sync InputGen Controller Test Input Send Generate Send Watch Test Payload Execution Report Send Case Info Test Payload Case
InputEval
Send Recv Test Exec Config Done Start Recv Schedule Sync Evaluation Headers DUT
Send All Exec Done Done Recv Payload
Figure 4.2: Module schematic of HW fuzzer with internal FSMs.
4.3.1 Schedule
The hardware fuzzer will generate packets independently and all packets are generated in
the same way using the same strategy. The scheduling provides each test configuration with
24
an identifier which is an integer that is increased for every test input. The scheduling module determines if a predefined number of tests to be run has been reached or not, if it has been reached then this module stalls the fuzzer, and signals that the fuzzing is done.
4.3.2 InputGen In the input generation module, test cases are generated based on the predefined grammar where different fields in the IPv4 datagram are filled with values that are either predefined values that are acceptable or with pseudo random values generated via a Linear-Feedback Shift Register (LFSR). The module steps through the length of the IPv4 header, 16 bits at a time. For each field in the step it compares a random value with a predefined threshold which gives a probability of randomizing that field. Along with the test case, a bit signalling if the test case is valid or should be dropped is set. The value of this bit indicates what the fuzzer think should happen with the test case. The bit set if the field that is randomized results in an invalid datagram. The generated test case is then transferred to the input evaluation module allowing the input generation to receive the next test configuration to generate a test while another test is being evaluated. While the header is being generated the header checksum is calculated to be able to set a valid value for the field since this field is calculated based on the other fields. The payload of the datagram is always generated by choosing random values, 32 bits at a time.
4.3.3 InputEval The input evaluation module consists of three blocks; control, input and evaluation, working together to evaluate a test case. The control block receives the test case, synchronizes the start of the other blocks and monitors the test. The input block is responsible for generating the proper signals to transmit the IPv4 packet into the DUT. The DUT requires the payload of the IPv4 datagram to be transmitted via an AXI stream and the input block handles this transmission. The Finite State Machine (FSM) states seen in Figure 4.2 handle the different input interfaces of the header of an IPv4 datagram and its payload. The same thing can be seen in the evaluation block with different FSM states for receiving the output instead.
The evaluation block evaluates the test case by collecting the output of the DUT. This col- lected output is used to determine if the DUT considered the input valid or invalid. The packet is determined to be invalid if the DUT reports an error signal or quietly drops the packet and the packet is considered valid if the packet can be read out in its entirety on the output side of the DUT. In order to determine if the packet was dropped, a watchdog timer is used and if the timer expires the packet is determined to have been dropped. This watchdog timer is set to wait for a number of clock cycles that is larger than the largest number of clock cycles a test case is expected to take. The evaluation signal is cross checked in the control block with the signal determining the validity of a packet generated in the input generation block. If there is a difference in these two evaluation signals it is considered a bug, when a valid package has been dropped or an invalid package has been passed through a bug is reported. This is the oracle, which is a combination of the input generation, control block and evaluation block.
When a test is done the control block reports the test case if a bug was found i.e. if the two evaluation signals are different. The test case is also reported every N-th test case and if the total coverage of the DUT has changed during the test. This coverage comes from the instrumentation and is not used by the fuzzer, only reported for measurements. The evaluation of the test case is then reported back to the scheduling block where the scheduler counts the test case as executed. It then orders the next test case or stalls the fuzzer if the test case iterations have been reached.
25
4.4 Implementation of software fuzzer
In order to have something to compare the hardware approach with, a software fuzzer is implemented in two different ways as described below.
4.4.1 Hardware setup Despite the fuzzer being implemented in software the DUT is still synthesized on hardware meaning that some extra hardware is required in order to correctly forward the inputs to the DUT and communicate the results back. This extra hardware, called a harness, around the DUT is responsible for receiving the test cases via Ethernet and forwards the test cases into the DUT. The harness then constructs a report of the test case and transmits that report back via Ethernet. Since there is a mismatch between the DUT input data width, which is eight bits, and the Ethernet system data width which requires 32 bits there is a scaling module that is placed directly before the DUT, see Figure 4.3. This scaling module takes an AXI stream interface with a 32-bit wide data bus as input and divides the one 32-bit word into four 8-bit words delivered serially via an 8-bit AXI stream output interface. Since these four 8-bit words are serialized and put on the data bus one after another it introduces a delay in clock cycles compared to modules receiving 32-bit words.
reset
ethernet header Coverage
EthAxis DUT Errors
32 ResetDut 32 32 AxiScale 8 8 Report TX 32
axi_tlast
axi_itvalid
Figure 4.3: Software fuzzing modules on the FPGA
Immediately after the DUT there is a report module and the responsibility of this module is to read out all the data from the DUT and see if any errors are issued by the DUT. The module collects the coverage according to Equation 4.1 and sends this information to the external computer via Ethernet. Depending on if any issues were raised, a byte indicating the validity of the packet according to the DUT is set to indicate either valid or invalid and appended to the information sent. In order to detect a dropped packet, this module requires an input signal taken before the DUT to indicate that a packet was sent in. The module then notices that a packet was sent in based on this signal, but none was delivered meaning the packet was considered invalid and dropped. For this signal, the AXI stream signal indicating that the last byte is sent (often called axi_tlast) is used. This signal indicates that an entire packet has been delivered to the DUT, which is used in the report module to trigger a watchdog counter. If nothing is received on the output side after a predefined amount of time the packet is considered to have been dropped by the DUT. In the case of a dropped packet a report packet is sent with an invalid indication and the coverage information at the time of the detection of a dropped packet. Inside the report module there is a submodule responsible for taking the collected information from the report module and putting it together into an Ethernet frame with a predefined Ethernet broadcast header and transmitting it via an AXI stream to the Ethernet system.
26
First in the chain of modules is a reset module responsible for resetting the DUT in between testcases. The module resets the DUT, then forwards a whole input test case and then blocks further input until the test case has been passed through the DUT. As seen in Figure 4.3, the module provides a reset signal directly to the DUT. It also uses the axi_itvalid signal from the report module that indicates that a test case has passed through the DUT. This signals that it is now allowed to reset the DUT without prematurely resetting data currently in the DUT.
The last module is an Ethernet parsing module. The input to this module is a raw Ethernet frame provided as an AXI stream. This raw frame includes a header with a Media Access Control (MAC) address and an EtherType value. The module removes the Ethernet header from the encapsulated packet and realigns the frame data to the AXI stream. Realigning in this case means that the first byte of the frame data is set to also be the first byte of the first word in the AXI stream. This means that the first word in the AXI output stream has four valid bytes instead of two valid bytes and two invalid bytes, which would be the result when only removing the header since the 14 bytes in the header does not properly align to an AXI data bus of four bytes. The header is sent in parallel to the DUT which takes the Ethernet header as input in case it would be used by later modules.
4.4.2 Software implementation In order to fuzz the target on the FPGA via software, an Ethernet socket is set up for the particular interface that the Operating System (OS) assigns to the network connection. In order to get a fair comparison the fuzzer is implemented in C code for fast execution speed. To represent the IPv4 packet, a 20 byte header struct is combined with a payload array such that the entire structure is 46 bytes in size. The header part of the structure is also divided into the different fields in an IPv4 header, see Figure 2.2. To send this structure, it is appended to an Ethernet header and transmitted via the link layer socket.
Once the IPv4 structure has been created it can be fuzzed via a function that steps through all the fields and sets a value for that field. The input generation step sets this value to either a random value or a default value depending on whether a random 8-bit value is larger or smaller than a predefined probability threshold. This threshold could for example be set to 128 and if the random value is larger than 128 a random value is set otherwise the default value is set. Setting the value to 128 should result in the field being set to a random value approximately 50 percent of the time and being set to the default value the other 50 percent of the time.
By changing these threshold values, the probability of fuzzing a certain field can be changed and that certain field can be set to be fuzzed all the time by setting the threshold to 0 or never be fuzzed by setting the threshold to 255. The default values that can be set result in a valid packet except the checksum where the logic is inverted compared to the other fields. This is because the checksum must be calculated based on the current fields for it to be valid, therefore the default value is chosen when the checksum field is fuzzed. The checksum is calculated when the field is not fuzzed according to Section 2.2.1, where the fields are looped through and added together using 1’s complement arithmetic. After the structure has been fuzzed it is transmitted via the Ethernet socket and the structure can then be reused and fuzzed again since all fields are set to a value, either default or random. The scheduling of inputs is done sequentially and independent of feedback. A test case is generated and transmitted to the FPGA and then the next test case is generated immediately after.
In its current state, the black-box software fuzzer does not evaluate weather or not a potential bug was found. It strictly fuzzes for a fixed number of inputs that are determined at compile time. The reason for strictly fuzzing and not reporting potential bugs is that the software implementation is used as a comparison to benchmark the hardware black-box fuzzer. For
27
this end, it is not a necessity to implement this input evaluation. Input evaluation could however, be done the same way as the hardware black-box fuzzer. It is possible to keep track of the validity of the generated input and compare against the result that the synthesized harness reports.
4.4.3 AFL++ The approach for the fuzzer in Section 4.4.2 is simple and uses no feedback. In order to test a more complex and sophisticated fuzzer, AFL++ [33] is used. AFL++’s fuzzing logic is already implemented and left untouched. AFL++ provides a proxy interface that starts the fuzzer and leaves open a pointer to area in memory that AFL++ assumes to contain the coverage information and also a buffer containing the data for the next test case with a main function and test loop. The user must provide the logic in the main loop for testing the target with the provided data and write the coverage back to AFL++ via the coverage area pointer. This way, the user only has to solve how to interface with their own target from the AFL++ system.
CPU
Input Buffer AFL proxy
FPGA
Send test
case Harness
AFL++ Receive Ethernet DUT
coverage
Write
Coverage coverage buffer to AFL++
Figure 4.4: AFL++ setup using the provided proxy interface between AFL++ engine and FPGA. Dotted line is data-path and solid line is state transitions.
Since a working setup via an Ethernet socket already exists as described in Section 4.4.2 it is repurposed for interfacing with AFL++. The difference with AFL++ is that the coverage feedback is needed by the fuzzer meaning that the resulting coverage by the test needs to be sent back to the fuzzer before a new test case can be issued. AFL++ provides the test case in a buffer which is appended to an Ethernet header and sent out via the network connection after which the proxy waits for the return data. Once a packet comes back and is read the return data is stored in a receive buffer. From there the packet can be parsed and stored back into coverage area and the test case has ended allowing AFL++ to create a new test input based on the new coverage, see Figure 4.4.
The input evaluation is done via the coverage feedback which is received as a bit-vector inside a packet among other information. But AFL++ expects the coverage to be a byte-vector which means that the bit-vector has to be restructured to be stored correctly. Once the byte-vector has been written to the coverage area, control is left to the AFL++ engine which checks the byte-vector if anything was altered compared to the previous iteration. If it has been changed the test case is added to the First In First Out (FIFO) used for scheduling seeds.
28
AFL++ requires a seed input that it uses as a starting point for its fuzzing and mutation. The seed is stored as a binary file and generated via a Python packet manipulation framework called Scapy [44] which makes it easy to build IPv4 packets with different payloads. The gen- erated seed packet is of the same fixed length as the packets generated by the simple software implementation and the hardware fuzzer. AFL++ can, compared to those approaches, vary the actual length of a packet. The length is something that is fixed in the hardware fuzzer and in the corresponding software approach that is supposed to work the same. The reason for setting the same length is to give a fair comparison despite AFL++ being able to change the size of the packet. The seeds are used as the first starting point in the scheduling FIFO where new interesting test cases are added as seeds when they are found in order to update the con- figuration. These found seeds can then be used in the input generation of AFL++ where the seed is mutated with bit flips of a varying number of sequential bits, addition or subtraction of integers and insertions of interesting numbers.
29
5 Results
This chapter presents the results of the implemented fuzzers. The results are structured ac- cording to the measured metric, beginning with the resource usage of the system and Device Under Test (DUT), then the timing of the DUT and last the coverage result from the fuzzers.
5.1 Resource usage
The Logic Element (LE) usage for the entire system is shown in Table 5.1. The results are taken after place and route is finished.
Table 5.1: Total system logic elements after place and route.
Config LE
HW with instrumentation 22930 (7,65 %)
HW without instrumentation 21656 (7,23 %)
SW with instrumentation 21286 (7,11 %)
The resource usage of the Ethernet subsystem, hardware fuzzer/software harness and DUT are presented in Table 5.2. For the hardware (HW) configurations the Fuzzer/Harness field represents the size of the fuzzer and for the software (SW) the harness. The numbers and results in this table were taken after synthesis. The resource usage is divided among the different modules of the system and measured in number of Lookup Tables (LUTs) and D Flip-Flops (DFFs). The total resource usage includes all the modules and is the final usage for the whole system since the input/output interface logic is included in the Ethernet subsys- tem. The resource usage for the modules that are relevant for the fuzzing and dependent on the fuzzing technique is presented in Figure 5.1. Figure 5.2 shows the resource usage for the whole systems as the sum of their modules.
30
Table 5.2: Hierarchical breakdown of resource usage for hardware fuzzer with and without
instrumentation, and software harness with instrumentation.
Config Total Ethernet subs. Fuzzer/Harness DUT
LUT DFF LUT DFF LUT DFF LUT DFF
HW w. instr. 21621 10607 18794 8379 2368 2002 459 226
HW wo. instr. 20517 10134 18505 8379 1662 1529 350 226
SW w. instr. 19858 9259 18495 8379 921 654 442 226
Config
HW w. instr.
2000 2000 HW wo. instr.
SW w. instr.
1500 1500
1000 1000
500 500
0 0
Fuzzer/Harness DUT Fuzzer/Harness DUT
Figure 5.1: Resource usage for fuzzer/harness and DUT.
Module
20000 20000 Ethernet subs.
Fuzzer/Harness
DUT
15000 15000
10000 10000
5000 5000
0 0
Figure 5.2: Total resource usage with module composition.
5.2 Timing with and without instrumentation
In Table 5.3 the timings and the corresponding highest clock frequencies are shown. The mea- surements were taken after place and route. These measurements include only the hardware fuzzer and DUT. The synthesis left out the Ethernet subsystem that otherwise would contain the critical path which left no results for finding the critical path and max frequency for the fuzzer and DUT. For both configurations, with and without instrumentation, the critical path is inside the DUT which allows conclusions about the instrumentation to be made.
31
LUT LUT
DFF DFF
HW w. instr. HW wo. instr. SW w. instr. HW w. instr. HW wo. instr. SW w. instr.
Table 5.3: Timing for fuzzer and DUT
Config Min. period Max. freq.
HW w. instr. 3,164 ns 316,1 MHz
HW wo. instr. 3,242 ns 308,5 MHz
5.3 Coverage metrics
To test the impact of choosing different values for the probabilities used in the black-box
fuzzers, three configurations were used, see Table 5.4. The chosen values are discussed fur-
ther in Section 6.1.3. The fields not listed in the table are always randomized for both the
hardware and software black-box fuzzers. These probability configurations were run to ob-
tain the final coverage reached and the frequency of test cases tested, see Table 5.5. AFL++
does not use these configurations as it uses its own mutation strategy instead of a random
generation strategy based on a grammar.
Table 5.4: For each configuration, the probability (independent per field) that the fuzzer as-
signs a randomly generated value to the listed IPv4 header fields. All other header fields and
the payload are always assigned randomly generated values.
Prob. conf. Version IHL Length Checksum
1 3,125 % 3,125 % 3,125 % 3,125 %
2 25 % 25 % 25 % 25 %
3 3,125 % 3,125 % 25 % 3,125 %
The results of different fuzzing configurations/setups are shown in Table 5.5. The Fuzzer
column lists which fuzzer setup is used, HW means hardware fuzzer and SW refers to the
software fuzzer using random generation strategy just as the hardware fuzzer. In the column,
AFL refers to the software fuzzer setup that uses AFL++. Then there is SW80MHz which is a
measurement with the random generation software fuzzer but running the synthesized logic
including the DUT and harness at 80 MHz instead of 200 MHz as the other cases.
Table 5.5: The test frequency and coverage obtained by the different coverage configurations. Fuzzer Prob. conf. Time/iterations Test cases per sec. Final coverage HW 1 100 M it. 1,72 M 91,9 % HW 2 100 M it. 1,21 M 91,9 % HW 3 100 M it. 1,46 M 94,5 % SW 1 1 M it. 18,3 k 94,5 % SW 2 1 M it. 17,7 k 94,5 % SW 3 1 M it. 18,5 k 96,1 % AFL valid 60 s 2,39 k 94,5 % AFL inv. seed 60 s 2,47 k 61,6 % SW80MHz 3 1 M it. 18,4 k 94,5 %
The Prob. conf. column in Table 5.5 shows which probability configuration is used for the dif-
ferent measurements. For AFL++ fuzzer the probability configuration is either valid meaning
that the seed used is a normal valid packet or inv. seed meaning that the seed used from the
start is invalid. All configurations were measured 5 times and the measurements that reached
the highest final coverage are shown in the table, the rest of the measurements can be seen
in Appendix A in Table A.1. The Time/iterations column have different units depending on
the fuzzer setup, the reason for this is that time limits were used for fuzzing with AFL++
and for the others iteration limits. With these particular limits the total time for fuzzing is
approximately 60 s for all the setups with some variation.
32
5.3.1 Hardware measurements
HW probability config 1
100
80
60
&
H
4
20
0 0 10 20 30
Time since start [s] 40 50 60
Figure 5.3: Coverage over time for HW fuzzer with random configuration 1.
Figure 5.3 plots the coverage over time with the hardware fuzzer using probability configu- ration 1. The fuzzer runs 100 M iterations in approximately 58 s. That is roughly 1,72 M test cases per second and the coverage reaches 91,9 %.
HW probability config 2
100
80
E60
&
8
S 4
20
0 0 20 40 60 80 Time since start [s]
Figure 5.4: Coverage over time for HW fuzzer with random configuration 2.
Figure 5.4 plots the coverage over time with the hardware fuzzer using probability configu- ration 2. The fuzzer runs 100 M iterations in approximately 83 s. That is roughly 1,21 M test cases per second and the coverage reaches 91,9 %.
33
100 HW p probabilityty config 9 3
80
60
&
©
H
4
20
0 0 10 20 30 40 50 60 70
Time since start [s]
Figure 5.5: Coverage over time for HW fuzzer with random configuration 3.
Figure 5.5 plots the coverage over time with the hardware fuzzer using probability configu-
ration 3. The fuzzer runs 100 M iterations in approximately 69 s. That is roughly 1,46 M test
cases per second and the coverage reaches 94,5 %.
5.3.2 Software measurements
SW probability config 1
100
80
60
&
©
H
40
20
0
0 10 20 30 40 50
Time since start [s]
Figure 5.6: Coverage over time for SW fuzzer with random configuration 1.
Figure 5.6 plots the coverage over time with the software fuzzer with random generation using probability configuration 1. The fuzzer runs 1 M iterations in approximately 55 s. That is roughly 18,3 k test cases per second and the coverage reaches 94,5 %.
34
SW probability config 2
100
80
60
g
40
20
0
0 10 20 30 40 50
Time since start [s]
Figure 5.7: Coverage over time for SW fuzzer with random configuration 2.
Figure 5.7 plots the coverage over time with the software fuzzer with random generation using probability configuration 2. The fuzzer runs 1 M iterations in approximately 56 s. That is roughly 17,7 k test cases per second and the coverage reaches 94,5 %.
SW probability config 3
100
80
60
g
©
3 40
20
0
0 10 20 30 40 50
Time since start [s]
Figure 5.8: Coverage over time for SW fuzzer with random configuration 3.
Figure 5.8 plots the coverage over time with the software fuzzer with random generation using probability configuration 3. The fuzzer runs 1 M iterations in approximately 54 s. That is roughly 18,5 k test cases per second and the coverage reaches 96,1 %.
35
AFL++ 60s
100
80
60
&
H
40
20
0 0 10 20 30 40 50 60
Time since start [s]
Figure 5.9: Coverage over time for SW fuzzer using AFL++.
Figure 5.9 plots the coverage over time with the software fuzzer using AFL++. The fuzzer runs for 60 seconds at approximately 2,39 k test cases per second and reaches a final coverage of 94,5 %. As can be seen in the figure there is a short pause of packets in the beginning which occurs due to AFL++ beginning with validating the seed and then observing that a feedback is received. After the seed validation the actual fuzzing starts which results in the pattern of a coverage point then a short pause followed by the rest of the coverage points.
AFL++ 60s, invalid seed
100
80
g 60
g
©
H
40
20
0 0 10 20 30 40 50 60 Time since start [s]
Figure 5.10: Coverage over time for SW fuzzer using AFL++ with an invalid seed.
Figure 5.10 plots the coverage over time with the software fuzzer using AFL++. The fuzzer runs for 60 seconds at approximately 2,47 k test cases per second and reaches a final coverage
36
of 61,6 %. The measurement shown in the figure uses a seed that consists of an invalid packet
due to an invalid header length value.
SW 80MHz probability configuration 3
100
80
60
¢
©
H
40
20
0
0 10 20 30 40 50
Time since start [s]
Figure 5.11: Coverage over time for SW fuzzer with random configuration 3 running the
FPGA at 80 MHz instead of 200 MHz.
Figure 5.11 plots the coverage over time with the software fuzzer with random generation using probability configuration 3. The fuzzer runs 1 M iterations in approximately 54 s. That is roughly 18,4 k test cases per second and the coverage reaches 94,5 %. This measurement runs the software harness and the DUT at 80 MHz in clock frequency instead of 200 MHz as the other measurements.
37
6 Discussion
In this chapter, the results are discussed first, then the method and last opportunities for improvements to the work.
6.1 Results
This section discusses the results obtained.
6.1.1 Resource usage Table 5.2 shows that there is a small difference in resource usage in modules between dif- ferent synthesis runs where the module has not been modified. This can be seen with the Lookup Tables (LUTs) in the Ethernet subsystem column and the number of LUTs for the Device Under Test (DUT) between hardware and software with instrumentation, which can be explained by the synthesis using optimization methods that can differ between different synthesis executions.
The resource usage presented in Table 5.1 shows that the system leaves a large part of the available resources of the Field Programmable Gate Array (FPGA) unused allowing for a larger design to be tested. This is further supported by the fact that the Ethernet subsystem is practically constant, see Table 5.2 and Figure 5.2. The Ethernet subsystem is separate from the fuzzer/harness since it is not responsible for input directly to the DUT, and separate from the DUT since it is not a part of the fuzzing target. This separation makes it clear that the subsystem is just a means for inter FPGA communication and will not scale with the complexity of the fuzzer/harness or DUT. Since the dominating resource usage currently comes from the subsystem, there is an even larger relative margin for adding complexity to the fuzzer/harness and DUT than what the values in Table 5.1 may suggest. Another thing to note is that the Ethernet subsystem takes up more resources than necessary for the task of establishing the Ethernet interface since it also includes modules used for debugging purposes, see Appendix B. These modules do not contribute to the main task of the system, and is not regarded as part of the main subject of this work, which is the fuzzer and DUT.
38
Therefore the system could be reduced, this however has not been a priority due to the system not being restricted by resources.
The impact of instrumentation on the DUT can be seen in Figure 5.1 based on the data in Ta- ble 5.2. There the DUT instances in the HW configurations with and without instrumentation show that the LUT usage of the DUT increases by 31 % with instrumentation but leaves the D Flip-Flop (DFF) usage unchanged. The increase in LUTs is expected since control signals in the design need to be rerouted and multiple input multiplexers separated to allow each control signal to be accessible. The DFF usage being unchanged is misleading because to get meaningful value from the coverage, the coverage signals need to be stored in registers. Equation 4.1 contains three vectors that have to be registered to not impact the timing. These registers are a part of the fuzzer/harness which explains the difference seen in Figure 5.1. The instrumentation overhead for the system both by LUTs and DFFs can be expected to be proportional to the resource usage of the DUT, given the assumption that the number of mux-signals is proportional to the size of the design. This overhead is significant and has to be considered.
Table 5.2 and Figure 5.1 also show the difference in resources used by the hardware black- box fuzzer and the software harness. The hardware fuzzer uses ~2x the number of LUTs and ~3x the number of DFFs compared to the software harness. The resource usage would increase for the hardware fuzzer if more options and customizations of the datagrams were to be implemented. The logic in the software harness however would not increase in the same scenario since that logic is implemented external to the FPGA. The resources used for detecting dropped packets and forwarding the data to the DUT would not change by much and therefore the resource usage would stay approximately the same if more fuzzing options are added.
Resource usage in relation to other FPGAs/projects For comparison the resource usage is approximated on other common FPGA boards. For this comparison it is assumed that the design synthesizes to the same number of LUTs and DFFs. This assumption is however unlikely to hold up in reality as the design is synthesized on a hardware with different basic hardware blocks or Logic Elements (LEs), see Section 2.4.1. Depending on how the LEs are designed, different logical functions can be implemented.
Table 6.1 shows the approximate number of LUTs and DFFs for a few FPGA boards and the percentage of resources that the hardware fuzzer would use if synthesized on that board with the same number of LUTs/DFFs. The total number of LUTs comes from the classical view of an LE consisting of a LUT and a register and is therefore equal to the total number of LEs. The number of LEs is a result of the basic building block used in the FPGA and how many classical LEs that building block is equivalent to according to the manufacturer. For the Altera board this number is provided [9], for the Efinix board the number of eXchangeable Logic and Routings (XLRs)/LEs is provided [11] and for the Nexys board, that uses Configurable Logic Blocks (CLBs), the number of slices/6-input LUTs/DFFs is provided [45] together with the multiplier 1,6 that says how many LEs a 6-input LUT is comparable to. For the Everest Dev Board, which is used in this work, the number is provided approximately in the data sheet [30] and specifically through numbers provided via synthesis. See Section 2.4.1 for details on the building blocks used in the different FPGAs.
As can be seen in Table 6.1 resource usage of the hardware fuzzer and harness excluding the DUT is relatively low both for the FPGA used in this project and the rough estimate for other FPGAs. If we compare those numbers to TurboFuzz [5] which reports a resource usage of 89394 LUTs (17,11 %) and 139477 registers (13,34 %) together with 23,03 % of block RAMs we see that that implementation uses a significant amount of the hardware resources.
39
Table 6.1: Approximate resource usage for other common FPGA boards. % uses HW LUT- s/DFFs. FPGA Board Total cells ~LEs Total LUTs % Total DFFs % Everest Dev Board 299,544 LEs 299,544 299,544 0,79 299,544 0,31 (used) Nexys A7-100T 15,850 slices 101,440 101,440 2,33 126,800 0,73 (63,400 6LUTs) Altera Cyclone V E 56,480 ALMs 149,500 149,500 1,58 149,500 0,62 Efinix Ti180 M484 172,800 XLRs 176,256 176,256 1,34 176,256 0,52
It is reasonable that TurboFuzz uses more resources since it implements more sophisticated strategies. However, if the DUT doesn’t require complex fuzzing strategies it results in a design that is unnecessarily large.
6.1.2 Timing The timing of the DUT seen in Table 5.3 is interestingly better for the instrumented DUT. This can be attributed to either a better optimizing with Yosys [38] and the FIRRTL compiler [39] or variations in the place and route. However, the difference is small which means that the instrumentation does not change the properties of the DUT in a way that is likely to affect how it is used.
6.1.3 Coverage metrics To compare the implemented fuzzing approaches two different metrics are used, mux- coverage and test case throughput.
Choice of probabilities For the black-box fuzzers the impact of the probability to choose random values was inter- esting to test which is why three configurations where chosen for comparison. These three configurations are presented in Table 5.4. The choices are made to test how a low or high probability of selecting random values or a tailored probability for random values affects the achieved coverage. Configuration 1 assigns low probabilities for fuzzing the fields that can invalidate the datagram, and the probability is the same for all those fields. This also applies to configuration 2 but uses a high probability of fuzzing the fields. The tailored probability, configuration 3, is chosen to have a small probability for a random value to replace fields known to be checked early by the DUT and high for fields that affect the parsing later.
Choice of seeds As can be seen in Table 5.5, Figure 5.9 and Figure 5.10, the AFL++ software fuzzer uses two different seeds where one is a valid Internet Protocol version 4 (IPv4) datagram and the other is invalid by having a wrong header length. See Section 6.2 for the reason why the number of seeds is limited. The reason for the choice of these seeds is similar to the reasoning with the probability configurations. The invalid seed is rejected early and tests if AFL++ recovers and finds a valid packet and in Figure 5.10 we can see that this is something that AFL++ struggles with. The other seed, which is valid, will reach deeper in the design and is more likely to mutate to a datagram that is rejected at a deeper stage in the design via for example wrong length for the packet. This highlights the importance of choosing a seed that actually reaches some deeper parts of the device.
40
Coverage results The coverage obtained by the hardware black-box fuzzer, seen in Table 5.5, is different based on the probability configuration. By following the coverage signals this difference can be attributed to the case where the IPv4 length header field is smaller than the length of the Ethernet payload. Since the hardware fuzzer always transmits an Ethernet payload of length 46 bytes, the required value for the length field is between 20 bytes which is the minimum header length, and 46 bytes. This case is reached by the software fuzzer for all three proba- bility configurations but only configuration three for the hardware fuzzer. The probability of generating such a value with 16 bits, which is the width of the length field, is approximately 0,04 %. This combined with the fact that the DUT rejects any packet that has a faulty version, header length (IHL) or checksum gives the probability to generate this packet 0,00104 %, 0,00386 % and 0,00832 % for each respective probability configuration. These probabilities might seem low but since a lot of packages are generated, 1 million for the software black- box fuzzer, the probability of generating at least one such packet is very close to 100 %. This means the hardware black-box fuzzer might have another reason for not covering this case, since it generates 100 million packets which should increase the probability even further. The reason might be that the Linear-Feedback Shift Register (LFSR) used as a pseudo random number generator has some attribute or is used in a way that inhibits generation of this spe- cific case.
The hardware and the software black-box fuzzer reach different coverages, as seen in Ta- ble 5.5. This difference is attributed to the difference in the harness and the hardware fuzzer. The harness supplies the DUT with the packet to test as it is received in the Ethernet subsys- tem to improve the testing speed. However this means that the packet might have to wait while receiving the packet, triggering interface logic. The hardware fuzzer will always have the next part of the packet valid as input and will thus not trigger this logic. Since this study is limited to testing the IPv4-protocol this coverage is not seen as interesting. The signals left to reach 100 % for all the fuzzers can be attributed to signals related to the interface and not the IPv4 logic. Through this testing and analysis of the remaining coverage points, the dis- covery of a state that is unreachable in one of the modules used from the verilog-ethernet [32] repository has been found. This means that no fuzzer could ever reach 100 % coverage on this DUT.
The rate of test cases being tested by the software black-box fuzzer is more than 6,4 times faster than the AFL++ fuzzer. The hardware black-box fuzzer is in turn more than 65 times faster than the software black-box fuzzer. With the right configuration of each fuzzer they all reach the maximum coverage.
Variation in final coverage From Table 5.5 and Table A.1 it is clear that the variation in final coverage for the different measurements is small as almost all measurements for a single configuration is the same. This is most likely a result caused by a limited DUT where the amount of coverage points and branches are fewer than optimal. This is discussed further in Section 6.2.
6.1.4 Hardware throughput As can be seen in Table 5.5 and Table A.1 there is a difference in throughput between the different configurations for the hardware fuzzer. This is a result of the watchdog timer men- tioned in Section 4.3.3. The second configuration statistically generates the largest number of invalid packets and therefore the watchdog timer expires more for this configuration than for the first configuration that generates the least amount of invalid packets. Each time an invalid packet is generated and the DUT drops it, the evaluation block stalls the fuzzer while
41
waiting for the timer to expire causing a lower throughput. Accordingly, the first configura- tion has the highest throughput because it generates the least amount of invalid test cases, configuration 3 has the second highest throughput as it is less likely than configuration 2 to generate invalid test cases and configuration 2 has the lowest throughput.
6.1.5 Software throughput The number of test cases per second varies significantly between the hardware and software setup where Table 5.5 shows that hardware fuzzer has a 65x-98x higher throughput than the software black-box fuzzer. In Table 5.5 it is also shown that the software setup clocking the DUT and harness on the FPGA at 80 MHz instead of 200 MHz results in approximately the same throughput of test cases per second. This indicates that the bottleneck limiting the software fuzzers’ throughput is not in the FPGA as clocking it faster does not improve the throughput.
We can conclude that the bottleneck should be in the actual software/ Operating System (OS). The reason for this is that the max number of bits moved through the link is lower than the capacity of the link. For the generated test cases the packets are small in size but for this scenario the max frame size of 1518 bytes (including header) will be used. For the number of test cases per second, the value 19 k is used which is the rounded upper limit of test cases per second for all the software implementations. The link is full duplex and capable of 1 Gb per second. According to Equation 6.1, these numbers result in a throughput less than the max capacity of the link. 1 Gb/s ą 19000 test cases/second ˆ 1518 bytes ˆ 8 bits (6.1) This means that the link is capable of moving that amount of frames per second even if the frames are max size, which they are not, and so the bottleneck should also not be in the physical link but must therefore be in the software/OS. Some improvements could possibly be made with more efficient code or setting higher priorities in the OS on that task to avoid interrupts and context switches.
Throughput variation Table 5.5 also shows that the software fuzzer with random generation is ~10x faster than running AFL++. This result is reasonable as the coverage information has to be received by AFL++ before moving on to the next test case. Data has to move in a complete roundtrip where the fuzzer stalls while waiting in comparison to the other setup where the data only has to be sent before the next test case can be fuzzed and then sent. The mutation engine in AFL++ runs in parallel in another process meaning that the mutation work is not necessarily part of the extra overhead. However, running multiple processes that communicates with each other can incur extra overhead with context switches handled by the OS which could slow down the AFL++ fuzzer compared to the other software setup.
Table 5.5 and Table A.1 show that there are smaller variations in throughput between mea- surements and configurations for software setups which can likely be attributed to interfer- ence from the OS discussed previously. For the black-box software fuzzer the throughput is in the range 16,3 k to 18,9 k test cases per second with an average of 17,755 k test cases per second. The software fuzzer using AFL++ has a throughput in the range 2,39 k to 2,48 k test cases per second with an average of 2,438 k test cases per second.
6.2 Method
One limit of this study is the simple DUT. It uses one of the few available open source Register Transfer Level (RTL) implementations of the IPv4 protocol stack. But to get an actual DUT,
42
it had to be assembled and implemented to a usable version. Since the study’s goal was to implement and test fuzzers, it was not a priority to spend time developing a design to be tested. Research like RFuzz [3] and TurboFuzz [5] uses processor implementations like Sodor, Rocket Chip, CVA6 and BOOM. These are large designs in which the fuzzers do not reach full coverage and the coverage over time is more gradual because of the deeper logic being harder to reach. The simple DUT limits the discovery of potential weaknesses of the fuzzers implemented.
Another result of the simple DUT is that AFL++ does not utilise its strengths. AFL++ is a mu- tation based fuzzer which works well when there are lots of nested branches to take because once a new branch has been found that input is used to find sub-branches as well. When the DUT is small with few branches this strength is nullified and the result is that AFL++ to a great extent randomly modifies a seed with little difference to a random generation fuzzer. There are still differences, one example is that AFL++ is not limited to the grammar of an IPv4 datagram and can mutate the length of a datagram but since there is not much difference in the coverage feedback AFL++ doesn’t generate many new seeds.
In regard to seeds, AFL++ was used with only a small number of seeds (one per configura- tion). The reason for this is because in the preprocess stage when AFL++ is seed trimming, see Section 2.6.3, AFL++ warns that seeds not contributing with new coverage may be useless. These seeds that were not contributing with new coverage were manually removed, leaving few seeds that actually find new coverage during the preprocess stage and later fuzzing. The other possibility is that seeds have to be set based on knowledge of the inner workings of the DUT and previous results which leaves minimal work for AFL++ since the DUT is small, leaving few inputs that increase coverage which AFL++ is supposed to find. This approach seems contradictory to the entire point of fuzzing since the fuzzer is supposed to find edge cases and not the user.
For the best possible measurements, the interference from the OS with other tasks should be minimized as the software fuzzers are running. Guaranteeing no interference is difficult for the most part but can be improved through setting priority of tasks/processes so that the OS will prioritize them. With Windows Subsystem for Linux (WSL), this is a bit more compli- cated as the priority has to be set for both the program in WSL and for WSL in Windows. This complication was not explored during measurements and therefore no guarantees of minimal interference can be made. However, it is unlikely that such an improvement would significantly change the results given the large difference in test cases per second as seen in Table 5.5.
Given the interesting result discussed in Section 6.1.3, a better source for true randomness in- stead of a LFSR that generates pseudo-random numbers can be of interest to use or evaluate. It seems to be a drawback of the hardware fuzzer and its capability of generating random numbers when the software fuzzer for all three probability configurations generates that spe- cific case but the hardware fuzzer cannot consistently generate that case. The pseudo random algorithm used in the software fuzzer display the expected behaviour which indicates that the approach is not the issue.
6.3 Future work
A significant improvement with a larger DUT containing multiple branches would be inter- esting to measure in order to see if the results would change or stay similar. Since this is a clear drawback of the work it is also a clear improvement in a future work and a task that would have been valuable to test. But unfortunately this could not be done due to limited
43
time and resources and that task having a lower priority compared to development and im- plementation of other systems.
Another future work suggestion would be to extend the fuzzer to handle custom options and header lengths in the IPv4 header which would make it more versatile. Extending the fuzzer to IPv6 datagrams and/or Ethernet frames is also an option for future work that would make it more flexible and usable since it would allow fuzzing of a broader range of modules and inputs at layer 2 or 3 in the Internet Protocol (IP) stack. One possible improvement could be to test a mutation based approach especially if the DUT is larger and contains more branches and see how that would perform against the current approach both in resource usage and performance.
As described in Section 2.10 there exist multiple approaches for hardware instrumentation in order to have access to a coverage metric. One interesting question to answer is how these approaches compare against each other, testing how much extra resources are used by each in different cases and resolving when it is a good idea to use one or another since undoubtedly all of the approaches will have strengths and weaknesses.
44
7 Conclusion
This chapter concludes the thesis project. First, with answers to the research questions and then the consequence of this work in regards to the aim.
7.1 Research questions
This section answers the research questions.
7.1.1 Viability of hardware fuzzing for IPv4 This work shows that a hardware fuzzer targeting Internet Protocol version 4 (IPv4) modules can be implemented with a small resource footprint with plenty of area to spare. Although it is larger than the software fuzzer harness, it remains a viable alternative in many cases. The main constraint is when the target design is at the limit of what could fit on the Field Programmable Gate Array (FPGA).
7.1.2 Instrumentation The technique of mux-coverage extracts coverage information from a design with some Lookup Table (LUT) overhead but no decrease in timing margins. With the assumption that the LUT overhead is proportional to the size of the design, the instrumentation can cause large designs to become to big for the FPGA.
7.1.3 Performance The three different fuzzers compared show significant difference in test case throughput. The hardware black-box fuzzer was 79 times faster than the software fuzzer using the configu- ration that reached the highest coverage and 613 times faster than the software fuzzer with AFL++. All the fuzzers reached the same coverage points that are of interest, showing that for this simple Device Under Test (DUT) all fuzzers are viable. Because the hardware and soft- ware black-box fuzzers use the same strategy, we can conclude that the synthesized hardware fuzzer offers substantial performance benefits due to its higher throughput.
45
7.2 Consequences
This work shows that there are multiple approaches that can be used to test Register Transfer Level (RTL) designs synthesized on FPGA. Depending on the constraints of the design and DUT the approach appropriate could be different. If the DUT is large and the instrumentation does not fit on the device then the hardware or software black-box fuzzers could be a good option with the hardware fuzzer being dependent on the complexity of the input which may cause it to exceed the size of the device. If the instrumentation fits and the DUT has deep logic the AFL++ could be a good choice although we did not see a difference with our experimental setup.
46
Bibliography
[1] Zhihao Hu and Zulie Pan. “A Systematic Review of Network Protocol Fuzzing Tech- niques”. In: 2021 IEEE 4th Advanced Information Management, Communicates, Electronic and Automation Control Conference (IMCEC). Vol. 4. 2021, pp. 1000–1005. DOI: 10.1109/ IMCEC51613.2021.9482063. [2] Raghul Saravanan and Sai Manoj Pudukotai Dinakarrao. “The Fuzz Odyssey: A Survey on Hardware Fuzzing Frameworks for Hardware Design Verification”. In: Proceedings of the Great Lakes Symposium on VLSI 2024. GLSVLSI ’24. Clearwater, FL, USA: Association for Computing Machinery, 2024, pp. 192–197. ISBN: 9798400706059. DOI: 10 . 1145 / 3649476.3658697. URL: https://doi.org/10.1145/3649476.3658697. [3] Kevin Laeufer, Jack Koenig, Donggyu Kim, Jonathan Bachrach, and Koushik Sen. “RFUZZ: Coverage-Directed Fuzz Testing of RTL on FPGAs”. In: 2018 IEEE/ACM In- ternational Conference on Computer-Aided Design (ICCAD). 2018, pp. 1–8. DOI: 10.1145/ 3240765.3240842. [4] Timothy Trippel, Kang G. Shin, Alex Chernyakhovsky, Garret Kelly, Dominic Rizzo, and Matthew Hicks. Fuzzing Hardware Like Software. 2021. arXiv: 2102 . 02308 [cs.AR]. URL: https://arxiv.org/abs/2102.02308. [5] Yang Zhong, Haoran Wu, Xueqi Li, Sa Wang, David Boland, Yungang Bao, and Kan Shi. TurboFuzz: FPGA Accelerated Hardware Fuzzing for Processor Agile Verification. 2025. arXiv: 2509.10400 [cs.AR]. URL: https://arxiv.org/abs/2509.10400. [6] Sadullah Canakci, Leila Delshadtehrani, Furkan Eris, Michael Bedford Taylor, Manuel Egele, and Ajay Joshi. “DirectFuzz: Automated Test Generation for RTL Designs us- ing Directed Graybox Fuzzing”. In: 2021 58th ACM/IEEE Design Automation Conference (DAC). 2021, pp. 529–534. DOI: 10.1109/DAC18074.2021.9586289. [7] James F. Kurose and Keith W. Ross. Computer Networking: A Top-Down Approach. En- glish. Seventh. Pearson, 2017. ISBN: 978-1-292-15359-9. [8] Hideharu Amano. Principles and Structures of FPGAs. English. 1st ed. Springer Singa- pore, 2018. ISBN: 978-981-13-0824-6. [9] Altera Corporation. Cyclone V Device Handbook Volume 1: Device Interfaces and Integration. Section 1.1.4-5 ALM Output, accessed: 2026-05-11. Altera Corporation. URL: https : / / docs . altera . com / r / docs / 683375 / current / cyclone - v - device - handbook-volume-1-device-interfaces-and-integration/alm-output.
47
[10] Advanced Micro Devices, Inc. 7 Series FPGAs Configurable Logic Block User Guide (UG474). Version 1.9. Accessed: 2026-05-11. AMD. 2025. URL: https://docs.amd. com/r/en-US/ug474_7Series_CLB/CLB-Slices. [11] Efinix, Inc. Ti180 Data Sheet. Tech. rep. DSTi180-v3.1. Accessed: 2026-05-11. Efinix, Inc., Sept. 2024. URL: https://www.efinixinc.com/docs/titanium180-ds-v3.1. pdf. [12] Efinix, Inc. Why the XLR Cell is a Big Deal White Paper. Tech. rep. Accessed: 2026-05-11. Efinix, Inc., 2026. URL: https : / / www . efinixinc . com / docs / wp - why - xlr - cell-is-a-big-deal-v2.0.pdf. [13] Barton P. Miller, Lars Fredriksen, and Bryan So. “An empirical study of the reliability of UNIX utilities”. In: Commun. ACM 33.12 (Dec. 1990), pp. 32–44. ISSN: 0001-0782. DOI: 10.1145/96267.96279. URL: https://doi-org.e.bibl.liu.se/10.1145/ 96267.96279. [14] Chen Chen, Baojiang Cui, Jinxin Ma, Runpu Wu, Jianchao Guo, and Wenqian Liu. “A systematic review of fuzzing techniques”. In: Computers & Security 75 (2018), pp. 118– 137. ISSN: 0167-4048. DOI: https : / / doi . org / 10 . 1016 / j . cose . 2018 . 02 . 002. URL: https : / / www . sciencedirect . com / science / article / pii / S0167404818300658. [15] Michał Zalewski and Google AFL Project. American Fuzzy Lop Technical Details. File: docs/technical_details.txt, accessed: 2026-05-11. 2019. URL: https://github.com/ google/AFL. [16] Andrea Fioraldi, Dominik Maier, Heiko Eißfeldt, and Marc Heuse. “AFL++ : Com- bining Incremental Steps of Fuzzing Research”. In: 14th USENIX Workshop on Offen- sive Technologies (WOOT 20). USENIX Association, Aug. 2020. URL: https : / / www . usenix.org/conference/woot20/presentation/fioraldi. [17] Valentin JM Manès, HyungSeok Han, Choongwoo Han, Sang Kil Cha, Manuel Egele, Edward J Schwartz, and Maverick Woo. “The art, science, and engineering of fuzzing: A survey”. In: IEEE Transactions on Software Engineering 47.11 (2019), pp. 2312–2331. [18] Michał Zalewski and Google AFL Project. AFL. accessed: 2026-05-11. 2019. URL: https://github.com/google/AFL. [19] Michał Zalewski and Google AFL Project. AFL Sister Projects Documentation. File: docs/- sister_projects.txt, accessed: 2026-05-11. 2019. URL: https://github.com/google/ AFL. [20] Jaewon Hur, Suhwan Song, Dongup Kwon, Eunjin Baek, Jangwoo Kim, and Byoungy- oung Lee. “DifuzzRTL: Differential Fuzz Testing to Find CPU Bugs”. In: 2021 IEEE Symposium on Security and Privacy (SP). 2021, pp. 1286–1303. DOI: 10.1109/SP40001. 2021.00103. [21] Sadullah Canakci, Chathura Rajapaksha, Leila Delshadtehrani, Anoop Nataraja, Michael Bedford Taylor, Manuel Egele, and Ajay Joshi. “ProcessorFuzz: Processor Fuzzing with Control and Status Registers Guidance”. In: 2023 IEEE International Sym- posium on Hardware Oriented Security and Trust (HOST). 2023, pp. 1–12. DOI: 10.1109/ HOST55118.2023.10133714. [22] Raghul Saravanan, Sudipta Paria, Aritra Dasgupta, Swarup Bhunia, and Sai Manoj P D. “PROFUZZ: Directed Graybox Fuzzing via Module Selection and ATPG-Guided Seed Generation”. In: 2025 IEEE/ACM International Conference On Computer Aided Design (ICCAD). 2025, pp. 1–9. DOI: 10.1109/ICCAD66269.2025.11240782. URL: https: //ieeexplore-ieee-org.e.bibl.liu.se/document/11240782. [23] Snyder, Wilson and Verilator Community. Verilator. Verilog/SystemVerilog simulator, accessed: 2026-05-11. 2026. URL: https://verilator.org/guide/latest/.
48
[24] Mohammed Bakiri, Christophe Guyeux, Jean-François Couchot, and Abdelkrim Kamel Oudjida. “Survey on hardware implementation of random number generators on FPGA: Theory and experimental analyses”. In: Computer Science Review 27 (2018), pp. 135–153. ISSN: 1574-0137. DOI: https : / / doi . org / 10 . 1016 / j . cosrev . 2018.01.002. URL: https://www.sciencedirect.com/science/article/ pii/S1574013716302271. [25] Texas Instruments. What’s an LFSR. 1996. URL: https : / / www . ti . com / lit / an / scta036a/scta036a.pdf. [26] Vishakha V Bonde and AD Kale. “A Review on Implementation of Random Number Generation based on FPGA”. In: Int. J. Sci. Res. 14.1 (2013), pp. 2319–7064. URL: https: //www.ijsr.net/archive/v4i1/SUB15960.pdf. [27] Siti Hazwani, Sheroz Khan, Mohammad Umar Siddiqi, Khalid AS Al-Khateeb, Mo- hamed Hadi Habaebi, and Zeeshan Shahid. “Randomness analysis of pseudo random noise generator using 24-bits lfsr”. In: 2014 5th International Conference on Intelligent Sys- tems, Modelling and Simulation. IEEE. 2014, pp. 772–774. [28] Arrow Electronics, Inc. Everest-DEV-Board PolarFire FPGA Evaluation Board. Accessed: 2026-05-11. 2026. URL: https://www.arrow.com/en/products/everest-dev- board/arrow-development-tools.html. [29] Microchip Technology Inc. PolarFire FPGA Datasheet. DS00003831G. Accessed: 2026- 05-11. Microchip Technology Inc. 2026. URL: https : / / ww1 . microchip . com / downloads / aemDocuments / documents / FPGA / ProductDocuments / DataSheets/PolarFire-FPGA-Datasheet-DS00003831.pdf. [30] Microchip Technology Inc. PolarFire FPGA Product Overview. DS60001657R. Accessed: 2026-05-11. Microchip Technology Inc. 2025. URL: https : / / ww1 . microchip . com / downloads / aemDocuments / documents / FPGA / ProductDocuments / ProductBrief/PolarFire-FPGA-Product-Overview-60001657.pdf. [31] Microchip Technology Inc. VSC8575 Device Datasheet. VMDS-10457. Version 4.2. Ac- cessed: 2026-05-11. Microchip Technology Inc. 2019. URL: https://ww1.microchip. com/downloads/en/DeviceDoc/VMDS-10457.pdf. [32] Alex Forencich. Verilog Ethernet. Open-source Verilog Ethernet MAC/IP/UDP stack, ac- cessed: 2026-05-11. 2025. URL: https://github.com/alexforencich/verilog- ethernet. [33] Marc Heuse, Heiko Eißfeldt, Andrea Fioraldi, and Dominik Maier. AFL++. Ver- sion 4.00c. Jan. 2022. URL: https://github.com/AFLplusplus/AFLplusplus. [34] Wireshark Foundation. Wireshark. Network protocol analyzer, accessed: 2026-05-11. 2026. URL: https://www.wireshark.org/. [35] Microsoft. Windows Subsystem for Linux (WSL). Accessed 2026-05-11. 2025. URL: https: //learn.microsoft.com/en-us/windows/wsl/. [36] Alex Forencich. Verilog Ethernet: ip_eth_rx.v. 2025. URL: https : / / github . com / alexforencich/verilog- ethernet/blob/master/rtl/ip_eth_rx.v. Ver- ilog Ethernet. Open-source Verilog Ethernet MAC/IP/UDP stack, accessed: 2026-05-11. 2025. URL: https://github.com/alexforencich/verilog-ethernet. [37] Arm Ltd. AMBA AXI-Stream Protocol Specification. ARM IHI 0051B. Accessed: 2026- 05-11. Arm Ltd. 2010. URL: https : / / developer . arm . com / documentation / ihi0051/a/. [38] YosysHQ. Yosys Open SYnthesis Suite. Open-source RTL synthesis tool, accessed: 2026- 05-11. URL: https://yosyshq.net/yosys/.
49
[39] A. Izraelevitz, J. Koenig, P. Li, R. Lin, A. Wang, A. Magyar, D. Kim, C. Schmidt, C. Markley, J. Lawson, and J. Bachrach. “Reusability is FIRRTL ground: Hardware con- struction languages, compiler frameworks, and transformations”. In: 2017 IEEE/ACM International Conference on Computer-Aided Design (ICCAD). Nov. 2017, pp. 209–216. DOI: 10.1109/ICCAD.2017.8203780. [40] Microchip Technology Inc. PolarFire FPGA 1G Ethernet Loopback Using IOD CDR Applica- tion Note. AN4623. Accessed: 2026-05-11. Microchip Technology Inc. 2025. URL: https: / / ww1 . microchip . com / downloads / aemDocuments / documents / FPGA / ApplicationNotes / ApplicationNotes / PolarFire _ FPGA _ 1G _ Ethernet _ Loopback_Using_IOD_CDR.pdf. [41] Microchip Technology Inc. PolarFire FPGA 1G Ethernet Solutions User Guide. UG0687. Version 5.0. Accessed: 2026-05-11. Microchip Technology Inc. 2021. URL: https : / / ww1 . microchip . com / downloads / aemDocuments / documents / FPGA / ProductDocuments / UserGuides / microsemi _ polarfire _ fpga _ 1g _ ethernet_solutions_user_guide_ug0687_v5.pdf. [42] Microchip Technology Inc. ZL30722 Packet Clock Network Synchronizer Product Brief. Accessed: 2026-05-11. Microchip Technology Inc. Jan. 2016. URL: https : / / ww1 . microchip . com / downloads / en / DeviceDoc / zl30722 _ short _ from _ ds _ january_2016.pdf. [43] Microchip Technology Inc. CoreTSE User Guide. DS50003245E. Accessed: 2026-05-11. Microchip Technology Inc. 2025. URL: https://ww1.microchip.com/downloads/ aemDocuments / documents / FPGA / ProductDocuments / UserGuides / ip _ cores/directcores/CoreTSE_HB.pdf. [44] Scapy Project. Scapy. Interactive packet manipulation tool, accessed: 2026-05-11. 2026. URL: https://scapy.readthedocs.io/en/latest/. [45] Advanced Micro Devices, Inc. 7 Series FPGAs Configurable Logic Block User Guide (UG474). Version 1.9. Accessed: 2026-05-11. AMD. 2025. URL: https://docs.amd. com/r/en-US/ug474_7Series_CLB/Device-Resources.
50
Appendix
51
A Measurements
Table A.1 shows the test frequency and final coverage for all measurements and configura- tions used.
52
Table A.1: The test frequency and coverage obtained by the different coverage configurations in the different measurements. Fuzzer Measurement Prob. conf. Time/iterations Test cases per sec. Final coverage HW 1 1 100 M it. 1,72 M 91,9 % HW 2 1 100 M it. 1,72 M 91,9 % HW 3 1 100 M it. 1,72 M 91,9 % HW 4 1 100 M it. 1,72 M 91,9 % HW 5 1 100 M it. 1,72 M 91,9 % HW 1 2 100 M it. 1,21 M 91,9 % HW 2 2 100 M it. 1,21 M 91,9 % HW 3 2 100 M it. 1,21 M 91,9 % HW 4 2 100 M it. 1,21 M 91,9 % HW 5 2 100 M it. 1,21 M 91,9 % HW 1 3 100 M it. 1,46 M 94,5 % HW 2 3 100 M it. 1,46 M 94,5 % HW 3 3 100 M it. 1,46 M 94,5 % HW 4 3 100 M it. 1,46 M 94,5 % HW 5 3 100 M it. 1,46 M 94,5 % SW 1 1 1 M it. 17,7 k 94,5 % SW 2 1 1 M it. 17,5 k 94,5 % SW 3 1 1 M it. 17,5 k 94,5 % SW 4 1 1 M it. 17,4 k 94,5 % SW 5 1 1 M it. 18,3 k 94,5 % SW 1 2 1 M it. 17,9 k 94,5 % SW 2 2 1 M it. 17,7 k 94,5 % SW 3 2 1 M it. 18,9 k 94,5 % SW 4 2 1 M it. 17,6 k 94,5 % SW 5 2 1 M it. 18,2 k 94,5 % SW 1 3 1 M it. 18,3 k 94,5 % SW 2 3 1 M it. 18,5 k 96,1 % SW 3 3 1 M it. 17,1 k 94,5 % SW 4 3 1 M it. 16,3 k 94,5 % SW 5 3 1 M it. 17,0 k 94,5 % AFL 1 - 60 s 2,42 k 91,9 % AFL 2 - 60 s 2,45 k 91,9 % AFL 3 - 60 s 2,45 k 91,9 % AFL 4 - 60 s 2,42 k 91,9 % AFL 5 - 60 s 2,39 k 94,5 % AFL 1 inv. seed 60 s 2,42 k 52,5 % AFL 2 inv. seed 60 s 2,42 k 52,5 % AFL 3 inv. seed 60 s 2,48 k 57,2 % AFL 4 inv. seed 60 s 2,47 k 61,6 % AFL 5 inv. seed 60 s 2,46 k 60,3 % SW80MHz 1 3 1 M it. 17,5 k 94,5 % SW80MHz 2 3 1 M it. 18,4 k 94,5 % SW80MHz 3 3 1 M it. 16,9 k 94,5 % SW80MHz 4 3 1 M it. 17,8 k 94,5 % SW80MHz 5 3 1 M it. 18,6 k 94,5 %
53
B Ethernet subsystem overview for the FPGA
Figure B.1 shows the top level of the Ethernet subsystem including the inputs/outputs and how the different modules are connected with each other.
54
PADN INBUF_DIFF_0 PADP CoreUARTapb_0 Y
PCLK INBUF_DIFF RXRDY pf_init_monitor_0_0 PRESETNRX TXRDY APB_bif PARITY_ERR SSDetect_0 FABRIC_POR_N CoreUARTapb_0 PCIE_INIT_DONE OVERFLOW USRAM_INIT_DONE FRAMING_ERR TX rx_data[9:0]rst_brck stream_start SRAM_INIT_DONE SSDetect BANK_4_VDDI_STATUS CORETSE_0 DEVICE_INIT_DONE USRAM_INIT_FROM_UPROM_DONE MTXHWM XCVR_INIT_DONE ZL30722_CTRL_0 USRAM_INIT_FROM_SPI_DONE nResetclk ZL30722_MISO TBI_TX_VALID USRAM_INIT_FROM_SNVM_DONE SYNCMDO SRAM_INIT_FROM_UPROM_DONE ZL30722_SCLK MDC SRAM_INIT_FROM_SNVM_DONE IF0IF1 ZL30722_MOSI AXI4S_TCLK MDOEN SRAM_INIT_FROM_SPI_DONE SCLK ZL30722_RSTn ZL30722_TEST AXI4S_ICLK AXI4S_TTREADY AUTOCALIB_DONE MOSI ZL30722_CSn TXCLK pf_init_monitor_0 CSn ZL30722_AC0 RXCLK AXI4S_ITVALID ZL30722_AC1 TBI_TX_CLK AXI4S_ITLAST TBI_RX_CLK TCG[9:0] MIV_RV32_C0_0 ZL30722_CTRL MISO SIGNAL_DETECTMDI ANX_STATE[9:0] ANX_STATE[9] CLK PCLK ANX_STATE[8] RESETN PRESETN ANX_STATE[7] DEBUG AXI4S_TTVALID ANX_STATE[6] JTAG_TCK AXI4S_TTLAST ANX_STATE[5] JTAG_TDI ANX_STATE[4] JTAG_TDO TIME_COUNT_OUT[63:0] AXI4S_ITREADY JTAG_TDO_DR EXT_RESETN RCG[9:0] ANX_STATE[3] APB_INITIATOR AXI4S_TTDATA[31:0] ANX_STATE[2] JTAG_TMS AXI4S_TTKEEP[3:0] ANX_STATE[1] APBSAXI4S_TRGT ANX_STATE[0] IRQJTAG_TRSTN TSM_CONTROL[31:0] EXT_IRQ TSM_TX_INTR[3:0] MIV_RV32_C0 TSM_RX_INTR[3:0] AXI4S_ITDATA[31:0] AXI4S_ITKEEP[3:0] AXI4S_INITR COREJTAGDEBUG_C0_0 PF_IOD_CDR_CCC_C0_0 CORETSE_0 AXI4S_ITUSER[7:0] JTAG_HEADERTCKTDI DEBUG_TARGET_0 REF_CLK PLL_LOCK TGT_TCK_0 ARST_N TX_CLK_G TDOTMS TGT_TDO_0 CDR_CLOCKS TGT_TMS_0 REFCLK_N TGT_TDI_0 PF_IOD_CDR_CCC_C0 REFCLK_P CoreAPB3_0_0 TRSTB TGT_TRSTN_0 COREJTAGDEBUG_C0 PHY_MDC REF_CLK_0 RESET_N RX_N APB3mmaster APBmslave0 PHY_MDIO RX APBmslave2 PF_CCC_0_0 PHY_RST RX_P APBmslave1 coma_modePCLK TCK CoreAPB3_0 REF_CLK_0 OUT0_FABCLK_0 ZL30722_AC0 ZL30722_MISO PF_CCC_0 PLL_LOCK_0 ZL30722_CSn TRSTB CORESPI_0_0 PF_IOD_CDR_C0_0 ZL30722_MOSI AXI4S_TTLAST TMS TDI PLL_POWERDOWN_N_0 ZL30722_AC1 AXI4S_TTVALID AXI4S_TTKEEP[3:0] ZL30722_RSTn AXI4S_TTDATA[31:0] A AND2_2 SPIRXAVAIL SPITXRFM RX_PRX_N TX_P ZL30722_TEST B AND2 PCLK SPISCLKO STREAM_START TX_N TXZL30722_SCLK AXI4S_ITREADY Y SPIINT SPIOEN SPISDO TX_DATA[9:0] OR2_0 RST_N RX_CLK_R TX_N AB Y PRESETN RX_VAL TX_P SPISSI SPISS[7:0] OR2 SPIMODE CDR_CLOCKS RX_DATA[9:0] LINK_OKTDO SPISDI PF_IOD_CDR_C0 AXI4S_TTREADY SPICLKI SPISS[7] AXI4S_ITLAST SPISS[6] AXI4S_ITVALID D BIBUF_0EY PAD APB_bif SPISS[5] AXI4S_ITKEEP[3:0] SPISS[3] AXI4S_ITUSER[7:0] BIBUF SPISS[4] AXI4S_ITDATA[31:0] SPISS[2] CORESPI_0 Core_reset_pf_0 SPISS[1] CLKEXT_RST_N SPISS[0] BANK_x_VDDI_STATUS BANK_y_VDDI_STATUS PLL_LOCK SS_BUSY PLL_POWERDOWN_B INIT_DONE FABRIC_RESET_N FF_US_RESTORE FPGA_POR_N Core_reset_pf
Figure B.1: Ethernet subsystem overview 55