SOURCE ARCHIVE
EXTRACTED CONTENT
100,407 chars HeteroFuzz: Fuzz Testing to Detect Platform Dependent
Divergence for Heterogeneous Applications
Qian Zhang Jiyuan Wang Miryung Kim
University of California, Los Angeles University of California, Los Angeles University of California, Los Angeles
USA USA USA
zhangqian@cs.ucla.edu wangjiyuan@g.ucla.edu miryung@cs.ucla.edu
ABSTRACT KEYWORDS As specialized hardware accelerators like FPGAs become a promi- Fuzz testing, heterogeneous applications, platform-dependent di- nent part of the current computing landscape, software applications vergence are increasingly constructed to leverage heterogeneous architec- tures. Such a trend is already happening in the domain of machine ACM Reference Format: learning and Internet-of-Things (IoT) systems built on edge devices. Qian Zhang, Jiyuan Wang, and Miryung Kim. 2021. HeteroFuzz: Fuzz Testing Yet, debugging and testing methods for heterogeneous applications to Detect Platform Dependent Divergence for Heterogeneous Applications. are currently lacking. These applications may look similar to reg- In Proceedings of the 29th ACM Joint European Software Engineering Confer- ular C/C++ code but include hardware synthesis details in terms ence and Symposium on the Foundations of Software Engineering (ESEC/FSE of preprocessor directives. Therefore, their behavior under hetero- ’21), August 23–28, 2021, Athens, Greece. ACM, New York, NY, USA, 13 pages. https://doi.org/10.1145/3468264.3468610 geneous architectures may diverge significantly from CPU due to hardware synthesis details. Further, the compilation and hardware 1 INTRODUCTION simulation cycle takes an enormous amount of time, prohibiting frequent invocations required for fuzz testing. There is a growing interest in computer architectures to incorporate We propose a novel fuzz testing technique, called HeteroFuzz, heterogeneity and specialization to improve performance [11, 13, designed to specifically target heterogeneous applications and to 14, 17]. FPGA is re-programmable hardware that often exceeds detect platform-dependent divergence. The key essence of Hetero- the performance of general-purpose CPUs by several orders of Fuzz is that it uses a three-pronged approach to reduce the long magnitude [6, 22, 46] and offers lower cost across a wide variety of latency of repetitively invoking a hardware simulator on a hetero- domains [5, 10, 15]. To support heterogeneous computing, hardware geneous application. First, in addition to monitoring code coverage vendors provide CPU+FPGA multi-chip packages such as Intel as a fuzzing guidance mechanism, we analyze synthesis pragmas in Xeon [23, 47], and cloud providers support virtual machines with kernel code and monitor accelerator-relevant value spectra. Second, FPGA development frameworks such as Amazon F1 [1]. In the we design dynamic probabilistic mutations to increase the chance context of this paper, we use a term, heterogeneous applications to of hitting divergent behavior under different platforms. Third, we refer to software that consists of both host code and kernel code and memorize the boundaries of seen kernel inputs and skip HLS simu- can offload its computation-intensive kernel from CPU to FPGA lator invocation if it can expose only redundant divergent behavior. under heterogeneous architectures. We evaluate HeteroFuzz on seven real-world heterogeneous appli- Platform Dependent Divergence. Although FPGAs are becom- cations with FPGA kernels. HeteroFuzz is 754X faster in exposing ing commercially available to a broad user base, they are associated the same set of distinct divergence symptoms than naive fuzzing. with a high development cost [50]. There has been work on high- Probabilistic mutations contribute to 17.5X speed up than the one level synthesis (HLS) [16], which takes C/C++ kernel code as input without. Selective invocation of HLS simulation contributes to 8.8X and automatically synthesizes a corresponding FPGA accelerator. speed up than the one without. With HLS, programmers can implement their heterogeneous ap- plications in C/C++; however, such C/C++ programs can produce CCS CONCEPTS different results on heterogeneous architectures compared to CPU • Software and its engineering → Software testing and debug- due to various platform-dependent characteristics—bitwidth, avail- ging; • Computer systems organization → Heterogeneous. able resources, memory access, recursion depth, dataflow mode, etc. For example, because an optimized FPGA kernel uses a custom bitwidth for integers and floating points, it could lead to overflows on FPGA. Thus, detecting platform-dependent divergent behavior is important because even when the application runs correctly on Permission to make digital or hard copies of part or all of this work for personal or CPU, it could still crash or produce a wrong output silently when classroom use is granted without fee provided that copies are not made or distributed the kernel is executed on the FPGA accelerator [40, 41]. In fact, for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. our investigation of Xilinx’s forum—a popular online Q/A forum For all other uses, contact the owner/author(s). for FPGA HLS development—shows numerous examples of such ESEC/FSE ’21, August 23–28, 2021, Athens, Greece platform-dependent divergence. Table 1 illustrates several such ex- © 2021 Copyright held by the owner/author(s). amples [65]. Programmers ask, “why is there a difference between ACM ISBN 978-1-4503-8562-6/21/08. https://doi.org/10.1145/3468264.3468610 C-code, RTL-simulation, and the hardware test runs?” [58]. In this
ESEC/FSE ’21, August 23–28, 2021, Athens, Greece Qian Zhang, Jiyuan Wang, and Miryung Kim
Table 1: Examples of Behavior Divergence Between CPU and FPGA
ID Description Time
894069 [57] Segmentation fault when allocating a big array int x[1920][1080] on FPGA yet no error on CPU 8.5h
595225 [58] Different outcome caused by HLS dataflow directive 47.7h 438446 [59] Different outcome caused by FPGA fetching incorrect struct vector training_set[MAXSZ] 10.6h 754676 [60] Different outcome caused by bitwidth typedef ap_fixed<25,1,AP_RND> s25f24_type 2.3h 785019 [61] Getting all zeros when shifting an array caused by #pragma HLS RESET 3.1h 907213 [62] Undecided output when overwriting a same variable within the loop yet no error on CPU 79.4h 1166264 [63] EMFILE error when loading 2048 files with #pragma HLS ARRAY_PARTITION yet no error on CPU 11.7h 1126600 [64] The 25-tap FIR filter bypassess some input multiplications with #pragma HLS PIPELINE 93.2h
post 894069 [57], due to limited stack size, running an image process- Second, we design dynamic probabilistic mutations to increase
ing application with HLS simulation leads to a segmentation fault, the chance of exposing platform-dependent behavior. HeteroFuzz
but no such error happens on the CPU Linux platform. Though gradually increases the activation probability of the current muta-
this programmer expressed a desire to test this platform-dependent tion if a new accelerator value spectrum is achieved. For example,
divergence, no tool exists to meet such needs. when a programmer uses #typedef ap_uint
HeteroFuzz: Fuzz Testing to Detect Platform Dependent Divergence for Heterogeneous Applications ESEC/FSE ’21, August 23–28, 2021, Athens, Greece
Figure 1: HeteroFuzz’s three-pronged approach: multi-dimensional guidance, probabilistic mutations, and selective invocation
As the current need and prevalence of developing heterogeneous for CPU defines integer arithmetics at 32 bits by default, individual applications are rising significantly in the new era of cloud-based bitwidths could be programmed in FPGA. For example, if an inte- hardware accelerator microservices [4], the software engineering ger variable 𝑎𝑔𝑒 has a maximum value of 83—it only needs 7 bits community must design a new automated testing technique to instead of 32 bits. The programmer can insert a pragma #typedef improve efficiency and effectiveness for this emerging category of ap_uint<7> bit7 to declare an arbitrary precision integer of 7 heterogeneous applications. To our knowledge, HeteroFuzz is the bits. With custom bitwidth, FPGA accelerators have much more fre- first end-to-end approach that re-invents and adapts traditional quent overflows that would not happen on CPU, leading to failures fuzzing to heterogeneous applications. It speeds up differential or wrong outputs in the host code. testing under heterogeneous hardware platforms, and overcomes Memory Management. FPGA has no capability of on-chip mem- the limitation of long hardware simulation latency during fuzz ory management. Function calls to memory allocation are replaced testing, while finding significantly more divergence errors. by pre-allocating a static array with an estimated size and managing 2 BACKGROUND data elements manually. Similarly, recursions must be converted into iterations using a stack with a finite estimated size [52, 67, 68]. 2.1 Heterogeneous Computing with FPGA HLS When an input or recursion depth exceeds the pre-estimated size, Heterogeneous architectures with FPGAs have shown the potential the FPGA kernel could crash or return a completely wrong output to improve performance under strict energy constraints [16]. A het- to the host code by accessing an unexpected address. erogeneous application can be decomposed into host code executed Parallelization. Reprogrammable hardware provides an inherent on CPU and kernel code executed on accelerators. potential for parallelizing computation. Such parallelization can High-Level Synthesis (HLS). HLS tools for FPGAs, such as Xilinx be done through pipelining of different computation stages and by Vivado HLS [66], have raised the abstraction of hardware develop- duplicating processing elements or data paths to achieve an effect ment by automatically generating RTL (Register-Transfer Level) similar to multi-threading. To guide such parallelization, a developer descriptions from C/C++ code. During HLS, the frontend schedules must write HLS pragmas such as #pragma HLS unroll—creating each operation from the kernel code to certain clock cycle time multiples copies of the loop body which allows some or all loop slots. Next, it allocates the number and type of hardware unit re- iterations to occur in parallel or #pragma HLS dataflow—enabling sources used for implementing functionality. Finally, the binding task-level pipelining, which allows functions to overlap their oper- stage maps all operations to the allocated hardware units. This en- ations. Such parallel execution can produce an outcome different tire HLS process can take several minutes for simulation and several from sequential execution on CPU, especially when a feedback path hours for synthesis, depending on the complexity of kernel logic. exists in two different functions or modules. To achieve good quality-of-results (QoRs), HLS developers must As described above, testing HLS code is different from C/C++ be- insert synthesis directives and pragmas manually [19]. This requires cause HLS directives can cause differential behavior on accelerators. having inter-disciplinary expert knowledge and knowing obscure 2.2 Fuzz Testing platform-dependent details. Testing HLS code is difficult because the Without loss of generality, the procedure of fuzzing [39, 69] can be resulting FPGA may produce an outcome different from CPU due described as follows. Starting with an initial set 𝐼 of seed inputs, the to the assumptions specified during accelerator synthesis. Below, fuzzing procedure randomly selects one input from 𝐼, and generates we describe a few examples of how HLS directives and pragmas new inputs by mutating several bits or bytes of the current input. In can produce a different execution outcome. terms of which mutations to apply, it selects an available mutation Custom Bitwidth. On-chip resource efficiency yields a higher from a given set of mutation operators. Commonly used input level of parallelization. HLS supports arbitrary bitwidth for integers mutations are either bit-level or byte-level mutations in which because reducing a variable’s bitwidth could lead to resource reduc- random bits (or bytes) are flipped. It then collects the guidance tion in FPGA directly. While the instruction set architecture (ISA) feedback such as branch coverage by executing the instrumented
ESEC/FSE ’21, August 23–28, 2021, Athens, Greece Qian Zhang, Jiyuan Wang, and Miryung Kim
1 int main(int argc, char 1 #include "ap_int.h" 1 #include "ap_int.h" *argv[]){ 2 int accumulate( 2 void kernel{ 2 int data[] = gradient(argv[1]); 3 int data[size],int size); 34 #pragma HLS dataflow 3 int sum; 4 typedef ap_uint<8> bit8; fifo a; fifo b; 4 float th = argv[2]; 5 funX(a,b); 5 for(i = 0 to data. 5 typedef ap_uint<9> bit9; 6 funY(a,b);} 7 void funX{ 6 size()) 6 #define M 400; 8 typedef ap_uint<1> bit1; 7 sum+=data[i]; 7 #pragma HLS INTERFACE 9 bit1 exist = b.read_nb(temp_b); 8 for(i = 0 to data.size()){ 8 m_axi port=data 10 if (exist and temp_b == 3){ 9 data[i] /= sum; 9 offset=slave bundle=gmem 11 a.write(1);} 10 if(data[i] > th) keep(argv[3]); 10 #pragma HLS INTERFACE 12 else{a.write(0);}} 11 else discard(argv[4]);}} 13 void funY{ 11 s_axilite port==data 14 temp_a = a.read(); (a) Original Application 12 bundle=control 15 b.write(temp_a+3);} 13 #pragma HLS INTERFACE 1 int main(int argc, char 14 m_axi port=size Figure 3: Example Kernel Program Showing Diver- *argv[]){ 15 offset=slave bundle=gmem gence: Dataflow Mode 2 int data[] = 16 bit8 data_fpga[M]; gradient(argv[1]); 17 bit9 i; As discussed in Section 2, when writing kernel code, HLS devel- 34 int sum; 18 bit8 sum; opers must insert directives and pragmas to expose layout regularity float th = argv[2]; 19 for (i = 0 to size) { 5 int size = data.size(); 20 #pragma HLS unroll and parallelism for FGPA accelerator synthesis. After analyzing 6 accumulate( 21 data_fpga[i] = (bit8)data[i];} the values of sample data sent to the kernel, Bob includes the ap_- 7 data[size],int size); 22 SUM_LOOP: for (i = 0 to size) int header file at line 1 in Figure 2c to use custom bitwidths of 8 8 for(i = 0 to data.size()){ 23 #pragma HLS unroll factor=2 9 data[i] /= sum; 24 data_fpga[i] = (bit8)data[i]; and 9 respectively in lines 4 and 5 instead of using 32-bit integers. 10 if(data[i] > th) 25 sum + = data_fpga[i];} He also pre-defines the max array size as 400 in line 6. Lines 7-15 keep(argv[3]); 26 sum = (int) sum; 11 else discard(argv[4]);}} 27 return sum;} define the data transfer interface between CPU and FPGA. Lines 19-21 offload data from CPU to data_fpga on FPGA. To exploit (b) Host in Heterogeneous Ap- (c) Kernel in Heterogeneous Appli- hardware-level parallelization, Bob inserts #pragma HLS unroll in plication cation line 23 to make two copies of the SUM_LOOP body. After compiling Figure 2: Example Program Showing Divergence: Bitwidth this heterogeneous application using Vivado HLS, CPU-side host program with new inputs [32, 39, 69]. All inputs that enhance a code in Figure 2b will invoke the kernel accumulate at lines 6-7, guidance metric—e.g., exercising a new branch or leading to a send data to the synthesized FPGA, and wait for the returned result. unique crash—are then saved to the working set, 𝑆. Then, with 𝑆, After converting this original application to a heterogeneous the fuzzing procedure moves onto the next step of selecting an application, Bob may want to test it by using handcrafted inputs input from 𝑆 and applying mutations to the input. or randomly generating data. Bob must check whether this het- This fuzzing procedure is based on two inherent yet over-sighted erogeneous application produces divergent behavior on the same assumptions: (1) it takes a minuscule amount of time to execute input. For example, when the kernel inputs [1,1,1,253] are sent the target program, and (2) arbitrary mutations are likely to yield to FPGA, Bob finds a divide-by-zero error in the host code at line 9 meaningful inputs. Such assumptions do not hold for heterogeneous of Figure 2b, which does not happen when running on CPU only. applications because hardware simulation takes several minutes and When the kernel inputs [2,1,1,253] are sent to FPGA, the FPGA random mutations cannot account for hardware synthesis intricacies. accelerator will return the sum as 1, instead of the correct sum 257 computed by CPU, leading to a completely wrong filtering output 3 MOTIVATING SCENARIO in lines 10-11 of Figure 2b. Finding such divergence-inducing inputs is extremely challenging—suppose Bob uses an AFL-like technique This section presents motivating examples for why testing platform- in an attempt to find such divergence-inducing inputs. AFL cannot dependent behavior is necessary. Suppose that Bob writes an image distinguish the above two inputs that produce distinct divergence denoising application shown in Figure 2a. This application filters symptoms because no such errors happen using 32-bit integers on the gradients based on their percentages with respect to the sum of CPU and these two inputs are identical in terms of their branch all gradients. Line 2 calculates the gradient vector of an input image coverage in host code. by taking the absolute difference between two adjacent pixels. Line Figure 3 describes another kernel example. In line 9, read_nb() 4 defines the filtering threshold. Lines 5-7 aggregate the gradients is a Boolean type HLS read function which returns false if data is across the entire vector. Lines 10-11 filter out the gradients less unavailable. When executing on CPU only, funX is executed before than the filtering threshold. Bob runs this C application on CPU funY sequentially, so fifo b is always empty and line 11 can never and finds that the loop at lines 5-7 is a hot code path responsible be executed. As a consequence, fifo a is full of 0 and fifo b for a significant execution time. becomes empty. However, when executing on FPGA, #pragma HLS Therefore, Bob decides to convert this original application in dataflow in line 2 enables parallel execution of funX and funY. C to a heterogeneous application. He refactors the loop at lines 5-7 Therefore, fifo b is not empty and the if condition at line 10 can in Figure 2a into an HLS kernel function accumulate in Figure 2c. evaluate to true, if the current data in b is 3. The original application is then converted to host code in Figure 2b HeteroFuzz is motivated by such challenge of testing heteroge- that still runs on CPU and communicates directly with a hardware neous applications described above. HeteroFuzz directly analyzes accelerator generated from the kernel code in Figure 2c. HLS pragmas in kernel code, monitors accelerator spectra and
HeteroFuzz: Fuzz Testing to Detect Platform Dependent Divergence for Heterogeneous Applications ESEC/FSE ’21, August 23–28, 2021, Athens, Greece
Table 2: Mapping from HLS Pragmas to Accelerator Value Spectra
FPGA-Relevant
Category HLS Directive /Pragma Description of Pragma Value Spectra
Data Type typedef ap_uint<x> bitx Define an unsigned Integer with a custom bitwidth The actual variable values
typedef ap_int<x> bitx Define a signed Integer with a custom bitwidth with this type
#pragma HLS array_partition Partition an array into smaller arrays or individual elements
Memory #pragma HLS array_reshape Combine array partitioning with vertical array mapping The set of accessed offsets
#define size M Define the estimated array size
Recursion N/A N/A The actual used size of
a stack and the number of iterations
Parallelization #pragma HLS dataflow Enable task-level pipelining The FIFO queue size
#pragma HLS pipeline Allow concurrent execution of operations if feedback path exists
#pragma HLS unroll Create multiples copies of the loop body
Loop #pragma HLS loop_tripcount Specify the total number of iterations The actual number of iterations
#pragma HLS loop_flatten Allow nested loops to be flattened into a single loop hierarchy
#pragma HLS loop_merge Merge consecutive loops into a single loop to reduce overall latency
Interface/ #pragma HLS INTERFACE Perform input and output operations using a specific I/O protocol N/A
Configuration etc. in the design interface
branch coverage in host code, adjusts the activation probabilities of multiplexers at the hardware level. Each branch in the synthesized mutation operations, and reduces unnecessary simulations to find hardware is executed in parallel as long as prior signals are ready; divergence errors. however, the output being produced by the hardware logic for an untaken branch is never used. Thus, branch coverage cannot be 4 APPROACH derived directly from the synthesized hardware [29]. Branch cover- HeteroFuzz contains three novel components that work in con- age collected from the host code only is ineffective as a guidance cert to detect platform-dependent differential behavior for hetero- feedback due to its inability to detect errors in heterogeneous ap- geneous applications. Figure 1 describe its overall architecture: plications, because inputs with the same coverage in the host code multi-dimensional guidance feedback based on accelerator spectra may exhibit different behavior, as discussed in Section 3. (Section 4.1), (B) dynamic probabilistic mutations (Section 4.2), and HeteroFuzz analyzes the inserted HLS pragmas and traces their (C) selective invocation to reduce the latency of repetitively invok- associated FPGA-relevant spectra as accelerator feedback. It cur- ing a hardware simulator (Section 4.3). Its three-pronged approach rently supports five kinds of value spectra, shown in Table 2. These is based on two key insights: platform-dependent behavior can be mappings are user-extensible by modifying a configuration file. exposed by monitoring accelerator-relevant spectra and accelera- Figure 2c is an accumulation FPGA kernel with ten HLS di- tor synthesis is optimized based on kernel input values; therefore rectives. It uses typedef ap_uint<8> bit8 and typedef ap_- multiple invocations within the same value range may expose re- uint<9> bit9 to customize 32-bit integer data into 8 and 9 bit dundant divergence symptoms. Although HeteroFuzz is designed integers respectively. HeteroFuzz monitors the actual values of vari- for FPGA accelerators, its key idea can be easily extended to other ables declared with these types, such as data_fpga, sum, and i in heterogeneous platforms by monitoring platform-specific spectra. lines 22-24 of Figure 2c. HeteroFuzz ignores the pragmas in lines 7-21 as HLS data transfer interface has no impact on kernel logic. 4.1 Accelerator Spectra Monitoring The pre-allocated array data_fpga in line 22 is declared with a maximum size of 400. HeteroFuzz monitors the accessed offsets in Inputs with the same branch coverage can still have distinct impacts case an unexpected address is accessed. The unroll pragma in line on hardware characteristics; HeteroFuzz augments branch cover- 29 makes two copies of the loop body. HeteroFuzz records the ac- age feedback with hardware-dependent characteristics to detect tual iteration counts, which can be used together with array access divergence symptoms when running host and accelerator together. offsets to detect potential divergence when the size of offloaded Prior studies [40, 41] find that numerous severe security problems data is not multiple of 2. Then HeteroFuzz instruments the kernel originate from such host-accelerator interaction. to record these three types of values spectra for Figure 2c: (1) the Coverage Feedback in Host Code. HeteroFuzz instruments the value range of fpga_data, output variable sum, and intermediate host program based on its extracted control flow graph (CFG) using variable i; (2) the set of accessed offsets in array data_fpga; and LLVM [36]. Each node in a CFG represents a basic code block and (3) the actual number of loop iterations. each edge (𝐴, 𝐵) represents a transition between two blocks 𝐴 and Similar to how AFL [69] keeps track of a single test’s branch 𝐵 in the CFG. Similar to AFL, HeteroFuzz initializes an array with coverage and cumulative branch coverage for all tests, HeteroFuzz binary bits called trace_bits, with zeros, in each fuzzing iteration. keeps track of value spectra for each test execution and cumulative Iteration here means one execution of a target program with a value spectra. In each test execution, it initializes an array acc_- generated test input. Each bit represents a branch in the program. If feedback, where each entry has four fields: <value spectra type, an edge (𝐴, 𝐵) is exercised, HeteroFuzz updates the corresponding name, min value, max value>. It then updates total_feedback entry from 0 to 1 in trace_bits. to record cumulative value spectra for all seen inputs that can safely Accelerator Value Spectra. Tracing branch coverage in a syn- execute on the synthesized FPGA. For example, when the kernel thesized hardware kernel is infeasible, because HLS takes kernel inputs are [1,1,1,253] , the tracked hardware feedback is shown code as input but implements branching logic using a pipeline of in column FPGA Accelerator Spectra in Table 3. While coverage-guided
ESEC/FSE ’21, August 23–28, 2021, Athens, Greece Qian Zhang, Jiyuan Wang, and Miryung Kim
Table 3: Example Execution of Generated Inputs
Act. Program Kernel FPGA Accelerator Spectra New Mutation Memorization
ID Mut. Inputs Inputs Invoke Type Name Min Max Branch Overflow Probability Range Size
seed N/A [1,2,1,2,11] [1,1,1,9] yes input_value fpga_data 1 9 yes no [0.17,0.17,0.17, (1,9) 4
variable_value i 0 3 0.17,0.17,0.17]
variable_value sum 2 12
mem_offset fpga_data [0,1,2,3]
loop SUM_LOOP N/A 2
1 M6 [1,2,1,2,2] [1,1,1,0] no N/A N/A N/A N/A no N/A [0.17,0.17,0.17, (1,9) 4
0.17,0.17,0.17]
2 M3 [1,2,1,2,255] [1,1,1,253] yes input_value fpga_data 1 253 no yes [0.16,0.16,0.22, (1,9) 4
variable_value i 0 3 0.16,0.16,0.16]
variable_value sum 0 3
mem_offset fpga_data [0,1,2,3]
loop SUM_LOOP N/A 2
3 M3 [0,2,1,2,255] [2,1,1,253] yes input_value fpga_data 1 253 no yes [0.16,0.16,0.22, (1,9) 4
variable_value i 0 3 0.16,0.16,0.16]
variable_value sum 1 4
mem_offset fpga_data [0,1,2,3]
loop SUM_LOOP N/A 2
4 M1 [1,2,1,2,11,36] [1,1,1,9,25] yes input_value fpga_data 1 25 no no [0.21,0.15,0.21, (1,25) 5
variable_value i 0 5 0.15,0.15,0.15]
variable_value sum 2 37
mem_offset fpga_data [0,1,2,3,4,5]
loop SUM_LOOP N/A 3
fuzzing would discard the inputs for not achieving new branch ( coverage, HeteroFuzz saves them for increasing the value spectra 1 if i is a kernel-sensitive input; of fpga_data and sum. Based on the collected branch coverage 𝑃𝑖 = 𝑎 otherwise (1) feedback and monitored FPGA accelerator spectra, an input will be kept in the generated tests, if it increases either kind of feedback. 𝑎That is, the kernel-sensitive inputs are always selected and mutated. is the probability of selecting a non-kernel input, as it is still necessary to mutate all inputs in the host code to exercise diverse 4.2 Probabilistic Mutations behavior and increase branch coverage in the host code. In our Designing mutations to detect platform dependent behavior in experiments, we use 𝑎 = 0.10 as default. heterogeneous applications is challenging because: (1) mutations Considering that data offloaded to a hardware kernel is often an modify the inputs of a host program, not just those host inputs array or a matrix, HeteroFuzz uses six basic mutations extended related to kernel inputs, and (2) different mutations contribute to from AFL to generate new inputs, shown as follows. Scala values are divergence-inducing inputs in different degrees. treated as one element array. The reason why we focus on arrays We propose dynamic probabilistic mutations to increase the and matrices is that hardware accelerators are designed to batch chance of detecting hardware-dependent behavior. These mutations process multiple elements in parallel. All divergence errors that represent input modifications to explore the input space during fuzz can be found by naive fuzzing should be all found by HeteroFuzz as testing, as opposed to code modifications used in mutation testing. well because HeteroFuzz ’s input mutations are a superset of naive Input mutations are directly applied to inputs in host code. We clas- input mutations. sify those inputs into kernel-sensitive inputs and kernel-irrelevant • Data Size Mutation (M1) inserts/deletes one or several inputs. Kernel-sensitive inputs refer to a subset of host program random elements if the input data is an array: e.g., from inputs that will be offloaded to an FPGA kernel, and all other inputs [1,2,3] to [1,2,3,4] or [1,3] are kernel-irrelevant inputs. To identify kernel-sensitive inputs, • Data Dimension Mutation (M2) adds/removes one or sev- HeteroFuzz uses static backward slicing [53, 55] of the argument eral columns if the input data has multiple dimensions: e.g., names, data and size of the kernel function, accumulate all the from [[1,2,3],[3,2,1]] to [[1,2,3]] . way to the original input arguments in the host code. For example, • Data Element Mutation (M3) mutates the value of one in Figure 2b, starting from the invocation of accumulate at lines 6 element based on its type: e.g., from [1,2,3] to [1,2,4] . and 7, we use backward slicing on its arguments, data and size, • Type Mutation (M4) modifies the type of a selected entry, in turn marking line 5 and line 2 where data and size are defined while keeping the same value: e.g., from Integer to Float. respectively. Starting from line 2, it then marks argv[1], which • Bit Mutation (M5) flips one or several bits. is an input to the host code’s main function in line 1. A gradient • Byte Mutation (M6) flips one or several bytes. threshold argv[2] is used in in line 4, argv[3] is used at keep in While conventional fuzzing does not update the activation proba- line 10, and argv[4] is used at discard in line 11, making them bilities of mutations and generally keeps them uniform, HeteroFuzz irrelevant to accumulate in Figure 2c. Therefore, we label the input assigns different activation probabilities to individual mutations and argument argv[1] as a kernel-sensitive input. HeteroFuzz initial- updates their probabilities based on accelerator spectra feedback. If izes the selection probability of individual host inputs to be mutated a new child input generated by mutation 𝑚 increases the monitored as follows: accelerator spectra, 𝑚 will be labeled as a favored mutation. Favored
HeteroFuzz: Fuzz Testing to Detect Platform Dependent Divergence for Heterogeneous Applications ESEC/FSE ’21, August 23–28, 2021, Athens, Greece
Table 4: Subject Programs
Subject # of Symptoms % of Div-inducing Inputs
ID Program Kernel Description HF WithoutSpectra HF WithoutSpectra
P1 Median Filter Bubble Sort Blur an image by replacing the pixel value to a median 6 1 68.2% 6.8%
P2 Median Filter Merge Sort Blur an image by replacing the pixel value to a median 6 1 37.3% 4.5%
P3 Image Denoising Accumulation Denoise an image based on analyzing image gradients 5 0 71.0% 0%
P4 KNN L2norm Finds the top-k most relevant elements 5 2 72.9% 14.5%
P5 Signal Transmission RGB2YUV Transform RGB signals to YUV signals 7 2 40.0% 4.5%
P6 3D Rendering Rendering Render an image based on 3D information 9 2 56.0% 10.0%
P7 Face Detection Detection Detect human faces in an image 9 2 52.2% 5.0%
mutations will then have higher activation probabilities. Given the that the accelerator designer uses to optimize the FPGA data type. activation probabilities of 𝑙 mutations 𝑃 = {𝑃0, 𝑃1, ...𝑃𝑙−1 }, they will Therefore, FPGA execution with inputs that fit the memorized range be updated dynamically: can either be fully correct or only report a redundant divergence 𝑃𝑚 + 𝛼 if 𝑚 is chosen and it symptom arising from trying out a new value exceeding the safe 𝑃𝑚 = increases spectra (2) range. Thus, we can safely obviate such executions to speed-up the 𝑃𝑚 − 𝑙𝛼−1 otherwise input generation process. What we mean by “safely obviate" is that, for the purpose of finding a new divergence symptom, it does not 𝑙 is the number of mutations. Since we have six mutations, here 𝑙 matter because you will get the same kind of divergence error. is 6. Every 𝑃𝑚 is initialized as 1/𝑙 and 𝛼 is the update factor that is pre-defined as 0.05. In column Mutation Probability of Table 3, 5 the activation probability for each mutation is initialized to 1/𝑙 = EVALUATION 0.17. In the second execution (ID 2), inputs created by mutation M3 We evaluate following research questions: increase the spectra of input_value and variable_value. Het- RQ1 How effective is HeteroFuzz’s accelerator spectra monitoring eroFuzz consequently increases M3’s activation probability from in generating divergence-inducing inputs? 𝑃𝑚 = 0.17 to 𝑃𝑚 + 𝛼 = 0.22, and adjusts the probabilities of other RQ2 How much speed-up is enabled by HeteroFuzz’s dynamic mutations to 𝑃𝑚 − 𝑙−1𝛼 = 0.16. probabilistic mutations? 4.3 Selective Invocation RQ3 How much speed-up is enabled by HeteroFuzz’s selective To reduce the long latency of repetitive hardware simulation, Het- RQ4 invocation? eroFuzz maintains the range of seen kernel input values that run How effective and efficient would HeteroFuzz be, in compari- correctly on FPGA Memorization/Range in Table 3, and their data RQ5 son to naïve fuzzing using an AFL-like technique? size, Memorization/Size in Table 3. We use a global variable enable_- Can input range checking obviate the need of fuzz testing sim to indicate whether hardware simulation should be invoked or and still find platform-dependent errors? not. This variable is set to be true if the range or the data size of Benchmarks. Our benchmarks include seven real-world and pub- seen kernel inputs grow beyond the current records. Suppose that licly available heterogeneous applications written in C/C++ with HeteroFuzz had seen a set of concrete values [2,3,5] for an integer FPGA kernels, listed in Table 4. P1-P3 are from OpenCV [42] ex- array x, we maintain the range of x as (2, 5). It is safe to record only amples, P4-P5 are from [30], and P6-P7 are from Rosseta [71]. Our the range (2, 5) instead of considering combinations of concrete subject selection criteria is based on whether the programs cover values as a set {2, 3, 5} for the purpose of finding a new, distinct different synthesis optimizations, and whether a diverse set of HLS divergence error symptom. If an error were to happen for an unseen pragmas is used for detecting platform-dependent divergence be- combination such as {2, 4, 5}. This error symptom will be identical havior. Seven programs in Table 4 cover all twenty four kinds of to the error symptoms that one would get for executing a value pragmas (e.g., custom bitwidth, parallelization, memory manage- less than 2 or greater than 5. This is due to the unique property of ment, etc.), and thus activate different kinds of synthesis optimiza- FPGA synthesis, where an integer or fixed-point variable maps to tions. a contiguous range of values. These subject programs may look small in size, but they are actu- In Table 3, for ID seed, column Memorization/Range is updated ally larger than kernel benchmarks the FPGA community uses [24, to (1,9) for variable fgpa_data, as the smallest item is 1 and the 49, 71] and real-world heterogeneous applications described in Xilinx largest item is 9 and its execution does not lead to any error on posts [57–59]. Building a hardware accelerator is similar to design- FPGA execution. For ID 1, since the kernel input [1,1,1,0] is still ing a new instruction type in CPU instruction set architecture (ISA). within the range of (1,9), HeteroFuzz skips hardware simulation. Most kernel code is in order of tens of lines, as it maps directly to For ID 2, since the kernel input [1,1,1,253] now includes a value 253 hardware circuits. In fact, in a usual FPGA development workflow, that goes beyond the range of (1,9), HeteroFuzz invokes hardware developers instrument software on CPU, find out its hot code path simulation and finds that an overflow signal is captured on FPGA, corresponding to tens of lines of code, and extract it as a separate indicating unsafe accelerator execution. Though the max value of kernel for FPGA synthesis. Therefore, our work cannot be judged this input is 253, HeteroFuzz keeps the range of safe execution as under the same scalability standard used for pure software research (1,9). (e.g., handling GitHub projects with millions of lines of code). Sim- As Memorization/Range is updated for FPGA-safe executions, it ply put, the current landscape of heterogeneous platforms cannot is guaranteed to be narrower than or equal to the range of data handle FPGA synthesis of such large kernel size.
ESEC/FSE ’21, August 23–28, 2021, Athens, Greece Qian Zhang, Jiyuan Wang, and Miryung Kim
P1 P2 P3 P4 P5 P6 P7
1086420 0 50 100 150 0 50 100 150 0 50 100 150 0 50 100 150 0 50 100 150 0 50 100 150 0 50 100 150
Time (min) Time (min) Time (min) Time (min) Time (min) Time (min) Time (min)
HeteroFuzz WithoutSpectra WithoutMutation WithoutSelective NaiveFuzz
Figure 4: Number of Unique Divergence Symptoms
P1 P2 P3 P4 P5 P6 P7
100 80 60 40 200 0 50 100 150 0 50 100 150 0 50 100 150 0 50 100 150 0 50 100 150 0 50 100 150 0 50 100 150 Time (min) Time (min) Time (min) Time (min) Time (min) Time (min) Time (min) HeteroFuzz WithoutSpectra & WithoutMutation WithoutSelective NaiveFuzz Figure 5: % Cumulative Branch Coverage in Host Code To answer the research questions above, we create the following WithoutSpectra is divergence-inducing. This is because Without- four baseline versions by downgrading HeteroFuzz. Spectra takes the FPGA accelerator as a black box and enlarges the • WithoutSpectra: This option disables accelerator spectra covered branches in host code only. On the contrary, HeteroFuzz monitoring from HeteroFuzz to measure how effectively monitors branches and accelerator spectra in tandem, leading to HeteroFuzz can find more divergence errors by monitoring 7.78X more divergence-inducing inputs. accelerator-relevant feedback. We then group divergence-inducing inputs into a set of unique • WithoutMutation: This option disables dynamic probabilis- symptoms, because different inputs may exhibit the same kind of tic mutations from HeteroFuzz to measure how fast Hetero- a divergence error. In Figure 4, Y axis is the average cumulative Fuzz can find the same divergence errors by increasing the number of detected symptoms. Within the same time budget, With- probabilities of divergence-inducing mutations. outSpectra detects 10 unique symptoms in total, while HeteroFuzz • WithoutSelective: This option disables selective invocation detects 47, almost 3.7x more divergence symptoms. It is also impor- from HeteroFuzz to measure how fast HeteroFuzz can find tant to note that existing inputs shipped with the original benchmark the same divergence errors by obviating the need to invoke do not find any divergence errors; in other words, HeteroFuzz has the an HLS simulator that finds redundant errors. potential to detect real-world platform-dependent errors proactively. • NaiveFuzz: This option enables only branch coverage as guid- We also assess speed-up enabled by HeteroFuzz by measuring ance and invokes an HLS simulator for every input. We the time taken to find the same set of divergence symptoms found estimate its running time by multiplying the number of invo- by WithoutSpectra. Across seven applications, WithoutSpectra cations required for WithoutSpectra with the average HLS takes total 21 hours to find 10 unique symptoms, while HeteroFuzz simulation time. takes only 0.52 hours, demonstrating 40X speed-up. Experimental Environment. All experiments are done by lever- Table 5 lists five sample symptoms found in P3. We describe why aging Vivado Design Suite 2018.03 to simulate kernel execution on these divergent behavior appear between CPU and FPGA and how Xilinx Virtex UltraScale+ XCVU9P FPGA. HeteroFuzz finds them in detail below. 5.1 RQ1: Benefit of Accelerator Spectra First, as with most hardware designs, the kernel in P3 uses op- timized bitwidths for data offloaded from CPU to FPGA and its To evaluate HeteroFuzz’s guidance strategy that monitors accel- intermediate variables. When a large number 2147483600 is sent to erator spectra in addition to branch coverage, we generate inputs the kernel, it only keeps eight most significant bits or least signifi- for P1-P7 by running HeteroFuzz and WithoutSpectra for three cant bits in the binary representation of 2147483600 and cuts off hours. With the generated inputs, we execute the program on CPU the others, leading to a wrong result. When the inputs [1,1,1,253] versus a heterogeneous platform that runs host on CPU and kernel and [2,1,1,253] are executed on FPGA kernel, runtime overflow on the FPGA/HLS similuator. We then measure the percentage of happens with variable sum in line 25 of Figure 2c, leading to a divide- divergence-inducing inputs and the number of unique divergence by-zero error in host code and a wrong returned result respectively. symptoms. This experiment is done over ten independent runs and Although integer overflow can happen in CPU as well, it shows up the rightmost two columns in Table 4 report the results (HeteroFuzz much more frequently in accelerators, and it is not easy to predict in column HF). On average, 56.8% of inputs generated by Hetero- the consequent impact. HeteroFuzz monitors the value ranges of Fuzz is divergence-inducing, while 6.48% of inputs generated by
HeteroFuzz: Fuzz Testing to Detect Platform Dependent Divergence for Heterogeneous Applications ESEC/FSE ’21, August 23–28, 2021, Athens, Greece
Table 5: Example Divergence Symptoms for P3
Input 105 HeteroFuzz HFWithoutInvocation
ID Symptom Description Checking
S1 Unexpected Memory overflow happens ×
Endless when FPGA attempts to write
Loop data_fpga at an unexpected 104
address in line 21 of Figure 2c.
S2 Kernel Host is offloading a large ✓
Offloading number 2147483600 to FPGA P1 P2 P3 P4 P5 P6 P7
Error in line 21 of Figure 2c, leading Figure 6: Number of Input Trials.
to a wrong returned result.
S3 Kernel The value of intermediate × WithoutMutation, HeteroFuzz finds the same 18 divergence symp-
Runtime variables in line 25 of Figure 2c toms reported by WithoutMutation 17.5 times faster, taking only
Overflow exceeds its bitwidth capacity, 1.2 hours as opposed to 21 hours. For example, in P3, while Hetero-
leading to a wrong result.
S4 Divide by FPGA returned result leads to × Fuzz finds five divergence errors, WithoutMutation finds only one
Zero in Host divide-by-zero error divergence error by generating input [1,1,1,311] that leads to
in line 9 of Figure 2. an error, because the max value to be offloaded to FPGA is 255. The
S5 Incorrect CPU and FPGA produce ✓ other four errors are not found by WithoutMutation, because it
Loop different results when the input wastes most fuzzing time on generating type-invalid data for the
Unrolling array size is not multiple of 2. FPGA kernel, such as [1,1,1#] , when an integer array is expected.
inputs and intermediate variables to increase the chance of showing HeteroFuzz leverages probabilistic mutations to increase the chance
differential behavior caused by accelerator integer overflows. of hitting divergence behavior by isolating kernel-sensitive inputs
Second, when a program attempts to access an invalid or illegal and prioritizing divergence-inducing mutations such as modifying
address in CPU, the memory management unit can give excep- the value of a specific element in the array. The achieved branch
tion signals such as Segmentation Fault. However, in FPGA, all coverage of WithoutMutation is the same with the coverage of
memory accesses are mapped to a legal physical address on BRAM, WithoutSpectra, as shown in Figure 5, because no accelerator spec-
which can result in severe security problems and unexpected accel- tra monitoring implies that dynamic mutations cannot be enabled.
erator behavior. HeteroFuzz generates input [1,...,0] wherein the (However, when turning off dynamic mutations, HeteroFuzz can
401st element is 0. Line 21 in Figure 2c then writes this over-ranged still use accelerator spectra as a guidance feedback.)
data_fpga[400] to the address of a loop iterator i, leading to an Summary 2
endless loop execution of lines 19-21.
Third, to further complicate the difficulty of finding divergence HeteroFuzz achieves 17.5X speed-up in detecting the same
errors, because P3 makes two copies of the loop body during FPGA set of errors by dynamically adjusting the activation prob-
synthesis to enable parallelization, a wrong result happens only if ability of divergence-inducing mutations.
the size of an input array is not multiples of the unroll factor 2.
HeteroFuzz and WithoutSpectra achieve similar coverage, as 5.3 RQ3: Benefit of Selective Invocation
shown in Figure 5. The Y-axis indicates the percentage of covered
branches in host code. For all applications except P7, HeteroFuzz’s To assess speed-up enabled by selective invocation of a hardware
coverage grows slightly slower than WithoutSpectra. This is be- simulator, we compare HeteroFuzz with a downgraded version
cause HeteroFuzz increases the activation probabilities of mutations WithoutSelective. We measure the number of inputs the tools can
that lead to a new accelerator feedback, pushing the fuzz engine to explore within the same 24-hour budget.
explore more platform-dependent divergence rather than enlarging In Figure 6, the Y-axis is the average number of input trials over
branch coverage in host code. ten independent runs. In P2, WithoutSelective can invoke a simu-
Summary 1 lator with 5760 inputs only, while HeteroFuzz can attempt over 51k
inputs, achieving 8.8X speedup. HeteroFuzz achieves such speedup
HeteroFuzz finds 7.78X divergence-inducing inputs (3.7X by skipping the repetitive simulation calls when inputs hit the mem-
unique divergence symptoms) by monitoring accelerator orized value range of seen kernel inputs. This selective invocation
spectra in addition to the branch coverage of host code. saves time but does not sacrifice fault detection capability, because
kernel inputs contained within already seen ranges can lead to
5.2 RQ2: Benefit of Probabilistic Mutation either correct FPGA executions or already discovered divergence
To evaluate the benefit of dynamic probabilistic mutations, we error symptoms. Collecting kernel input values does not incur ad-
create a downgraded version WithoutMutation that disables prob- ditional hardware level instrumentation, as such information can
abilistic mutations from HeteroFuzz. We assess how fast Hetero- be extracted from HLS simulation results.
Fuzz and WithoutMutation detect divergence symptoms within the Summary 3
three-hour limit. We repeat the experiments ten times and report By reducing unnecessary hardware simulation calls, Het-
average results in Figure 4. eroFuzz speeds up differential testing by 8.8X without sac-
In total, HeteroFuzz detects 47 unique symptoms, while With- rificing fault detection capability.
outMutation reports 18, given the same time limit. Compared to
Input Trials
ESEC/FSE ’21, August 23–28, 2021, Athens, Greece Qian Zhang, Jiyuan Wang, and Miryung Kim
5.4 RQ4: Comparison Against Naive Fuzzing accelerator is trying to process data with an invalid value or size. Fuzz testing is often built on an implicit assumption that the pro- However, such input checking is still inadequate and does not gram under test can be executed millions of times within a matter of prevent the platform-dependent error S1 and kernel runtime errors hours. However, such assumption does not hold for heterogeneous S3 and S4. For example, S4 is caused by inputs that satisfy the applications due to the long latency of hardware simulation. We guard condition, [1,1,1,253] . In other words, assertions inserted estimate the time required for NaiveFuzz by multiplying the number by a defensive developer in the host code may not fully prevent of iterations required to find the same kind of divergence symp- runtime errors coming from hardware accelerators due to varying tom using WithoutSpectra (—where one iteration means running resource availability of individual FPGA devices. To our knowledge, a program on a new input) with an average hardware simulation HeteroFuzz is the only testing tool that can detect such platform- time for each program. Figure 4 shows comparison between the dependent errors missed by input checking in host code. running time of HeteroFuzz and the estimated time of NaiveFuzz. Summary 5 Please note that though we report an estimated time not an actual running time, because HW simulation time does not vary much for Our case study shows evidence that even if a developer each input (the standard deviation 𝜎 = 1.62), this estimated should manually constructs and inserts assertions in host code, be highly similar to the actual time of using an AFL-like technique kernel errors from accelerators cannot be fully prevented. that directly invokes an HLS simulator with a new input. Within 6 THREATS TO VALIDITY three hours, NaiveFuzz finds only one divergence symptom for P4, P5 and P7, but could not find any in other programs. For P2, Naive- We discuss the threats to validity as follows. Fuzz requires at least 2262 hours to detect all symptoms detected Device Dependence. We simulate all the kernel executions on a by HeteroFuzz within three hours, leading to 754X speed-up. In single Xilinx Virtex UltraScale+ XCVU9P FPGA, which is currently addition to the results in Figure 4, we ran HeteroFuzz for 24 hours the widely used FPGA. This setup may restrict the generalizability and compare against what NaiveFuzz would find within the same of our results to other devices, because the detected divergence time budget. HeteroFuzz detects 61.8X more error-inducing inputs symptoms could vary for different platforms, e.g. Intel’s Altera. with the same budget of 24 hours, compared to NaiveFuzz. Though the absolute numbers of execution time and symptoms are Summary 4 dependent on a detailed configuration, we believe that HeteroFuzz Using an AFL-like technique to repetitively invoke a hard- would retain the overall benefits of speedup and divergence-finding ware simulator would be too slow and insufficient to reveal capability when it is applied to different platforms. Since HeteroFuzz platform-dependent errors in heterogeneous applications. uses FPGA simulation, it does not find mechanical failures caused by temperatures, aging of devices, and radiation on FPGA. Such hardware in-field testing is often done by device physicists. 5.5 RQ5: Comparison Against Input Checking Time Limit. We empirically set three hours as the time limit for One may question whether input validity checking on the side fuzzing. Longer execution time may expose more divergence errors of the host code is feasible and adequate for preventing platform- or more execution paths as suggested in [27]; however, this time dependent errors in kernel code to be executed on a hardware limit is reasonable, as we did not see any increase in new types of accelerator. Unlike pure software systems where a caller function divergence errors with a higher time limit for subjects P1-P7. can prevent errors by checking the pre-condition of its callee prior Input Mutations. In terms of mutations, HeteroFuzz refers to in- to invocation, such input validity checking is not always feasible in put modifications to explore the input space during fuzz testing, as heterogeneous applications [30, 40]. The reason is that it is nearly opposed to injecting code faults in mutation testing. HeteroFuzz impossible to identify the precise pre-condition in advance due to the not only is faster than naive fuzzing but is safe—i.e., HeteroFuzz difficulties of modeling individual FPGA devices [12, 28, 70], because can find all errors that can be found by naive fuzzing, as Hetero- the pre-condition is dependent on the resource availability on a Fuzz’s input mutations are a super-set of low-level input mutations. specific platform. For example, when a merge-sort kernel requires Designing new kinds of input mutations could affect the efficiency a 5MB array for dynamic block memory, Xilinx-Zynq-7030 with of fuzz testing. Currently, there are no equivalent high-level input 9.3MB BRAM will work fine, but Xilinx-Zynq-7020 with 4.9MB mutations in HeteroFuzz. Low-level bit or byte mutations retained BRAM will produce an incorrect sorting result silently. by HeteroFuzz could subsume other high-level input mutations, be- To further substantiate this argument, we conducted a case study cause combinations of low-level mutations could map to high-level on several divergent symptoms found by HeteroFuzz in the example mutations. application shown in Figure 2. Table 5 summarizes a divergence 7 RELATED WORK symptom, a detailed error description, and whether input range checking could have prevented such error. After analyzing the Fuzz Testing. Fuzzing has gained popularity in both academia and HLS pragmas in line 4, line 6, and line 23, we manually extract industry due to its black/grey box approach with a low barrier to the pre-condition of kernel code in terms of a range check and entry [43]. The key idea of fuzz testing originates from random insert it an input guard: an integer array whose size is multiple of test generation where inputs are incrementally produced with the two but no larger than 400, and each element in this array should hope to exercise previously undiscovered behavior [18, 20, 44]. For be less than 256. After inserting this guard into the original host example, AFL mutates a seed input to discover previously unseen code, divergence symptoms S2 and S5 are prevented because the coverage profiles [69]. To carefully explore a vast space of inputs
HeteroFuzz: Fuzz Testing to Detect Platform Dependent Divergence for Heterogeneous Applications ESEC/FSE ’21, August 23–28, 2021, Athens, Greece
and unbounded program paths, Lemieux et al. create custom muta- as Verilog, VHDL, etc., using formal verification [26, 48] and run- tions so that the generated inputs gravitate toward exercising rare time verification [29]. For example, RFUZZ [29] is a circuit runtime branches [33]. Other approaches incorporate symbolic execution verification tool for FIRRTL IR (UC Berkeley’s own version of RTL with fuzzing to guide selection and mutation of the inputs to invoke language). RFUZZ invents a new notion of MUX toggle cooverage for unique program paths [7, 51]. Padhye et al. incorporate the seman- circuit testing at gate level and employs a rapid memory resetting tic validity of input mutations in Zest [45] to reduce the search on FPGA for RTL circuit verification. As another example, Qin and space of inputs by mapping low-level, bit-level mutations to valid Mishra [48] present a scalable test generation technique [48] for structural changes in the high-level input representation. All these hardware kernels in Verilog by interleaving concrete and symbolic fuzzing techniques are built on the assumption that the program execution to bridge the gap between model checking and testing. under test can be executed millions of times within a matter of As opposed to these techniques that find crashes on kernels only, hours. However, in the domain of heterogeneous applications, a HeteroFuzz targets end-to-end application code testing and reveals single invocation of a hardware simulator may take several minutes, differential behavior of the entire heterogeneous application (i.e., which is the exact problem that HeteroFuzz addresses. host and accelerator together) under different platforms. In other Instead of using coverage as guidance, several techniques have in- words, it is not feasible to directly compare HeteroFuzz against vestigated how to use custom guidance mechanisms. PerfFuzz [32] these circuit testing techniques [29, 48], because they do not have uses the execution counts of exercised instructions together with capability to test host code together with kernel code, and their branch coverage as fuzzing guidance to explore pathological perfor- input languages are Verilog variants, not C variants. mance behavior. UAFL [54] incorporates typestate properties and Revealing Precision Errors. FPGen [21] uses symbolic execution information flow analysis to detect the use-after-free vulnerabilities. to generate inputs to trigger large numerical floating-point errors. MemLock [56] employs both coverage and memory consumption It defines inaccurate precision loss checks and injects these checks metrics to guide the fuzzing process. AFLgo [2] extends AFL to at strategic program locations to construct specialized branches to direct fuzzing towards user-specified target sites. However, none induce floating-point errors. Different from FPGen that focuses on explicitly monitors hardware-level accelerator spectra and metrics floating-point overflows and errors, HeteroFuzz has such a broad to reveal platform-dependent divergence like HeteroFuzz. scope in generating inputs that lead to variable overflow on kernels, Another angle to optimize fuzz testing is to update which muta- kernel exceptions, incorrect returned result, etc. tion operations to apply. SymFuzz [8] uses symbolic execution to de- Mediating Host-Accelerator Interactions. Interaction and com- termine the number of bits to be mutated in a seed input. Angora [9] munication between accelerators and the host can pose severe secu- updates mutation operations to be aware of taint-level observations. rity problems.Crossing Guard [40] is a coherence interface between SDF [35] uses seed properties to guide mutation in web-browser the host and accelerators. It prevents potential bugs caused by host- fuzz testing. In grammar-based fuzzing, Saffron [31] repairs the accelerator communication. Border Control [41] is a sandboxing given grammar based on whether the program accepts unexpected mechanism that guarantees that the memory access permissions inputs outside of the provided grammar, and then it adaptively re- are respected by accelerators, regardless of design errors or ma- fines the probabilities of every production rule. MOPT [38] finds an licious intent. While the above work focuses on preventing bugs optimal probability distribution for mutation operators to discover caused by host-accelerator interactions, HeteroFuzz on the other vulnerabilities more efficiently. To our knowledge, none designs hand is a test generation tool to detect bugs when running host probabilistic mutations by associating monitored accelerator spec- and hardware accelerator together. tra with the probability of activating specific mutation operators. 8 Testing in HLS and Hardware Accelerators. HLS tools automat- CONCLUSION ically generate RTL descriptions from C/C++ programs. Yann et As hardware specialization, energy efficiency, and flexible re-program- al. [25] test HLS by randomly generating programs and verifying mability are becoming increasingly important, a new type of cloud- the equivalence between the synthesized design and the original based hardware accelerator microservices based on FPGA has emerged. code. Christopher et al. [34] investigate many-core compiler fuzzing Major service providers such as Amazon F1 and Microsoft Azure in the context of heterogeneous computing with OpenCL kernels. have begun to support heterogeneous application development to They report more than 50 OpenCL compiler bugs. Silver [37] pro- enable acceleration with customizable hardware. poses a single end-to-end correctness theorem about running a HeteroFuzz makes three key contributions in automated testing verified compiler on a verified FPGA platform. It generates ma- of heterogeneous applications by incorporating multi-dimensional chine code for Silver based on a high-level executable specification, guidance, dynamic probabilistic mutations, and selective invocation. and the synthesized FPGA hardware will have the observable be- In total, the speed-up achieved by HeteroFuzz’s three-pronged havior of the original high-level specification. Different from HLS approach in finding the same set of errors is up to 754X, compared compiler testing that finds bugs in compilers, HeteroFuzz detects to using an AFL-like technique naively. HeteroFuzz is the first end- platform-dependent behavior in heterogeneous applications. end technique that significantly improves testing effectiveness and HeteroFuzz focuses on testing software applications with host efficiency for this new breed of heterogeneous applications. code and kernel code combined together. In other words, Hetero- 9 Fuzz’s problem concerns C-like code testing, where a sub region of ACKNOWLEDGEMENT code could be offloaded to FPGA accelerators. On the other hand, the The participants of this research are in part supported by NSF grants hardware design community targets circuit verification in the form CHS-1956322 CCF-1764077, CCF-1723773, ONR grant N00014-18-1- of bitstream and/or hardware description languages (HDL) such 2037, and Intel CAPA grant.
ESEC/FSE ’21, August 23–28, 2021, Athens, Greece Qian Zhang, Jiyuan Wang, and Miryung Kim
REFERENCES [23] Prabhat Gupta. 2021. Xeon+FPGA Platform for the Data Center. https://www.
[1] Amazon.com. 2021. Amazon EC2 F1 Instances: Run Custom FPGAs in the AWS archive.ece.cmu.edu/~calcm/carl/lib/\exe/fetch.php?media=carl15-gupta.pdf.
Cloud. https://aws.amazon.com/ec2/instance-types/f1. [24] Yuko Hara, Hiroyuki Tomiyama, Shinya Honda, Hiroaki Takada, and Katsuya
[2] Marcel Böhme, Van-Thuan Pham, Manh-Dung Nguyen, and Abhik Roychoud- Ishii. 2008. Chstone: A benchmark program suite for practical c-based high-level
hury. 2017. Directed Greybox Fuzzing. In Proceedings of the 2017 ACM SIGSAC synthesis. In 2008 IEEE International Symposium on Circuits and Systems. IEEE,
Conference on Computer and Communications Security, David Evans, Tal Maklin, 1192–1195.
and Dongyan Xu (Eds.). Association for Computing Machinery (ACM), United [25] Yann Herklotz and John Wickerson. 2020. Finding and Understanding Bugs
States of America, 2329–2344. https://doi.org/10.1145/3133956.3134020 ACM in FPGA Synthesis Tools. In The 2020 ACM/SIGDA International Symposium on
Conference on Computer and Communications Security 2017
, CCS 2017 ; Field-Programmable Gate Arrays (Seaside, CA, USA) (FPGA ’20). Association for
Conference date: 30-10-2017 Through 03-11-2017. Computing Machinery, New York, NY, USA, 277–287. https://doi.org/10.1145/
[3] Gary Bradski and Adrian Kaehler. 2008. Learning OpenCV: Computer vision with 3373087.3375310
the OpenCV library. O’Reilly Media, Inc. [26] Roope Kaivola, Rajnish Ghughal, Naren Narasimhan, Amber Telfer, Jesse Whitte-
[4] Mary Branscombe. 2017. FPGAs and the New Era of Cloud-based Hardware more, Sudhindra Pandav, Anna Slobodová, Christopher Taylor, Vladimir Frolov,
Microservices. https://thenewstack.io/developers-fpgas-cloud/. Erik Reeber, and Armaghan Naik. 2009. Replacing Testing with Formal Verifica-
[5] Jared Casper and Kunle Olukotun. 2014. Hardware acceleration of database tion in Intel® CoreTM I7 Processor Execution Engine Validation. In Proceedings
operations. In Proceedings of the 2014 ACM/SIGDA international symposium on of the 21st International Conference on Computer Aided Verification (Grenoble,
Field-programmable gate arrays. ACM, 151–160. France) (CAV ’09). Springer-Verlag, Berlin, Heidelberg, 414–429.
[6] Adrian M Caulfield, Eric S Chung, Andrew Putnam, Hari Angepat, Jeremy Fowers, [27] George Klees, Andrew Ruef, Benji Cooper, Shiyi Wei, and Michael Hicks. 2018.
Michael Haselman, Stephen Heil, Matt Humphrey, Puneet Kaur, Joo-Young Kim, Evaluating Fuzz Testing. In Proceedings of the 2018 ACM SIGSAC Conference on
et al. 2016. A cloud-scale acceleration architecture. In The 49th Annual IEEE/ACM Computer and Communications Security (Toronto, Canada) (CCS ’18). Association
International Symposium on Microarchitecture. IEEE Press, 7. for Computing Machinery, New York, NY, USA, 2123–2138. https://doi.org/10.
[7] Sang Kil Cha, Maverick Woo, and David Brumley. 2015. Program-Adaptive 1145/3243734.3243804
Mutational Fuzzing. In Proceedings of the 2015 IEEE Symposium on Security and [28] D. Koeplinger, R. Prabhakar, Y. Zhang, C. Delimitrou, C. Kozyrakis, and K. Oluko-
Privacy (SP ’15). IEEE Computer Society, USA, 725–741. https://doi.org/10.1109/ tun. 2016. Automatic Generation of Efficient Accelerators for Reconfigurable
SP.2015.50 Hardware. In 2016 ACM/IEEE 43rd Annual International Symposium on Computer
[8] S. K. Cha, M. Woo, and D. Brumley. 2015. Program-Adaptive Mutational Fuzzing. Architecture (ISCA). 115–127. https://doi.org/10.1109/ISCA.2016.20
In 2015 IEEE Symposium on Security and Privacy. 725–741. [29] Kevin Laeufer, Jack Koenig, Donggyu Kim, Jonathan Bachrach, and Koushik Sen.
[9] Peng Chen and Hao Chen. 2018. Angora: Efficient fuzzing by principled search. 2018. RFUZZ: Coverage-Directed Fuzz Testing of RTL on FPGAs. In Proceedings
In 2018 IEEE Symposium on Security and Privacy (SP). IEEE, 711–725. of the International Conference on Computer-Aided Design (San Diego, California)
[10] Zhe Chen, Hugh T Blair, and Jason Cong. 2019. LANMC: LSTM-Assisted Non- (ICCAD ’18). Association for Computing Machinery, New York, NY, USA, Article
Rigid Motion Correction on FPGA for Calcium Image Stabilization. In Proceedings 28, 8 pages. https://doi.org/10.1145/3240765.3240842
of the 2019 ACM/SIGDA International Symposium on Field-Programmable Gate [30] Jason Lau, Aishwarya Sivaraman, Qian Zhang, Muhammad Ali Gulzar, Jason
Arrays. 104–109. Cong, and Miryung Kim. 2020. HeteroRefactor: Refactoring for Heteroge-
[11] Andrew A Chien, Allan Snavely, and Mark Gahagan. 2011. 10x10: A general- neous Computing with FPGA. In Proceedings of the ACM/IEEE 42nd Interna-
purpose architectural approach to heterogeneity and energy efficiency. Procedia tional Conference on Software Engineering (Seoul, South Korea) (ICSE ’20). As-
Computer Science 4 (2011), 1987–1996. sociation for Computing Machinery, New York, NY, USA, 493–505. https:
[12] Young-Kyu Choi, Jason Cong, Zhenman Fang, Yuchen Hao, Glenn Reinman, and //doi.org/10.1145/3377811.3380340
Peng Wei. 2019. In-Depth Analysis on Microarchitectures of Modern Hetero- [31] Xuan-Bach D. Le, Corina S. Pasareanu, Rohan Padhye, David Lo, Willem Visser,
geneous CPU-FPGA Platforms. ACM Trans. Reconfigurable Technol. Syst. 12, 1, and Koushik Sen. 2019. Saffron: Adaptive Grammar-based Fuzzing for Worst-
Article 4 (Feb. 2019), 20 pages. https://doi.org/10.1145/3294054 Case Analysis. ACM SIGSOFT Software Engineering Notes 44, 4 (2019), 14. https:
[13] Eric S Chung, Peter A Milder, James C Hoe, and Ken Mai. 2010. Single-chip hetero- //doi.org/10.1145/3364452.3364455
geneous computing: Does the future include custom logic, FPGAs, and GPGPUs?. [32] Caroline Lemieux, Rohan Padhye, Koushik Sen, and Dawn Song. 2018. Perffuzz:
In 2010 43rd annual IEEE/ACM international symposium on microarchitecture. Automatically generating pathological inputs. In Proceedings of the 27th ACM
IEEE, 225–236. SIGSOFT International Symposium on Software Testing and Analysis. ACM, 254–
[14] Jason Cong, Mohammad Ali Ghodrat, Michael Gill, Beayna Grigorian, Karthik 265.
Gururaj, and Glenn Reinman. 2014. Accelerator-rich architectures: Opportunities [33] Caroline Lemieux and Koushik Sen. 2018. FairFuzz: a targeted mutation strategy
and progresses. In Proceedings of the 51st Annual Design Automation Conference. for increasing greybox fuzz testing coverage. In Proceedings of the 33rd ACM/IEEE
ACM, 1–6. International Conference on Automated Software Engineering, ASE 2018, Montpellier,
[15] Jason Cong, Licheng Guo, Po-Tsang Huang, Peng Wei, and Tianhe Yu. 2018. France, September 3-7, 2018, Marianne Huchard, Christian Kästner, and Gordon
SMEM++: A Pipelined and Time-Multiplexed SMEM Seeding Accelerator for Fraser (Eds.). ACM, 475–485. https://doi.org/10.1145/3238147.3238176
Genome Sequencing. In 2018 28th International Conference on Field Programmable [34] Christopher Lidbury, Andrei Lascu, Nathan Chong, and Alastair F. Donaldson.
Logic and Applications (FPL). 210–2104. 2015. Many-Core Compiler Fuzzing. In Proceedings of the 36th ACM SIGPLAN
[16] Jason Cong, Bin Liu, Stephen Neuendorffer, Juanjo Noguera, Kees Vissers, and Conference on Programming Language Design and Implementation (Portland, OR,
Zhiru Zhang. 2011. High-level synthesis for FPGAs: From prototyping to de- USA) (PLDI ’15). Association for Computing Machinery, New York, NY, USA,
ployment. IEEE Transactions on Computer-Aided Design of Integrated Circuits and 65–76. https://doi.org/10.1145/2737924.2737986
Systems 30, 4 (2011), 473–491. [35] Ying-Dar Lin, Feng-Ze Liao, Shih-Kun Huang, and Yuan-Cheng Lai. 2015. Browser
[17] Jason Cong, Vivek Sarkar, Glenn Reinman, and Alex Bui. 2010. Customizable fuzzing by scheduled mutation and generation of document object models. In
domain-specific computing. IEEE Design & Test of Computers 28, 2 (2010), 6–15. 2015 International Carnahan Conference on Security Technology (ICCST). IEEE,
[18] Christoph Csallner and Yannis Smaragdakis. 2004. JCrasher: an automatic robust- 1–6.
ness tester for Java. Software: Practice and Experience 34, 11 (2004), 1025–1050. [36] LLVM. 2021. https://llvm.org/.
[19] Jeferson Santiago da Silva, François-Raymond Boyer, and JM Langlois. 2019. [37] Andreas Lööw, Ramana Kumar, Yong Kiam Tan, Magnus O. Myreen, Michael
Module-per-Object: a Human-Driven Methodology for C++-based High-Level Norrish, Oskar Abrahamsson, and Anthony Fox. 2019. Verified Compilation
Synthesis Design. arXiv preprint arXiv:1903.06693 (2019). on a Verified Processor. In Proceedings of the 40th ACM SIGPLAN Conference on
[20] Gordon Fraser and Andrea Arcuri. 2011. EvoSuite: Automatic Test Suite Gen- Programming Language Design and Implementation (Phoenix, AZ, USA) (PLDI
eration for Object-Oriented Software. In Proceedings of the 19th ACM SIGSOFT 2019). Association for Computing Machinery, New York, NY, USA, 1041–1053.
Symposium and the 13th European Conference on Foundations of Software Engi- https://doi.org/10.1145/3314221.3314622
neering (Szeged, Hungary) (ESEC/FSE ’11). Association for Computing Machinery, [38] Chenyang Lyu, Shouling Ji, Chao Zhang, Yuwei Li, Wei-Han Lee, Yu Song, and
New York, NY, USA, 416–419. https://doi.org/10.1145/2025113.2025179 Raheem Beyah. 2019. MOPT: Optimized Mutation Scheduling for Fuzzers. In 28th
[21] Hui Guo and Cindy Rubio-González. 2020. Efficient Generation of Error-Inducing USENIX Security Symposium (USENIX Security 19). USENIX Association, Santa
Floating-Point Inputs via Symbolic Execution. In Proceedings of the ACM/IEEE Clara, CA, 1949–1966. https://www.usenix.org/conference/usenixsecurity19/
42nd International Conference on Software Engineering (Seoul, South Korea) (ICSE presentation/lyu
’20). Association for Computing Machinery, New York, NY, USA, 1261–1272. [39] Valentin Manes, HyungSeok Han, Choongwoo Han, sang cha, Manuel Egele,
https://doi.org/10.1145/3377811.3380359 Edward Schwartz, and Maverick Woo. 2019. The Art, Science, and Engineering
[22] Licheng Guo, Jason Lau, Zhenyuan Ruan, Peng Wei, and Jason Cong. 2019. Hard- of Fuzzing: A Survey. IEEE Transactions on Software Engineering PP (10 2019),
ware acceleration of long read pairwise overlapping in genome sequencing: A 1–1. https://doi.org/10.1109/TSE.2019.2946563
race between fpga and gpu. In 2019 IEEE 27th Annual International Symposium [40] Lena E. Olson, Mark D. Hill, and David A. Wood. 2017. Crossing Guard: Mediating
on Field-Programmable Custom Computing Machines (FCCM). IEEE, 127–135. Host-Accelerator Coherence Interactions. SIGARCH Comput. Archit. News 45, 1
(April 2017), 163–176. https://doi.org/10.1145/3093337.3037715
HeteroFuzz: Fuzz Testing to Detect Platform Dependent Divergence for Heterogeneous Applications ESEC/FSE ’21, August 23–28, 2021, Athens, Greece
[41] Lena E. Olson, Jason Power, Mark D. Hill, and David A. Wood. 2015. Border Con- [53] Frank Tip. 1994. A survey of program slicing techniques. (1994). trol: Sandboxing Accelerators. In Proceedings of the 48th International Symposium [54] Haijun Wang, Xiaofei Xie, Yi Li, Cheng Wen, Yuekang Li, Yang Liu, Shengchao on Microarchitecture (Waikiki, Hawaii) (MICRO-48). Association for Comput- Qin, Hongxu Chen, and Yulei Sui. 2020. Typestate-Guided Fuzzer for Discovering ing Machinery, New York, NY, USA, 470–481. https://doi.org/10.1145/2830772. Use-after-Free Vulnerabilities. In Proceedings of the 42nd International Conference 2830819 on Software Engineering (ICSE). [42] OpenCV. 2021. https://opencv.org/. [55] Mark Weiser. 1984. Program slicing. IEEE Transactions on software engineering 4 [43] Alessandro Orso and Gregg Rothermel. 2014. Software Testing: A Research (1984), 352–357. Travelogue (2000–2014). In Proceedings of the on Future of Software Engineering [56] Cheng Wen, Haijun Wang, Yuekang Li, Shengchao Qin, Yichang Liu, Zhiwu Xu, (Hyderabad, India) (FOSE 2014). Association for Computing Machinery, New Hongxu Chen, Xiaofei Xie, Geguang Pu, and Ting Liu. 2020. MemLock: Memory York, NY, USA, 117–132. https://doi.org/10.1145/2593882.2593885 Usage Guided Fuzzing. In ICSE 2020. [44] C. Pacheco, S. K. Lahiri, M. D. Ernst, and T. Ball. 2007. Feedback-Directed [57] Xilinx. 2021. https://forums.xilinx.com/t5/High-Level-Synthesis-HLS/Dynamic- Random Test Generation. In 29th International Conference on Software Engineering memory-allocation-in-Vivado-HLS-and-segmentation-faults/td-p/894069. (ICSE’07). 75–84. [58] Xilinx. 2021. https://forums.xilinx.com/t5/High-Level-Synthesis-HLS/Problems- [45] Rohan Padhye, Caroline Lemieux, Koushik Sen, Mike Papadakis, and Yves with-simple-program-and-dataflow-directive/m-p/595225. Le Traon. 2019. Semantic Fuzzing with Zest. In Proceedings of the 28th ACM SIG- [59] Xilinx. 2021. https://forums.xilinx.com/t5/High-Level-Synthesis-HLS/HLS- SOFT International Symposium on Software Testing and Analysis (Beijing, China) design-problem-The-result-of-CSim-and-C-RTL-cosimulation-is/m-p/438446. (ISSTA 2019). Association for Computing Machinery, New York, NY, USA, 329–340. [60] Xilinx. 2021. https://forums.xilinx.com/t5/High-Level-Synthesis-HLS/Fixed- https://doi.org/10.1145/3293882.3330576 point-arithmetic/m-p/754676. [46] Andrew Putnam, Adrian M Caulfield, Eric S Chung, Derek Chiou, Kypros Con- [61] Xilinx. 2021. https://forums.xilinx.com/t5/High-Level-Synthesis-HLS/HLS- stantinides, John Demme, Hadi Esmaeilzadeh, Jeremy Fowers, Gopi Prashanth Synthesis-Difference-in-C-Simulation-and-C-RTL-Cosimulation/m-p/785019. Gopal, Jan Gray, et al. 2014. A reconfigurable fabric for accelerating large-scale [62] Xilinx. 2021. https://forums.xilinx.com/t5/High-Level-Synthesis-HLS/Issue- datacenter services. ACM SIGARCH Computer Architecture News 42, 3 (2014), about-overwrite-in-a-loop-HLS-coding-style/m-p/907213. 13–24. [63] Xilinx. 2021. https://forums.xilinx.com/t5/High-Level-Synthesis-HLS/HLS-error- [47] Weikang Qiao, Jieqiong Du, Zhenman Fang, Michael Lo, Mau-Chung Frank with-co-sim/m-p/1166264. Chang, and Jason Cong. 2018. High-throughput lossless compression on tightly [64] Xilinx. 2021. https://forums.xilinx.com/t5/High-Level-Synthesis-HLS/Simple- coupled CPU-FPGA platforms. In 2018 IEEE 26th Annual International Symposium top-function-skipping-inner-loop-logic-entirely/m-p/1126600. on Field-Programmable Custom Computing Machines (FCCM). IEEE, 37–44. [65] Xilinx. 2021. https://forums.xilinx.com/t5/High-Level-Synthesis-HLS/bd-p/hls. [48] Xiaoke Qin and Prabhat Mishra. 2014. Scalable Test Generation by Interleaving [66] Xilinx. 2021. Vivado High-Level Synthesis. https://www.xilinx.com/products/ Concrete and Symbolic Execution. In Proceedings of the 2014 27th International design-tools/vivado/integration/esl-design.html. Conference on VLSI Design and 2014 13th International Conference on Embedded [67] Zeping Xue and David B Thomas. 2015. SysAlloc: A hardware manager for Systems (VLSID ’14). IEEE Computer Society, USA, 104–109. https://doi.org/10. dynamic memory allocation in heterogeneous systems. In 2015 25th International 1109/VLSID.2014.25 Conference on Field Programmable Logic and Applications (FPL). IEEE, 1–7. [49] Brandon Reagen, Robert Adolf, Yakun Sophia Shao, Gu-Yeon Wei, and David [68] Zeping Xue and David B Thomas. 2016. SynADT: Dynamic Data Structures Brooks. 2014. Machsuite: Benchmarks for accelerator design and customized in High Level Synthesis. In 2016 IEEE 24th Annual International Symposium on architectures. In 2014 IEEE International Symposium on Workload Characterization Field-Programmable Custom Computing Machines (FCCM). IEEE, 64–71. (IISWC). IEEE, 110–119. [69] Michał Zalewski. 2021. American Fuzz Loop. http://lcamtuf.coredump.cx/afl/. [50] Kyle Rupnow, Yun Liang, Yinan Li, and Deming Chen. 2011. A study of high-level [70] Jieru Zhao, Liang Feng, Sharad Sinha, Wei Zhang, Yun Liang, and Bingsheng synthesis: Promises and challenges. In 2011 9th IEEE International Conference on He. 2017. COMBA: A Comprehensive Model-Based Analysis Framework for ASIC. IEEE, 1102–1105. High Level Synthesis of Real Applications. In Proceedings of the 36th International [51] Nick Stephens, John Grosen, Christopher Salls, Andrew Dutcher, Ruoyu Wang, Conference on Computer-Aided Design (Irvine, California) (ICCAD ’17). IEEE Press, Jacopo Corbetta, Yan Shoshitaishvili, Christopher Kruegel, and Giovanni Vigna. 430–437. 2016. Driller: Augmenting Fuzzing Through Selective Symbolic Execution. https: [71] Yuan Zhou, Udit Gupta, Steve Dai, Ritchie Zhao, Nitish Srivastava, Hanchen Jin, //doi.org/10.14722/ndss.2016.23368 Joseph Featherston, Yi-Hsiang Lai, Gai Liu, Gustavo Angarita Velasquez, Wenping [52] David B Thomas. 2016. Synthesisable recursion for C++ HLS tools. In 2016 IEEE Wang, and Zhiru Zhang. 2018. Rosetta: A Realistic High-Level Synthesis Bench- 27th International Conference on Application-specific Systems, Architectures and mark Suite for Software-Programmable FPGAs. Int’l Symp. on Field-Programmable Processors (ASAP). IEEE, 91–98. Gate Arrays (FPGA) (Feb 2018).