2024 Design, Automation & Test in Europe Conference (DATE 2024)



LLM-based Processor Verification: A Case Study for
Neuromorphic Processor

Chao Xiao1, Yifei Deng2, Zhijie Yang1,Renzhi Chen3, Hong Wang2, Jingyue Zhao3, Huadong Dai3, Lei Wang3∗, Yuhua Tang1 and Weixia Xu1 1 National University of Defense Technology, Changsha, China 2 University of Electronic Science and Technology of China, Chengdu, China 3 Defense Innovation Institute, AMS, Beijing, China

Abstract—With the increasing complexity of the hardware [2], coverage-directed test generation (CDG) [3], and coverage- design, conducting verification before the tapeout is of utmost directed test selection (CDS) [4], are proposed. Their success importance. Simulation-based verification remains the primary in achieving higher coverage is largely thanks to the design- method owing to its scalability and flexibility. A comprehensive verification of modern processors usually requires numerous specific optimizations. However, the next decade will witness effective tests to cover all possible conditions and use cases, the new golden age for computer architecture [5], and emerging leading to significant time, resource, and manual effort even with domain specific architectures (DSAs), such as neuromorphic the EDA. Moreover, novel domain specific architecture (DSA), processors [6], [7], pose challenges due to the lack of readily such as neuromorphic processors, will exacerbate the challenge of available off-the-shelf tests and random testing EDA tools verification. Fortunately, emerging large language models (LLMs) have been demonstrating a powerful ability to complete specific like RAVEN [8] for Arm processor and RISCV-DV [9] for tasks assigned by human instructions. In this paper, we explore the RISC-V processor. Besides, to achieve high coverage, existing challenges and opportunities encountered when using the LLMs simulation-based methods require the expertise and involve- to accelerate the DSA verification using the proposed LLM-based ment of experienced verification engineers which means much workflow consisting of test generation, compilation&simulation, manual effort. and result collection&processing. By verifying a RISC-V core and a neuromorphic processor, we examine the capabilities and limi- To generate tests efficiently and reduce manual efforts, a tations of the LLMs when using them for the function verification promising new approach is the application of large language of traditional processors and emerging DSA. In the experiment, models (LLMs) to automatically generate tests. LLMs are 36 C programs and 128 assembly snippets for the RISC-V core trained on a large amount of text data and can complete a and the neuromorphic processor are generated using an advanced LLM to demonstrate our claim. The experimental results show wide range of tasks such as text summarization and language that the code coverage based on the LLM test generation can translation. The well-known LLMs include GPT-2/3/4 from reach 89% and 91% for the above two architectures respectively, OpenAI [10] and have been already applied for software code showing a promising research direction for the future processor generation [11], [12] and hardware design [13]–[16]. verification in the new golden age for computer architecture. Index Terms—processor function verification, large language DAVE is the first work to derive Verilog snippets from natural model (LLM), test generation, neuromorphic processor language via fine-tuned GPT-2 [13]. Both Chip-Chat [14] and ChipGPT [15] utilized the state-of-the-art ChatGPT to help I. INTRODUCTION designers generate hardware logic designs. Thakur et al. [16] examined fine-tuning five different pre-trained LLMs models Due to the ongoing scaling of the semiconductor manufactur- and created a collection of Verilog coding problems to check ing process, the complexities of microprocessors have increased the functional correctness of generated code. To the best of our significantly, leading to continuously increasing verification knowledge, the use of LLMs to generate tests for processor complexity. In 2022, as indicated in the Wilson Research Group verification has not been explored yet. Functional Verification Study [1], a substantial 40%∼50% of Therefore, in this paper, we aim at the following question: the entire hardware design cycle is dedicated to verification What are the potential advantages and obstacles to integrating efforts. the LLMs into traditional processor and DSA verification Simulation-based methods co-simulate the design under test processes? To explore this question, we perform the LLM- (DUT) with a reference model and check the deviation of based verification for Xuantie-C910 [17], a traditional RISC-V their outputs given the same input stimuli. In the last few processor, and a neuromorphic processor based on the custom decades, in order to perform thorough coverage tests for com- neuromorphic instruction set extension using GPT-3.5. Besides, prehensive verification, simulation-based approaches including we construct an LLM-based verification workflow for DSA random test generation, constrained random generation (CRG) which consists of test generation, compilation&simulation, and result collection&processing, where the only part of human ef- This work was supported in part by the National Natural Science Foundation fort is to ‘converse’ with GPT-3.5. The test cases are written in of China under Grants 62372461, 62032001 and 62203457, and in part by the Key Laboratory of Advanced Microprocessor Chips and Systems. two programming languages, C and assembly, for the Xuantie- * Corresponding author: leiwang@nudt.edu.cn C910 and neuromorphic processor, respectively using LLM.

                                                                             979-8-3503-4859-0/DATE24/© 2024 EDAA

       1   Prompts: Create a C program to calculate the sum of two integers.
       2   Responses:
       3    #include <stdio.h>
       4    int main(){
       5        int num1, num2, sum;
       6        printf (”Enter the first integer :”);
       7        scanf(”%d”, &num1);
       8        printf (”Enter the second integer:”);
       9        scanf(”%d”, &num2);
      10        sum = num1 + num2;
      11        printf (”The sum of %d and %d is %d.\n”, num1, num2, sum);
      12        return 0;
      13    }
Fig. 1.         An example of prompts to LLM and its response for generating a C
program which performs the function of integers adding.
                       The LLM-generated tests are then compiled into executable
                 binary files (ELFs) and serve as stimuli in the RTL-level simu-                         Stall_idu_pipe1                mux
                lator. After collecting and analyzing the simulation results, we    Neuromorphic     mux     Neuromorphic iid/wb_data/cmplt
                  prompt LLM to generate the next round of test cases, targeting                         APU
                  the uncovered parts of the DUT. The main contributions of this        ID           ins/iid

paper are: • We present the first attempt, to our best knowledge, to Fig. 2. Block diagram of tightly-coupled neuromorphic processor extended demonstrate the ability of conversational LLM for tradi- from Xuantie-C910 using ISA-extension method. tional processor and DSA function verification. B. Large Language Models (LLMs) • We perform the function verification on both a RISC- LLMs, consisting of hundreds of billions (or even more) of V processor and a neuromorphic processor to explore parameters, are generally based on transformer architectures. the potential advantages and obstacles of LLM-based GPT-4 is the latest version of the GPT family. The inputs to processor verification. LLMs are commonly referred to as ‘prompts’, where users can • We share our experience and observation in using LLM ‘program’ LLMs to successfully complete a specific task. A for functional verification of traditional processors and case where we ask GPT-3.5 to write a C program to calculate DSA such as neuromorphic processors, as well as potential the sum of two integers is shown in Figure 1. It is apparent that future research directions of processor verification based GPT-3.5 not only understands the intent of the human prompts on LLM. but also responds correctly and writes high-quality code. In the experiment, 36 C programs and 128 assembly snippets C. The Processor under Verification for the RISC-V core and the₁ neuromorphic processor are Neuromorphic computing aims to design and build computer generated using an advanced LLM to demonstrate our claim. systems inspired by the structure and function of the human The experimental results show that the code coverage based on brain. As the running platform of neuromorphic algorithms, the LLM test generation can reach 89% and 91% for the above neuromorphic processors such as TrueNorth [6] and Loihi [7] two architectures respectively, showing a promising research are proposed these years, showing unforgettable energy effi- direction for the future processor verification. ciency when compared with traditional processor architectures such as CPU, GPU, and DNN accelerators. II. BACKGROUND In order to accelerate neuromorphic computing, we extend the neuromorphic instruction set based on the RISC-V instruc- A. Processor Verification tion set and implement tightly coupled neuromorphic comput- ing units in the CPU core based on neuromorphic instructions. Processor verification is indeed crucial for ensuring the Our neuromorphic instruction set defines 10 types of extension functionality and reliability of the processor. Testing programs registers, including weight vector, spike vector, output spike include bare metal programs, Uboot, operating systems, and vector, neuron type/voltage threshold value parameters, and so more complex applications. However, it is challenging to obtain on. In addition, the neuromorphic instruction set uses 3 in- a suitable testing program and complete as much verification of struction formats to define 8 types of 33 instructions, including the processor as possible within a limited time. For ARM archi- wide vector loading&storing, data movement, neuron current tecture processors, the RAVEN [8] testing tool can be used. For and accumulation (vertical/horizontal), and so on. RISC-V, the RISCV-DV [9] tool can be used. However, these We have implemented our neuromorphic instruction set- approvals are not designed for Domain Specific Architecture based micro-architecture extended on the high-performance (DSA) like neuromorphic processors, requiring a lot of manual open-source RISC-V processor Xuantie-C910 [17]. The archi- effort for the verification. Code coverage is widely adopted by tecture of the extended neuromorphic processor is shown in simulation-based verification, about 75% for ASIC projects and Figure 2. The original Xuantie-C910 processor core adopts a 55% for FPGA projects [1]. Therefore, code coverage is one of superscalar architecture with 3 decoding and 8 execution, with a the most important indexes to evaluate the verification progress pipeline depth of 9-12 levels and a working frequency of 1GHz. and quality. It can support various functions, such as cache prefetching,

    Test Generation              Generate  C / Assembly  Compile Compilation & Silmulation
                                              Tests

    Input                 LLM                                  RISC-V binarySimulate
                                             Branch
                                             Block
                                     Analyze Expression               Collect
  Prompt                         Module       Coverage  Simulator
generator                        filter       reports

                          Pass       Terminate
                                 Result colletion & processing                    (a)    (b)

Fig. 3. The overview of LLM-based processor verification components and Fig. 4. Prompts to GPT-3.5 in RISC-V processor test generation. (a) an workflow which consists of test generation, compilation&simulation, and result example of generating C programs for testing RISC-V ISA. (b) an example of collection&processing. our prompts for specific function unit test generation and the LLM responses cache path prediction, branch prediction, unordered memory for testing the branch prediction. access, etc., to improve performance and reduce the power 1) ISA Prompts: The fact that instruction set architecture consumption of the processor. The new neuromorphic proces- defines the interface between the software and hardware com- sor core supports the execution of neuromorphic instructions ponents motivates us to use the ISA as the prompt to GPT-3.5. mentioned above. To do this, we take the function description of each RISC-V We have extended the vector register to store spikes, weights, instruction implemented in the target processor as the prompt. and neuron states related to the neuron model; We extend Figure 4(a) shows the content of the conversation with GPT-3.5. the memory access system to implement ScratchPad Memory 2) Prompts for Hardware-aware Test Generation: Modern (SPM) and load store unit to improve the memory bandwidth; processors consist of many specific function units such as We implement the computing components to achieve two branch direction prediction in Xuantie-C910, to achieve high- computing methods (”synapse-wise and neuron-wise”) and a performance computing. For these specific function units, we parallel computing neuron array consisting of 512 neurons provide GPT-3.5 with the corresponding prompts. Figure 4(b) units. shows how we prompt GPT-3.5 to generate tests for the veri- fication of the branch prediction module. In the conversation, III. METHODOLOGY GPT-3.5 first calls the rand() function to initialize the array, A. Overview complicating the following branches. Then, it creates a loop body with 1x106 iterations, which contains two branches. Figure 3 shows the overview of how we apply LLMs Clearly, GPT-3.5 has understood our prompts and effectively to perform processor-level function verification. It generally generated the testing programs. contains three parts: test generation, compilation&simulation, C. Test Generation for the Neuromorphic Architecture and result collection&processing. Since the neuromorphic instruction set is unknown to GPT- The workflow of our LLM-based test generation is as fol- 3.5, the first step before test generation is to teach GPT-3.5 lows. Test generation focuses on creating new tests based on how to write a neuromorphic application based on the custom the processed results to cover more possible conditions. The neuromorphic instruction set. test cases are written in C and assembly for RISC-V ISA The testing process is performed as follows. Firstly, we teach and neuromorphic ISA, respectively. Then, the generated tests GPT-3.5 the function and usage of each neuromorphic instruc- are compiled into executable binary files, which are served as tion. For example, Figure 5 shows the prompts and responses stimuli in the simulator. After the simulation, different types of when teaching GPT-3.5 the lw.wv instruction, an instruction to coverage information are collected and processed. Based on the load synaptic weights from memory to the extended Weight processed results, verification engineers examine the uncovered Vector Register (WVR). Each synaptic weight is stored in 4 conditions along with their underlying reasons. Subsequently, bits and WVRs are the 128x32bits weight vector registers. As they provide feedback to the LLMs regarding the strategies shown in Figure 6, since the LLM has already understood the generated by the subsequent round of tests. Then, the next usage of the instruction we teach, a simple assembly program iteration of verification begins. code segment with cyclic weights and spike loading can be B. Test Generation for RISC-V Processor generated by the LLM then. The details of the testing results and our analysis are as In this work, we are committed to exploring the use of LLM follows. Regarding this novel neuromorphic instruction, gener- to generate tests. We need to prompt the LLM on what test ating the correct code is very challenging for GPT-3.5. It often programs we need, and how LLMs generate the corresponding requires iterative refinement and collaboration with humans to tests is out of scope for the purposes of this paper. We use ensure the generated code aligns with the intended functionality. GPT-3.5 as the basic LLM. As shown in Figure 5, remarkably, despite the vagueness of our

                                   Listing 1: cccc                                                                                                            Listing 1: cccc

1 Prompts: Now we have defined a new neuromorphic extension instruction lw.wv. Its mnemonic is ‘lw.wv rd 1 Prompts: ok, we now need to load weights and spike sequences. You can splice two pieces of code together. , imm(rs1)’. It represents using the rs1 general register as the base address and the imm immediate 2 Responses: as the offset to generate the address for accessing memory, and loading the 32b memory access data 3 li x10, 0 # Counter for accessing WVRs. into the weight vector extension register wvr indexed indirectly by the rd general register . There are 4 li x11, wvr base address # Base address for accessing WVRs. a total of 128 wvrs, each of which is 32b in size . Please generate a loop that traverses and 5 li rd, 0 # Register to index SVRs. accesses all wvrs. 6 li rs1 , svr base address # Base address for accessing SVRs. 2 Responses: 7 li imm, spike offset # Offset for the spike sequence address. 3 li x10, 0 # Counter for accessing WVRs. 8 loop: There should be two loops for loading synaptic weights and spike sequences respectively. 4 li x11, 0 # Base address for accessing WVRs. 9 lw.wv x12, 0(x11) # Load WVR data referenced by x10 into x12. 5 # Loop to traverse and access WVRs. 10 addi x10, x10, 1 # Increment counter for WVR access. 6 loop: 11 addi x11, x11, 4 # Increment base address for the next WVR. 7 lw.wv x12, 0(x11) # Load WVR data referenced by x10 into x12. 12 lw.sv rd, imm(rs1) # Load SVR data into the indexed SVR register. 8 # Perform operations on the loaded WVR data as needed. 13 addi rs1 , rs1 , 4 # Increment rs1 by 4 to access the next spike sequence. 9 # Increment counter and update base address for the next WVR. 14 addi rd, rd, 1 # Increment rd to index the next SVR. Failed 10 addi x10, x10, 1 destination WVR register not increased 15 blt rd, 32, loop # Branch back to the loop if rd is less than 32. attempt 11 addi x11, x11, 4 # Assuming each WVR size is 32 bits (4 bytes). Failed 16 Prompts: You made a few mistakes. (1) The x12 in lw.wv x12, 0(x11) does not increase; (2) Lw.wv cycles 12 bne x10, x0, loop # Branch back to the loop if the counter is not zero. attempt 128 times, lw.sv cycles 32 times. 13 # End of loop. Jump condition error 17 Responses: 14 Prompts: There are a total of 128 WVRs. 18 li x10, 0 # Counter for accessing WVRs. 15 Responses: 19 ...... 16 li x10, 0 # Counter for accessing WVRs. 20 li imm, spike offset # Offset for the spike sequence address. 17 li x11, 0 # Base address for accessing WVRs. 21 wvr loop: destination WVR register not increased 18 # Loop to traverse and access WVRs. 22 lw.wv x12, 0(x11) # Load WVR data referenced by x10 into x12. 19 loop: 23 addi x11, x11, 4 # Increment base address for the next WVR. 20 lw.wv x12, 0(x11) # Load WVR data referenced by x10 into x12 24 addi x10, x10, 1 # Increment counter for WVR access. 21 # Perform operations on the loaded WVR data as needed. 25 blt x10, 128, wvr loop # Branch back to the loop if the counter is less than 128. 22 # Increment counter and update base address for the next WVR. 26 svr loop : 23 addi x10, x10, 1 27 lw.sv rd, imm(rs1) # Load SVR data into the indexed SVR register. 24 addi x11, x11, 4 # Assuming each WVR size is 32 bits (4 bytes). Failed 28 addi rs1 , rs1 , 4 # Increment rs1 by 4 to access the next spike sequence. 25 blt x10, 128, loop # Branch back to the loop if the counter is less than 128. attempt 29 addi rd, rd, 1 # Increment rd to index the next SVR. Failed 26 # End of loop. 30 blt rd, 32, svr loop # Branch back to the loop if the rd is less than 32. attempt 27 Prompts: The destination register should also be incremented. 31 Prompts: During each wvr loop, the value of the x12 register should increase by 1 to point to the next 28 Responses: wvr extension register . 29 li x10, 0 # Counter for accessing WVRs. 32 Responses: 30 li x11, 0 # Base address for accessing WVRs. 33 li x10, 0 # Counter for accessing WVRs. 31 # Loop to traverse and access WVRs. 34 ...... 32 loop: 35 li imm, spike offset # Offset for the spike sequence address. 33 lw.wv x12, 0(x11) # Load WVR data referenced by x10 into x12 36 wvr loop: 34 # Perform operations on the loaded WVR data as needed. 37 lw.wv x12, 0(x11) # Load WVR data referenced by x10 into x12. 35 # Increment counter and update base address for the next WVR. 38 addi x11, x11, 4 # Increment base address for the next WVR. 36 addi x10, x10, 1 39 addi x12, x12, 1 # Increment x12 to point to the next WVR extension register. 37 addi x11, x11, 4 # Assuming each WVR size is 32 bits (4 bytes). 40 addi x10, x10, 1 # Increment counter for WVR access. 38 # Increment destination register for the next WVR. 41 blt x10, 128, wvr loop # Branch back to the loop if the counter is less than 128. 39 addi x12, x12, 1 Pass 42 svr loop : 40 blt x10, 128, loop # Branch back to the loop if the counter is less than 128. 43 lw.sv rd, imm(rs1) # Load SVR data into the indexed SVR register. 41 # End of loop. 44 ...... Pass 45 blt rd, 32, svr loop # Branch back to the loop if the rd is less than 32. Fig. 5. Example of teaching GPT-3.5 the lw.wv extended neuromorphic instruction. After two attempts, LLM understands the instruction and generates Fig. 6. Example of prompts to GPT-3.5 for generating code segment with the the correct program for testing different conditions of the instruction execution.₁ loop of loading synaptic weights and spike train operands, which can be seen as part of a simple neuromorphic application. This example shows the ability¹ of LLM for understanding unknown knowledge and generating tests for DSA.

further prompt, GPT-3.5 demonstrates an impressive ability to D. Compilation & Simulation pinpoint the source of the error directly. After the initial prompt, The generated tests are compiled into executable binary files, we further provide two vague prompts to correct errors in the which are subsequently utilized as stimuli within the simulator. initial program. To our surprise, GPT-3.5 swiftly identifies the Once binary files are obtained, we identify the instructions precise location of the mistake. For instance, in the second contained within and proceed to search for instructions that prompt, where we mention a total of 128 WVR registers, GPT- have not been covered. Throughout the simulation process, 3.5 not only comprehends the need to iterate 128 times but also we diligently record three coverage metrics for each module. accurately modifies the error by changing ’bne x10 x0 loop’ to These metrics facilitate a comprehensive evaluation of the ’blt x10 128 loop’. This remarkable capability showcases GPT- verification progress, allowing us to identify coverage gaps and 3.5’s programming language understanding and its ability to drive improvements in the test suite for each module. address errors even in ambiguous contexts. As shown in Figure 6, once GPT-3.5 has acquired a solid E. Result Collection & Processing understanding of the fundamental usage and functionality of 1) Coverage Collection: After the simulation, we parse the each extended neuromorphic instruction, we engage it in com- log files to obtain the different types of coverage, including bining multiple instructions to create complex test programs. block coverage, expression coverage, and toggle coverage for As a result, GPT-3.5 generates the assembly program to load further understanding and re-prompt to LLM. synaptic weights and spike sequences. The lw.sv instruction, 2) Module Filter: Module Filter is proposed in this work similar to the lw.wv, is to load spike sequences from memory to timely find modules whose coverage achieves satisfactory to the 32 extended Spike Vector Registers (SVRs). Regardless thresholds in the test generation loop, thereby terminating the of the errors in the initially generated programs, it reflects the loop. capability of LLMs to retain and effectively apply previously In detail, during the verification process, certain modules of acquired knowledge. Based on this fact, we prompt GPT- the DUT may quickly achieve high coverage, even reaching 3.5 to generate a set of combinational test programs with 100%. This occurrence is typically observed when the module’s different neuromorphic instructions, which are then translated functionality is simple, and the designed tests effectively cover into machine code using an assembler and subsequently sent the designated functionality. For example, the ct iu alu module to the simulator as stimuli. This process leverages GPT-3.5’s in Xuantie-C910 is responsible for handling simple operations knowledge to synthesize intricate sequences of neuromorphic like left shift and addition. The coverage for this module instructions and generate comprehensive test conditions. quickly achieves a high level by incorporating arithmetic opera-

                         TABLE I                                                    Block coverage                      Expression coverage  Toggle coverage
 ISA COVERAGE USING LLM             TO GENERATE THE TEST PROGRAM         100.0%                                                                         95.4%
                                                                         90.0%                                                                   91.1%           89.0%88.0% 91.0%
 Instruction Type     Num. of Ins.  Covered  Coverage                                           86.5%     87.6%                        89.1%  90.5%
      RV64I              63            57     90.48%                         78.8%                  87.8%    83.8%                      85.4%               85.5%
                                                                         80.0%     71.6%     70.2%
      RV64M              13            12     92.31%                     70.0%         64.3%                 65.3%                                     66.5%
      RV64A              22            21     95.45%                     60.0%                                                          57.5%                60.3%    58.1%
      RV64F              30            29     96.67%                     50.0%                      48.0%                                                        43.0%
      RV64D              32            31     96.88%                     40.0%     37.8%
      RV64C              37            29     78.38%
 Neuromorphic ISA        33            33    100.00%                     30.0%              19.2%
                                                                         20.0%
tions. For such ‘simple’ modules, once the coverage reaches the          10.0%
defined threshold, they are deemed as passed in our workflow.             0.0%     ct_vfpu  ct_cp0  ct_iu    ct_lsu                     ct_rtu   ct_idu      ct_ifu ct_core neuromorphic
Consequently, generating  additional           tests specifically for                                                                                            APU

these passed modules is no longer pursued, taking into account Fig. 7. Block, expression, and toggle coverage the LLM-generated test can considerations such as diminishing returns, resource optimiza- achieve in RISC-V processor and neuromorphic processor verification. tion, and time constraints. This approach allows for the efficient neuromorphic instructions because the tests are written in allocation of verification resources, prioritizing the assessment assembly directly. For the RISC-V ISA, it’s important to note of more complex and critical modules while still ensuring that that the tests are written in the C programming language. Due the entire system undergoes thorough verification. to the nature of the C language and its compilation process, it If, despite multiple rounds of testing, the coverage of certain can be challenging to achieve full coverage. The six uncovered modules remains below the set threshold, it may indicate that instructions in RV64I are csrrc, csrrci, csrrs, csrrsi, csrrw, and the LLM is unable to generate effective tests targeting the ebreak. The first five instructions are used for interacting with uncovered parts of the DUT based on the currently provided CSRs (Control and Status Registers). The ebreak is commonly prompts. The reasons for this failure can be multi-faceted used as a debugging or breakpoint instruction. It is apparent and include factors such as neglecting edge cases, intricate that achieving coverage of these six special instructions in a dependencies or interactions between modules, and more. compiled C program can be challenging. Consequently, to mitigate testing costs, it becomes necessary 2) Block, Expression, and Toggle Coverage: The block, to terminate test generation specifically for these ‘difficult’ expression, and toggle coverage are depicted in Figure 7. modules. The ct vfpu, ct cpu0, ct iu, ct lsu, ct rtu, ct idu, and ct ifu In addition to the previously mentioned two types of mod- are sub-modules of ct core. The total block and expression ules, the remaining modules will continue the iterative testing coverage of ct core reach 88.98% and 88.01%. Notably, the until the ‘termination’ or ‘pass’ conditions are met. block coverage of the neuromorphic extension reaches 91.02%. This can be attributed to the SIMD nature of the neuromorphic IV. EXPERIMENT architecture, which facilitates a more straightforward verifica- A. Experiment Setup tion process and consequently yields higher block coverage. In this paper, we utilize a native LLM, i.e., GPT-3.5, specif- It is crucial to highlight that the toggle coverage remains ically without fine-tuning it for processor function verification. generally low for all modules. This can be mainly attributed Our approach involves generating tests using GPT-3.5 based to the intricacies involved in evaluating signal flips within the on the given prompts. For the RISC-V ISA, we generate 36 design code. While toggle coverage proves useful in gate-level C programs, while for the neuromorphic ISA, we generate testing, controlling and managing these signal flips becomes 128 assembly programs. To assess the effectiveness and quality particularly challenging when dealing with high-level language of the generated tests, we have evaluated the following three programs such as C programs. metrics: ISA coverage: This metric measures the coverage of V. D ISCUSSION the ISA by the generated tests. It determines the extent to which different instructions are tested. Block coverage: It measures Speed and scale of LLM-based verification: Although we the percentage of basic blocks executed by the generated have automated crucial stages such as result collection and tests, helping identify areas that may have been skipped or processing, one significant aspect remains manual—the need insufficiently tested. Expression coverage: It is a mechanism for manual engagement in each conversation with GPT-3.5. The that factorizes logical expressions and monitors them during manual participation process involves human experts examining the simulation run. It provides metrics to quantify the degree the coverage outcomes and formulating prompts or queries that of verification completeness. Toggle coverage: It measures the can effectively guide GPT-3.5 toward addressing the uncovered activity of various signals in a design and provides information parts of the DUT. To address this challenge and increase on untoggled signals or signals that remain constant during the the speed and scale of LLM-based verification, future efforts simulation run. should prioritize exploring techniques and tools for automatic B. Experiment Results conversion of coverage results into natural language prompts. The objective of this automated conversion process should be

1. ISA Coverage: The ISA coverage is shown in Table creating prompts specifically targeted at undiscovered parts of I. For the neuromorphic ISA, the generated tests cover all the DUT.

Scalability for novel DSA: Our research showcases the complete HDL for a tape out using AI. However, to the best of tremendous potential of LLMs in the realm of functional our knowledge, the use of LLMs to generate tests for processor verification for novel DSA. With the appropriate teaching verification has not been explored yet. and deep understanding of new architectural features, GPT- VII. C 3.5 demonstrates its capability to generate tests that effectively ONCLUSION perform the functionality verification of these novel processors. In this work, we have applied GPT-3.5 to conduct the It should be noted that in this study, we use a dialogue-based functional verification on both traditional and novel processor approach to facilitate GPT-3.5 in acquiring an understanding architecture. To test and validate new processor architectures, of the intricate nature of neuromorphic ISA and this process the LLMs can efficiently generate effective test cases through still needs human participation, which can be considered as a the process of learning and adaptation. Collaborating with limitation. To address this limitation, future endeavors should LLMs allows verification engineers to focus more on identi- consider exploring the integration of deep learning methods fying potential deficiencies in the current iteration, rather than to fine-tune LLMs. By doing so, LLMs can be trained to investing excessive effort in test generation. Our future direc- autonomously adapt and comprehend the novel processor ar- tion aims at fully automating the entire verification process. chitecture. This autonomous adaptation empowers the LLMs It includes two key aspects. Firstly, eliminating the human to generate tests and perform functional verification without involvement to analyze and identify defects in each iteration heavy reliance on human involvement. of the verification process. Secondly, when confronted with Effectiveness of generated tests: Based on the conversation new architectures, automatically teaching the LLMs on a new with GPT-3.5 and the experimental results, it is evident that the instruction set architecture. generated tests exhibit a high level of effectiveness and are well- REFERENCES suited for addressing uncovered conditions. This effectiveness can be attributed to the careful consideration and planning used [1] Foster et al. The 2022 wilson research group functional verification by the verification engineers when preparing the prompts for study. https://blogs.sw.siemens.com/verificationhorizons/2022/10/10/ prologue-the-2022-wilson-research-group-functional-verification-study/, each conversation. [2] 2022. VI. RELATED WORKS Kitchen et al. Stimulus generation for constrained random simulation. In 2007 IEEE/ACM International Conference on Computer-Aided Design, To promote the efficiency and quality of verification, many pages 258–265, 2007. test generation methods have been proposed. Constrained ran- [3] Fine et al. Coverage directed test generation for functional verification using bayesian networks. In Proceedings 2003. Design Automation dom generation (CRG) [2] is a test generation technique, which Conference (IEEE Cat. No.03CH37451), pages 286–291, 2003. specifies and solves constraints to generate new tests thus [4] Masamba et al. Supervised learning for coverage-directed test selection creating better tests than pure random generation. Coverage- in simulation-based verification. In 2022 IEEE International Conference On Artificial Intelligence Testing (AITest), 2022. directed test generation (CDG) [3] leverages artificial intel- [5] Jeff et al. A new golden age in computer architecture: Empowering the ligence (AI) techniques to automate test biasing during the machine-learning revolution. IEEE Micro, 38(2):21–29, 2018. verification process. However, one of the challenges with CDG [6] Merolla et al. A million spiking-neuron integrated circuit with a scalable communication network and interface. Science, 345(6197):668 – 673, is that it requires significant domain knowledge to be encoded 2014. into the AI model. Coverage-directed test selection (CDS) [7] Davies et al. Loihi: A neuromorphic manycore processor with on-chip [4], based on supervised learning from coverage feedback, [8] learning. IEEE Micro, pages 82–99, 2018. Andrea et al. Raven- a real-time analysis and verification environment. selects effective tests from the large set of generated tests and Proc. of DSP Deutschland 97, 1997. prioritizes them for simulation. Novelty-driven verification [18] [9] Michalak et al. Riscv-dv. https://github.com/chipsalliance/riscv-dv, 2023. constructs an unsupervised model of similarity from previous [10] OpenAI. Gpt-4 technical report, 2023. [11] Chen et al. Evaluating large language models trained on code. arXiv tests and ranks newly generated tests, with preference given to preprint arXiv:2107.03374, 2021. the most dissimilar. [12] Nijkamp et al. Codegen: An open large language model for code with Recently, researchers have been exploring and applying [13] multi-turn program synthesis, 2023. Pearce et al. DAVE: Deriving automatically verilog from english. In LLMs for various professional uses. Codex, fine-tuned on mil- Proceedings of the 2020 ACM/IEEE Workshop on Machine Learning for lions of open-source software repositories from GitHub, solves CAD. ACM, nov 2020. 28.8% of the programming problems on the HumanEval dataset [14] Blocklove et al. Chip-chat: Challenges and opportunities in conversational hardware design, 2023. [11]. Nijkamp et al. [12] have trained and released a family of [15] Chang et al. Chipgpt: How far are we from natural language hardware LLMs up to 16.1B parameters for program synthesis. Despite design, 2023. the application of LLMs in the software domain, there are also [16] Thakur et al. Benchmarking large language models for automated verilog rtl code generation. In 2023 Design, Automation Test in Europe many works focused on the hardware domain. DAVE [13], the Conference Exhibition (DATE), pages 1–6, 2023. first work exploring the use of LLMs in the hardware design, is [17] Chen et al. Xuantie-910: A commercial multi-core 12-stage pipeline out- fine-tuned from the GPT-2 model over synthetically generated of-order 64-bit high performance risc-v processor with vector extension : Industrial product. In 2020 ACM/IEEE 47th Annual International ‘English-Verilog’ pairs. Given that LLMs perform effectively Symposium on Computer Architecture (ISCA), pages 52–64, 2020. in an interactive manner, Blocklove et al. [14] conducted a [18] Blackmore et al. Novelty-driven verification: Using machine learning to conversational case study where a hardware engineer collabo- identify novel stimuli and close coveragen. In Proceedings of the 2021 Design and Verification Conference (Virtual), 2021. rated to co-design an 8-bit accumulator-based microprocessor architecture, using ChatGPT-4. It is the first time to write the