SOURCE ARCHIVE
EXTRACTED CONTENT
116,281 charsIEEE TRANSACTIONS ON COMPUTER-AIDED DESIGN OF INTEGRATED CIRCUITS AND SYSTEMS 1
FLAME: Enhancing Functional Coverage in
Processor Verification via Large Language Models
Xiaopeng Li , Junjie Chen , Ming Yan , Dong Wang , Xingyu Fan , Zhentao Tang , Shixiong Kai ,
Jianye Hao , Mingxuan Yuan , Senior Member, IEEE, and Zan Wang
1 Abstract—Processor functional verification plays a crucial larger number of functional points (FPs) indicates higher test 58
2 role in ensuring the quality of processor designs. Traditional e↵ectiveness, which serves as a crucial indicator of functional 59 techniques like constrained random verification (CRV) struggle 34 to achieve high functional coverage due to the vast instruction verification automation in practice. 60 5 space of processors. While large language model (LLM)-based In the last decade, various techniques based on simulation 61 6 techniques show potential, merely instructing LLMs has notable have been developed to cover more FPs automatically, such as 62 7 limitations, especially when addressing functional points (FPs) constrained random verification (CRV) [3], [6] and coverage- 63 8 that require deep semantic understanding. To tackle these chal- guided fuzzing (CGF) methods [7], [8], [9], [10]. However, 64 9 lenges, we propose a novel technique, FLAME, which specially many FPs remain uncovered due to the inherent complexity 65 10 designs and integrates retrieval-augmented generation (RAG), of their triggering conditions (often requiring test cases with 66 11 chain-of-thought (CoT), and functional-coverage-guided feedback specific semantics). Due to the vast space of instruction types 67 12 strategies. This technique establishes semantic mappings between and parameters, it is dicult for simulation-based techniques 68 13 FPs and instructions, enabling the iterative generation of valid to search for e↵ective test cases that satisfy such complex 69 and e↵ective test cases. Evaluation on four widely used open- 1415 source processor designs shows that FLAME outperforms the triggering conditions. Particularly, most of these existing 70 16 typical and state-of-the-art baselines in functional coverage techniques require extensive involvement from experienced 71 17 improvement with an average of 34.25%⇠220% while drastically engineers to write instruction constraints for ensuring test 72 18 reducing the time required to achieve the same functional validity, causing significant manual e↵ort in implementing 73 19 coverage by up to 86.13%. Moreover, ablation analysis highlights these techniques. 74 20 the vital role of each component in the framework’s overall Large language models (LLMs) possess powerful seman- 75 21 e↵ectiveness. This work demonstrates the superiority of our tic understanding capabilities, and recent advances have 76 22 LLM-based technique FLAME in enhancing processor functional brought forth some LLM-based techniques to processor func- 77 23 verification. tional verification [11], [12], [13], [14]. In general, they 78 24 Index Terms—Functional coverage, large language models instructed LLMs to understand the semantics of uncov- 79 25 (LLMs), processor verification, test generation. ered FPs and then generate test cases to cover them. The 80 results have demonstrated the superiority of such LLM-based 81
ing LLMs still exhibits significant limitations, particularly 83
26 P I. INTRODUCTION techniques over traditional ones. However, simply instruct- 82
27 ROCESSOR verification is a vital step in processor devel- in addressing the FPs that demand comprehensive semantic 84
28 opment that ensures design correctness, particularly as understanding. For instance, in a real-world RISC-V processor 85
29 modern hardware designs grow more complex [1], [2]. Manual (i.e., IBEXv2 [15]), the design document specifies an FP 86 30 verification is quite costly as it requires significant domain- named auto_InstrCategoryCSRIllegal, defined as 87 31 specific expertise. The main automatic verification method “any instruction attempting a CSR access that is not allowed” 88 32 involves simulating and functionally verifying the processor [15]. Achieving this FP requires the generated test cases to 89 33 design through test cases to ensure all functionalities work include specific combinations of control and status register 90 34 as expected [3], [4]. Functional coverage is a key metric for (CSR)-related instructions (e.g., csrw and csrr), as well 91 35 evaluating processor verification e↵ectiveness [5]. Covering a as the proper parameters of these instructions and CSRs’ 92 36 Received 26 May 2025; revised 23 October 2025; accepted 9 February addresses. FPs like this often necessitate the generation of 93 37 2026. This work was supported in part by the National Key Research and highly specific instructions or instruction combinations to pro- 94 38 Development Program of China under Grant 2024YFB4506300, in part by duce valid test cases—an area where the current LLM-based 95 39 the National Natural Science Foundation of China under Grant 62322208, techniques fall short. Indeed, our experiments confirm that the 96 40 in part by the Huawei Project Funding, in part by the National Natural state-of-the-art LLM-based techniques [12] su↵er from low 97 Science Foundation of China under Grant 62232001, and in part by the CCF 4142 Young Elite Scientists Sponsorship Program (by CAST). This article was test validity (only up to 26.13% on a large-scale processor, 98 43 recommended by Associate Editor M. Sonza Reorda. (Corresponding author: i.e., CVA6 [16]) and yields only marginal improvements 99 44 Junjie Chen.) in covering FPs in practical scenarios (as demonstrated in 100 45 Xiaopeng Li, Junjie Chen, Ming Yan, Dong Wang, Xingyu Fan, and Section V-B). 101 46 Zan Wang are with the School of Computer Software, Tianjin Univer- To sum up, the main reasons for the significant challenge 102 sity, Tianjin 300350, China (e-mail: xpl@tju.edu.cn; junjiechen@tju.edu.cn; 4748 yanming@tju.edu.cn; dong w@tju.edu.cn; fxyqaq@tju.edu.cn; wangzan@ of covering more FPs are twofold. 103 49 tju.edu.cn. 1) These FPs represent specific processor scenarios and 104 50 Zhentao Tang, Shixiong Kai, Jianye Hao, and Mingxuan Yuan are behaviors under test, requiring special instructions to 105 with the Huawei Noah’s Ark Laboratory, Beijing 100085, China (e-mail: 5152 tangzhentao1@huawei.com; kaishixiong@huawei.com; haojianye@huawei. trigger corresponding test conditions. Notably, mapping 106 53 com; Yuan.Mingxuan@huawei.com). abstract FPs to concrete input instructions is challeng- 107 54 Digital Object Identifier 10.1109/TCAD.2026.3666802 ing, which requires a deep understanding of both the 108 55 1937-4151 © 2026 IEEE. All rights reserved, including rights for text and data mining, and training of artificial intelligence and 56 similar technologies. Personal use is permitted, but republication/redistribution requires IEEE permission. 109 57 See https://www.ieee.org/publications/rights/index.html for more information. 110
2 IEEE TRANSACTIONS ON COMPUTER-AIDED DESIGN OF INTEGRATED CIRCUITS AND SYSTEMS
111 instruction set architecture (ISA) and processor-related the-art LLM-based techniques on average. This demonstrates 174
112 documentation. that FLAME significantly enhanced functional coverage in 175
113 2) Complex interactions between instructions require pre- processor verification by leveraging LLMs’ semantic under- 176
114 cise modeling of dependencies and execution order, standing and addressing the mapping and validity challenges 177
115 making it challenging to generate valid instruction as aforementioned. Our experiments also confirmed the con- 178
116 sequences as test cases. Existing LLM-based techniques tribution of retrieval-based semantic reasoning, source-level 179
117 struggle to generate semantically desirable and correct test case generation, and iterative coverage refinement to the 180
118 instruction sequences due to their tendency to halluci- overall e↵ectiveness of FLAME, and investigated the impact 181
119 nate. of di↵erent LLMs (including GPT-3.5-Turbo, CodeLlama-7B, 182
120 In this work, we aim to harness LLMs’ capabilities more CodeLlama-13B, and Phind CodeLlama-34B) on FLAME. 183
121 e↵ectively to enhance functional coverage in processor ver- The main contributions of our work are as follows. 184
122 ification, addressing the FP bottlenecks that hinder existing 1) We propose a novel LLM-based technique to enhance 185
techniques. To achieve this, we introduced FLAME (Func-
123 tional coverage using large language model), o↵ering the functional coverage in processor verification, reducing 186
124 following novel and impactful capabilities. expert knowledge and manual e↵ort. 187
125 2) We build ecient semantic mappings between FPs and 188
126 1) Mapping Semantics Between FPs and Instructions: instructions, and generate e↵ective test cases from the 189
127 For uncovered points, FLAME performs static anal- source code level, in order to address the search space 190
128 ysis to extract instruction-related information. Using and validity challenges. 191
129 retrieval-augmented generation (RAG), it retrieves rel- 3) We conduct extensive experiments on four widely used 192
130 evant descriptions from design documents, such as real-world processors, demonstrating the performance 193
131 processor documentation and FPs definitions. These and generalizability of FLAME. We have made the 194
132 descriptions establish semantic links between FPs and replication package publicly available on our homepage 195
133 the corresponding instructions desired by test cases. This [26]. 196
134 method e↵ectively addresses the challenges of complex
135 semantic conditions required for achieving nontrivial
136 functional coverage and the extensive search space asso- II. BACKGROUND AND RELATED WORK 197
137 ciated with input instructions. A. Processor Functional Verification 198
138 2) Generating Valid and E↵ective Test Cases: FLAME To ensure the quality and correctness of processor designs, 199
leverages RAG to obtain instruction details (e.g.,
139 instruction types and usage constraints) from the ISA processor functional verification plays a critical role. This 200
140 specification, and then feeds them into LLMs to guide process aims to confirm that the design meets specified func- 201
141 the test generation. In particular, to ensure test validity, tional and performance requirements. For this task, functional 202
142 FLAME generates tests at the source code level (i.e., coverage is a key metric, which measures how thoroughly ver- 203
143 generating C programs and then compiling them to ification tests explore the design’s functionality requirements, 204
144 instructions for verification) rather than the instruction rather than just measuring code execution paths like code 205
145 level, since LLMs have demonstrated superior perfor- coverage does [5]. Functional coverage tracks those scenarios, 206
146 mance on the former [17], [18], [19] (as investigated events, and sequences specified in the design documentation, 207
147 in Section V-A1). Also, analyzing patterns of invalid ensuring all intended behaviors are tested. The functional 208
148 or ine↵ective test cases is helpful to ensure constraint coverage models are structured hierarchically by processor 209
149 compliance and maximize validity. Therefore, FLAME design developers into FPs and groups (a group of FPs with 210
150 enhances its test generation process by incorporating relevant functionalities) [5]. An FP typically represents a fine- 211
151 some counterexamples, specific test cases that previously grained design feature or instruction behavior, such as the 212
152 failed to cover certain FPs, into its prompts. execution of a specific branch instruction or the update of 213
153 3) Functional-Coverage-Guided Iterative Testing: To CSRs [5]. For example, in a RISC-V processor, FPs may 214
154 enhance improvement, FLAME collects functional include “jump and link instruction (JAL) execution” or “CSR 215
155 coverage results and refreshes uncovered FPs as well write protection check.” These FPs are grouped into higher 216
156 as counterexamples after executing the generated level categories (e.g., control flow and memory access) as 217
157 test cases. This information guides subsequent test functional groups (FGs) to cover the intended functional 218
158 generation rounds by updating the target FPs. behavior of the processor systematically. 219
159 In the literature, there are a number of techniques targeting 220
160 To investigate the e↵ectiveness of FLAME, we conducted this task, which fall into two general categories: static formal 221
161 extensive experiments on four real-world processor designs verification methods [27], [28], [29], [30], [31], [32] and 222 162 based on the open-source ISA (RISC-V), including IBEX dynamic simulation-based verification methods [8], [9], [10], 223 163 (with two di↵erent versions) [15], CVA6 [16] and CV32E40P [23], [25], [33], [34], [35], [36], [37], [38], [39]. The latter, 224 164 [20], which have been widely used in the studies on pro- which includes both directed and random testing, is an e↵ec- 225 165 cessor verification [9], [11], [21], [22], [23], [24], [25]. Our tive and widely used approach. In particular, directed testing 226 166 experimental results demonstrate the significant superiority is an engineer-driven approach in which testers carefully craft 227 167 of FLAME over the five typical or state-of-the-art baselines specific scenarios to validate known functionalities and verify 228 168 (including the widely-studied CRV and CGF techniques, one that features work as intended [36]. However, as designs 229 169 state-of-the-art processor fuzzing technique, and two state-of- become more complex, directed testing alone becomes costly 230 170 the-art LLM-based techniques). For instance, FLAME covered and insucient to cover all scenarios, especially those at the 231 171 34.25%⇠220% more target FPs than them on average across periphery of expected behaviors [36]. Random or constraint- 232 172 the four studied processor designs. Particularly, FLAME random testing, known as CRV, fills this gap by generating a 233 173 improves test validity by 28.13% compared to the state-of- large number of random inputs guided by specific constraints 234
LI et al.: FLAME: ENHANCING FUNCTIONAL COVERAGE IN PROCESSOR VERIFICATION VIA LLMs 3
235 to test the design. This technique allows a broader exploration RAG first retrieves relevant information from external sources, 297 236 of the design’s functionality space, making it e↵ective for which is then used to augment the original prompt, providing 298 237 complex designs [40]. However, CRV still leaves a portion the model with a richer context. Finally, the model generates 299 238 of FPs ineciently covered. a response with more accurate, contextually relevant, and 300 239 To alleviate this problem, numerous CGF methods have factually grounded outputs. By leveraging external informa- 301 240 been proposed [7], [8], [9], [10], [23], [38]. CGF is a well- tion, RAG significantly improves the performance of LLMs, 302 241 established verification technique that tests processors by particularly in tasks requiring specialized domain knowledge 303 242 adaptively generating random inputs and mutating seed inputs [49]. 304 243 to simulate unpredictable usage scenarios, guided by various Within the software domain, LLMs have shown particular 305 244 coverage metrics. This approach excels at exposing edge cases, promise in generating test cases [18], [50], [51], [52], [53], 306 245 making it particularly e↵ective at uncovering hidden design [54], [55]. By analyzing code and natural language descrip- 307 246 issues and improving reliability before production. However, tions, LLMs can understand programming structures and logic 308 247 because these traditional techniques have only limited interac- to generate test cases targeting specific functionalities and edge 309 248 tion with and understanding of the functional coverage model, cases. This automation helps ensure software meets functional 310 249 they often hit bottlenecks in functional coverage growth. These requirements more comprehensively than traditional random or 311 250 uncovered FPs represent the scenarios that are particularly manual testing approaches. 312 251 challenging to test comprehensively in processor design. Building on LLMs’ advances in software testing, 313 252 To improve functional coverage e↵ectively, several other researchers are now exploring their applications for processor 314 253 advanced approaches, especially learning-based approaches, verification [11], [12], [13], [14]. In general, they instructed 315 254 have been proposed [21], [22], [33], [41], [42], [43]. For exam- LLMs to understand the semantics of uncovered FPs and then 316 255 ple, Design2Vec [22] constructed control data flow graphs and generate test cases to cover them. In particular, Zhang et al. 317 256 learned their syntactic and semantic representations using a [11] proposed an LLM evaluation framework LLM4DV for 318 257 graph neural network (GNN), which in turn generates test processor verification that evaluates the e↵ects of di↵erent 319 258 cases based on this representation. LMT [42] improved the models and prompts on module-level designs. Xiao et al. 320 259 coverage capability at last-mile FPs by learning the relevant [12] proposed an LLM-based test method for processors 321 260 variables and constraints using random forest (RF) and gen- with di↵erent ISAs through zero-shot prompting. They 322 261 erative adversarial network (GAN). However, most of them demonstrated the method’s e↵ectiveness on both traditional 323 262 focus on the submodules or abstract representations of the processors and emerging domain-specific architectures. 324 263 processor design and are not applicable to the processor ChatFuzz [13], trained on large machine language datasets, 325 264 architecture itself. That is, these methods do not analyze the leverages LLMs to understand processor language and 326 265 impact of instruction inputs on the functional coverage from generate data/control flow entangled yet random machine 327 266 the processor-level view, making it dicult to migrate these code sequences to generate input instructions needed for 328 267 methods to the verification of the entire processor design [10]. processor verification. VerilogReader [14] used LLMs to 329 268 Particularly, the growing complexity of modern processors understand Verilog code and coverage, and generate test 330 269 and their diverse instruction sets makes improving functional cases for Verilog modules (module-level) through zero-shot 331 270 coverage an increasingly urgent challenge. prompting. 332 These LLM-based approaches demonstrate the promising 333 271 B. LLM-Based Processor Test Generation role of LLMs for processor verification, but simply instructing 334 272 LLMs have rapidly emerged as transformative tools across LLMs makes them face significant limitations in practice. 335 273 diverse domains, o↵ering impressive capabilities in natural In particular, these approaches struggle to improve func- 336 274 language and programming language understanding, code tional coverage e↵ectively and eciently due to the lack of 337 275 generation, and complex reasoning. Generic LLMs, such as in-depth semantic understanding between the functionalities 338 276 ChatGPT [44] and Llama 3 [45], have exhibited superiority to be explored and the input instructions. To address this, our 339 277 in a range of natural language processing tasks. Meanwhile, approach makes better use of ICL strategies by leveraging 340 278 code-focused LLMs such as CodeLlama [46] are developed RAG to improve the ability to model the instructions—FPs 341 279 to facilitate programming-related tasks. Various prompting mapping and utilizing the advanced CoT to improve the 342 280 strategies have been developed to invoke LLMs e↵ectively, like validity of the test cases generated by LLMs. 343 281 zero-shot prompting, few-shot learning [47], chain-of-thought 282 (CoT) [48], and RAG [49]. Zero-shot prompting allows the III. FLAME FRAMEWORK 344 283 model to perform a verification or test generation task solely Fig. 1 shows the overview of our novel technique FLAME 345 284 based on the given instruction, without any prior examples. for processor functional verification, which is dedicated 346 285 Few-shot learning, a representative form of in-context learning to covering more FPs automatically. FLAME is divided 347 286 (ICL), provides several representative input–output pairs as into three parts: knowledge base construction, LLM-assisted 348 287 demonstrations in the prompt, enabling the model to infer the test generation, and functional-coverage-guided feedback. 349 288 underlying task pattern and generate more accurate or con- FLAME begins by collecting extensive processor-design- 350 289 sistent outputs. CoT refers to a series of intermediate natural related information to build a comprehensive knowledge 351 290 language reasoning steps that lead to a final output, which base that provides essential background information. Then, 352 291 enables LLMs to provide more reliable answers by breaking FLAME uses the RAG technique to retrieve information 353 292 down their reasoning process into clear, sequential steps. RAG related to the target FPs from the established knowledge 354 293 enhances existing models by incorporating knowledge from base and generate high-quality test cases based on our 355 294 external databases through a common paradigm containing devised documents—instructions—programs CoT. Finally, a 356 295 three main steps: information retrieval, data augmentation, functional-coverage-guided feedback mechanism is utilized, 357 296 and final generation. Di↵erent from simple prompt looping, where the previously-generated test cases and coverage result 358
4 IEEE TRANSACTIONS ON COMPUTER-AIDED DESIGN OF INTEGRATED CIRCUITS AND SYSTEMS
functionality, and implementation, serving as the foun- 397
dational reference for understanding the design and FPs. 398
For example, the IBEX documentation [56] provides 399
information about the coverage plan. 400
2) FPs Definitions: This information contains the defini- 401
tions for FPs, documenting the design principles and 402
structures behind their implementation. It also outlines 403
the targets for verification, ensuring systematic test 404
coverage across relevant scenarios. For instance, the 405
definitions in [15] primarily record the IBEX micro- 406
architectural events and their behaviors. 407
3) ISA Specification: These specifications define the sup- 408
ported instructions, operational semantics, and encoding 409
formats—essential elements for generating valid and 410
meaningful test cases. For example, this documentation 411
[57] contains the instruction set for RISC-V, detailing 412
the usage of various instructions. 413
The collected processor-design-related information is sys- 414
tematically processed to facilitate ecient retrieval during 415
LLM-assisted test generation. For design documentation and 416
ISA specification, textual information is divided into chunks 417
359 Fig. 1. Overview of our proposed FLAME. using a structured segmentation pipeline that leverages section 418
boundaries, hierarchical headings, and token-length constraints 419
to preserve semantic coherence. A deterministic index is then 420
360 information could be provided to LLMs as a reference for automatically constructed through a hybrid rule-based and 421 361 the iterative generation process. Note that, due to the cost LLM-assisted linking process, where a controlled vocabulary 422 362 of LLMs, our technique is not applied to FPs that are easily of FP identifiers and instruction names is first generated, 423 363 addressed in practice, specifically those covered eciently by and an LLM refines and verifies associations between these 424 364 widely used CRV methods in our work following the existing identifiers and the most relevant documentation or specifi- 425 365 study [42]. In other words, FLAME focuses on addressing FP cation chunks identified by the keywords. Similarly, for FP 426 366 bottlenecks in order to achieve cost-e↵ectiveness. definitions, source code is segmented into chunks according 427 367 Note that fine-tuning LLMs entails substantial additional to the hierarchical structure of FPs and indexed with their 428 368 costs and may not generalize well across diverse architectures related information. This process culminates in the creation of 429 369 and processor designs. Hence, we focus on employing more a comprehensive knowledge base, which integrates processor 430 370 ecient prompting techniques through well-designed strate- design documentation, FP definitions, and ISA specification. 431 371 gies. While RAG and CoT are common prompting methods 372 in the area of LLMs, their e↵ectiveness is unexplored in the 373 context of processor verification, making their adoption in this B. LLM-Assisted Test Generation 432 374 challenging domain nontrivial. To address these challenges, we This phase consists of three stages: interpretation, analyza- 433 375 devised a novel approach by carefully leveraging the intrinsic tion, and generation, where the LLM serves as interpreter, 434 376 nature of processors rather than relying solely on simple analyzer, and generator, as shown in Fig. 1. In the first 435 377 logical reasoning. two stages, RAG is employed to retrieve external knowl- 436 edge from the constructed knowledge base (detailed in 437 378 A. Knowledge Base Construction Section III-A). This enhances the LLMs’ understanding of 438 target FPs and improves the semantic mapping between FPs 439 379 This phase involves collecting and encoding processor- and instructions, which facilitates addressing the challenges of 440 380 design-related information to ensure that the following test complex semantic conditions required for achieving nontrivial 441 381 generation process can leverage comprehensive and structured functional coverage and the extensive search space associated 442 382 information. Thoroughly exploiting specification information with input instructions. 443 383 related to target FPs is crucial for improving complex semantic To improve the validity of generated test cases, we specifi- 444 384 conditions in uncovered FPs and reducing instruction input cally devise a documents—instructions—programs CoT for the 445 385 space. This information helps reduce the LLM’s hallucination workflow of three stages: 1) the interpretation stage, which 446 386 threat when invoking RAG, enhancing the relevancy and will retrieve information about the target FPs and enrich 447 387 accuracy of generated responses. it with LLMs to get detailed information (documents) and 448 388 Three types of information related to processor design instruction types; 2) the analyzation stage, which will retrieve 449 389 were collected for knowledge base construction, as shown information about the instructions and analyze it by LLMs 450 390 in Fig. 1: processor design documentation, FPs definitions, based on the functionalities and behaviors of the target FPs 451 391 and ISA specification. The first two help LLMs understand to get the specific instruction usages (instructions) that cover 452 392 the processor design under test, while the ISA Specification the FPs; and 3) the generation stage, which will generate a C 453 393 helps LLMs understand input instructions. Their details are program (programs) with the corresponding semantic structure 454 394 described as follows. based on the instruction usages from the analyzation Stage. 455 395 1) Processor Design Documentation: These documents This prompting technique exploits the logical reasoning ability 456 396 provide descriptions of the processor’s architecture, of LLMs, thereby improving their performance in test case 457
LI et al.: FLAME: ENHANCING FUNCTIONAL COVERAGE IN PROCESSOR VERIFICATION VIA LLMs 5
458 generation for processor functional verification. Below, we will tailored as an input for target FPs. This test case is designed to 521 459 detail the three stages, respectively. evaluate target FPs comprehensively by exercising the detailed 522 460 1) Interpretation: At this stage, FLAME first retrieves behaviors, edge cases, and specific conditions identified in 523 461 relevant information from the design documentation and FPs previous stages. 524 462 definitions of the target FPs to help the LLM better understand 463 the context. Retrieval is performed through exact keyword C. Functional-Coverage-Guided Feedback 464 lookup using the prebuilt deterministic index constructed 525 465 during the knowledge base construction stage. Since exact To enhance test eciency and achieve comprehensive func- 526 466 matching alone can fail due to naming inconsistencies or tional coverage, FLAME implements a functional-coverage- 527 467 incomplete matches in the ISA specification and design doc- guided feedback mechanism. This mechanism involves 528 468 umentation, FLAME applies Levenshtein distance [58] to executing test cases (including compilation and simulation), 529 469 resolve multiple near matches or naming variants and select collecting functional coverage data, selecting target FPs, and 530 470 the most similar documentation chunks. In contrast to cosine utilizing counterexamples to refine the testing process. We now 531 471 similarity, which depends on semantic representations and outline the involved key components. 532 472 thus performs poorly on short, lexically similar identifiers 1) Compilation and Simulation: To verify the e↵ectiveness 533 473 [59], Levenshtein distance operates directly on token-level of the test cases generated in the prior phase, FLAME 534 474 edit operations, making it particularly e↵ective for short FP simulates them on the processor designs. Initially, the 535 475 and instruction names that di↵er only by a few characters. generated C programs are compiled into instruction 536 476 This retrieval mechanism leverages the o✏ine-constructed sequences using an ISA-specific compiler (e.g., RISC- 537 477 index to ensure accurate and consistent mapping between V GCC for RISC-V ISA processors). These instruction 538 478 FPs, instruction names, and their corresponding documentation sequences are then fed into the processor for simulation 539 479 segments, while avoiding the computational cost and stochastic using commercial electronic design automation (EDA) 540 480 variability of on-the-fly embedding models. These contents, tools like Synopsys VCS [60]. 541 481 along with the target FPs, are then fed into the LLM for 2) Functional Coverage Collection: To analyze the func- 542 482 deeper interpretation and improved understanding of the FPs. tional coverage results and guide subsequent test gener- 543 483 This stage strengthens the semantic understanding between the ation, FLAME collects functional coverage information 544 484 target FPs and their associated design documentation, serving after simulation. During this process, the aforementioned 545 485 as a fundamental component. EDA tools generate detailed functional coverage reports, 546 486 2) Analyzation: In the second stage, FLAME first segments which are automatically parsed to extract functional cov- 547 487 the LLM’s response generated in the first stage, and extracts erage results. The collected data is analyzed to identify 548 488 the keywords related to the instruction types as candidate uncovered FPs. 549 489 instructions based on the instruction types defined in the 3) Target FP Selection: To refine the testing process 550 490 ISA. For each candidate, retrieval from the ISA specification progressively for achieving comprehensive functional 551 491 follows the same procedure as in the interpretation stage: coverage, FLAME establishes a rule for selecting target 552 492 FLAME first performs exact keyword lookup using the pre- FPs for the next iteration. In particular, FLAME first 553 493 built deterministic index constructed during the knowledge calculates the proportion of uncovered FPs within each 554 494 base construction stage, and when multiple near matches functional group (defined by design developers based 555 495 or minor naming variants exist, applies Levenshtein distance on functionality relevance as presented in Section II-A). 556 496 [59] to select the most similar specification chunks. The The FGs are then sorted based on this proportion, prior- 557 497 retrieved contents, which contain the correct usage of the target itizing the group with the highest number of uncovered 558 498 instructions, are combined with the detailed FPs information FPs. To summarize, after each iteration, the current 559 499 generated in the first stage. The combined information is then target functional group is moved to the end of the 560 500 fed back into the LLM for deeper analysis and improved queue, and the remaining groups are recalculated and 561 501 understanding of the instructions. By synthesizing the LLM’s resorted based on their uncovered point proportions. 562 502 initial response with the retrieved ISA instruction specifi- This strategy prevents repetitive exploration of a single 563 503 cations, the LLM can analyze the target FPs with a focus functional group and ensures that subsequent iterations 564 504 on identifying key attributes, e.g., the specific instruction target new areas, thus improving coverage in a structured 565 505 types, possible combinations, and applicable operation modes and ecient manner. 566 506 associated with the target points. The outcome of this stage 4) Counterexample Utilization: The feedback mechanism 567 507 is refined information on instruction types, parameters, and plays a crucial role by incorporating coverage informa- 568 508 usage scenarios for the target FPs. This enables the LLM to tion from previously-generated test cases. In particular, 569 509 comprehend how the target FPs interact within the design’s to make use of past test cases and provide valuable 570 510 functional context and establish a precise semantic mapping. insights for future iterations, FLAME selects the test 571 511 3) Generation: In the third stage, FLAME incorporates the cases from the history that can cover FPs within the same 572 512 specific instruction details obtained in the second stage to functional group as the target uncovered FPs. These 573 513 generate an e↵ective test case at the source code level (i.e., a selected test cases act as counterexamples, encouraging 574 514 C program that can be compiled to instructions for processor the LLM to generate more diverse test cases. 575 515 verification). Note that FLAME chooses the source code level In addition, FLAME records information from previously- 576 516 rather than the instruction level in order to improve test generated invalid test cases, including error messages and 577 517 validity, since LLMs have demonstrated superior performance related statements. This information is also utilized as coun- 578 518 on source code generation [17], [18], [19]. By integrating terexamples during the generation process, helping the LLM 579 519 instructions with the syntactic and operational properties of the avoid generating test cases with similar errors. The counterex- 580 520 C language, the LLM will produce a test case that is explicitly amples information is then integrated into the generation stage 581
6 IEEE TRANSACTIONS ON COMPUTER-AIDED DESIGN OF INTEGRATED CIRCUITS AND SYSTEMS
582 TABLE I
583 BASIC INFORMATION ABOUT STUDIED PROCESSOR DESIGNS
584 in the form of few-shot learning as shown in Fig. 1. By
585 iteratively incorporating these counterexamples, the LLM is
586 guided to produce more diverse and less error-prone test cases,
587 enhancing the overall testing eciency and e↵ectiveness.
Fig. 2. Functional coverage growth under CRV. 635
588 IV. EXPERIMENTAL SETUP
589 We formulate three research questions to evaluate FLAME. Their research showed that these FPs can be identified by 636
590 1) RQ1: How do di↵erent LLMs and test case formats analyzing the growth curve of functional coverage. Following 637
591 perform in FLAME? their work, we performed the widely used CRV method [37] 638
592 2) RQ2: How does FLAME perform compared to exist- (detailed in Section IV-C) on each of the four processor 639
593 ing test generation techniques in processor functional designs to observe the growth of FPs. When the iteration 640
594 verification? of CRV comes to 100, the growth curve of the four studied 641
595 3) RQ3: How does each key component of FLAME con- processor designs becomes nearly flat, as confirmed in Fig. 2. 642
596 tribute to the overall e↵ectiveness? More specifically, from this figure, the choice of 100 itera- 643
tions demonstrates a tradeo↵ between the LLM overhead and 644
597 A. Studied Subjects overall performance. Using too few iterations may result in 645
598 1) Processor Designs Under Test: To evaluate the per- unnecessary LLM overhead (which can be addressed by the 646
599 formance of FLAME in improving functional coverage, it CRV method), while too many iterations only yield marginal 647 600 is essential that the processor designs under test include improvements. Accordingly, we ran CRV for 100 iterations 648 601 functional coverage models. These models define FPs and per processor (a consistent budget across the four designs for 649 602 groups, enabling the measurement of functional coverage generalizability) and took the uncovered FPs at the end of this 650 603 improvements. Based on this requirement, we collected all the run as the target set for FLAME. In other words, after CRV 651 604 open-source processor designs used in prior work on improv- reaches its budgeted saturation, the CRV-covered points (e.g., 652 605 ing functional coverage [11], [21], [22], including IBEXv1, 6072 FPs on CVA6) are excluded, and the remaining points 653 606 IBEXv2, CVA6, and CV32E40P. are defined as the target FPs for subsequent evaluation. 654 607 IBEX is a production-quality, open-source 32-bit RISC-V B. Studied LLMs 655 CPU core that supports various extensions [15]. It has under- 608 gone extensive verification and multiple tape-outs, making it We selected four popular LLMs, considering source 656 609 a reliable benchmark for evaluating verification methods on availability (open-source versus closed-source) and their spe- 657 610 real-world designs [11], [21]. We used two versions with cialization (general-purpose versus code-specific). 658 611 612 diverse complexity of functional coverage models (IBEXv1 1) GPT-3.5-Turbo (GPT3.5) is one of the influential 659 613 and IBEXv2). CVA6 is a 64-bit RISC-V CPU, which imple- closed-source LLMs, trained on large amounts of natural 660 614 ments the RISC-V ISA including integer, multiplication, and language text and code snippets, with reinforcement 661 615 division, atomic and compressed extensions, as well as three learning to follow human instructions [44]. We did not 662 616 privilege levels [16], o↵ering insights into verifying high choose the latest versions of ChatGPT (i.e., GPT-4 and 663 617 complexity designs [9], [23]. CV32E40P is an industrially- GPT-4o) due to their high cost, and because FLAME do 664 618 supported open-source RISC-V core that implements base not require their extended context window. Nevertheless, 665 619 integer instructions, featuring both a hardware multiplier and GPT-3.5-Turbo has shown superiority in various tasks 666 620 compressed instruction support. [61], [62], [63]. 667 621 The diversity in design scale, architectural complexity, and 2) CodeLlama is one of the most popular LLMs for code 668 622 feature sets across these processors ensures a comprehensive generation and infilling derived from Llama 2 models 669 623 assessment of FLAME in addressing broad verification chal- [64]. In particular, we studied the two sizes: Codellama 670 624 lenges. Table I shows the statistical summary of the studied 7B (CL7B) and Codellama 13B (CL13B). 671 625 processor designs. The three columns in this table represent 3) Phind CodeLlama (PCL34B) is from Phind models, the 672 626 the number of FGs, the number of FP, and the number of lines fine-tuned versions of CodeLlama-34B, leveraging an 673 627 of code (LOC) for each design. For example, CVA6 contains internal Phind corpus and instruction-tuning on 1.5B 674 628 the most FPs (23 734), while CV32E40P contains the least tokens of high-quality programming data [65]. This 675 629 (775). makes it a state-of-the-art open-source model for code 676 630 2) Target FPs: Inspired by the existing work [42], we generation. 677 631 also focused on the FPs that remain uncovered when random 632 test case generation reaches a functional coverage growth C. Baselines 678 633 bottleneck. As presented in Section III, FLAME focuses on To comprehensively evaluate the e↵ectiveness of FLAME, 679 634 addressing FP bottlenecks for achieving cost-e↵ectiveness. we compared its performance with five existing test generation 680
LI et al.: FLAME: ENHANCING FUNCTIONAL COVERAGE IN PROCESSOR VERIFICATION VIA LLMs 7
681 techniques, covering a traditional CRV technique, a coverage- E. Implementation and Environment 742
682 guided technique, a latest fuzzing technique, and two latest 1) Studied LLMs: The open-source LLMs, i.e., Phind 743
683 LLM-based techniques. CodeLlama (Phind-CodeLlama-34B-v2 version) [65] and 744
684 1) CRV is the kind of constrained random verification CodeLlama (CodeLlama-13b-Instruct-hf and CodeLlama- 745
685 method, which is the most widely used in industry 7b-Instruct-hf versions) [46], were downloaded from 746
686 and academia [3], [6], [40]. In our verification sce- Huggingface, while the closed-cource GPT-3.5-Turbo 747
687 nario, we choose the most-studied random instruction (gpt-3.5-turbo-0125 version) was accessed through its ocial 748
688 generator RISCV-DV as an implementation, which is an API [44]. To balance diversity and coherence in the generated 749
689 open-source instruction generator for RISC-V processor test cases, following the existing work [67], [68], [69], 750
690 verification [37]. RISCV-DV requires configuration to [70], and considering the exploratory nature of our task, 751
691 use (we used its default configuration in our experiments we set the temperature to 0.7 for these LLMs. The balance 752
692 following the existing work [9], [38]) and generates is critical for test generation tasks: we expect the LLM to 753
693 constrained random test cases based on the constraints produce correct outputs, yet also explore diverse edge-case 754
694 of its internal implementation. scenarios that enhance coverage. The studied LLMs have a 755
695 2) DifuzzRTLfca is our adapted version of DifuzzRTL [8], maximum context length of 16K tokens. The prompts for 756
696 widely studied CGF method.1 In our adaptation, LLMs employ universally recognizable delimiters, including 757
697 based on our target scenario, we used the desired symbolic markers and structured field labels, which are 758
698 functional coverage as the guidance for the mutation naturally interpretable by di↵erent model architectures. 759
699 process for fair comparison, thus mutating the instruc- These delimiters clearly separate prompt segments and 760
700 tion sequences generated by the CRV and generating maintain a consistent logical structure across models. Since 761
701 instruction sequence mutants as input to the processor modern LLMs are well trained to recognize such structural 762
702 design. keywords, this format ensures reliable parsing and consistent 763
703 3) Cascade is the latest processor fuzzing technique pro- behavior without requiring model-specific adaptations. All 764
704 posed by Solt et al. [25]. Due to the CGF’s limitation the prompt templates can be found on our homepage 765
705 of generating too simple test cases, Cascade constructs [26]. 766
706 test cases with more complex data and control flows by 2) Studied Processor Designs: For the processor-design- 767
707 utilizing the instruction set simulator (ISS) to entangle related information in knowledge base construction, we 768
708 the control flow with the randomized data flow during collected the documentation of processor designs (including 769
709 test case generation. Cascade outputs binary format IBEX CPU, CVA6, and CV32E40P), and the FPs defini- 770
710 (ELF) programs as input to the processor design. tions in their functional coverage model, as well as their 771
711 4) LLM-PVISA and LLM-PVFunc, denoted by us, are the ISA specification (i.e., RISC-V Instruction Set Manual). 772
712 two latest LLM-based processor functional verification The targeted RISC-V ISA is RV32IMC by default [15], 773
713 strategies proposed by Xiao et al. [12]. LLM-PVISA [16], [20]. 774
714 takes the functional description of each ISA instruction 3) Baselines: We implemented DifuzzRTLfc based on 775
715 implemented in the target processor as the prompt to the DifuzzRTL implementation by replacing its original 776
716 LLMs, while LLM-PVFunc provides LLMs with specific coverage guidance with our target functional coverage. 777
717 functionality of the target processor. The output of both We adopted the open-source Cascade implementation and 778
718 LLM-PVISA and LLM-PVFunc is C programs. extended it to our studied processor designs. We implemented 779
719 In addition, there are other LLM-based processor verifi- the LLM-PVISA and LLM-PVFunc methods based on their 780
720 cation techniques, including LLM4DV [11], VerilogReader paper [12]. 781
721 [14], and ChatFuzz [13]. However, LLM4DV and Verilo- More detailed environment specifications are available 782
722 gReader primarily focus on module-level processor verifica- on our homepage [26] upon request for replication 783 723 tion, whereas our scenario involves full processor design at the studies. 784 724 architecture level. Furthermore, ChatFuzz requires extensive 725 training datasets, which are unavailable for reproduction. As a V. RESULTS AND ANALYSIS 785 726 result, these methods are not well-suited to our scenario and 727 were not included as baselines in our study. A. RQ1: Influence of LLMs and Test Formats 786 1) Validity Comparison of Test Case Formats: 787 728 D. Evaluation Metrics a) Process: A processor receives input as a sequence of 788 instructions, typically consisting of assembly code. High-level 789 729 We evaluated the performance of FLAME and baselines programming languages (e.g., C programs) can be compiled 790 730 using these commonly used metrics [8], [9], [10], [11], [21], into these instruction sequences. For processor verification, 791 731 [42], [66]. we have two choices: either generating instruction sequences 792 732 1) Functional Coverage: The number of target FPs covered directly or creating C programs and then compiling them 793 733 during verification, measuring e↵ectiveness. into assembly code as part of the verification process. The 794 734 2) Pass Rate: The ratio of successfully executed tests choice of test case format may influence how LLMs inter- 795 735 (passing the compilation and simulation) to the total pret and process information. Therefore, we investigated the 796 736 number of generated test cases, reflecting test validity. e↵ect of the two test case formats on all four studied LLMs 797 737 3) Time Reduction: The reduction in time when achieving (i.e., GPT3.5, CL7B, CL13B, and PCL34B) across the 798 738 the same functional coverage, measuring eciency. four processor designs (i.e., IBEXv1, IBEXv2, CVA6, and 799 739 1We did not adapt the state-of-the-art CGF technique, i.e., MorFuzz [9], CV32E40P). 800 740 since it requires significant e↵ort to apply it to our studied processors through For each LLM and processor design, we used the fixed 801 741 communications with its authors. prompt template and varied only the command text to steer 802
8 IEEE TRANSACTIONS ON COMPUTER-AIDED DESIGN OF INTEGRATED CIRCUITS AND SYSTEMS
803 TABLE II
804 PASS RATE COMPARISON OF TEST CASE FORMATS
805 generation toward either C programs or instruction sequences.
806 For example, for instruction-sequence generation, we used
807 the same template as C-program generation but replaced “C
808 programs” with “instruction sequences.” For each setup, we
809 generated an equal number (500) of C programs and instruc-
810 tion sequences as test cases of the processor designs. The pass
811 rate metric was evaluated as the proportion of test cases that
812 successfully passed both compilation and simulation.
813 b) Results: Table II presents the average pass rates
814 for generating C programs and instruction sequences using
815 various LLMs across the four processor designs. As observed, Fig. 3. Functional coverage comparison across LLMs. 861
816 generating C programs consistently achieves higher pass rates TABLE III 862
817 compared to generating instruction sequences for all the stud- NUMBER OF VALID TEST CASES GENERATED BY LLMS 863
818 ied LLMs. For instance, the average pass rates for instruction
819 sequences range from only 1.63%–11.80% across the studied
820 processor designs. In contrast, the average pass rates for
821 C programs range from 67.40% to 76.60%. Our manual
822 investigation of the LLM-generated instructions and execution
823 information revealed that the primary reason for the low pass
824 rates of instruction sequences is their higher likelihood of TABLE IV 864
containing incorrect syntax or unstructured assembly code.
825 This, in turn, results in a higher probability of compilation PASS RATE COMPARISON OF DIFFERENT LLMS 865
826
827 and simulation failures. The results further confirm the per-
828 formance of LLMs in generating high-level source code (C
829 programs here) due to training on a considerable amount of
830 source code data. In the future, incorporating assembly code
831 for fine-tuning LLMs may improve the validity of generated
832 instruction sequences. Based on current results, we selected C
833 programs as the test case format for subsequent experiments.
834 2) E↵ectiveness Comparison of LLMs: and CL7B (967 on CVA6) in just 13.50, 9.29, and 6.98 h, 866
835 a) Process: To examine the impact of LLMs on the respectively. This represents time reductions of 71.88%, 867
836 e↵ectiveness of FLAME, the same prompt was used for all 80.65%, and 85.46%, highlighting the superior eciency of 868 837 four LLMs, and the test cases were generated in C programs. PCL34B in these scenarios. 869 838 All experiments were performed on the same machine under Table III shows the number of valid test cases generated 870 839 identical conditions, with a fixed duration of 48 h, encompass- by di↵erent LLMs. Among the LLMs, GPT3.5 generates the 871 840 ing both test case generation and simulation. We analyzed the largest number of valid test cases (e.g., 1087 and 995 for 872 841 number of target FPs covered by the generated test cases and IBEXv1 and CV32E40P). However, the number of valid test 873 842 measured the pass rate to evaluate test validity. cases generated does not directly correlate with improvements 874 843 b) Results: Fig. 3 presents the comparative results across in functional coverage. For instance, the PCL34B model does 875 844 di↵erent LLMs in terms of the number of covered target func- not generate or execute the largest number of valid test cases 876 845 tional points. Among the four LLMs, the open-source PCL34B on any of the processor designs, yet it achieves the most 877 846 demonstrates the best performance (except on IBEXv2, where significant improvement in functional coverage e↵ectiveness. 878 847 all four LLMs perform similarly), covering the most target Table IV further presents the performance of LLMs in terms 879 848 functional points, within the 48-h time budget. It indicates of pass rate. The results show that the test cases generated by 880 849 that PCL34B outperforms the other open-source Code Llama- PCL34B achieved the highest average pass rates among the 881 850 series models (i.e., CL13B and CL7B) and the closed-source four processor designs, with rates of 64.62%, 43.89%, 39.48%, 882 851 GPT3.5 across most processor designs, because PCL34B was and 59.21%, respectively. Considering the tradeo↵ between 883 852 fine-tuned on the Phind dataset and achieved 73.8% pass@1 e↵ectiveness (functional coverage and pass rate) and eciency 884 853 on HumanEval (outperforming GPT-4 and GPT-3.5) [65], (number of valid test cases generated), we chose the best- 885 854 highlighting the importance of fine-tuning LLMs specifically performing PCL34B model for subsequent experiments. 886 855 on high-quality programming-related data. RQ1 Summary: Key results show that generating test cases 887 856 In terms of time eciency, PCL34B achieves the maximum at the source code level (i.e., C programs) using LLMs 888 857 functional coverage obtained by the other models within sig- achieves a higher pass rate. In addition, PCL34B achieves 889 858 nificantly shorter durations. For instance, on IBEXv1, CVA6, the best tradeo↵ among the four LLMs, reaching the highest 890 859 and CV32E40P, PCL34B matches the maximum coverage number of FPs within the controlled time and reducing the 891 860 achieved by GPT3.5 (56 on IBEXv1 and 10 on CV32E40P) time to reach the same number by up to 85.46%. 892
LI et al.: FLAME: ENHANCING FUNCTIONAL COVERAGE IN PROCESSOR VERIFICATION VIA LLMs 9
TABLE V 935
NUMBER OF VALID TEST CASES GENERATED BY TECHNIQUES 936
TABLE VI 937
PASS RATE COMPARISON OF FLAME AND BASELINES 938
893 Fig. 4. Functional coverage comparison between techniques. TABLE VII 939
TOKENS REQUIRED FOR LLM-BASED TECHNIQUES 940
894 B. RQ2: Comparison Between FLAME Versus Existing
895 Techniques
896 1) Process: To compare FLAME with existing processor
897 verification techniques, we selected five typical or state-of- 898 the-art techniques as baselines, including CRV, DifuzzRTLfc, 899 Cascade, LLM-PVISA, and LLM-PVFunc, as detailed in 900 Section IV-C. We used the optimal experiment setting based 901 on the conclusion from Section V-A, i.e., we chose PCL34B 902 as the LLM and used C programs as the format of test case diculty of our target problem. Similarly, as presented in the 941 903 generation (for both FLAME and the two LLM-based base- existing work [25], the type of functional coverage we target 942 904 lines). Each technique was executed under identical conditions is more sophisticated, requiring significantly greater e↵ort to 943 905 on the same set of target FPs, with a fixed time budget of 48 enhance. The results show that although these existing pro- 944 906 h/run. We calculated the number of covered target FPs and the cessor fuzzing techniques can improve other coverage metrics 945 907 pass rate of generated test cases to assess their e↵ectiveness (e.g., multiplexer coverage and control register coverage) [8], 946 908 and validity. [25], they do not necessarily yield similar improvements in 947 909 2) Results: Fig. 4 shows the number of covered target FPs functional coverage. 948 910 achieved by each technique over time, highlighting the acceler- Tables V and VI present the number of valid test cases and 949 911 ated coverage gains obtained through FLAME. For the studied the pass rate for each studied technique. As shown in Table V, 950 912 processor designs, FLAME achieved the best functional cov- FLAME generates a relatively smaller number of valid test 951 913 erage (i.e., the largest number of target FPs), outperforming cases across the four studied processor designs compared to 952 914 the best baseline (i.e., LLM-PVISA on IBEXv1, LLM-PVFunc most of the baselines (except Cascade). Notably, for IBEXv1 953 915 on IBEXv2, LLM-PVFunc on CVA6, and LLM-PVFunc on and IBEXv2, FLAME achieved the largest number of FPs 954 916 CV32E40P) by 135.14%, 34.25%, 64.67%, and 220.00%, using the smallest number of test cases (except Cascade), 955 917 respectively. Also, FLAME achieved the maximum coverage with 554 and 178 test cases, respectively. Cascade aims to 956 918 obtained by these baselines within significantly shorter times: generate significantly more complex test cases [25], which 957 919 6.66, 34.27, 8.52, and 6.93 h for the four processor designs, consequently require substantially longer execution times. As 958 920 representing time reductions of 86.13%, 28.60%, 82.25%, and a result, within the fixed time budget of our experiments, 959 921 85.56%, respectively. These results indicate that our proposed all evaluated processors were able to execute only a limited 960 922 technique, FLAME, demonstrates superior e↵ectiveness and number of valid test cases produced by Cascade. 961 923 eciency in addressing bottleneck FPs. For the validity of test case generation, the test cases 962 924 The existing processor fuzzing techniques, i.e., DifuzzRTLfc generated by the CRV method (RISCV-DV) and Cascade 963 925 and Cascade, have demonstrated limited e↵ectiveness in are all correct (100% in pass rate), due to the reliance on 964 926 improving functional coverage. A likely reason is their inabil- manual configuration as well as the rule-based nature. The 965 927 ity to capture the high-level design intent underlying FPs. As DifuzzRTLfc is also easier to generate valid test cases, due to 966 928 a result, the test cases they generate often fail to exercise the the fact that they were mutated from valid seeds of CRV. The 967 929 specific instructions and combinations associated with these pass rate of DifuzzRTLfc is lower than that of CRV because the 968 930 FPs. Particularly, our study focuses on the FPs that remain mutation process leads to the creation of incorrect test cases. 969 931 uncovered when random test generation reaches a plateau in Nonetheless, as shown in Table VI, FLAME achieves 970 932 functional coverage growth, indicating that these uncovered the highest pass rate for generating test cases across 971 933 points are much more challenging. The poor performance all processor designs when compared to the other two 972 934 of these studied baselines on them further underscores the LLM-based techniques (LLM-PVISA and LLM-PVFunc), with 973
10 IEEE TRANSACTIONS ON COMPUTER-AIDED DESIGN OF INTEGRATED CIRCUITS AND SYSTEMS
974 pass rates of 69.42%, 67.68%, 52.86%, and 68.53%. While
975 LLM-PVISA demonstrates a pass rate slightly lower than
976 FLAME on IBEXv1, IBEXv2, and CV32E40P, its perfor-
977 mance drops significantly on CVA6 (the most complicated
978 processor design), with a pass rate of only 5.42%. On the other
979 hand, LLM-PVFunc shows even lower pass rate performance
980 on IBEXv1, IBEXv2, and CV32E40P, and its pass rate on
981 CVA6 is only 26.13%. These findings suggest that FLAME is
982 comparatively capable of generating a smaller number of test
983 cases while maintaining relatively high validity. Particularly,
984 FLAME is a↵ected less by the complexity of processor designs
985 in terms of pass rate than the other two LLM-based techniques.
986 We further assessed the token usage of all LLM calls to
987 gain an understanding of the computational and monetary
988 costs. Table VII reports the average and maximum number
989 of tokens per iteration for FLAME and the LLM-based base-
990 lines. Because FLAME employs a three-stage CoT prompt
991 construction, it naturally incurs higher token consumption than
992 baseline methods that rely on nearly zero-shot prompting. Nev-
993 ertheless, its average token usage (4.9K per iteration) remains Fig. 5. Functional coverage comparison across variants. 1036
994 modest relative to modern inference capacities (16K con- TABLE VIII
995 text window for LLMs). The additional token cost primarily 1037
996 reflects the deeper reasoning context encoded in the multistage NUMBER OF LOC OF FP DEFINITIONS IN PROCESSOR DESIGNS 1038
997 prompts, which in turn yields higher-quality test generation 998 and improved coverage (up to 220.00%), demonstrating that 999 FLAME achieves a favorable balance between token eciency 1000 and coverage e↵ectiveness. 1001 RQ2 Summary: FLAME demonstrates the best functional 1002 coverage improvement compared to the baselines and signifi- 1003 cantly reduces the time required to achieve the same functional Their performance was assessed using two key metrics: 1004 coverage, with a reduction of up to 86.13%. Furthermore, functional coverage and pass rate of generated test cases. This 1039 1005 FLAME tends to generate a smaller number of test cases while systematic analysis reveals the impact of each component on 1040 1006 maintaining relatively high validity. the overall e↵ectiveness and validity of FLAME and assesses 1041 1042
used industry baseline. 1044
1007 C. RQ3: Contributions of Key Components in FLAME whether the ablated versions still outperform the commonly 1043
1008 1) Process: FLAME consists of two main components: 1) 2) Results: Fig. 5 presents the comparison results between 1045
1009 the constructed knowledge base, which is used by RAG to FLAME and its five variants. The results show varying 1046 1010 retrieve context information of target FPs during the test gener- degrees of decline after the removal of individual compo- 1047 1011 ation process, and 2) the functional-coverage-guided feedback nents; however, these variants still outperform CRV (except 1048 1012 mechanism, which enhances testing eciency. The constructed FLAMEnoDOC on CV32E40P). In particular, the removal of 1049 1013 knowledge base is derived from three key sources: 1) processor certain components has the most significant negative impact 1050 1014 design documentation; 2) functional points definitions; and on functional coverage improvement for di↵erent processor 1051 1015 3) ISA specification. The functional-coverage-guided feedback designs. For instance, removing the ISA specification results 1052 1016 mechanism comprises two primary components: 4) feedback in a decrease of 36 FPs on IBEXv1, while the removal 1053 1017 functional coverage information and 5) previously-generated of counterexamples leads to a reduction of 110 FPs on 1054 1018 test cases as counterexamples. These five components [1)⇠5)] IBEXv2. Similarly, removing the feedback mechanism causes 1055 1019 collectively form the overall representation and functionality a loss of 673 FPs on CVA6, and the removal of processor 1056 1020 of FLAME, enabling it to improve the e↵ectiveness and design documentation decreases FPs by 16 on CV32E40P. 1057 1021 eciency of processor functional verification. As shown in Table VIII, we observe that CV32E40P contains 1058 1022 To evaluate the contributions of these key components significantly fewer lines of code (LOC) in its FP definitions 1059 1023 in FLAME, we constructed five variants by systematically (14.84 on average and 36.00 on maximum), likely provid- 1060 1024 removing one component at a time and measuring the ing limited contextual information for LLMs’ interpretation. 1061 1025 performance, and additionally included the widely used CRV Consequently, the design documentation plays a more critical 1062 1026 baseline for practical comparison. 1) FLAMEnoDOC (which role in conveying the semantics and relationships of FPs 1063 1027 removed the processor design documentation information and their corresponding instructions. When this supplementary 1064 1028 from FLAME); 2) FLAMEnoDEF (which removed the FPs documentation is removed, the LLM lacks sucient context 1065 1029 definitions information from FLAME); 3) FLAMEnoISA (which to accurately interpret the target FPs, potentially leading to a 1066 1030 removed the ISA specification information from FLAME); more pronounced drop in functional coverage. This indicates 1067 1031 4) FLAMEnoFB (which removed the functional-coverage- di↵erent qualities of various kinds of information within 1068 1032 guided feedback mechanism from FLAME); and di↵erent designs and also demonstrates the stable performance 1069 1033 5) FLAMEnoCE (which removed the previously-generated of FLAME across diverse process designs. 1070 1034 test cases as counterexamples from FLAME). The other Tables IX and X present the number of valid test cases 1071 1035 experiment settings were the same as those in Section V-B. generated by FLAME and its variants, along with their cor- 1072
LI et al.: FLAME: ENHANCING FUNCTIONAL COVERAGE IN PROCESSOR VERIFICATION VIA LLMs 11
1073 TABLE IX
1074 NUMBER OF VALID TEST CASES GENERATED BY FLAME AND VARIANTS
1075 TABLE X Fig. 6. Extended experimental results. (a) Functional coverage comparison 1115
1076 PASS RATE COMPARISON OF FLAME AND VARIANTS between FLAME and LLM-PVFunc on IBEXv2 in 96 h. Functional coverage 1116
comparison between FLAME using GCC and LLVM as compiler on CVA6. 1117
LLM-PVFunc continue to increase the number of covered target 1118
FPs within the 48-h window on IBEXv2, suggesting that this 1119
budget may be insucient to assess saturation or relative 1120
performance. To investigate whether FLAME maintains its 1121
advantage beyond the time limit, we extended the verification 1122
process to 96 h [see Fig. 6(a)], where the baseline reached clear 1123
1077 responding pass rates. As shown in Table IX, FLAME and saturation. While eventual convergence cannot be ruled out 1124 1078 its variants generate a comparable number of test cases across totally, FLAME not only sustained stable growth but also con- 1125 1079 the four studied processor designs. Table X demonstrates that tinued to outperform LLM-PVFunc, ultimately achieving higher 1126 1080 the variants FLAMEnoISA and FLAMEnoCE exhibit significant final functional coverage. This demonstrates FLAME’s ability 1127 1081 declines in pass rates. For IBEXv1, IBEXv2, and CV32E40P, to eciently exploit additional verification time, maintaining 1128 1082 the removal of the ISA specification has the most substantial consistent growing progress. 1129 1083 negative impact on pass rates, demonstrating the importance 1084 of learning correct usages from specifications in general. B. Generality Across Compiler Environments 1130 1085 Conversely, on CVA6, the removal of counterexamples results Another factor that may influence FLAME’s performance 1086 in the largest reduction in the pass rate, indicating the necessity is the choice of compiler, as the LLM-generated test pro- 1131 1087 of learning to generate valid test cases from counterexam- grams are written in C and compiled into executable binaries 1132 1088 ples for the most complicated processor design. Although before simulation. Di↵erent compiler toolchains (e.g., GCC 1133 1089 FLAMEnoDOC, FLAMEnoDEF, and FLAMEnoFB generated rel- and LLVM) may optimize instruction generation di↵erently, 1134 1090 atively more valid test cases than FLAME on CVA6 and 1135 1091 CV32E40P, we observe that the test cases produced by these potentially a↵ecting instruction distribution and thus impacting 1136 1092 variants are overly simplistic, resulting in a smaller number of functional coverage. To evaluate whether such di↵erences 1137 1093 achieved target FPs. materially influence the FLAME’s performance, we conducted 1138 1094 RQ3 Summary: The experiments show that each com- an additional experiment using the CVA6 processor as a rep- 1139 1095 ponent of FLAME has an important role in the overall resentative case, comparing results obtained from the LLVM 1140 1096 framework’s e↵ect on functional coverage enhancement. In toolchain with those from GCC-based compilation. As shown 1141 1097 addition, the two modules ISA specification and counterexam- in Fig. 6(b), both configurations produced highly consistent 1142 1098 ples in FLAME contribute the most to the overall pass rate of coverage curves, with FLAMEGCC and FLAMELLVM achiev- 1143 1099 test case generation. ing 1319 and 1367 covered FPs within 48 h, respectively, 1144 which has only a di↵erence of less than 4%, suggesting that 1145
compiler-specific code generation details. This demonstrates 1147
1100 VI. DISCUSSION the coverage achieved by FLAME is largely insensitive to 1146
1101 To gain deeper insights into the robustness and extensi- the strong generality of FLAME across compilation envi- 1148
1102 bility of FLAME, we designed two complementary analyses ronments, reinforcing its robustness in practical verification 1149 1103 addressing its temporal scalability and environmental general- pipelines that may employ di↵erent toolchains. 1150 1104 ity, which often vary across scenarios in practice. In particular, 1105 we 1) examined how FLAME behaves under an extended test- VII. THREATS TO VALIDITY 1151 1106 ing time budget to evaluate its convergence characteristics and 1107 long-term scalability, and 2) assessed its sensitivity to com- A. External Threat 1152 1108 piler variation to understand whether compilation toolchains This limitation arises from the restricted diversity of proces- 1153 1109 materially influence functional coverage outcomes. sor designs, which may constrain the generalizability of the 1154 results to other processor designs with di↵erent architectures 1155
experiments on four widely used open-source processors, these 1157
1110 A. Scalability Overextended Testing Time or custom extensions. While we conducted comprehensive 1156
1111 The timing scalability is crucial in industrial verification do not fully represent processor designs with diverse ISAs, 1158
1112 workflows, where testing budgets often vary dynamically, and leaving certain architectural variations untested. Addressing 1159 1113 methods must retain eciency across both short and prolonged larger and more intricate instruction sets may pose a significant 1160 1114 verification cycles. As shown in Fig. 4, both FLAME and challenge, especially in terms of computational overhead with 1161
12 IEEE TRANSACTIONS ON COMPUTER-AIDED DESIGN OF INTEGRATED CIRCUITS AND SYSTEMS
1162 LLMs. Drawing inspiration from the research on LLMs in underscores the critical contribution of each component to the 1220
1163 code generation [71], [72], [73], one potential method for overall e↵ectiveness. Future work will focus on expanding 1221
1164 relieving LLM’s overhead could involve using LLMs to e↵ec- the evaluation to encompass a broader range of processor 1222
1165 tively summarize processor information, thereby optimizing designs and exploring more ecient LLM strategies for further 1223
1166 prompt constraints. We take it as our future work. enhancement. 1224
1167 B. Internal Threat ACKNOWLEDGMENT 1225
1168 One of these limitations stems from baseline imple- The authors would like to thank the anonymous reviewers 1226
1169 mentation details and chosen evaluation metrics that may for their constructive suggestions to help improve the quality 1227
1170 bias performance comparisons. To address this, we strictly of this article. They confirm that no generative AI tools were 1228
1171 adhered to ocial documentation for baseline reproduction used in the preparation of this article. 1229
1172 and employed widely accepted metrics to ensure validity and
1173 reliability. The configuration settings of LLMs may also intro- REFERENCES 1230
1174 duce potential biases. To mitigate this, we followed established [1] J. Nurmi, Processor Design: System-on-Chip Computing for ASICs and 1231 1175 practices [67], [68], [69], [70] by adopting a temperature of FPGAs. Cham, Switzerland: Springer, 2007. 1232 1176 0.7 to encourage diversity in the generated outputs while [2] H. D. Foster, “Trends in functional verification: A 2014 industry study,” 1233 1177 preserving coherence, which is an appropriate tradeo↵ for in Proc. 52nd ACM/EDAC/IEEE Design Autom. Conf. (DAC), Jun. 2015, 1234 1178 code-related generation tasks that require both correctness [3] pp. 1–6. 1235 I. Wagner, Post-Silicon and Runtime Verification for Modern Processors. 1236 1179 and exploratory coverage. To further improve reproducibil- Cham, Switzerland: Springer, 2014. 1237 1180 ity, we fixed random seeds and model versions and used a [4] K. Laeufer, V. Iyer, D. Biancolin, J. Bachrach, B. Nikoli´c, and K. Sen, 1238 1181 preprocessed, deterministic knowledge base, ensuring consis- “Simulator independent coverage for RTL hardware languages,” in Proc. 1239 1182 tent retrieval behavior across runs. The third internal threat 28th ACM Int. Conf. Architectural Support Program. Lang. Operating 1240 Syst., Volume 3, Mar. 2023, pp. 606–615. 1241 1183 concerns potentially invalid instructions in LLM-generated [5] A. Piziali, Functional Verification Coverage Measurement and Analysis. 1242 1184 sequences. To address this, our simulation harness detects such Cham, Switzerland: Springer, 2007. 1243 1185 cases by raising exceptions, terminating execution, recording [6] Y. Naveh et al., “Constraint-based random stimuli generation for hard- 1244 1186 results, and continuing subsequent iterations. While ensur- ware verification,” in Proc. AAAI, 2006, pp. 1720–1727. 1245 1187 ing evaluation robustness, our study focuses on RV32IMC [7] K. Laeufer, J. Koenig, D. Kim, J. Bachrach, and K. Sen, “RFUZZ: 1246 Coverage-directed fuzz testing of RTL on FPGAs,” in Proc. IEEE/ACM 1247 1188 functional coverage (excluding vector extensions), unlike prior Int. Conf. Comput.-Aided Design (ICCAD), Nov. 2018, pp. 1–8. 1248 1189 RVV exception-based negative testing [74], and extending our [8] J. Hur, S. Song, D. Kwon, E. Baek, J. Kim, and B. Lee, “DifuzzRTL: 1249 1190 framework to such scenarios remains future work. Di↵erential fuzz testing to find CPU bugs,” in Proc. IEEE Symp. Secur. 1250 Privacy (SP), May 2021, pp. 1286–1303. 1251 [9] J. Xu, Y. Liu, S. He, H. Lin, Y. Zhou, and C. Wang, “MorFuzz: Fuzzing 1252 1191 C. Construct Threat processor via runtime instruction morphing enhanced synchronizable co- 1253 simulation,” in Proc. USENIX Secur., 2023, pp. 1307–1324. 1254 1192 This limitation arises from the reliance on CRV techniques [10] S. Canakci et al., “ProcessorFuzz: Processor fuzzing with control and 1255 1193 for collecting target FPs, which may not comprehensively status registers guidance,” in Proc. IEEE Int. Symp. Hardw. Oriented 1256 1194 capture all critical scenarios. Nevertheless, this approach aligns Secur. Trust (HOST), May 2023, pp. 1–12. 1257 1195 with the method adopted in the recent study [42]. Another [11] Z. Zhang et al., “LLM4DV: Using large language models for hardware 1258 test stimuli generation,” 2023, arXiv:2310.04535. 1259 1196 potential threat arises from the hardware configuration dif- [12] C. Xiao et al., “LLM-based processor verification: A case study for 1260 1197 ferences between LLM-based and non-LLM methods, which neuromorphic processor,” in Proc. Design, Autom. Test Eur. Conf. Exhib. 1261 1198 could a↵ect execution time measurements. To mitigate this, (DATE), Mar. 2024, pp. 1–6. 1262 1199 we deployed only the LLM inference component on GPUs, [13] M. Rostami, M. Chilese, S. Zeitouni, R. Kande, J. Rajendran, and 1263 A.-R. Sadeghi, “Beyond random inputs: A novel ML-based hardware 1264 1200 which is necessary for ecient decoding and meeting memory fuzzing,” in Proc. Design, Autom. Test Eur. Conf. Exhib. (DATE), Mar. 1265 1201 requirements, while keeping all other pipeline stages on CPUs, 2024, pp. 1–6. 1266 1202 consistent with the non-LLM baselines. Our experiments fol- [14] R. Ma et al., “VerilogReader: LLM-aided hardware test generation,” 1267 1203 lowed each method’s standard and practical setup without 2024, arXiv:2406.04373. 1268 1204 applying any unfair acceleration. Moreover, since our analysis [15] (2024). Ibex RISC-V Core. [Online]. Available: https://github.com/ 1269 lowRISC/ibex 1270 1205 emphasizes end-to-end verification eciency rather than raw [16] F. Zaruba and L. Benini, “The cost of application-class processing: 1271 1206 inference speed, the comparison remains fair and reflects each Energy and performance analysis of a linux-ready 1.7-GHz 64-bit RISC- 1272 1207 paradigm’s practical computational cost in realistic deploy- V core in 22-nm FDSOI technology,” IEEE Trans. Very Large Scale 1273 1208 ment scenarios. [17] Integr. (VLSI) Syst., vol. 27, no. 11, pp. 2629–2640, Nov. 2019. 1274 A. Mastropaolo et al., “On the robustness of code generation techniques: 1275 An empirical study on GitHub copilot,” in Proc. IEEE/ACM 45th Int. 1276 1209 VIII. CONCLUSION [18] Conf. Softw. Eng. (ICSE), May 2023, pp. 2149–2160. 1277 J. Liu, C. S. Xia, Y. Wang, and L. Zhang, “Is your code generated by 1278 1210 We propose a novel LLM-based test generation frame- ChatGPT really correct? Rigorous evaluation of large language models 1279 1211 work, FLAME, to cover more FPs in processor verification for code generation,” in Proc. NeurIPS, 2023, pp. 21558–21572. 1280 1212 automatically. By leveraging RAG, CoT, and a functional- [19] X. Du et al., “Evaluating large language models in class-level code 1281 1213 coverage-guided feedback mechanism, FLAME establishes generation,” in Proc. IEEE/ACM 46th Int. Conf. Softw. Eng., Apr. 2024, 1282 pp. 1–13. 1283 1214 semantic mappings between FPs and instructions, enabling the [20] M. Gautschi et al., “Near-threshold RISC-V core with DSP extensions 1284 1215 iterative generation of valid and e↵ective test cases. Evalua- for scalable IoT endpoint devices,” IEEE Trans. Very Large Scale Integr. 1285 1216 tion of four widely used processor designs demonstrates that (VLSI) Syst., vol. 25, no. 10, pp. 2700–2713, Oct. 2017. 1286 1217 FLAME surpasses baselines in functional coverage improve- [21] A. Nazi et al., “Adaptive test generation for fast functional 1287 coverage closure,” in Proc. DVCON, 2022. [Online]. Available: 1288 1218 ment while significantly reducing the time required to achieve https://dvcon-proceedings.org/wp-content/uploads/Adaptive-Test- 1289 1219 the same functional coverage. In addition, ablation analysis Generationfor-Fast-Functional-Coverage-Closure-1.pdf 1290
LI et al.: FLAME: ENHANCING FUNCTIONAL COVERAGE IN PROCESSOR VERIFICATION VIA LLMs 13
1291 [22] S. Vasudevan et al., “Learning semantic representations to verify hard- [47] P. Sahoo, A. K. Singh, S. Saha, V. Jain, S. Mondal, and A. Chadha et al., 1367 1292 ware designs,” in Proc. NeurIPS, vol. 34, 2021, pp. 23491–23504. “A systematic survey of prompt engineering in large language models: 1368 1293 [23] A. Tyagi et al., “TheHuzz: Instruction fuzzing of processors Techniques and applications,” 2024, arXiv:2402.07927. 1369 1294 using golden-reference models for finding software-exploitable [48] J. Wei et al., “Chain-of-thought prompting elicits reasoning in large 1370 1295 vulnerabilities,” in Proc. USENIX Secur., 2022, pp. 3219–3236. language models,” in Proc. NeurIPS, 2022, pp. 24824–24837. 1371 1296 [24] C. Chen et al., “Xuantie-910: A commercial multi-core 12-stage pipeline [49] P. Lewis et al., “Retrieval-augmented generation for knowledge-intensive 1372 1297 out-of-order 64-bit high performance RISC-V processor with vector NLP tasks,” in Proc. NeurIPS, vol. 33, 2020, pp. 9459–9474. 1373 1298 extension: Industrial product,” in Proc. ACM/IEEE 47th Annu. Int. Symp. [50] J. Wang, Y. Huang, C. Chen, Z. Liu, S. Wang, and Q. Wang, “Software 1374 1299 Comput. Archit. (ISCA), May 2020, pp. 52–64. testing with large language models: Survey, landscape, and vision,” IEEE 1375 1300 [25] F. Solt, K. Ceesay-Seitz, and K. Razavi, “Cascade: CPU fuzzing Trans. Softw. Eng., vol. 50, no. 4, pp. 911–936, Apr. 2024. 1376 1301 via intricate program generation,” in Proc. USENIX Secur., 2024, [51] X. Hou et al., “Large language models for software engineering: 1377 1302 pp. 5341–5358. A systematic literature review,” ACM Trans. Softw. Eng. Methodol., 1378 1303 [26] (2025). Homepage. [Online]. Available: https://superpung.github.io/ vol. 33, no. 8, pp. 1–79, Nov. 2024. 1379 1304 flame [52] A. M. Dakhel, A. Nikanjam, V. Majdinasab, F. Khomh, and M. C. Des- 1380 1305 [27] A. Jayasena and P. Mishra, “HIVE: Scalable hardware-firmware co- marais, “E↵ective test generation using pre-trained large language 1381 1306 verification using scenario-based decomposition and automated hint models and mutation testing,” Inf. Softw. Technol., vol. 171, Jul. 2024, 1382 1307 extraction,” IEEE Trans. Comput.-Aided Design Integr. Circuits Syst., Art. no. 107468. 1383 1308 vol. 43, no. 10, pp. 3278–3291, Oct. 2024. [53] Y. Chen, Z. Hu, C. Zhi, J. Han, S. Deng, and J. Yin, “ChatUniTest: 1384 1309 [28] P. Mishra and N. Dutt, “Specification-driven directed test generation for A framework for LLM-based test generation,” in Proc. 32nd ACM Int. 1385 1310 validation of pipelined processors,” ACM Trans. Design Autom. Electron. Conf. Found. Softw. Eng., Jul. 2024, pp. 572–576. 1386 1311 Syst., vol. 13, no. 3, pp. 1–36, Jul. 2008. [54] C. Yang, J. Chen, B. Lin, Z. Wang, and J. Zhou, “Advancing code 1387 1312 [29] H.-M. Koo and P. Mishra, “Functional test generation using property coverage: Incorporating program analysis with large language models,” 1388 1313 decompositions for validation of pipelined processors,” in Proc. Design ACM Trans. Softw. Eng. Methodol., Jul. 2025. 1389 1314 Autom. Test Eur. Conf., 2006, pp. 1–6. [55] L. Yang et al., “On the evaluation of large language models in unit test 1390 1315 [30] R. Mukherjee, M. Purandare, R. Polig, and D. Kroening, “Formal tech- generation,” in Proc. 39th IEEE/ACM Int. Conf. Automated Softw. Eng., 1391 1316 niques for e↵ective co-verification of hardware/software co-designs,” in Oct. 2024, pp. 1607–1619. 1392 1317 Proc. 54th ACM/EDAC/IEEE Design Autom. Conf. (DAC), Jun. 2017, [56] (2024). Ibex Documentation. [Online]. Available: https://ibex- 1393 1318 pp. 1–6. core.readthedocs.io/en/latest 1394 1319 [31] K. Laeufer, B. Fajardo, A. Ahuja, V. Iyer, B. Nikoli´c, and K. Sen, “RTL- [57] (2024). The RISC-V Instruction Set Manual. [Online]. Available: https:// 1395 1320 repair: Fast symbolic repair of hardware design code,” in Proc. 29th github.com/riscv/riscv-isa-manual 1396 1321 ACM Int. Conf. Architectural Support Program. Lang. Operating Syst., [58] Y. Li et al., “A normalized Levenshtein distance metric,” IEEE Trans. 1397 1322 Apr. 2024, pp. 867–881. Pattern Anal. Mach. Intell., vol. 29, no. 6, pp. 1091–1095, Jun. 2007. 1398 1323 [32] A. Ferraiuolo, R. Xu, D. Zhang, A. C. Myers, and G. E. Suh, [59] Y. Song and D. Roth, “Unsupervised sparse vector densification for short 1399 1324 “Verification of a practical hardware security architecture through static text similarity,” in Proc. Conf. North Amer. Chapter Assoc. Comput. 1400 1325 information flow analysis,” in Proc. Twenty-Second Int. Conf. Architec- Linguistics, Hum. Lang. Technol., 2015, pp. 1275–1280. 1401 1326 tural Support Program. Lang. Operating Syst., Apr. 2017, pp. 555–568. [60] (2024). VCS—The Industry’s Highest Performance Simulation Solution. 1402 1327 [33] C. Ioannides and K. I. Eder, “Coverage-directed test generation auto- [Online]. Available: https://www.synopsys.com/verification/simulation/ 1403 1328 mated by machine learning - a review,” ACM Trans. Design Autom. vcs.html 1404 1329 Electron. Syst., vol. 17, no. 1, pp. 1–21, Jan. 2012. [61] Z. Rasheed, M. Waseem, K. Syst¨a, and P. Abrahamsson, “Large lan- 1405 1330 [34] W. Chen, L.-C. Wang, J. Bhadra, and M. Abadir, “Simulation knowledge guage model evaluation via multi AI agents: Preliminary results,”, 1406 1331 extraction and reuse in constrained random processor verification,” in arXiv:2404.01023. 1407 1332 Proc. 50th ACM/EDAC/IEEE Design Autom. Conf. (DAC), May 2013, [62] L. Chen, M. Zaharia, and J. Zou, “How is ChatGPT’s behavior changing 1408 1333 pp. 1–6. over time?,” 2023, arXiv:2307.09009. 1409 1334 [35] K. Campbell, L. He, L. Yang, S. Gurumani, K. Rupnow, and D. Chen, [63] W. C. Ou´edraogo et al., “Test smells in LLM-generated unit tests,” 2024, 1410 1335 “Debugging and verifying SoC designs through e↵ective cross-layer arXiv:2410.10628. 1411 1336 hardware–software co-simulation,” in Proc. 53rd Annu. Design Autom. [64] H. Touvron et al., “Llama 2: Open foundation and fine-tuned chat 1412 1337 Conf., Jun. 2016, pp. 1–6. models,” 2023, arXiv:2307.09288. 1413 1338 [36] A. Jayasena and P. Mishra, “Directed test generation for hardware [65] (2023). Phind. [Online]. Available: https://huggingface.co/Phind/Phind- 1414 1339 validation: A survey,” ACM Comput. Surv., vol. 56, no. 5, pp. 1–36, CodeLlama-34B-v2 1415 1340 May 2024. [66] J. Xu et al., “DejaVuzz: Disclosing transient execution bugs with 1416 1341 [37] C. Alliance. (2024). RISC-V DV: A RISC-V Randomized Instruction dynamic swappable memory and di↵erential information flow track- 1417 1342 Generator. [Online]. Available: https://github.com/chipsalliance/riscv-dv ing assisted processor fuzzing,” in Proc. 30th ACM Int. Conf. 1418 1343 [38] Y. Xu, S. Wang, D. Tang, N. Sun, and Y. Bao, “PathFuzz: Broadening Architectural Support Program. Lang. Operating Syst., Aug. 2025, 1419 1344 fuzzing horizons with footprint memory for CPUs,” in Proc. 61st [67] pp. 64–80. 1420 1345 ACM/IEEE Design Autom. Conf., Jun. 2024, pp. 1–6. X. Wang et al., “Self-consistency improves chain of thought reasoning 1421 1346 [39] G. Zhang, P. Wang, T. Yue, D. Liu, Y. Guo, and K. Lu, “Instiller: Toward [68] in language models,” 2022, arXiv:2203.11171. 1422 1347 ecient and realistic RTL fuzzing,” IEEE Trans. Comput.-Aided Design X. Wan, R. Sun, H. Dai, S. O. Arik, and T. Pfister, “Better zero-shot 1423 1348 Integr. Circuits Syst., vol. 43, no. 7, pp. 2177–2190, Jul. 2024. [69] reasoning with self-adaptive prompting,” 2023, arXiv:2305.14106. 1424 1349 [40] N. Kitchen and A. Kuehlmann, “Stimulus generation for constrained X. Wan et al., “Universal self-adaptive prompting,” in Proc. 1425 1350 random simulation,” in Proc. IEEE/ACM Int. Conf. Comput.-Aided Conf. Empirical Methods Natural Lang. Process., 2023, 1426 1351 Design, Nov. 2007, pp. 258–265. [70] pp. 7437–7462. 1427 1352 [41] P. Mishra and N. Dutt, “Functional coverage driven test generation for Z. Tian, J. Chen, and X. Zhang, “Fixing large language models’ 1428 1353 validation of pipelined processors,” in Proc. Design, Autom. Test Eur., specification misunderstanding for better code generation,” in 1429 1354 2005, pp. 678–683. Proc. IEEE/ACM 47th Int. Conf. Softw. Eng. (ICSE), Apr. 2025, 1430 1355 [42] M. Yan et al., “Achieving last-mile functional coverage in testing chip [71] pp. 1514–1526. 1431 1356 design software implementations,” in Proc. IEEE/ACM 45th Int. Conf. X. Jiang et al., “Self-planning code generation with large language 1432 1357 Softw. Eng., Softw. Eng. Pract. (ICSE-SEIP), May 2023, pp. 343–354. models,” ACM Trans. Softw. Eng. Methodol., vol. 33, no. 7, pp. 1–30, 1433 1358 [43] Z. Aref, R. Suvarna, B. Hughes, S. Srinivasan, and N. B. Man- [72] Sep. 2024. 1434 1359 dayam, “Advanced reinforcement learning algorithms to optimize design J. Li, G. Li, Y. Li, and Z. Jin, “Structured chain-of-thought prompting 1435 1360 verification,” in Proc. 61st ACM/IEEE Design Autom. Conf., Jun. 2024, for code generation,” ACM Trans. Softw. Eng. Methodol., vol. 34, no. 2, 1436 1361 pp. 1–6. [73] pp. 1–23, Feb. 2025. 1437 1362 [44] OpenAI. (2022). ChatGPT: Optimizing Language Models for Dialogue. J. Li, Y. Zhao, Y. Li, G. Li, and Z. Jin, “AceCoder: An e↵ective 1438 Eng. Methodol., vol. 33, no. 8, pp. 1–26, Nov. 2024. 1440 1363 [Online]. Available: https://openai.com/index/chatgpt prompting technique specialized in code generation,” ACM Trans. Softw. 1439 1364 [45] A. Dubey et al., “The llama 3 herd of models,” 2024, arXiv:2407.21783. [74] M. Schlaegl and D. Grosse, “Single instruction isolation for RISC-V 1441 1365 [46] B. Rozi´ere et al., “Code llama: Open foundation models for code,” 2023, vector test failures,” in Proc. 43rd IEEE/ACM Int. Conf. Comput.-Aided 1442 1366 arXiv:2308.12950. Design, Oct. 2024, pp. 1–9. 1443