Generation of various programs in

Verilog/SystemVerilog for testing EDA tools

    Luiza de Melo Gomes Fernando Magno Quint˜ao Pereira
    UFMG UFMG
    Belo Horizonte, Brazil Belo Horizonte, Brazil
    luizademelo@dcc.ufmg.br fernando@dcc.ufmg.br

Abstract—Testing Electronic Design Automation (EDA) tools unique tokens and production rules to ChiGen1, a ”bottom- hinges on the availability of benchmarks—programs written up” Verilog fuzzer initially developed to test Cadence De- in Hardware Description Languages (HDLs) like Verilog, Sys- sign Systems’ Jasper Formal Verification Platform and re- temVerilog, and VHDL. While benchmark collections exist, their diversity remains limited. This limitation is increasingly problem- leased as open-source in 2024. ChiGen generates Verilog de- atic given the growing demand for training large language models signs through a three-stage process: skeleton generation using in the EDA domain. In order to address this challenge, this paper a probabilistic grammar, mock identifier replacement with introduces enhancements in the variety of programs produced scope-compliant names and type inference using the Hindley- by ChiGen, a tool for synthesizing realistic Verilog designs. Milner algorithm. Our current work significantly enhances Originally developed to test Cadence Design Systems’ Jasperᵀᴹ Formal Verification Platform, ChiGen has demonstrated its ChiGen’s capabilities by expanding its token set and produc- capability to uncover zero-day bugs in tools such as Verible, tion rules to include modern SystemVerilog constructs like Verilator, and Yosys. This work expands ChiGen’s capabilities to classes, interfaces, and packages, as well as formal verification include SystemVerilog constructs such as classes, interfaces, and primitives such as assertions. packages, as well as formal verification primitives like assertions, Our experiments demonstrate that ChiGen outperforms top- sequences, and properties. These additions significantly increase both structural diversity and the semantic representativity of the down fuzzers like Verismith, VlogHammer, and TransFuzz generated programs. in terms of structural diversity and bug-finding effectiveness. Index Terms—Verilog, Synthesis, Testing, Fuzzing. Since its release, ChiGen has identified at least five confirmed issues in prominent open-source EDA tools such as Yosys, I. INTRODUCTION Icarus Verilog [19], and Verible [6]. II. RELATED WORK Fuzzing is an automated testing technique that generates This paper presents techniques for building Verilog fuzzers, random, often unexpected, inputs to uncover bugs, vulner- with a specific focus on enhancing the variety of generated abilities, or unintended behaviors in software. Many Elec- designs. Several other Verilog fuzzers are available as open- tronic Design Automation (EDA) tools, including YOSYS source tools [10], [15], [20]. Unlike our approach, these tools [21], VJ ERILATOR [14], MODELSIMTM [7], XceliumTM [5], primarily expand a core set of Verilog syntax incrementally, ASPERTM[4], can benefit from fuzzers capable of automati- ensuring each expansion results in a valid design. In contrast, cally generating Verilog designs. Such tools enable early bug ChiGen’s probabilistic grammar approach enables the genera- discovery, performance optimization, compliance verification, tion of a richer variety of tokens while maintaining syntactic and general validation. correctness. During the development of ChiGen, we engaged Existing open-source Verilog fuzzers, such as with the authors of Verismith, gaining valuable insights into VlogHammer[20], Verismith[10], and TransFuzz[15], follow differing methodologies. a top-down approach. They start with minimal valid Verilog In the past two years, the emergence of large language syntax and expand it using various techniques, ensuring models (LLMs) has introduced new methods for Verilog code the generation of semantically valid designs. However, our generation [8], [12], [17], [18]. While ChiGen is not an experience testing the Jasper Formal Verification Platform LLM, it offers unique advantages by focusing on probabilistic reveals that this strict adherence to validity limits test case grammar-based generation. Unlike LLMs, which aim to shape diversity. Interestingly, semantically invalid Verilog designs code toward specific semantics, ChiGen prioritizes diversity in can be just as effective in identifying issues in EDA tools. generated tokens and production rules. Specifically, it models Additionally, these tools often generate constructs that differ the probabilities of production rules as k-grams, allowing for significantly from human-written Verilog, covering fewer than more comprehensive exploration of Verilog’s syntax space 40% of the production rules in the Verilog-2005 grammar without assigning probabilities to token sequences directly. (IEEE 1364-2005), as described in section V. This paper introduces enhancements in the quantity of 1 Available at https://github.com/lac-dcc/chimera

            ChiGen’s enhanced token generation capability makes it
well-suited for creating diverse Verilog benchmarks. However,                                                             0

it differs from existing benchmark collections [1], [2], [3], 8 ChiBench programs probabilistic grammar generation [11], [13], [16], which are static and limited to a predefined set of designs. Unlike these immutable collections, ChiGen dynamically generates new benchmarks, integrating them into its library of Verilog programs. This dynamic approach not only broadens the scope of available designs but also ensures that benchmarks can adapt to new requirements, enriching the yd goes 2 testing landscape for hardware tools and methodologies. III. CHIGEN OVERVIEW module m1() ChiGen is a stochastic tool for generating random Verilog input wire 1d_1; programs, designed to diversify the input space for testing chiGen selects grammar ules based on xed Electronic Design Automation (EDA) software. A high-level endmodule overview of ChiGen is shown in Figure 1. The tool takes as input the JSON grammar derived from ChiBench programs, Fig. 1. ChiGen overview randomly selects grammar rules based on predefined probabil- ities, and generates a Verilog program as output. At its core, ChiGen operates in three distinct phases, each contributing to in the partial syntax tree. This stochastic process ensures high its robust program generation process. The following sections variability in the generated syntax trees while maintaining a provide a detailed explanation of these phases. statistically representative structure based on the ChiBench A. Syntax Generation via Probabilistic Grammars corpus. The result is a syntactic skeleton of a Verilog program, ready for subsequent refinement. The core of ChiGen’s functionality lies in its use of n-gram In order to obtain the production rules of a program, we language models to guide the synthesis of Verilog programs. used Verible’s parser. This is possible by using the trace of These models require a comprehensive corpus of Verilog Verible’s parser, which outputs production rules in the order programs to estimate the probabilities of various language in which they are processed. Then, we can count how many constructs, such as module declarations, always blocks, and times each rule appear given the context of n preceeding rules. specific operators. For this purpose, we leverage ChiBench These rules and the frequency of each of rule is then stored [16], a collection of Verilog programs meticulously mined in a JSON file. Once the probabilities of each construct are from diverse open-source repositories, serving as our primary established in a JSON file, ChiGen synthesizes programs by training dataset. assembling syntax trees guided by the language model. To extract the production rules and their frequencies from ChiBench, we utilize Verible’s parser. Verible, an open-source B. Variable Renaming and Scope Creation Verilog parsing and linting tool, provides a detailed trace of Following the initial syntax generation phase, the abstract the production rules it applies during the parsing process. By syntax trees (ASTs) produced by ChiGen contain generic analyzing these traces across the entire ChiBench corpus, we placeholders. These placeholders mark locations where spe- can count the occurrences of each grammar rule. Crucially, we cific, unique, and contextually appropriate identifiers—such model the probabilities of these production rules as k-grams, as variable names, module names, port names, or instance meaning the probability of a rule being selected depends on names—are required. Proper management of these identifiers the context of the k preceding rules. This approach allows for a and their visibility within defined scopes is critical for gener- more nuanced and realistic generation of syntax, as it captures ating syntactically valid and semantically meaningful Verilog common structural patterns and dependencies found in human- and SystemVerilog programs. Incorrect naming conventions or written Verilog. This differs significantly from approaches that improper scope resolution can lead to compilation errors, un- assign probabilities to token sequences directly, which can lead intended behavioral ambiguities, or even crashes in Electronic to less structurally coherent designs. Design Automation (EDA) tools. The collected rules and their observed frequencies, condi- In this second core phase of ChiGen’s code generation, a tioned on their k-gram context, are then compiled into a JSON dedicated scope management engine, which we refer to as file. This file effectively serves as ChiGen’s ”probabilistic the ”scope delimiter,” performs a comprehensive traversal of grammar.” During the program generation phase, ChiGen the generated syntactic skeleton. This traversal is typically reads this JSON file. Starting from a root production rule executed in a depth-first manner, allowing the engine to metic- (e.g., ”source text”), ChiGen iteratively expands non-terminal ulously identify, enter, and exit various scope regions defined symbols by randomly selecting a production rule from the by the Verilog and SystemVerilog grammar. These regions are available options, weighted by their probabilities defined in fundamental to the language’s structure and include, but are the JSON grammar and conditioned on the k preceding rules not limited to, module declarations, function and task

            bodies, always blocks, fork-join blocks, begin-end         scope delimiter executes a critical cleanup step. It re-

blocks, class definitions, and package declarations. moves from the set of in-scope elements all variables, The ”scope delimiter” meticulously maintains a dynamic parameters, and other identifiers that were declared symbol table, which is conceptually structured as a stack of specifically within that region. Conceptually, this opera- hash maps. Each map within this stack represents the set tion corresponds to popping the current scope’s symbol of in-scope elements (i.e., declared identifiers) for the table off the stack. This mechanism rigorously adheres to current lexical context. As the tree traversal proceeds, the the lexical scoping rules of Verilog and SystemVerilog, variable renaming process operates through three precise and preventing erroneous references to identifiers that are interconnected steps: no longer in scope and ensuring that names are only 1) Declaration Renaming and Entry into Scope: When visible where they are legally declared. This rigorous the scope delimiter encounters a placeholder repre- cleanup prevents accidental cross-scope references and senting the declaration of a new identifier (e.g., a helps maintain the integrity of the generated design. wire, reg, logic, parameter, or a new class/module This variable renaming and scope creation phase is es- name), it triggers the generation of a new, unique sential to ChiGen’s ability to produce high-quality, diverse, symbol. These symbols are systematically generated and syntactically correct Verilog and SystemVerilog programs. (e.g., id_0, id_1 for variables, module_0 for mod- By dynamically managing identifier uniqueness and visibility ules, instance_0 for instances) to ensure global throughout the generation process, it ensures that the re- uniqueness within the generated program. This newly sulting designs are parsable and elaboratable by EDA tools. minted identifier, denoted as s, then replaces the generic This enables effective and targeted testing of critical EDA placeholder in the Abstract Syntax Tree (AST), and functionalities such as symbol resolution, name binding, and s is immediately inserted into the symbol table that hierarchical elaboration. Failures in these aspects of EDA corresponds to the currently active scope. This step tools, manifesting as name conflicts, unresolved references, or guarantees that every declared entity receives a distinct unexpected shadowing behavior, are common sources of bugs and resolvable name within its defined visibility. and can be effectively uncovered by the controlled randomness // Before renaming (fragment of AST): and strict adherence to scoping rules introduced during this logic GenericIdentifier; phase. module GenericIdentifier (GenericIdentifier, GenericIdentifier); IV. SYSTEMVERILOG AND FORMAL CONSTRUCTS // After declaration renaming (illustrative The overarching objective of this work is to significantly transformation): enhance the diversity and realism of the Verilog and Sys- logic id_0; // ’id_0’ is a newly generated temVerilog designs generated by ChiGen. This is achieved moduleunique name by strategically expanding the set of tokens and production module_1 (id_1, id_2); // ’module_1’, ’id_1’, ’id_2’ are new unique names rules that ChiGen’s probabilistic grammar can utilize, thereby better reflecting the constructs found in real-world hardware 2) Usage Resolution and Replacement: As the traversal designs. Accomplishing this goal involves a multifaceted ap- continues into the body of the generated code, the proach, centered on three core tasks: systematically identify- scope delimiter encounters placeholders that represent ing absent or under-represented tokens and production rules uses or references to previously declared identifiers. within ChiGen’s current generation capabilities, meticulously At each such instance, it randomly selects an already- incorporating these missing constructions into its grammar declared symbol from the set of currently active and generation logic, and diligently correcting any erroneous in-scope elements. This selection is performed programs that might arise from initial implementations of new from all visible scopes, prioritizing the innermost scope features. to simulate standard HDL name resolution rules (e.g., Beyond simply increasing linguistic coverage, a crucial local variables shadow global ones). The placeholder enhancement for ChiGen is the introduction of formal verifica- in the AST is then replaced with the chosen symbol. tion constructs. These include SystemVerilog assert, property, The inherent probabilistic nature of this selection in- and sequence blocks. For instance, ChiGen can now generate troduces significant syntactic and semantic variety in assertions such as: how variables are referenced, which is exceptionally assert property (@(*) id_1 == id_2); valuable for fuzzing. This controlled randomness can generate unusual, yet syntactically valid, connections The inclusion of these elements is vital for two primary and expressions that effectively challenge the parsing, reasons: first, they dramatically increase the language cover- elaboration, and semantic analysis capabilities of EDA age, moving ChiGen beyond basic Verilog-2005 to encom- tools. pass more modern and complex design patterns. Second, 3) Scope Exit and Cleanup: Upon the completion and perhaps more critically, they enable the generation of of processing a scope region (e.g., encountering an sophisticated test cases specifically designed to probe and endmodule, endfunction, or end keyword), the challenge the formal engines of Electronic Design Automation

(EDA) tools. This allows for more targeted and effective             SystemVerilog Assertions (SVAs), ChiGen can now produce
fuzzing campaigns against formal verification platforms,             highly specific and semantically rich test cases that probe the
A. Addition of packages                                              deepest layers of formal analysis.
         Packages in SystemVerilog serve as containers for sharing   1) Assertion Generation Strategy:           ChiGen employs a so-

declarations (e.g., parameters, types, functions, tasks, classes) phisticated, heuristic-based approach to strategically insert across multiple modules or interfaces without global scope assert statements into the generated Verilog/SystemVerilog pollution. In ChiGen, modules are generated iteratively until code. This strategy is designed to create assertions that are con- a specified program size or complexity threshold is met. To textually relevant to the surrounding design logic. A common integrate packages, the generation process was augmented: and robust pattern in hardware description languages is the instead of exclusively generating module declarations, ChiGen assignment of a value to a variable, which typically represents now probabilistically decides whether to generate a module, a data path, a control signal, or an intermediate computation. package, or interface as a top-level construct. ChiGen intelligently identifies locations within the Abstract The process for assigning unique names to generated pack- Syntax Tree (AST) where such assignment statements occur, ages mirrors that used for modules. After all primary struc- for instance: tural units (modules, packages, interfaces) are generated, a assign id_1 = id_2; systematic renaming pass ensures unique identifiers, typically following a pattern like module 1, module 2, package 1, Immediately following the identification of such an assign- interface 1, and so on. This ensures clarity and avoids name ment within the AST, ChiGen probabilistically determines collisions within the generated design. Critically, ChiGen’s whether to insert a corresponding concurrent assertion. The scope management system was updated to understand pack- simplest and most fundamental form of this generated asser- age scopes, allowing declarations within a package to be tion checks for an equality condition: correctly referenced (package_name::item_name) by assign id_1 = id_2; other generated constructs that import or explicitly reference assert property (@() id_1 == id_2); the package. SystemVerilog interfaces abstract communication between This specific assertion, triggered by the @() event sensi- design blocks, simplifying port declarations and promoting tivity list, validates that id 1 holds the exact value of id 2 at reusable verification environments. Similar to packages, in- every point in time when either id 1 or id 2 changes. Such an terfaces are now part of ChiGen’s top-level generation op- assertion is invaluable for detecting unexpected value propaga- tions. ChiGen generates interface declarations, including their tions, subtle functional discrepancies, or unintended temporal modports (module ports) and interface variables. The type shifts in data. Its simplicity makes it a robust candidate for inference engine (Section 3.3) was extended to correctly automated generation, yet its failure can indicate critical bugs deduce and propagate interface types when instances of these in formal analysis tools or design elaboration. interfaces are declared within modules, ensuring that the The power of this dynamic assertion generation extends be- connections adhere to SystemVerilog’s strict typing rules. This yond simple variable-to-variable assignments. ChiGen’s mech- enables ChiGen to produce designs that utilize more modern anism is designed to handle more complex right-hand side and complex interconnection patterns, challenging EDA tools expressions. If the assignment involves an arithmetic, logical, to correctly parse and elaborate such hierarchical structures. or bitwise expression, the generated assertion will accurately Packages can also be imported according to the Verilog reflect that expression, directly testing the equivalence: grammar. For instance, below there is a program which produces a package and imports it: assign id_1 = 1 + 2; assert property (@() id_1 == 1 + 2); package package_0; typedef logic id_1; This approach ensures that the generated assertions are not endpackage merely syntactically correct but also semantically tied to the package package_1; typedef logic id_2; surrounding design logic. This contextual relevance signifi- import package_0::; cantly increases their efficacy in identifying meaningful issues endpackage within formal verification tools, as they represent realistic B. Addition of Assertions verification challenges. This dynamic generation of assertions based on existing code patterns is a key strength, ensuring Formal verification is an indispensable phase in modern that the assertions are semantically related to the generated hardware design, heavily relying on the precise specification design logic, which consequently increases their relevance and of design behavior through constructs like assertions and effectiveness for formal verification. While ChiGen currently properties. ChiGen’s enhanced capability to generate these focuses on these assignment-driven assertions, future work formal constructs significantly bolsters its utility for rigorous could explore more sophisticated assertion patterns, including testing of formal verification engines within Electronic Design those involving more complex temporal operators to capture Automation (EDA) tools. By integrating the generation of intricate design behaviors.

2) Expanded Assertion Coverage:                 Beyond these immedi-                              V. EVALUATION
ate, context-driven assertions, ChiGen also supports the gen-                            This section evaluates ChiGen’s performance in addressing
eration of more elaborate assert property statements that can           four key research questions:
incorporate temporal aspects, even without explicitly defined
property or sequence blocks. These assertions can directly               •                   RQ1: How diverse are the designs generated by ChiGen?
embed temporal expressions to specify complex behavioral                 •   RQ2: What types        of bugs can be uncovered                 using
checks over time. This capability broadens the spectrum of                   ChiGen-enabled fuzzing?
formal verification scenarios that ChiGen can generate, mov-             •                 RQ3: How the different techniques listed in Section III
ing beyond simple combinatorial checks to include sequential                 increase the diversity of Verilog designs?
behaviors. For example, ChiGen can produce an assertion with                 a) Baselines:                To train ChiGen’s probabilistic grammar,
a specific clocking event and a condition that must hold true:                         we used 10,000 Verilog designs from the ChiBench collection
BLOCK_0 :                                                                              [16]. The performance of ChiGen is then compared with other
assert property     (id_1) @(negedge     id_5);                                      fuzzer collections, including Verismith [10], TransFuzz [15],
                                                                                             and VlogHammer [20]. Note that VlogHammer generates a
C. Addition of classes                                                           fixed set of 3,000 designs, which limits its scope of evaluation.
          The introduction of classes in SystemVerilog significantly    A. RQ1: Diversity
extended  its   capabilities for object-oriented         programming         1) Syntactical Diversity:            We evaluated the syntactical di-
(OOP), enabling higher-level abstraction and testbench devel-                          versity of ChiGen-generated designs by analyzing the number
opment. ChiGen now incorporates the generation of class dec-                         of unique production rules in the Verible grammar required to
larations and associated constructs, primarily focusing on static                   parse these designs. The Verible grammar includes 456 distinct
members which can be accessed without object instantiation,             production rules.
making them more amenable to automated generation within                     a) Results:              Figure 3 shows the syntactical diversity for
a fuzzer context.                                                                        populations of varying sizes. Unique production rules are

1. Generation of Class Declarations: ChiGen’s grammar counted only once, regardless of their frequency in the designs. was updated to include production rules for class declara- As the number of generated designs increases, the diversity tions, including member variables (logic, int, etc.), methods approaches that of ChiBench, which uses 406 unique rules (function, task), and class-specific keywords (static, local, pro- for 10,000 designs. In comparison, Verismith exercises 179 tected). This allows ChiGen to create diverse class definitions, rules, TransFuzz uses 151, and VlogHammer, limited to 3,000 reflecting various class structures found in SystemVerilog. designs, employs only 137 rules. 2) Addition of calls to functions of classses: A key en-

hancement    is ChiGen’s     ability to  generate  calls to   static             vs.                    of
                                                                                 Total Productions    Number
functions declared within a class. Static functions, unlike             wl       designs
regular member functions, belong to the class itself rather than
an instance of the class, allowing them to be called directly           400
using the scope resolution operator (::).                               2350    ==                                     —    2gram
               ChiGen’s generation process includes a phase where it
traverses the partially built syntax tree. If a class definition                                                       i
containing a static function is found, ChiGen will probabilis-          |                                               ii
tically insert a call to this function in an appropriate context,
such as within an always block or a task/function. The program          2001                                            x   Transfuzz
below illustrates this concept:
                                                                        mol
class id_1;
static function     logic id_2(id_3);                                                 Numberof     designs
    id_3        <=     1;
endclassendfunction                                                     Fig. 2. Syntactical diversity of ChiGen designs, measures as the number of
                                                                           unique production rules in the Verilog grammar exercised when parsing a
always @* id_1::id_2(id_3); // Call          to static                  population of generated files.
function                                                                                 ChiGen’s performance varies slightly with the size of the

This capability challenges EDA tools’ parsers and elab- probabilistic context K : for K = 1, 2, 3, 4, 5, 6, the number orators to correctly resolve static method calls and their of unique rules exercised are 336, 355, 364, 374, and 377, arguments. While ChiGen currently prioritizes static function respectively for 1-gram. calls for their simpler integration, future developments could The results for unique token diversity parallel the posi- explore the generation of class instances, constructors, and tive trends observed for production rules. While a ”reduced calls to non-static methods, which would require more com- ChiBench” dataset (a specific subset of the full ChiBench plex object lifecycle management within the fuzzer. corpus tailored for this particular token analysis) contains 246

     of programs                                                                                      TABLE I
     Total     Tokens vs. Number
 300                                     +                                                                SUMMARY OF ISSUES IDENTIFIED IN EDA TOOLS
                                                                         Issue   Tool             Description
 0                                                                       2181    Verible          Crashes instead of reporting syntax errors related to instantiation type.

                                                    20m                  2233    Verible          Incorrectly accepts Verilog code with mismatched  program and
                                               — 1am                     2189 |  Verible          Crashes with syntactically valid input.
£go                                            —                                                  endmodule keywords.
                                               — om                      1174    Icarus Verilog   Crashes when assigning to parameters in a procedural block.
                                                                         4598    Yosys            Crashes while simplifying program.
150                                             x   Reduced Chisench
                                                x   ViogHammer
 -                                                  Compre chisench         These     findings      highlight key    weaknesses     in              the              tested
                                                                                                              EDA tools. Specifically, Verible experienced several crashes,
     x                                   x                              including     one     related to     instantiation     type     errors,                     another
     2°      ES      of      29          ES                             with valid syntax, and a third with mismatched                                              program
    and endmodule keywords. Additionally, Icarus Verilog and

Fig. 3. Syntactical diversity of ChiGen designs, measures as the number of Yosys encountered crashes in scenarios involving parameter unique tokens in the Verilog grammar exercised when parsing a population assignments and program simplifications, respectively. These of generated files. results demonstrate ChiGen’s effectiveness in identifying crit- ical issues that affect the stability and reliability of EDA tools. unique tokens, ChiGen successfully generated designs exer- C. RQ3: Evolution cising 202 unique tokens. This was observed with populations In this section, we examine the evolution of ChiGen by di- of 214 (16,384) designs, specifically when generated using a viding it into three distinct versions, each marking a significant 1-gram probabilistic context (i.e., K = 1). This demonstrates step in its development and capabilities. that ChiGen produces designs with a substantial variety of In this section, we discuss the development trajectory of lexical elements, ensuring comprehensive testing of the scan- ChiGen, presenting its evolution through different versions. ner and parser components within EDA toolchains. The ability This chronological analysis highlights the systematic improve- to generate such a high number of unique tokens, even with ments implemented to broaden the diversity of generated a basic 1-gram context, underscores ChiGen’s foundational designs and to incorporate more advanced Verilog and Sys- strength in producing rich and varied HDL code at the most temVerilog constructs. Each version represents a significant granular level. milestone, marked by specific enhancements that collectively In conclusion, ChiGen’s superior syntactical diversity, as contribute to ChiGen’s current robust state as a sophisticated evidenced by its extensive coverage of both production rules fuzzer for EDA tools. and unique tokens, is a direct consequence of its innovative Figure 4 visually encapsulates this progression, illustrat- bottom-up, probabilistic grammar approach. This broad and ing how each iteration has incrementally contributed to the deep coverage is critical for effective fuzzing, as it enables the increased diversity and complexity of the generated outputs, generation of highly varied and often unconventional test cases as measured by the number of unique tokens and production that are significantly more likely to uncover subtle, unforeseen rules exercised. Our analysis is based on consistent samples bugs and vulnerabilities in EDA tools, thereby improving their of 10,000 programs generated for each version, providing a overall robustness and reliability in handling the vast landscape reliable basis for comparison. of Verilog and SystemVerilog designs. We delineate ChiGen’s evolution through several key ver- B. RQ2: Bug Detection sions: Initial Baseline (Pre-Version 22): This period represents ChiGen was specifically designed to uncover bugs in the ChiGen’s state when this paper started, prior to the major Jasper Formal Verification Platform and has been integrated enhancements detailed in this paper. It served as the initial into Cadence Design Systems’ development methodology. Due point for measuring token and production rule coverage, to confidentiality agreements, the effectiveness data related to primarily focusing on core Verilog constructs. Although ca- this integration cannot be disclosed. pable of generating diverse designs, its coverage of modern To demonstrate ChiGen’s bug-finding capabilities, we con- SystemVerilog features and formal constructs was limited. ducted extensive campaigns targeting three open-source EDA Version 22: Introduction of Packages and Minor Fixes. tools: Verible (v0.0-3808), Yosys (v0.45), and Verilator (Re- Version 22 marks the beginning of the significant enhance- lease 159). In each campaign, 3,000 designs were gener- ments detailed in this monograph. The primary focus of ated—500 for each production context—and submitted to the this iteration was the integration of SystemVerilog package respective tools. The compiled designs were analyzed, and any constructs. As discussed in Section IV, packages provide a crashes or failed assertions were flagged as issues. crucial mechanism for sharing common declarations in Sys- a) Results: The bug-finding campaigns revealed several temVerilog. Their addition required modifications to ChiGen’s issues across the tested EDA tools. The following table sum- grammar and its scope management system to correctly han- marizes the identified issues: dle package declarations and their usage. This version also

incorporated several minor bug fixes that addressed previously                       Sum of all Tokens  present in        valid  programs
identified syntactic inaccuracies in the generated programs.              £ go
The impact of these changes was a noticeable expansion in              5
the types of structural organization ChiGen could produce,             £   ook
laying the groundwork for more complex designs.                        2
        Version 23: Initial Integration of Classes. A major under-     2
taking in Version 23 was the initial introduction of SystemVer-            20k
ilog         class constructs. This was a significant step towards
enabling the generation     of more complex,       object-oriented                    Percentage of valid designs
test cases. However, the initial integration of classes proved
to be particularly challenging. The complexities associated                260
with class syntax, scope rules, and type interactions led to              5
a dramatic decrease in the percentage of syntactically valid     ~~        £40
designs. This phenomenon is a common characteristic during                &
the integration of highly complex language features into fuzzer           g  2                   v                        |        |
grammars, as subtle interactions can lead to widespread pars-
ing errors if not meticulously handled. Although groundbreak-                     Average size (in    number of tokens)     of valid designs
ing in its scope, this version highlighted the need for rigorous      H
error correction and refinement in subsequent iterations.
           Version 24: Major Syntax Error Resolution and Stability     §     4
Improvement. Version 24 was primarily dedicated to address-            <
ing the major syntax errors caused by the initial addition                §165
of classes in Version 23. This involved a thorough review                 $160
of the newly added class-related production rules, refinement                     v22     va3                               v24      v25
of their probabilities, and significant debugging of the type         2                               Versions (v22 to v25)
inference and variable renaming phases to correctly handle                                       lgram
class-specific contexts. The concerted effort in this version
led to a substantial increase in the number of valid designs,                     Fig. 4. Evolution of the number of tokens and productions.

indicating a greater maturity in ChiGen’s ability to produce well-formed SystemVerilog code that incorporates classes. total number of unique tokens and production rules generated. This improvement was crucial for ensuring that a higher This measured and iterative approach to development allows proportion of generated programs could be successfully parsed ChiGen to progressively cover more of the Verilog and Sys- and processed by EDA tools, thus increasing the efficiency of temVerilog language, ensuring that it remains a cutting-edge bug detection campaigns. tool for rigorous EDA tool testing. Version 25: Addition of Static Class Method Calls and Further Refinements (Current Version). The most VI. FUTURE WORK recent iteration, Version 25, represents the current state of Despite ChiGen’s significant achievements in generating ChiGen. A key enhancement in this version was the specific diverse and effective Verilog and SystemVerilog designs for integration of calls to static functions within classes. While testing Electronic Design Automation (EDA) tools, several this feature significantly expanded the functional coverage of key areas remain open for further exploration and potential SystemVerilog classes, its intricate nature, particularly con- enhancement. Notably, unlike some fuzzers such as Verismith cerning argument passing and return types, caused a slight, [9], which prioritize the generation of 100% syntactically and albeit manageable, decrease in the immediate percentage of semantically correct designs, ChiGen’s current probabilistic, perfectly valid designs. This transient reduction reflects the bottom-up approach means it does not consistently produce ongoing challenge of integrating advanced language features entirely correct programs. This inherent trade-off, which con- into a stochastic fuzzer while maintaining high levels of tributes to ChiGen’s ability to expose obscure bugs, also points syntactic correctness. However, the benefit of generating more to specific directions where its capabilities could be expanded realistic and challenging test cases for EDA tools, especially by subsequent research. those related to static class member resolution and formal One primary area for future investigation involves refining verification, far outweighs this minor fluctuation. This version ChiGen’s generation mechanisms to potentially increase the also includes continued minor refinements to existing grammar rate of syntactically and semantically correct outputs without rules and probabilistic models based on ongoing feedback and unduly sacrificing the structural diversity that is its hallmark. further analysis of the ChiBench corpus. This might entail more sophisticated post-generation validation The data presented in Figure 4 visually reinforce this and correction passes, or deeper integration of semantic checks narrative. The progression illustrates a consistent effort to during the generation process itself, to preemptively avoid expand the breadth of ChiGen’s output, steadily increasing the certain classes of errors. Achieving higher correctness rates

could broaden ChiGen’s applicability, particularly for EDA                      ChiGen has further enhanced its ability to generate structurally
tools that are less tolerant of malformed inputs or require                             diverse and realistic Verilog programs, making it a more
consistently valid designs for advanced analysis.                    powerful tool for testing EDA tools.
              Beyond merely improving correctness, a wealth of Sys-                   The addition of SystemVerilog and formal verification con-
temVerilog features remain to be explored for integration                          structs increases the relevance of ChiGen for modern hardware
into ChiGen’s generation framework. For instance, concerning                            development workflows, positioning it as a comprehensive
SystemVerilog classes, the current implementation primarily          benchmark generator for structural and formal EDA testing.
focuses on static members and basic class structures. Future                                  A
work could implement the generation of non-static class fields,                                   CKNOWLEDGMENTS
enabling object instantiation with constructors (new), and the                              This project is sponsored by Cadence Design Systems.
modeling of basic inheritance. Such additions would allow                                Additionally, Luiza de Melo acknowledges the support of
ChiGen to create more complex object-oriented test environ-                          Rafael Sumitani, Jo˜ao Victor Amorim, Mirlaine Crepalde and
ments, which are increasingly vital in modern verification           Augusto Mafra.
methodologies.                                                                                    REFERENCES
         Similarly, within the domain of formal verification, while   [1]  Luca Amaru, Pierre-Emmanuel Gaillardon, Eleonora Testa, and Gio-
ChiGen effectively generates assert property statements, the
comprehensive generation of named property and sequence                    vanni De Micheli.    The epfl combinational benchmark suite, February
                                                                           2019.
blocks, along with their varied instantiations, represents a          [2]  J. Babb, M. Frank, V. Lee, E. Waingold, R. Barua, M. Taylor, J. Kim,
significant unaddressed area. The framework is also capa-                  S. Devabhaktuni, and A. Agarwal.      The raw benchmark suite: compu-
ble of incorporating the generation of               covergroup and        tation structures for general purpose computing. In   FCCM, page 134,
                                                                           USA, 1997. IEEE Computer Society.
coverpoint     constructs for     functional coverage,        which   [3]  Franc Brglez, David Bryan, and Krzysztof Kozminski.     Combinational
would enhance its utility for testing coverage analysis tools.             profiles of sequential benchmark circuits. In ISCAS, pages 1929–1934,
Furthermore, exploring the generation of designs that leverage        [4]  New York, USA, 1989. IEEE.
                                                                           Cadence    Design  Systems, Inc.     Jasper     formal   verification
SystemVerilog’s interface and modport features for more real-              platform.                   https://www.cadence.com/en US/home/tools/
istic inter-module communication presents another opportunity              system-design-and-verification/formal-and-static-verification.html,
to deepen its language coverage.                                      [5]  2025. Accessed: 2025-06-23.
                                                                           Cadence    Design  Systems, Inc. Xcelium     logic  simulator. https:
        Another potential direction for future development involves        //www.cadence.com/en US/home/tools/system-design-and-verification/
investigating mechanisms for more targeted design generation.              simulation-and-testbench-verification/xcelium-simulator.html,   2025.
Although ChiGen’s stochastic nature excels at exploring the           [6]  Accessed: 2025-06-23.
                                                                           CHIPS Alliance. Verible. https://github.com/chipsalliance/verible, 2025.
design space broadly, subsequent research could explore ways               Accessed: 2025-06-23.
to guide the generator to produce designs with specific char-         [7]  Xilinx Edition II. Modelsim®. 1990.
acteristics, such as a minimum number of modules, specific            [8]  Mingzhe Gao, Jieru Zhao, Zhe Lin, Wenchao Ding, Xiaofeng Hou,
                                                                           Yu Feng,   Chao   Li, and  Minyi Guo.     AutoVCoder:    A Systematic
class types, or a particular density of assertions. This added             Framework for Automated Verilog Code Generation using LLMs, July
control would facilitate more focused testing campaigns for                2024. arXiv:2407.18333.
particular tool features or verification methodologies.               [9]  Yann Herklotz.   Personal communication regarding the design of veri-
    smith, received on november 13th, 2024.

By addressing these challenges and expanding its linguistic [10] Yann Herklotz and John Wickerson. Finding and understanding bugs in breadth, ChiGen’s framework offers a robust foundation for fpga synthesis tools. In FPGA, page 277–287, New York, NY, USA, continued development, promising further insights into the [11] 2020. Association for Computing Machinery. Krzysztof Ko´zmi´nski. Benchmarks for layout synthesis—evolution and stability and robustness of next-generation EDA tools. current status. In DAC, page 265–270, New York, NY, USA, 1991. Association for Computing Machinery. VII. CONCLUSION [12] Mingjie Liu, Nathaniel Pinckney, Brucek Khailany, and Haoxing Ren. Verilogeval: Evaluating large language models for verilog code genera- This paper presented significant advancements to ChiGen, tion, 2023. a ”bottom-up” Verilog fuzzer, through the expansion of its [13] Kevin E. Murray, Scott Whitty, Suya Liu, Jason Luu, and Vaughn Betz. token set and improvements in program correctness. ChiGen Timing-driven titan: Enabling large benchmarks and exploring the gap between academic and commercial cad. ACM Trans. Reconfigurable generates Verilog designs by constructing a syntactic skeleton, Technol. Syst., 8(2), mar 2015. inferring names and types, and injecting additional constructs [14] Wilson Snyder. Verilator 4.0: open simulation goes multithreaded. In using a probabilistic grammar and the Hindley-Milner type [15] Open Source Digital Design Conference (ORConf), 2018. Flavien Solt and Kaveh Razavi. Lost in Translation: Enabling Confused inference. Although these techniques are well-established in- Deputy Attacks on EDA Software with TransFuzz. In USENIX Security, dividually, their integration within ChiGen represents a unique [16] August 2025. approach to Verilog fuzzing. Rafael Sumitani, Jo˜ao Victor Amorim, Augusto Mafra, Mirlaine Crepalde, and Fernando Magno Quint˜ao Pereira. Chibench: a benchmark A key design philosophy of ChiGen is balancing valid and suite for testing electronic design automation tools, 2024. invalid program generation, with approximately 70% of its [17] Shailja Thakur, Baleegh Ahmad, Hammond Pearce, Benjamin Tan, outputs being valid Verilog designs. This deliberate inclusion Brendan Dolan-Gavitt, Ramesh Karri, and Siddharth Garg. Verigen: A large language model for verilog code generation. ACM Trans. Des. of semantically and syntactically invalid programs has proven Autom. Electron. Syst., 29(3), April 2024. effective in uncovering zero-day bugs in several EDA tools, [18] Ning Wang, Bingkun Yao, Jie Zhou, Xi Wang, Zhe Jiang, and Nan as highlighted in Section III. The addition of new tokens to Guan. Large language model for verilog generation with golden code feedback. arXiv preprint arXiv:2407.18271, 2024.

[19] Stephen Williams and Michael Baxter. Icarus verilog: open-source verilog more than a year later. Linux Journal, 2002(99):3, 2002. [20] Claire Wolf. Vloghammer. https://github.com/YosysHQ/VlogHammer, 2021. Accessed: 2024-10-16. [21] Clifford Wolf, Johann Glaser, and Johannes Kepler. Yosys-a free verilog synthesis suite. In Proceedings of the 21st Austrian Workshop on Microelectronics (Austrochip), volume 97, 2013.