Natural language is not enough: Benchmarking multi-modal generative AI for Verilog generation

Natural language interfaces have demonstrated their potential in automating Verilog generation from high-level specifications using large language models which receives much attention. However, this paper reveals that for spatially complex hardware structures, visual representations provide additional context critical for design intent, which may outperform only natural language input. Building upon this insight, our paper presents a benchmark of multi-modal generative models for Verilog synthesis from visuallinguistic inputs, encompassing both single modules and complex modules. Additionally, we introduce a visual and natural language Verilog query language to facilitate efficient and user-friendly multi-modal queries. To evaluate the performance of the proposed multi-modal hardware generative AI in Verilog generation tasks, we compare it with a popular method that relies solely on natural language. Our results demonstrate a significant accuracy improvement in the multi-modal generated Verilog compared to queries based solely on natural language. We hope to reveal a new field in the large hardware design model era, thereby fostering a more diversified and efficacious approach to hardware design.

Recommended citation: Chang K, Chen Z, Zhou Y, et al. Natural language is not enough: Benchmarking multi-modal generative AI for Verilog generation. https://yyh-sjtu.github.io/files/natural_language_is_not_enough.pdf