../

FIXME: Towards End-to-End Benchmarking of LLM-Aided Design Verification

Year
2026
Authors
Gwok-Waa Wan, SamZaak Wong, Shengchu Su, Chenxu Niu, Ning Wang, Xinlai Wan, Qixiang Chen, Mengnv Xing, Jingyi Zhang, Jianmin Ye, Yubo Wang, Rongchang Song, Tao Ni, Qiang Xu, Nan Guan, Zhe Jiang, Xi Wang, Yong Chen, Jun Yang
DOI
10.1609/aaai.v40i2.37079

Related

Persistent Notes

In-text annotations

“In this context, a fine-grained evaluation of the capabilities of LLMs in FV is crucial for guiding future research. Unfortunately, existing LAD benchmarks are inadequate for this task: 1. Existing works, such as RTLLM (Lu et al. 2024) and VerilogEval (Liu et al. 2023), mainly evaluate HDL generation and do not involve verification. The hardware designs provided in them are mostly sampled from beginner websites like HDLBits (hdlbits.01xz.net 2024), which cannot reflect the complexity of FV challenges in the real world, so these designs remain insufficient even after modifications. 2. Recently emerging FVEval (Kang et al. 2024) and AssertionBench (Pulavarthi et al. 2025) focus solely on specific scenarios in FV. This scope fails to capture the full breadth and complexity of real-world FV, which includes critical tasks such as specification comprehension (Li et al. 2024), testbench (Yang, Wille, and Drechsler 2014) generation, reference model (Swan 2006), assertion (Lu et al. 2025) design, and debugging (Wang et al. 2025). The time allocated to these tasks in FV is significant, as shown in Figure 1." Page 1087

“Benchmarking has become a key tool for assessing the capabilities of LLMs in hardware design. Most existing benchmarks focus on HDL code generation (e.g., RTLLM (Lu et al. 2024), VerilogEval (Liu et al. 2023), VeriGen (Thakur et al. 2024), and GenBen (Wan et al. 2025)). However, benchmarks specifically targeting the functional verification process remain limited. Although some efforts address formal verification (e.g., FVEval (Kang et al. 2024)), there is a significant gap in comprehensive evaluation on practical, end-to-end simulation-based verification tasks." Page 1088

%% Import Date: 2026-07-11T13:52:02.127-04:00 %%