Files and links (1)
Preprint (Author's original)arXiv.org - Non-exclusive license to distribute, Open
Preprint
DynaCode: A Dynamic Complexity-Aware Code Benchmark for Evaluating Large Language Models in Code Generation
13 Mar 2025
Abstract
The rapid advancement of large language models (LLMs) has significantly
improved their performance in code generation tasks. However, existing code
benchmarks remain static, consisting of fixed datasets with predefined
problems. This makes them vulnerable to memorization during training, where
LLMs recall specific test cases instead of generalizing to new problems,
leading to data contamination and unreliable evaluation results. To address
these issues, we introduce DynaCode, a dynamic, complexity-aware benchmark that
overcomes the limitations of static datasets. DynaCode evaluates LLMs
systematically using a complexity-aware metric, incorporating both code
complexity and call-graph structures. DynaCode achieves large-scale diversity,
generating up to 189 million unique nested code problems across four distinct
levels of code complexity, referred to as units, and 16 types of call graphs.
Results on 12 latest LLMs show an average performance drop of 16.8% to 45.7%
compared to MBPP+, a static code generation benchmark, with performance
progressively decreasing as complexity increases. This demonstrates DynaCode's
ability to effectively differentiate LLMs. Additionally, by leveraging call
graphs, we gain insights into LLM behavior, particularly their preference for
handling subfunction interactions within nested code.
Metrics
16 Record Views
Details
- Title
- DynaCode: A Dynamic Complexity-Aware Code Benchmark for Evaluating Large Language Models in Code Generation
- Creators
- Wenhao HuJinhao DuanChunchen WeiLi ZhangYue ZhangKaidi Xu
- Resource Type
- Preprint
- Language
- English
- Academic Unit
- Computer Science
- Other Identifier
- 991022040181304721