Yapay Zeka · 8 dk okuma · 17 Nisan 2026
LLMs hit formal reasoning ceiling; Chomsky Hierarchy reveals efficiency gap
New benchmark shows large language models struggle with structured complexity tasks and require prohibitive compute to achieve reliability in formal reasoning.
Kaynak: arxiv/cs.AI · Yihong Dong, Jianha Xiao, Xue Jiang, Xuyuan Guo, Zhiyuan Fan, Jiaru Qian, Kechi Zhang, Jia Li, Zhi Jin, Ge Li · orijinali aç ↗ ↗
ChomskyBench reveals LLMs face severe efficiency barriers in formal reasoning tasks, with performance tied directly to computational hierarchy levels.
- — ChomskyBench systematically tests LLM formal reasoning across Chomsky Hierarchy levels using language recognition and generation tasks.
- — Performance stratifies clearly by task complexity level, showing LLMs grasp hierarchical structure but with steep efficiency costs.
- — Larger models and advanced inference methods yield relative gains but demand prohibitive computational resources for practical reliability.
- — LLMs are substantially less efficient than traditional algorithms for formal tasks, revealing inefficiency rather than capability limits.
- — Current systems demonstrate that hybrid approaches combining LLMs with symbolic tools remain necessary for robust formal reasoning.
Sık sorulanlar
- LLMs lack the deterministic, step-by-step constraint satisfaction that formal reasoning requires. ChomskyBench shows that as task complexity increases through the Chomsky Hierarchy levels, LLMs require exponentially longer inference sequences and still fail to match the reliability of traditional algorithms. The core issue is efficiency: LLMs solve formal problems through pattern matching rather than logical deduction, making them computationally wasteful for these tasks.