Logo image
Back
A benchmark of expert-level academic questions to assess AI capabilities
Journal article   Open access   Peer reviewed

A benchmark of expert-level academic questions to assess AI capabilities

Long Phan, Alice Gatti, Nathaniel Li, Adam Khoja, Ryan Kim, Richard Ren, Jason Hausenloy, Oliver Zhang, Mantas Mazeika, Dan Hendrycks, …
Nature (London), v 649(8099), pp 1139-1146
01 Jan 2026
DOI: https://doi.org/10.1038/s41586-025-09962-4
PMID: 41606155
url
https://doi.org/10.1038/s41586-025-09962-4View
Published, Version of Record (VoR) Open

Abstract

Multidisciplinary Sciences Science & Technology Science & Technology - Other Topics ESI Highly Cited Paper (Incites)
Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve more than 90% accuracy on popular benchmarks such as Measuring Massive Multitask Language Understanding(1), limiting informed measurement of state-of-the-art LLM capabilities. Here, in response, we introduce Humanity's Last Exam (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be an expert-level closed-ended academic benchmark with broad subject coverage. HLE consists of 2,500 questions across dozens of subjects, including mathematics, humanities and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading. Each question has a known solution that is unambiguous and easily verifiable but cannot be quickly answered by internet retrieval. State-of-the-art LLMs demonstrate low accuracy and calibration on HLE, highlighting a marked gap between current LLM capabilities and the expert human frontier on closed-ended academic questions. To inform research and policymaking upon a clear understanding of model capabilities, we publicly release HLE at https://lastexam.ai.

Metrics

1 Record Views
8 citations in Web of Science

Details

InCites Highlights

Data related to this publication, from InCites Benchmarking & Analytics tool:

Highly Cited Paper 
Collaboration types
Industry collaboration
Domestic collaboration
International collaboration
Web of Science research areas
Computer Science, Artificial Intelligence
Logo image