The Complete Guide to LLM Benchmarks (2026): What They Measure, How They Work, and Which One to Use

If you are new to AI development, here is the awkward truth about LLM leaderboards: most of them are wrong about half the time, every model on them is “the best” at something, and the number that actually matters for what you are building is rarely the number on the front page.

This guide is a walking tour through every major LLM benchmark used in 2026 — what it measures, how it scores, what a high number really means, where the measurement quietly falls apart, and most importantly, which benchmark you should care about for your specific use case. There are no marketing slides here. Each section comes with primary-source links so you can verify what I am claiming.

We will move from the simplest knowledge tests, through reasoning and instruction following, into long-context and code, into agents that browse the web and click around in Excel, into vision and video and medical imaging, and finally into the human-preference arenas where users vote on which model they prefer. By the end you should be able to read any leaderboard with calibrated skepticism and pick the two or three benchmarks that actually matter for your project.

This is the third time in three years that the benchmark landscape has reshuffled. MMLU was the dominant frontier test in 2023; by 2024 it was saturated. GPQA replaced it; by mid-2025 the frontier had climbed past PhD-level human performance there too. HLE (Humanity’s Last Exam) was constructed specifically because the frontier was already eating everything else. The pattern will continue. So I will not just hand you a leaderboard snapshot — I will hand you the conceptual tools to read whatever the next set of benchmarks turns out to be.

Pour the tea. This is going to take a while.

Part 0: Why benchmarks exist (and why every one of them lies a little)

A benchmark is a standardized test. You take a frozen set of questions or tasks, you run a model against them, you compute a score, and now everyone can compare models on the same footing. The point is reproducibility — without benchmarks, every claim of “our model is better” is unfalsifiable.

But benchmarks have four structural problems that no amount of careful design fully solves:

1. Saturation. Every useful benchmark eventually gets solved. Once the top ten frontier models all score within 2 points of each other, the benchmark has lost discriminative power — it tells you which models are “in the club”, not which is best. Stanford’s 2026 AI Index found that nearly half of the 60 most-cited LLM benchmarks are now saturated¹.
2. Contamination. If your test questions appear in the model’s training data, the model can simply memorize them. For static, publicly-available benchmarks this is almost impossible to fully prevent — frontier labs scrape the entire indexable web. A 2024 review of MMLU found roughly 6.5% of its questions had ground-truth errors, and 57% of the Virology subset was flagged for problems².
3. The single-number problem. A benchmark gives you one accuracy figure. But a model scoring 84% on a video benchmark might be excellent on 10-second clips and terrible on hour-long videos. The headline number hides everything.
4. Agent-harness dependence. For any benchmark that lets the model use tools — browse the web, run code, edit files — the “score” is partly a measure of the agent scaffolding (prompt template, retry logic, memory) rather than the model itself. Two organizations can report the same model at 60% and 80% on the same benchmark because their scaffolds differ.

Hold these four caveats in your head for the rest of the guide. Every benchmark below has one or more of them.

A practical implication: there is no single “best” benchmark. The right move is to assemble a small portfolio of benchmarks that map to your use case, watch them over time, and treat any individual leaderboard score as a noisy estimate, not a verdict.

Part 1: The anatomy of a benchmark

Before we get into specific tests, let us pin down the vocabulary you will see throughout the guide.

Task format. Benchmarks vary by what they ask the model to produce. The four common shapes are:

• Multiple choice (MCQ). The model sees a question and four (or five, or ten) options and must pick one. Easy to grade automatically. MMLU, GPQA, MMMU and most knowledge benchmarks live here.
• Exact-match short answer. The model emits a string and a script checks whether it matches the reference answer. Used for math benchmarks like AIME where the answer is an integer.
• Open-ended generation, graded by a script. The model writes prose or code, and an automated checker (regex, unit tests, AST comparison) evaluates whether it satisfies the constraints. IFEval, HumanEval, and LiveCodeBench work this way.
• Open-ended generation, graded by a judge. Sometimes the judge is another LLM (“LLM-as-judge”), sometimes a human annotator with a rubric. Used for translation quality (WMT), Chatbot Arena, and any benchmark where the “right” answer is subjective.

Scoring metrics. The metric depends on the task:

• Accuracy — fraction of questions answered correctly. The most common.
• Pass@k — for code, the probability that at least one of $k$ sampled solutions passes all unit tests. Pass@1 means a single attempt; pass@10 means you get ten tries and the best one counts. Introduced by OpenAI for HumanEval³.
• F1 / BLEU / ROUGE / METEOR / TEDS — string-overlap or structure-similarity metrics used for summarization, translation, table extraction, etc.
• MQM (Multidimensional Quality Metrics) — a human-annotation framework for translation that distinguishes accuracy, fluency, style, and terminology errors.
• Elo / Bradley-Terry rating — for head-to-head comparisons (Chatbot Arena), an aggregate strength rating computed from win/loss pairs.
• Task Success Rate (TSR) — binary success on multi-step tasks, used by agent benchmarks (OSWorld, AndroidWorld).

Evaluation regime.

• Zero-shot. The model sees only the question. No examples.
• Few-shot. The model is given a handful of solved examples before the real question. MMLU is canonically evaluated 5-shot.
• Chain-of-Thought (CoT). The model is prompted to write out its reasoning before giving a final answer. Modern frontier models almost always run with CoT enabled for reasoning-heavy benchmarks; for MMLU-Pro this reverses the previous pattern where direct answers were stronger⁴.
• With/without tools. Tool-using runs (search, code execution, file system) are categorically different from plain-text runs.

When you read a leaderboard, always check which regime is being reported. A 5-shot CoT score and a 0-shot direct score are not comparable.

Part 2: Knowledge benchmarks — what does the model know?

These are the oldest and best-known category. They mostly test broad factual coverage in multiple-choice format. They have largely become baselines rather than frontier differentiators, but understanding them is foundational.

MMLU — Massive Multitask Language Understanding

The classic, now a baseline. MMLU was released in September 2020 by Dan Hendrycks and collaborators at UC Berkeley. It contains 15,908 multiple-choice questions across 57 subjects — from elementary mathematics to US history to international law to nutrition. Four options per question, evaluated typically 5-shot²⁵.

When MMLU launched, GPT-3 175B scored 43.9% and most language models hovered near the 25% random-chance baseline. By 2024, frontier models were sitting at 88%, brushing against the estimated human-expert ceiling of 89.8%. That is the saturation signal — once the headline number is at the human ceiling, the benchmark has stopped measuring anything new.

How it is measured. Each question is presented as Q: ... A) ... B) ... C) ... D) .... The model’s predicted letter is compared to the ground truth. The aggregate score is the average accuracy across all 57 subjects.

What a high score means. Broad factual knowledge across academic and professional domains — the “encyclopedic assistant” capability.

What it does not tell you. Whether the model can reason. Whether the answers were memorized. Whether the questions themselves are even correct (a 2024 review found ~6.5% of MMLU questions contain errors, with 57% of Virology questions flagged for ambiguity or incorrect answer keys²).

Use it for: quickly checking that a model is not broken or under-trained. Do not use it for: comparing frontier models.

MMLU-Pro

The harder successor. MMLU-Pro keeps the multi-discipline format but makes three changes: each question now has 10 options instead of 4 (cutting guessing rate), trivial questions are removed, and more questions require multi-step reasoning rather than recall⁴.

Scores drop 16–33 percentage points compared to MMLU. The pattern with respect to CoT also flips — MMLU-Pro rewards CoT prompting, confirming that the benchmark genuinely tests reasoning. Prompt-sensitivity also drops from 4–5% on MMLU to 2% on MMLU-Pro, making evaluations more stable.

Use it for: mid-tier model comparison and as a smoke test that does not saturate as quickly as MMLU.

MMLU-ProX — multilingual extension

MMLU-Pro questions translated into 29 languages by a multi-LLM translation pipeline with expert human review. Each language has 11,829 questions (or 658 in the “lite” subset)⁶.

The benchmark exists because multilingual performance is asymmetric — performance gaps of up to 24.3 percentage points exist between the highest- and lowest-resource language versions of the same question across 36 evaluated models. The translation step itself introduces artifacts; culturally-bound concepts often translate poorly.

Use it for: sanity-checking whether your model’s “intelligence” is truly multilingual or just English wearing translated clothes.

C-Eval — Chinese academic knowledge

The first comprehensive Chinese-language evaluation suite for LLMs. Built in 2023 by Tsinghua, Edinburgh, and HKUST. Multiple-choice across four difficulty levels (middle school, high school, college, professional) and 52 disciplines. A C-Eval Hard subset isolates the most demanding subjects⁷.

Use it for: models being deployed into Chinese-language production use cases. Western-pretrained models often score poorly here despite high MMLU scores.

GPQA — Graduate-level Google-proof Q&A

The benchmark that replaced MMLU as the frontier differentiator. 448 multiple-choice questions written by PhD-level domain experts in biology, physics, and chemistry. The defining design constraint: skilled non-experts with unrestricted web access can only solve 34% of them in 30 minutes. The answers cannot be Googled — you have to reason them out⁸⁹¹⁰.

The most commonly reported subset is GPQA Diamond — the 198 hardest questions.

When GPQA launched, GPT-4-based baselines scored 39% and PhD-level human experts averaged 65% (74% if you discount their self-admitted mistakes). By 2026 the top frontier models are above 90% on Diamond, and the benchmark is now itself approaching saturation¹⁰.

Use it for: evaluating any model intended to do scientific or technical reasoning. Treat with caution: the narrow STEM focus (just bio/phys/chem) is a real limitation, and the 448-question size means sampling variance is non-trivial.

SuperGPQA — 285 disciplines

GPQA’s broader cousin. Same expert-level approach, but expanding to 285 graduate-level disciplines from art history to engineering to linguistics. Built using a human–LLM collaborative filtering loop with 80+ expert annotators¹¹.

The best-performing model at publication (DeepSeek-R1) scored 61.82%, indicating substantial headroom compared to GPQA’s near-saturation. SuperGPQA covers the breadth that GPQA misses, but the human–LLM annotation pipeline can introduce subtle biases if the filter-LLMs share training data with the test-takers.

Use it for: measuring genuine encyclopedic expertise across the full academic spectrum.

TruthfulQA, HellaSwag, WinoGrande — the older suite

These predate the modern frontier era and are mostly saturated, but you still see them on every leaderboard:

• TruthfulQA — 817 questions across 38 categories designed to elicit confident hallucinations (health, law, conspiracy theories). Reported hallucination rates above 50% for baseline LLMs, but the benchmark itself is now contaminated and contains some incorrect “gold” answers. Frontier models still produce confident falsehoods, but TruthfulQA may not be the right tool to detect them anymore¹².
• HellaSwag — sentence-completion commonsense. The model picks the most plausible continuation from four options. Saturated at 95%+ for frontier models.
• WinoGrande — 44,000 adversarially-filtered Winograd-style pronoun-resolution problems. Tests commonsense inference; also largely saturated.

Use them for: confirming a new model has not regressed. Do not use them for: frontier comparison.

Part 3: Reasoning and Math — what does the model figure out?

The single fastest-moving category of 2024–2026. Knowledge benchmarks measure what a model has stored; reasoning benchmarks measure what it can construct on the fly.

HLE — Humanity’s Last Exam

Released January 2025 by the Center for AI Safety and Scale AI. Built precisely because MMLU, MMLU-Pro, and GPQA had all saturated. HLE contains 2,500 public questions (plus a private held-out set) hand-written by nearly 1,000 domain experts at over 500 institutions¹³¹⁴.

The defining constraint: every question had to stump all frontier models available at submission time.

Subject mix: mathematics (41%), biology/medicine (11%), CS/AI (10%), physics (9%), humanities/social science (9%), chemistry (7%), engineering (4%), other (9%). About 14% require multimodal (image + text) understanding; 24% are multiple choice, the rest demand exact short-answer responses.

How it is measured. Automated exact-match for short-answer questions, MCQ accuracy for the rest.

What scores look like. Even with reasoning-optimized frontier models, top public-leaderboard scores in early 2026 sit in the 30–35% range¹⁵. Anything above ~35% currently represents the cutting edge.

Use it for: distinguishing frontier reasoning models from one another. Limitations: small per-subject sample sizes, the exact-match grader can penalize correct answers with formatting differences, and the benchmark will eventually saturate as the field improves (the Nature paper explicitly acknowledges this).

AIME — American Invitational Mathematics Examination

The second-tier US high school math olympiad qualifier. Each year’s exam has 15 problems whose answers are integers 0–999. AIME 2025 and AIME 2026 are now used as direct LLM benchmarks because the fresh annual release reduces contamination risk¹⁶¹⁷.

Humans get 3 hours; LLMs are typically uncapped on time. Strong high-school competitors solve 5–7 of 15. Frontier models in 2026 sit near perfect scores on AIME 2025, so AIME 2026 is the current fresh variant.

Use it for: measuring creative multi-step mathematical reasoning across algebra, geometry, number theory, combinatorics, and probability. Caveat: the integer-answer format means partial credit is impossible — you either nailed the final number or you did not.

MATH-500

A 500-problem subset of the Hendrycks MATH dataset, a classic dataset of high-school competition math. MATH-500 was popularized by OpenAI as a reasoning-focused subset balanced across difficulty levels. Frontier models in 2026 score in the high 90s. Use it as a quick sanity-check rather than a frontier differentiator.

HMMT and IMO — the hardest competition math

The Harvard-MIT Mathematics Tournament (HMMT) and the International Mathematical Olympiad (IMO) represent the hardest pre-graduate mathematics in the world. Solutions usually require elegant multi-step proofs, not just final-answer matching. IMO problems in particular have been the benchmark that broke many reasoning models in 2024–2025.

Scoring relies partly on LLM-as-judge for partial proof credit, which introduces inconsistency. The small problem count (IMO has 6 problems per year) makes variance high. But a model that performs well here is genuinely doing creative theorem-prover-level reasoning.

FrontierMath — beyond the olympiad

Built by Epoch AI explicitly because olympiad math was getting solved. FrontierMath contains hundreds of unpublished research-level problems authored by professional mathematicians. Problems are tiered 1–4 by difficulty; tier 4 problems can take a research mathematician multiple days¹⁸.

A typical FrontierMath problem requires multiple hours of effort from a domain specialist. As of April 2026, top frontier models sit just over 50% on tiers 1–3 and roughly 30% on tier 4 — substantial headroom by any measure¹⁹.

There is also a “FrontierMath Open Problems” track containing problems mathematicians have actually tried and failed to solve. An AI system solved one of them in 2026 — a genuinely novel mathematical result, not a benchmark score.

Use it for: evaluating any model marketed as a “research-grade” or “PhD-level” reasoning system.

ARC-AGI-2 — fluid intelligence, not knowledge

The benchmark from François Chollet’s ARC Prize. The format is unusual: each task is a handful of input-output pairs on a colored-square grid. The model must infer the transformation rule and apply it to a new input. The benchmark explicitly targets fluid intelligence — solving genuinely novel problems from minimal examples — rather than accumulated knowledge²⁰.

ARC-AGI-2 tightens three things over ARC-AGI-1: symbolic interpretation (assigning meaning to visual elements), compositional reasoning (multiple interacting rules), and contextual adaptation (the same visual symbol meaning different things in different tasks). Average human performance on a calibrated study with 400+ participants in San Diego was around 60%.

In 2026, frontier general-purpose LLMs are climbing fast on ARC-AGI-2; meta-systems built specifically to attack the benchmark have pushed past 95%. The leaderboard now distinguishes “private (no internet, fixed compute)” runs from open ones, because compute scaling matters enormously here²¹.

Use it for: measuring novel-problem reasoning. Treat with caution: general LLM scores depend heavily on how the model is prompted to think.

Part 4: Instruction Following — does the model do what it is told?

In production systems, instruction adherence often matters more than raw intelligence. A brilliant model that ignores your formatting constraints will break your pipeline; a less brilliant model that obeys constraints will not.

IFEval — verifiable instructions

Built by Google Research to escape the LLM-as-judge bias problem. IFEval focuses on instructions that a simple script can verify programmatically: “write more than 400 words”, “mention ‘machine learning’ at least 3 times”, “format as a JSON list”, “do not use the word ‘however’”. About 500 prompts, each embedding one or more of 25 verifiable instruction types²²²³.

Scoring is at two levels — prompt-level (did all constraints in this prompt pass?) and instruction-level (what fraction of individual constraints across all prompts were met?).

Use it for: evaluating reliability for production pipelines, tool-use, structured-output APIs, and any place where output format failure breaks downstream systems.

Limitations: the 25 instruction types only cover what is programmatically verifiable. Subjective qualities (“be friendly”, “explain at a 10th-grade level”) are out of scope. Single-turn only.

IFBench and MultiChallenge — harder instruction following

These extend IFEval by stacking conflicting constraints, using longer prompts, and testing more realistic instruction shapes. MultiChallenge specifically tests scenarios where instructions change mid-conversation or create logical tension (“be concise, but cover all five points fully”). Some metrics fall back to LLM-as-judge for qualitative constraints, which reintroduces judge bias.

Use them for: evaluating instruction robustness under enterprise-grade complexity.

Part 5: Long Context — does the model use all that context window?

Context windows expanded from 4K tokens in 2022 to 1M+ tokens by 2025. A new benchmark class emerged to ask: when you stuff a million tokens in, does the model still actually use them?

LongBench v2

The current standard. 503 challenging multiple-choice questions with contexts from 8,000 to 2 million words across six task categories: single-document QA, multi-document QA, long in-context learning, long-dialogue history, code repository understanding, and long structured data²⁴²⁵.

The calibration is brutal: human experts given 15 minutes with in-document search scored 53.7%. The best direct-answer model scored 50.1%. Reasoning-optimized models (the o1-preview class) cracked the human baseline at 57.7%.

Use it for: any application involving long documents — RAG pipelines, codebase analysis, novel/script understanding, multi-document research workflows. Caveats: evaluation is expensive (you are processing millions of tokens per question), and the “lost-in-the-middle” problem varies unpredictably by model and context type.

Older long-context probes

You will still see “Needle in a Haystack” and “RULER” referenced. These are stress tests rather than reasoning benchmarks — they insert a fact at a specific position in a long context and check whether the model retrieves it. Useful to detect catastrophic failure (the model truly cannot see that part of the context); not a replacement for LongBench v2.

Part 6: Code — can the model actually program?

This category has evolved farthest from its origins. The 2021-era benchmarks (HumanEval, MBPP) tested single-function completion. The 2026 benchmarks test whether the model can navigate a 500-file repository, understand a GitHub issue written by a human in three sentences, and produce a multi-file patch that passes the test suite.

HumanEval and MBPP — the classics

HumanEval is OpenAI’s 164 hand-crafted programming problems, each specified by a function signature, a docstring, and hidden unit tests. The model writes the function body. Scoring is pass@k, the probability that at least one of $k$ generated samples passes all tests. Pass@1 with greedy decoding is the canonical reported metric³.

def has_close_elements(numbers: List[float], threshold: float) -> bool:
    """
    Check if in given list of numbers, are any two numbers closer to each
    other than given threshold.
    >>> has_close_elements([1.0, 2.0, 3.0], 0.5)
    False
    >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)
    True
    """

That is a real HumanEval prompt. The model’s output is dropped into a sandbox and the unit tests run.

MBPP (Mostly Basic Python Problems) is the Google equivalent — 974 entry-level Python tasks. Same pass@k methodology.

Both benchmarks are thoroughly saturated. Frontier models score in the high 90s. HumanEval+/MBPP+ (the EvalPlus extensions²⁶) added stronger test cases that cut early reported scores substantially, but those too are now near-saturation.

Use them for: smoke testing a new model can write Python at all. Do not use them for: frontier comparison.

LiveCodeBench — contamination-resistant code

LiveCodeBench solves the contamination problem by continuously collecting fresh competitive-programming problems from LeetCode, AtCoder, and Codeforces. Every problem is tagged with its release date. You evaluate the model only on problems released after its training cutoff, making contamination detectable by construction²⁷²⁸.

Beyond code generation, LiveCodeBench tests four distinct capabilities: (1) code generation, (2) self-repair (given failing test cases, fix the code), (3) test output prediction, and (4) code execution simulation.

LiveCodeBench v6 is the current version in 2026 evaluations.

Use it for: algorithmic problem-solving capability — competitive programming, interview-prep assistants, algorithm-heavy production code. Caveat: competitive programming is not the same as professional software engineering; a great LiveCodeBench model may be mediocre at refactoring a messy 50-file codebase.

SWE-bench Verified — real GitHub issues

The most cited software-engineering benchmark of 2025–2026. The full SWE-bench dataset contains 2,294 issues from 12 popular open-source Python repositories (Django, Flask, scikit-learn, Sphinx, requests, sympy, and so on). SWE-bench Verified is a 500-issue subset that OpenAI hand-validated to ensure each issue is solvable, unambiguous, and has fair test coverage²⁹³⁰.

The task: given a codebase snapshot and a GitHub issue description, produce a patch that passes all the hidden tests associated with the issue. Pass@1 is the metric.

Scores climbed from 4.4% in 2023 to the high 80s and low 90s for frontier models in 2026³¹. But Verified is now hitting saturation concerns: research found that 60.83% of successfully resolved SWE-bench Lite issues contain “solution leakage” (hints embedded in the issue description), and 47.93% of resolved issues pass tests too weak to verify correctness³². When weak-test issues are filtered out, resolution rates drop substantially.

Use it for: evaluating real coding-assistant quality on Python. Look beyond it for: non-Python languages and harder, less-leaky tasks.

SWE-bench Pro — the harder successor

OpenAI’s response to SWE-bench Verified saturation. SWE-bench Pro contains 1,865 tasks spanning multiple languages, drawn from a much broader repository pool. Tasks require coordinated changes across an average of 4.1 files per issue — testing whether the agent understands cross-file dependencies, not just local code patches³³.

The gap between Verified and Pro scores is striking. The same model can score in the 80s on Verified and the 40s–60s on Pro. As of mid-2026, Pro is the frontier signal for coding-agent quality.

Use it for: any 2026-and-beyond evaluation of coding-agent products.

SWE-bench Multilingual

Extends SWE-bench to 9 languages: C, C++, Go, Java, JavaScript/TypeScript, PHP, Ruby, and Rust. 300 curated tasks from 42 repositories³⁴. Models heavily-trained on Python often crash here — the 2025 SWE-PolyBench analysis found language-specific scores varying by 30+ percentage points within the same model³⁵.

Use it for: any polyglot codebase. If your team works in Go or Rust, do not trust Python SWE-bench scores as a proxy.

Terminal Bench 2 and SecCodeBench

Terminal Bench 2 evaluates models as agents operating in a real Unix shell — executing commands, managing files, writing scripts, navigating directories with only the terminal. This is closer to real DevOps and backend engineering work than either LeetCode-style problems or GitHub issue patches.

SecCodeBench tests whether the model generates secure code by default — knowledge of common vulnerability patterns (injection, buffer overflows, insecure deserialization), best-practice habits, and ability to identify vulnerabilities in existing code. As AI-generated code lands in production, this matters more every quarter.

Part 7: Tool Use and Agents — can the model do things?

A growing class of benchmarks tests whether a model can take an instruction, decide which tool to call, call it correctly, parse the result, decide what to do next, and ultimately complete a multi-step real-world task. These benchmarks are uniquely sensitive to agent-scaffold engineering.

BFCL — Berkeley Function Calling Leaderboard

The de facto standard for tool-use evaluation. Published by the Gorilla team at UC Berkeley. BFCL evaluates: invoking single functions correctly, parallel function calls, sequential chained calls, reasoning about when not to call a function, and stateful multi-turn agentic conversation. Coverage spans Python, Java, JavaScript, REST APIs, and over 2,000 question–function–answer pairs³⁶³⁷³⁸.

How it scores. Abstract Syntax Tree (AST) comparison — the benchmark checks the structure of generated tool calls rather than executing them, which lets it scale to thousands of functions efficiently. Both native function-calling APIs and prompt-only modes are evaluated. The overall accuracy is the unweighted average across sub-categories.

Use it for: evaluating tool-using application reliability — APIs, code-execution environments, database query agents.

Limitations: AST evaluation does not execute tools. A model that generates structurally-correct calls might still fail at runtime. Function-description quality matters enormously — poor docstrings inflate apparent model weakness. And BFCL does not test planning under uncertainty or error recovery, since it is not running tools in a real environment.

TAU2, VITA, DeepPlanning, MCP-Mark

Newer benchmarks attacking specific dimensions of agentic behavior:

• τ-bench / TAU2 — multi-domain end-to-end task completion using real APIs (airlines, e-commerce, customer service simulators). Tests goal maintenance, error recovery, and multi-turn dialogue with both a user and a system.
• VITA-Bench — simulated environments requiring complex planning, resource allocation, and sequential decision-making.
• DeepPlanning — long-horizon planning specifically. Can the model formulate and stick to a multi-step plan without getting distracted or making irreversible early mistakes?
• MCP-Mark — tool-use within the Model Context Protocol (MCP) ecosystem, evaluating performance with real MCP servers (GitHub, code tools, browsers). Directly relevant if you are building MCP-connected agents.

All four use task success rate (TSR) as the headline metric, sometimes with partial credit for completed sub-goals.

Use them for: evaluating autonomous-operation viability for complex real-world workflows.

Limitations: agent-scaffold dependence is severe. The same model with different prompts/memory strategies can swing 20+ points.

Part 8: Search Agents — can the model use the web?

As AI assistants gain real browser access, a distinct benchmark class emerged to evaluate web research.

BrowseComp and BrowseComp-zh

OpenAI’s BrowseComp measures one specific skill: answering questions that cannot be answered from training knowledge alone. The model must plan a search strategy, open and inspect sources, filter noisy or misleading pages, and synthesize a final answer. BrowseComp-zh applies the same design to the Chinese web, where the information ecosystem differs fundamentally³⁹⁴⁰.

BrowseComp-zh contains 289 multi-hop questions across 11 domains. Most models score below 10%; the best published system in 2025 reached 42.9%.

Use it for: research-assistant applications — deep-research agents, fact-checkers, information discovery.

Limitations: scores depend heavily on the search API used, its index freshness, rate limits, and context management. The same model can score 10+ points apart with a premium vs basic search API. BrowseComp-Plus was introduced to address this by using a fixed, curated corpus⁴¹.

WideSearch and Seal-0

Extend the web-search paradigm. WideSearch broadens domain coverage; Seal-0 tests multi-step synthesis where the answer requires combining results across multiple pages over multiple reasoning steps. Both share BrowseComp’s harness-dependence problem.

Part 9: Multilingual

Beyond the English-centric benchmarks above, evaluating LLMs across languages requires culturally authentic questions and tasks that reflect real multilingual use.

MMMLU and INCLUDE

MMMLU is the multilingual MMLU. INCLUDE goes further — rather than translating English questions, it sources questions from local academic and cultural contexts in each language. INCLUDE catches the failure mode where a model “speaks” a language but doesn’t know that language’s culture.

WMT24++ — translation quality

WMT (Workshop on Machine Translation) is the long-running annual MT evaluation campaign. WMT24++ extended the 2024 evaluation to additional language pairs and integrated LLM-based systems alongside traditional neural MT. Human assessment uses MQM (Multidimensional Quality Metrics) — a detailed error taxonomy distinguishing accuracy, fluency, style, and terminology⁴²⁴³.

Automatic metrics (COMET, xCOMET, MetricX) are used for fast evaluation, with correlation to human judgment as a quality check. The WMT25 evaluation used GPT-4.1 and CommandA as LLM-as-judges alongside reference-based and quality-estimation metrics.

Use it for: translation-focused products. Caveat: the official MQM results lag by months because human annotation is slow. COMET-based numbers are faster but can mislead for LLM outputs that differ stylistically from references.

PolyMATH

Tests whether a model can do math in the language the problem is presented in, without internally translating to English first. Important for educational and STEM support tools in non-English-speaking markets.

Part 10: Vision-Language — STEM, VQA, and reasoning over images

Multimodal benchmarks test the fusion of visual and textual understanding. The category has fragmented into many sub-types as models have become genuinely competent.

MMMU — the multimodal MMLU

MMMU (Massive Multi-discipline Multimodal Understanding and Reasoning) is 11,500 college-exam-derived questions across six disciplines (Art & Design, Business, Science, Health & Medicine, Humanities/Social Science, Tech & Engineering) and 30 subjects. Image types span charts, diagrams, maps, tables, music sheets, and chemical structures⁴⁴⁴⁵.

When MMMU launched, GPT-4V scored 56% and Gemini Ultra 59%. The Stanford 2026 AI Index reports that MMMU performance improved 18.8 percentage points from 2023 to 2024 alone¹.

Use it for: broad multimodal reasoning capability.

Caveat: research has shown models can score 42.7% on MMMU without seeing any images (GeminiPro with images blanked). This means the language model’s knowledge alone partially drives the score. The MMStar benchmark was created specifically to fix this⁴⁶.

MathVista and MathVision

MathVista — 6,141 examples derived from 28 multimodal math datasets plus three new ones (IQTest for visual IQ puzzles, FunctionQA for graph analysis, PaperQA for math in research papers). GPT-4V scored 49.9% at launch; o1 set the bar at 73.9% in 2024; frontier models now exceed average human worker performance⁴⁷.

MathVision (MATH-V) — addresses MathVista’s diversity gap by sourcing 3,040 problems directly from real math competitions across 16 disciplines and 5 difficulty levels. As of mid-2025, Seed1.5-VL led at 68.7% and the best open-source model at 49.7%, indicating this benchmark still has frontier headroom⁴⁸.

Use them for: any product involving math over diagrams, charts, or visual representations — educational tools, scientific paper assistants, dashboard interpreters.

MMBench, MMStar, HallusionBench

MMBench — broad VQA covering objects, scenes, attributes, spatial relations.

MMStar — filters out MMBench-style questions answerable without vision. 1,500 human-curated samples benchmarking six core visual capabilities and 18 detailed axes. This is the right benchmark when you care whether the model is actually looking at the image⁴⁶.

HallusionBench — explicitly targets visual hallucination: fabrication of details not in the image, or contradiction of the image with plausible-sounding falsehoods. A high HallusionBench score indicates a model grounded in what it sees rather than what it expects to see.

SimpleVQA and RealWorldQA

SimpleVQA — basic factual VQA with straightforward answers derived from internet images. Distinguishes factual accuracy from reasoning fluency.

RealWorldQA — spatial and physical understanding of everyday photographic scenes. Not academic or synthetic images, but real-world environments.

Part 11: Vision-Language — Documents, Spatial, Video

OmniDocBench

CVPR 2025 benchmark for document parsing — academic papers, financial reports, newspapers, textbooks, handwritten notes, slides. Version 1.5 covers 1,651 PDF pages with 28 block-level and 4 span-level annotation categories. Evaluates layout detection, table recognition, formula recognition, text OCR, and reading-order recovery⁴⁹⁵⁰.

Multiple metrics are used simultaneously: Normalized Edit Distance for text, BLEU/METEOR for paraphrase quality, TEDS for table structure, COCODet (mAP/mAR) for layout. Lower edit distance is better.

Use it for: any enterprise PDF-processing pipeline. This is one of the most immediately commercially valuable vision capabilities.

OCRBench and CharXiv

OCRBench — focused OCR evaluation across text styles, fonts, distortions, low-resolution images, and handwriting.

CharXiv — chart-understanding specifically. Bar charts, line graphs, scatter plots, tables-as-images. Tests combined visual perception and arithmetic reasoning (“What was the second-highest value in 2022?”, “Which series shows the fastest growth?”).

V*, ERQA, CountBench — spatial intelligence

V* — referring image segmentation and grounding. Locate objects in a scene based on natural-language descriptions including spatial relationships (“the cup behind the laptop”).

ERQA — egocentric (first-person) referring QA. Relevant for AR/VR and robotics.

CountBench — object counting in cluttered scenes. Most models are surprisingly bad at this.

VideoMME — the video standard

900 videos totaling 254 hours, 2,700 human-annotated QA pairs across six visual domains and 30 subfields. Critically, video durations span from 11-second clips to 1-hour videos, testing both short and long video understanding⁵¹⁵².

VideoMME has become the de facto industry standard — GPT-4.1 called it “an industry standard measure of multimodal long context ability”. Frontier models in 2026 sit above 85% on the overall score, though performance declines on the longest-video subset.

Use it for: video summarization, meeting analysis, tutorial extraction, sports analysis, multimodal content moderation.

Caveat: “video evaluation” usually samples only 8–384 frames from an hour-long video. Performance at true high-temporal-resolution is not what is being tested.

VideoMMMU, MLVU, MVBench, LVBench

• VideoMMMU — multi-discipline academic video questions.
• MLVU — long-form (30+ minute) video tasks including plot summarization, anomaly detection, and needle-in-a-haystack retrieval.
• MVBench — 20-task benchmark for temporal reasoning, action recognition, scene understanding.
• LVBench — hours-long videos with questions requiring integration across distant temporal positions.

Part 12: Vision-Language — Visual Agents (Computer Use)

The hottest category of 2025–2026. “Can this model use a computer?”

ScreenSpot Pro

GUI grounding — locating specific UI elements in high-resolution professional software screenshots. 23 applications across 5 categories (development tools, creative apps, CAD, engineering, office software), 3 operating systems. Tasks annotated by professional users⁵³⁵⁴.

Localization accuracy from natural-language instructions (“click the ‘Format Cells’ button”). Best models at launch were under 20%; frontier 2026 models are in the mid-80s.

Use it for: any “computer use” or RPA product. If the model cannot ground UI elements, it cannot reliably automate anything.

OSWorld-Verified

Realistic open-ended desktop tasks across Ubuntu, Windows, and macOS — Chrome, LibreOffice, GIMP, VS Code. Hand-validated subset of the original OSWorld benchmark, addressing 300+ community-identified issues⁵⁵⁵⁶.

Task success rate. When OSWorld launched, humans completed 72.36% of tasks while the best model managed 12.24%. Substantial progress since, but the human-AI gap on complex multi-application tasks remains real.

Use it for: any autonomous desktop productivity agent.

AndroidWorld

Mobile equivalent. 116 core tasks parameterized dynamically across 20 real Android applications. Binary success rewards validated by ADB system-state inspection — avoiding the trap of “the screenshot looks right but the underlying state is wrong”⁵⁷⁵⁸.

When AndroidWorld launched, the best agent completed 30.6% of tasks. By mid-2025 some agents reached 90%+, motivating the harder MobileWorld benchmark (27.8-step average tasks vs 14.3 for AndroidWorld, 62.2% multi-app tasks vs 9.5%).

Use it for: mobile-automation products.

Part 13: Medical Vision-Language

MedXpertQA

Among the most clinically realistic medical AI benchmarks. 4,460 questions across 17 medical specialties and 11 body systems. Two subsets: MedXpertQA Text and MedXpertQA MM (multimodal, with X-rays, ECGs, histology slides, dermatology photos). Questions sourced from USMLE, COMLEX-USA, and specialty board exams — the highest-stakes medical licensing tests⁵⁹⁶⁰.

MCQ accuracy with 5 options. Adaptive Brier Score filtering ensures questions are genuinely difficult. Multiple rounds of licensed-physician review.

Use it for: clinical decision support quality measurement.

Critical caveat: strong MedXpertQA performance does not translate to clinical safety. Real medical AI requires appropriate uncertainty expression, deference-to-humans calibration, and robustness to incomplete information — none of which MedXpertQA tests. Regulatory requirements for clinical deployment go far beyond benchmark scores.

SLAKE and PMC-VQA

SLAKE — bilingual (English/Chinese) medical VQA pairing clinical images with structured QA on anatomy, modality-specific interpretation, and disease recognition.

PMC-VQA — questions sourced from PubMed Central research papers about figures and clinical images as they appear in published medical literature. Useful for research-assistant and evidence-synthesis applications.

Part 14: Human Preference — Chatbot Arena

Every benchmark so far has had an objective right answer. Chatbot Arena (formerly LMSYS, now also branded LMArena and Arena AI) takes the opposite approach: humans compare two anonymous model responses side-by-side and vote for the one they prefer. The platform now has over 6 million votes and 37 months of history⁶¹⁶².

How it scores. Originally the classic Elo system; since late 2023 the platform migrated to the Bradley-Terry model, a maximum-likelihood estimator over the full match history. Bradley-Terry is more stable than online Elo when models are static (their performance is not changing) and produces meaningful confidence intervals.

The frontier rating range in 2026 sits roughly between 1,450 and 1,560 (Arena ratings have no fixed ceiling; they drift with the population).

Use it for: judging conversational quality, response style, and aggregate user preference — things that no objective benchmark can capture.

Limitations: vote bias is real. Users tend to prefer longer responses, more confident-sounding responses, and responses with markdown formatting. Models can be optimized for arena-pleasing without actually being more accurate or useful — there is open debate in 2026 about whether the leaderboard is “vibes-based”⁶³. Several models with apparently strong Arena scores underperform on objective benchmarks in their domain.

A useful heuristic: a model that ranks well on both Chatbot Arena and an objective benchmark for your use case is a safer bet than one that wins only on Arena.

Part 15: Holistic Frameworks — HELM and friends

The benchmarks above are individual tests. HELM (Holistic Evaluation of Language Models) from the Stanford Center for Research on Foundation Models is a framework that runs many tests against a model under standardized conditions and reports seven metrics for each: accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency⁶⁴⁶⁵.

The original HELM paper evaluated 30 prominent models on 42 scenarios, 21 of which had not previously been used in mainstream evaluation. The motivation was that before HELM, models were on average evaluated on just 17.9% of HELM’s core scenarios — and some prominent models shared zero benchmark scenarios in common. HELM improved this to 96.0% coverage⁶⁶.

HELM has extended to vision-language (VHELM), text-to-image (HEIM), and medical (MedHELM).

Use it for: systematically comparing model portfolios when you need defensible multi-dimensional evidence rather than a single-number ranking.

lm-evaluation-harness

The other major framework, from EleutherAI. It is the backend for Hugging Face’s Open LLM Leaderboard and is used internally by NVIDIA, Cohere, BigScience, BigCode, Nous, and Mosaic, among others⁶⁷.

# Install
git clone https://github.com/EleutherAI/lm-evaluation-harness
cd lm-evaluation-harness
pip install -e .

# Run MMLU on a HuggingFace model
python -m lm_eval \
    --model hf \
    --model_args pretrained=meta-llama/Llama-3.2-3B-Instruct,trust_remote_code=True \
    --tasks mmlu \
    --num_fewshot 5 \
    --device cuda:0 \
    --batch_size 4 \
    --output_path ./results/llama32-mmlu \
    --log_samples

You can also use it as a Python library:

from lm_eval import evaluator
from lm_eval.models.huggingface import HFLM

model = HFLM(pretrained="meta-llama/Llama-3.2-3B-Instruct", device="cuda:0")

results = evaluator.simple_evaluate(
    model=model,
    tasks=["mmlu", "gpqa_diamond_cot_zeroshot", "ifeval"],
    num_fewshot=5,
    batch_size=4,
)
print(results["results"])

Use it for: reproducible internal evaluations of open-weights models you are deploying yourself. Use the official benchmark harness (BFCL’s, SWE-bench’s, OSWorld’s, etc.) for any benchmark that is not in lm-eval-harness’s task registry — agent benchmarks especially require their own runners.

Part 16: Cross-cutting issues — what every benchmark gets wrong

Returning to the four structural problems from Part 0, now with more vocabulary.

Saturation

The headline finding from Stanford’s 2026 AI Index analysis of 60 major benchmarks: nearly half are saturated⁶⁸. SWE-bench Verified, MMLU, MMLU-Pro, HumanEval, MBPP, HellaSwag, WinoGrande, AIME 2025 — all have top models clustered within 2–3 percentage points. This is why MMLU → MMLU-Pro → SuperGPQA → HLE happens; the field constantly invents harder benchmarks to escape saturation, and the next saturation arrives within 18 months.

A practical implication: track multiple “generations” of benchmarks simultaneously. MMLU saturation just means everyone is competent at MMLU. It does not tell you who is best. For that, look at HLE, FrontierMath, ARC-AGI-2, SWE-bench Pro.

Data contamination

If a model has seen the test questions during training, the benchmark measures memorization rather than capability. Public benchmarks are especially vulnerable — frontier labs scrape essentially all indexable text.

A 2025 ICML paper systematically evaluated 20 contamination-mitigation strategies and found none effectively balanced fidelity preservation against contamination resistance⁶⁹. A counterpoint paper showed that sufficiently large-scale training (5× Chinchilla-optimal) may cause models to “forget” contaminating examples in proportion to scale — making the picture more nuanced⁷⁰.

The practical defenses are: (a) live benchmarks like LiveCodeBench that continuously add fresh problems; (b) annual benchmarks like AIME that release new problem sets yearly; and (c) private held-out sets like HLE’s private portion. Trust live and private benchmarks more than purely public ones.

Agent-framework dependence

For BFCL, OSWorld, SWE-bench, BrowseComp, and every other agentic benchmark, the agent harness — the system prompt, tool description quality, memory management, retry logic, context management — accounts for a substantial portion of the score. Different organizations using different scaffolds will report wildly different numbers for the same underlying model.

When you see an agent-benchmark score, ask: “what scaffold produced this?” If the scaffold is not documented, the number is approximate at best.

The single-number trap

A score of 84% on VideoMME hides that the model crushes 1-minute clips and fails on 30-minute ones. A score of 90% on SWE-bench Verified hides that 48% of the resolved issues have weak tests. A score of 94% on GPQA hides that minor question rephrasings can flip many answers.

Always read the sub-task breakdowns. The leaderboard headline is for marketing; the table on page 4 of the technical report is for engineering.

Part 17: The decision matrix — which benchmark for which use case?

The point of all this is to choose well. Here is the practical mapping.

Building a conversational assistant or chatbot. Primary: Chatbot Arena, MT-Bench (or successors), IFEval. Secondary: MMLU-Pro for baseline competence, MultiChallenge for instruction robustness, HallusionBench for visual grounding (if multimodal).

Building a coding assistant or autonomous coding agent. Primary: SWE-bench Pro, LiveCodeBench v6, BFCL. Secondary: SWE-bench Multilingual (if not Python-only), Terminal Bench 2 (if shell-heavy), SecCodeBench (if security-sensitive).

Building a RAG/document-chat pipeline. Primary: LongBench v2, OmniDocBench (for PDF parsing), IFEval. Secondary: BrowseComp / WideSearch (if web retrieval), MMStar (if document images).

Building a deep-research agent. Primary: BrowseComp / BrowseComp-Plus, HLE, GPQA Diamond. Secondary: SuperGPQA for domain breadth, MMLU-Pro for general grounding, LongBench v2 for synthesis.

Building a math or STEM tutor. Primary: MATH-500 baseline, AIME 2026 (or successor) for competition, MathVista for visual math. Secondary: FrontierMath for upper-bound, PolyMATH for non-English, GPQA for science.

Building a clinical decision-support tool. Primary: MedXpertQA MM, MMMU (Health & Medicine subset), HallusionBench. Mandatory secondary: domain-specific clinical safety evaluation that benchmarks alone cannot replace.

Building a desktop or mobile automation agent (computer-use). Primary: OSWorld-Verified, ScreenSpot Pro, AndroidWorld / MobileWorld. Secondary: BFCL, MMStar, IFEval for instruction adherence.

Building a translation product. Primary: WMT24++ MQM scores, COMET-based metrics for fast iteration. Secondary: MMLU-ProX cross-lingual scores for whether the model “thinks in” the target language.

Building a multilingual customer-service or content product. Primary: MMLU-ProX, MMMLU, INCLUDE (for cultural authenticity). Secondary: WMT for translation fluency, C-Eval if Chinese-facing.

Building a vision/video product (general). Primary: MMMU, VideoMME (overall), MMStar for genuine visual reasoning. Secondary: CharXiv for charts, RealWorldQA for spatial, HallusionBench for hallucinations.

A useful principle: pick three or four benchmarks max for your evaluation portfolio. More than that and you stop being able to read changes; fewer than that and you are flying blind on dimensions that matter.

Part 18: Running benchmarks yourself

You will sometimes need to evaluate a model that is not on any public leaderboard — a fine-tuned variant, a model behind a private API, your own model in development. The starting point is lm-evaluation-harness for any benchmark in its task registry (MMLU, MMLU-Pro, GPQA, IFEval, HellaSwag, WinoGrande, TruthfulQA, ARC, BBH, and many more).

A minimal end-to-end recipe:

import json
from lm_eval import evaluator
from lm_eval.models.huggingface import HFLM

# 1. Load the model
model = HFLM(
    pretrained="your-org/your-model",
    device="cuda:0",
    batch_size=8,
    dtype="bfloat16",
)

# 2. Choose your task portfolio
tasks = [
    "mmlu_pro",            # general knowledge baseline
    "gpqa_diamond_cot_zeroshot",  # graduate-level science
    "ifeval",              # instruction following
    "bbh_cot_zeroshot",    # multi-step reasoning
]

# 3. Run
results = evaluator.simple_evaluate(
    model=model,
    tasks=tasks,
    num_fewshot=0,         # zero-shot; set 5 for fair MMLU-Pro comparison
    log_samples=True,
)

# 4. Persist
with open("eval_results.json", "w") as f:
    json.dump(results["results"], f, indent=2)

print(json.dumps(results["results"], indent=2))

For benchmarks not in lm-eval-harness, you will need the official runner:

For coding benchmarks especially, always run in an isolated container or VM. The model’s generated code will be executed in your sandbox, and badly-aligned or simply buggy code can do unexpected things to your filesystem.

A common mistake: comparing scores across runs that used different evaluation regimes. Always pin (a) the model’s decoding settings (temperature, top-p, max-tokens), (b) the few-shot count, (c) the prompt template, and (d) the evaluation harness version. Two of the same model on the same benchmark can differ by 5+ points with different settings.

Part 19: How leaderboard scores have moved over time

If you read benchmark leaderboards casually, it is easy to miss how violently fast the curve is bending. Here are five reference snapshots, all from open public leaderboards:

• MMLU. GPT-3 in 2020: 43.9%. Frontier 2024: 88%. Human expert ceiling: ~89.8%. Saturated.
• GPQA Diamond. GPT-4 baseline in 2023: 39%. PhD humans: 65%. Frontier 2026: 90%+. Approaching saturation.
• HLE. Launch in early 2025: best models below 10%. Frontier early 2026: 30–35%. Long way to go.
• SWE-bench Verified. 2023: 4.4%. 2026: high 80s and low 90s. Saturated.
• AIME 2025. Strong humans: 5–7 of 15. Frontier 2026: near perfect. AIME 2026 introduced for fresh evaluation.

The pattern: every benchmark that becomes the frontier marker gets eaten within 12–24 months. This is the strongest argument for evaluating against a benchmark portfolio and watching trends, not a single snapshot.

Part 20: A working practitioner’s checklist

When you are about to choose a model based on benchmark scores, run through this:

1. Do the benchmarks match your use case? Coding assistant → SWE-bench Pro, not MMLU. RAG pipeline → LongBench v2, not HumanEval.
2. Is the headline benchmark saturated? If all top-10 models are within 2 points, the benchmark is telling you “everyone is competent”, not “this one is best”.
3. Are you reading the sub-task breakdown, not just the headline? A model excellent overall might be terrible on the dimension you care about.
4. Are you accounting for agent-scaffold effects? For BFCL, OSWorld, SWE-bench — different scaffolds can move scores 10–20 points. Compare like-for-like or self-evaluate.
5. Is the benchmark live or static? Live benchmarks (LiveCodeBench, AIME 2026) are more trustworthy than static public ones for contamination.
6. Have you cross-referenced human-preference with objective benchmarks? Arena strength without backing objective benchmarks for your domain is a yellow flag.
7. Have you run your own eval on a small private holdout? This is the only way to know what your model actually does on your data.
8. Have you watched the trend, not just the snapshot? Models that climb fast on emerging benchmarks usually beat models that peaked on saturated ones.

If you can answer all eight thoughtfully, you are already in the top 1% of LLM evaluators. Most teams pick a vendor based on the marketing slide, deploy, and find out about the gaps months later.

Appendix A: Benchmark quick-reference table

Closing thoughts

Benchmarks are the closest the field has to a shared empirical language. They make claims falsifiable. They let small teams compare against frontier labs. They surface failure modes that marketing slides hide. They are, on the whole, a force for honesty in an industry that does not always reward it.

But they are not science. They are measurement tools — and measurement tools are only as good as the questions you bring to them. A model that wins every benchmark in this guide can still be the wrong model for your product. A model that loses on every benchmark can still be the right one, if the dimensions it loses on do not matter for your users.

The skill to develop is benchmark literacy: reading a leaderboard with the same calibrated skepticism you would bring to an academic paper. Which benchmark are we looking at? Who built it, and what were their incentives? When was it last refreshed? What does saturation tell us about the spread? Is the harness identical across the entries being compared? What is the sub-task breakdown?

If this guide has helped you read leaderboards with sharper eyes — and pick the right benchmarks for your work — it has done its job.

The benchmark landscape will keep shifting. The ten benchmarks at the frontier in 2027 will not all be the ones at the frontier today. But the conceptual frame — task format, scoring metric, saturation, contamination, harness dependence, single-number trap, decision-matrix-to-use-case — that frame is durable. Carry it forward, and you will adapt to whatever comes next.

References

Sources are listed in the order they appear above.