Benchmarks

Real-world coding edits across 6 programming languages — measures whether the model produces a correct edit accepted on second attempt.

Real GitHub issues solved end-to-end by the model.

Factual grounding evaluation — measures whether a model's answer is supported by the provided source documents.

Massive Multitask Language Understanding — 57-subject multiple-choice exam.

Harder reformulation of MMLU with 10 answer choices and deeper reasoning.

Automatic proxy for LMArena Elo — pairwise head-to-head comparisons graded by GPT-4-Turbo against a curated set of 500 hard prompts. Strong predictor of human preference ratings.

Multi-step instruction-following across diverse tasks (math, coding, writing, reasoning) — measures aggregate capability breadth.

Long-context understanding across documents of 32K–2M tokens — tests whether the model can retrieve and reason over facts deep in the input.

Multi-Round Conversational Reasoning — tests whether a model can maintain facts and context across long multi-turn dialogues.

American Invitational Mathematics Examination, 2024 problems.

Grade-school math word problems.

Clinical reasoning and medical knowledge benchmark — evaluates a model's ability to answer healthcare questions with accuracy and safety.

Graduate-level, Google-proof science reasoning questions.