Scientific AI evaluation | James V. Roggeveen

Frontier language models have improved rapidly at code, partly because software tasks come with unusually clear correctness signals and mature evaluation workflows. Continued progress in mathematics, science, and engineering requires evaluations built by domain experts, grounded in real practice, and graded by symbolic, numerical, or programmatic checks without relying on other language models as judges.

My work in scientific AI evaluation focuses on building domain-centered benchmarks and the infrastructure that makes them possible. With the right programmatic evaluation tools, experts can focus on writing, testing, and refining authentic problems while the evaluation system handles model execution, collaborative review, answer parsing, grading, and failure analysis.

One lesson from this work is that flexible but constrained evaluation interfaces help experts scale. Rather than asking every author to build an arbitrary grading harness from scratch, a shared set of programmatic checks lets contributors generate more problems faster while keeping the benchmark reproducible. In this setting, constraints often breed creativity.

Example evaluation pipeline for HARDMath2: domain experts authored and reviewed problems in shared Google Sheets and Colab notebooks, while the evaluation system handled model runs, parsing, grading, and live feedback directly on the Google Sheet.

Benchmark Infrastructure

My two published benchmarks relied on infrastructure I developed to integrate model querying and grading directly into Google Sheets. Problem authors could write prompts and symbolically graded LaTeX answers in a shared sheet, run live model evaluations on their own problems, and receive immediate feedback on parsing, prompt clarity, and difficulty.

This let authors systematically fix problems in real time and ensured that the final benchmark contained many problems already compatible with the evaluation pipeline. Authors could contribute many tasks without needing to manage model APIs, grading harnesses, or evaluation bookkeeping themselves, which was especially important because many contributors were not software experts by training.

HARDMath2

I co-led HARDMath2 (Roggeveen et al., 2025), a 211-problem benchmark for graduate-level applied mathematics accepted to NeurIPS Datasets & Benchmarks 2025. The benchmark was built with more than 40 graduate students in a Harvard applied mathematics course, who developed and peer-reviewed original problems across asymptotics, perturbation methods, PDEs, and related topics as part of their weekly homework assignments. All problems were written with symbolic LaTeX answers that were parsed and evaluated programmatically by my custom LaTeX parsing script.

For this project, I built the collaborative authoring, testing, and automated symbolic-verification infrastructure. The system helped students test problems against frontier models, refine solutions, and produce machine-checkable answers without needing to directly manage the evaluation harness. The course structure created an iterative task-design loop: students tested candidate problems against increasingly capable models and refined them until they reached the desired difficulty level.

HARDMath2 evaluation results: solve rates across applied-mathematics categories show substantial variation by topic and model, with frontier systems still missing many graduate-level applied math problems.

CMT-Benchmark

I deployed this evaluation infrastructure for CMT-Benchmark (Pan et al., 2025), a condensed-matter theory benchmark built by expert researchers. The benchmark contains 50 original research-level problems evaluated across 17 frontier language models. This benchmark also required enhancing the programmatic verifier to support non-commutative operators, which are critical in many condensed-matter theory tasks.

Evaluation Philosophy

Expert creativity scales best when the evaluation interface is constrained enough to be reliable, but flexible enough to express real domain problems. In practice, this means giving experts clean ways to interact with models and programmatic verifiers before they begin writing questions. Predefined verifiers can limit some forms of expression, but they also make it possible to build difficult, reproducible tasks that reveal model behavior.

Public Resources

Large language models (LLMs) have shown remarkable progress in mathematical problem-solving, but evaluation has largely focused on problems that have exact analytical solutions or involve formal proofs, often overlooking approximation-based problems ubiquitous in applied science and engineering. To fill this gap, we build on prior work and present , a dataset of 211 original problems covering the core topics in an introductory graduate applied math class, including boundary-layer analysis, WKB methods, asymptotic solutions of nonlinear partial differential equations, and the asymptotics of oscillatory integrals. This dataset was designed and verified by the students and instructors of a core graduate applied mathematics course at Harvard. We build the dataset through a novel collaborative environment that challenges students to write and refine difficult problems consistent with the class syllabus, peer-validate solutions, test different models, and automatically check LLM-generated solutions against their own answers and numerical ground truths. Evaluation results show that leading frontier models still struggle with many of the problems in the dataset, highlighting a gap in the mathematical reasoning skills of current LLMs. Importantly, students identified strategies to create increasingly difficult problems by interacting with the models and exploiting common failure modes. This back-and-forth with the models not only resulted in a richer and more challenging benchmark but also led to qualitative improvements in the students’ understanding of the course material, which is increasingly important as we enter an age where state-of-the-art language models can solve many challenging problems across a wide domain of fields.

Large language models (LLMs) have shown remarkable progress in coding and math problem-solving, but evaluation on advanced research-level problems in hard sciences remains scarce. To fill this gap, we present CMT-Benchmark, a dataset of 50 problems covering condensed matter theory (CMT) at the level of an expert researcher. Topics span analytical and computational approaches in quantum many-body, and classical statistical mechanics. The dataset was designed and verified by a panel of expert researchers from around the world. We built the dataset through a collaborative environment that challenges the panel to write and refine problems they would want a research assistant to solve, including Hartree-Fock, exact diagonalization, quantum/variational Monte Carlo, density matrix renormalization group (DMRG), quantum/classical statistical mechanics, and model building. We evaluate LLMs by programmatically checking solutions against expert-supplied ground truth. We developed machine-grading, including symbolic handling of non-commuting operators via normal ordering. They generalize across tasks too. Our evaluations show that frontier models struggle with all of the problems in the dataset, highlighting a gap in the physical reasoning skills of current LLMs. Notably, experts identified strategies for creating increasingly difficult problems by interacting with the LLMs and exploiting common failure modes. The best model, GPT5, solves 30% of the problems; average across 17 models (GPT, Gemini, Claude, DeepSeek, Llama) is 11.4\pm2.1%. Moreover, 18 problems are solved by none of the 17 models, and 26 by at most one. These unsolved problems span Quantum Monte Carlo, Variational Monte Carlo, and DMRG. Answers sometimes violate fundamental symmetries or have unphysical scaling dimensions. We believe this benchmark will guide development toward capable AI research assistants and tutors.

Benchmark Infrastructure

HARDMath2

CMT-Benchmark

Evaluation Philosophy

Public Resources

References

2025