Reliable Language Model Evaluation

Live Leaderboards

MathArena: Uncontaminated Math Competitions

BaxBench: Secure and Correct Backends

SWT-Bench: Assessing Test-writing Capabilities

EU AI Act Compliance Leaderboard

Publications

2025

MathArena: Evaluating LLMs on Uncontaminated Math Competitions

Mislav Balunović, Jasper Dekoninck, Nikola Jovanović, Ivo Petrov, Martin Vechev

NeurIPS Datasets and Benchmarks 2025

Website

Paper

Code

Adaptive Generation of Bias-Eliciting Questions for LLMs

Robin Staab, Jasper Dekoninck, Maximilian Baader, Martin Vechev

ArXiv 2025

Paper

Code

IMProofBench: Benchmarking AI on Research-Level Mathematical Proof Generation

Johannes Schmitt, Gergely Bérczi, Jasper Dekoninck, Jeremy Feusi, Tim Gehrunger, Raphael Appenzeller, Jim Bryan, Niklas Canova, Timo de Wolff, Filippo Gaia, Michel van Garrel, Baran Hashemi, David Holmes, Aitor Iribar Lopez, Victor Jaeck, Martina Jørgensen, Steven Kelk, Stefan Kuhlmann, Adam Kurpisz, Chiara Meroni, Ingmar Metzler, Martin Möller, Samuel Muñoz-Echániz, Robert Nowak, Georg Oberdieck, Daniel Platt, Dylan Possamaï, Gabriel Ribeiro, Raúl Sánchez Galán, Zheming Sun, Josef Teichmann, Richard P. Thomas, Charles Vial

ArXiv 2025

Website

Paper

BrokenMath: A Benchmark for Sycophancy in Theorem Proving with LLMs

Ivo Petrov, Jasper Dekoninck, Martin Vechev

ArXiv 2025

Website

Paper

Code

BaxBench: Can LLMs Generate Secure and Correct Backends?

Mark Vero, Niels Mündler, Victor Chibotaru, Veselin Raychev, Maximilian Baader, Nikola Jovanović, Jingxuan He, Martin Vechev

ICML 2025 CC BY 4.0 by @fontawesome - https://fontawesome.com

Spotlight

Website

Paper

Code

Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad

Ivo Petrov, Jasper Dekoninck, Lyuben Baltadzhiev, Maria Drencheva, Kristian Minchev, Mislav Balunović, Nikola Jovanović, Martin Vechev

AI4Math@ICML 2025

Website

Paper

Code

The Open Proof Corpus: A Large-Scale Study of LLM-Generated Mathematical Proofs

Jasper Dekoninck, Ivo Petrov, Kristian Minchev, Mislav Balunovic, Martin Vechev, Miroslav Marinov, Maria Drencheva, Lyuba Konova, Milen Milenov Shumanov, Kaloyan Tsvetkov, Nikolay Drenchev, Lazar D. Todorov, Kalina Nikolova, Nikolay Georgiev, Vanesa Kalinkova, Margulan Ismoldayev

AI4Math@ICML 2025

Website

Paper

Code

MathConstruct: Challenging LLM Reasoning with Constructive Proofs

Mislav Balunović*, Jasper Dekoninck*, Nikola Jovanović, Ivo Petrov, Martin Vechev

ICML 2025 * Equal contribution

Paper

Code

Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation

Jasper Dekoninck, Maximilian Baader, Martin Vechev

ICLR 2025

Paper

Code

2024

A Synthetic Dataset for Personal Attribute Inference

Hanna Yukhymenko, Robin Staab, Mark Vero, Martin Vechev

NeurIPS Datasets and Benchmarks 2024

Paper

Code

ConStat: Performance-Based Contamination Detection in Large Language Models

Jasper Dekoninck, Mark Niklas Müller, Martin Vechev

NeurIPS 2024

Website

Paper

Code

SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents

Niels Mündler, Mark Niklas Müller, Jingxuan He, Martin Vechev

NeurIPS 2024

Website

Paper

Code

COMPL-AI Framework: A Technical Interpretation and LLM Benchmarking Suite for the EU Artificial Intelligence Act

Philipp Guldimann, Alexander Spiridonov, Robin Staab, Nikola Jovanović, Mark Vero, Velko Vechev, Anna Gueorguieva, Mislav Balunović, Nikola Konstantinov, Pavol Bielik, Petar Tsankov, Martin Vechev

arXiv 2024

Website

Paper

Code

Evading Data Contamination Detection for Language Models is (too) Easy

Jasper Dekoninck, Mark Niklas Müller, Maximilian Baader, Marc Fischer, Martin Vechev

arXiv 2024

Paper

Code