Auto-judge: a cross-task benchmark for comparing LLM judges for citation-grounded RAG systems

We present the Auto-Judge resource for the meta-evaluation of automated LLM judges, especially judges that evaluate Retrieval-Augmented Generation (RAG) systems that ground their response with citations. The resource couples (i) a data release of topics, pooled RAG responses, and human judgments, with (ii) a standardized protocol and software infrastructure for implementing "LLM-as-a-judge" methods in a reproducible and extensible way, including support for parameter sweeps and variant tracking. Auto-Judge is designed to support rigorous meta-evaluation by comparing LLM judges against human assessments and against each other across different evaluation tasks, and by examining known vulnerabilities of LLM judging such as circularity, overfitting, self-preference, and content manipulation. We describe the dataset, the submission and execution workflow (including TIRA-based execution), and we provide baseline judge implementations and correlation-based results to illustrate how the benchmark can be used to develop and diagnose new judge methods.