Arguing the Safety of Agentic AI using LLM-as-Judge Evaluations

To use LLMs for safety-critical tasks requires evaluation of how well those tasks are performed. For critical tasks performed at scale, such as clinical triage, some form of automated evaluation is usually necessary. This paper considers a common approach in agent-based LLM implementations known as LLM-as-judge (LaJ), that uses another LLM to perform this evaluation. Task evaluations usually take the form of a ‘basket of metrics’ that spread the risk of a single incorrect judgment. In this paper we look at whether typical comparative metrics used in LaJ evaluations of LLM task outputs are sufficient for safety assurance. We consider how task evaluations by LaJ in agentic AI settings can be structured and justified and provide a safety argument pattern for those LaJ evaluations. We discuss how the saftey argument pattern could be instantiated for a clinical example of an LLM-based perioperative risk assessment for surgery.