LLM judges of summaries fail basic perturbation tests

Many strong LLM-as-judge metrics fail basic perturbation tests of informational content, according to a new paper posted to arXiv. That's a problem for anyone building an AI summarizer for a specific reader. A biomedical researcher and a family doctor reading the same vaccine literature need different summaries, the paper argues.

The authors argue persona, not query, works better as the signal for what a summary needs to cover. Users rarely state everything relevant to them, so describing the reader beats asking a summarizer to answer one stated question.

The authors tested how sensitive different metrics are to both informational content and persona differences. Expert human evaluators judging whether a summary satisfies a specific reader agreed poorly with both traditional metrics and LLM-as-judge scores, per the paper.

Human experts and LLM judges disagree this much on satisfaction, so most summarization leaderboards are likely scoring the wrong thing. Worth watching whether any evaluation suite adopts persona-conditioned testing as a result.