Score a repository SKILL.md by reverted merged PRs, not synthetic tasks, because a capable agent saturates synthetic tasks with no document at all

arXiv 2609.12742·high signalXBlueskyLinkedInCopy link

Prior work on auto-synthesizing SKILL.md files optimizes against benchmarks a bare repo does not have, and the synthetic tasks used instead are easy enough that a capable agent solves them without any document, so the optimizer has nothing to measure. This work mines harder tasks by taking merged pull requests and reverting them at a single frozen base commit, then scores a candidate document purely by whether the same agent does better with it than without. On three Kotlin repositories GEPA-found documents gained 4.9 pp on average while SkillOpt-found documents gained 0.1 pp over seed, and the authors are candid that at one repository's data volume the GEPA gain cannot be separated from run-to-run variance — though a maintainer said the documents contained knowledge you only get by working in the project.