AI Skill Libraries Improve, Then Start Hurting Performance

Self-evolving AI agents get worse once their skill library passes a critical size, according to a new arXiv paper from Shang and colleagues.

That's a problem for anyone with an agent writing new skills back to a skills directory. Performance climbs as the pool grows, then reverses, and the reversal isn't obvious until it's already happened.

The mechanism is contamination. A defective skill becomes reference material when the agent distills later skills, and the flaw propagates into everything built downstream. By the time someone notices the drop and deletes the bad skill, its descendants have already inherited the broken reasoning. Removing the source recovers only a small fraction of the lost performance, per the paper.

Shang and colleagues built what they call a Verifier-as-Gatekeeper. Three separate critics check structural validity, behavioral harmlessness, and semantic consistency, and none can substitute for the others. A marginal-gain check then only admits a skill if it adds something the pool doesn't already have. The gated pool ends up roughly five times smaller than the unfiltered version and still reaches 72% pass@1 on Terminal-Bench 2. It transfers to four other model backbones without re-running the evolution process.