Skills
Comments help LLM code generation only when they leak correct solution content, and comments from a different problem cut pass@1 by 20.8%
On LiveCodeBench, neither comment frequency nor comment intent predicted pass@1. Prefilling weaker models with comment blocks taken from stronger models' passing solutions raised recipient pass@1 by 17.2% on average, comments from failed solutions gave no reliable gain, and comments written for a different problem dropped pass@1 by 20.8%. Across many models and prompt variants the best recovery of the external-comment gain through prompting alone was 24%, so the benefit is transferred solution content, not the act of commenting. For anyone stuffing a coding agent's context with nearby commented code, mismatched comments are actively worse than none.
↳ Follow the thread