Delegate harness-improvement proposals to a cheap model — generation capability is flat across model sizes

Lil'Log·medium signal

Lilian Weng's harness engineering survey reports a counterintuitive result: harness-updating capability measures flat across models from Qwen2-32B to Opus 4.6 — the bottleneck is utilization, not generation. The immediate cost lever: the agent that proposes harness edits does not need to be your frontier model, so route proposal generation to a small/cheap model and reserve top-tier capacity for judging and applying the edits. The survey also notes harness improvements transfer across benchmarks, implying they encode reusable engineering patterns rather than benchmark-specific tricks — so a harness tuned on one task family is worth porting rather than rebuilding.