AI2 published what broke while building its Shippy agent

AI2 detailed what broke while building Shippy, its production agent, in a post-mortem on Hugging Face. That kind of honesty is rare. Most agent content from labs and vendors reads like marketing in a lab coat: launch dates, benchmark charts, never the failures.

For anyone building agents, that gap matters. Teams are mostly guessing at failure modes because the companies further along don't publish what actually went wrong. A research lab with nothing to sell writing down its own mistakes is closer to ground truth than most agent best-practices content in circulation.

Read next to the AI2 post, Dex Horthy's interview with The Pragmatic Engineer makes a sharper claim. Teams blaming the model for bad agent output are usually looking at a context problem instead. What gets loaded, in what order, what gets evicted when the window fills up. That reframes a lot of complaints about model capability as fixable engineering problems, not capability ceilings you wait out.

Put the two pieces together and the pattern is that agent quality is systems work, not model work. Better prompting and bigger context windows won't help if the plumbing around retrieval, ordering, and eviction is wrong.

I'd watch whether that holds. AI2 has nothing to sell yet, which is why this post exists at all. The test is whether the lab keeps publishing this kind of detail once Shippy ships, or whether the write-ups quietly stop.