Datadog Just Showed Us What AI in Production Actually Looks Like

69% of all input tokens in production LLM traces are system prompts. Let that sink in for a second.

Datadog's State of AI Engineering 2026 dropped yesterday, and it's the best empirical snapshot we have of how companies actually use LLMs. Not how they demo them. Not how they pitch them to investors. How they run them in prod, at scale, across thousands of deployments.

The numbers that matter: 5% of all LLM spans report errors, with 60% of those coming from rate limits. Framework adoption (LangChain, Pydantic AI, Vercel AI SDK) doubled from 9% to 18% of organizations year-over-year. Anthropic Claude grew 23 percentage points of provider share while OpenAI maintains 63%. And 69% of companies now use 3+ models in production.

But that 69% system prompt stat is the one I keep coming back to. If you're optimizing for cost and latency, system prompts are where the money is burning. Two-thirds of your input tokens aren't user queries or retrieved context. They're the instructions you wrote once and send every single call. This means prompt compression, caching strategies, and system prompt architecture aren't premature optimization. They're the first thing you should look at.

The framework adoption doubling is interesting because it tells us the "just use the raw API" phase is ending for most teams. When you're running 3+ models with error handling, fallbacks, and observability, you need an abstraction layer. The question is which one wins. LangChain's been losing mindshare to lighter alternatives, but Datadog's data shows it's still growing in absolute terms.

The Anthropic market share jump (23pp) is the competitive signal here. OpenAI still dominates at 63%, but that dominance is eroding fast. A year ago it was closer to 80%. Claude is eating into that gap primarily through coding and agent use cases, which is exactly what you'd expect given Claude Code's adoption curve.

What to do about it: Audit your system prompts today. If you're sending 2,000+ tokens of instructions on every call, look at Anthropic's prompt caching (which gives you 90% off cached input tokens) or restructure your prompts to minimize repetition. The Datadog data says this is where most production spend actually lives.