Simon Willison releases condense-json 1.0 for LLM logs

Simon Willison shipped condense-json 1.0, a library that hash-references repeated JSON strings so LLM interaction logs take less disk space, per the release. Anyone logging agent traces at volume hits the same wall: identical prompt fragments and tool schemas get written to disk on every row. Willison built the tool to fix that for his own SQLite logs, without discarding any data.

The library scans a JSON structure for duplicate strings and substrings from a supplied replacements object. Matches get swapped for compact references. Condensed regions are marked with a $r key, and each reference points to its replacement value with a $ key and ID. Calling uncondense_json() restores the original structure exactly, so it's a storage trick, not lossy summarization.

The catch: you have to supply the replacements object yourself, the strings you expect to repeat, like prompt templates and tool schemas. It's not automatic duplicate detection across an entire log file.

The tool itself is small, but the confession behind it is the real point. LLM logging at any real volume duplicates the same prompt boilerplate on every row. Most people running agent traces haven't checked how much of their storage bill that boilerplate accounts for.