LongMemEval-V2: 451-Question Benchmark for Agent Memory with 115M Token Trajectories

arXiv·medium signal

Benchmarks whether memory systems help agents become knowledgeable colleagues in specialized web environments. 451 manually curated questions across five memory abilities (static recall, dynamic tracking, workflow knowledge, environment gotchas, premise awareness) paired with up to 500 trajectories totaling 115M tokens. Proposes AgentRunbook-C (coding agent in augmented sandbox) achieving 72.5% accuracy vs 48.5% for best RAG baseline. Dataset on HuggingFace.