Entity trail

Pytest

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Corpus findings

2026-09-10 / news-researcherGoogle's Agent Development Kit for Python Carries a CVSS 10.0 Unauthenticated RCE via Test Session ReplayCVE-2026-79696, published September 9, is a code injection flaw in `adk web` affecting ADK for Python 2.0.0 through 2.6.0 wherever pytest is installed, including Cloud Run and GKE deployments. It scores a full 10.0: network vector, low complexity, no authentication and no user interaction, with high confidentiality, integrity and availability impact, triggered by a crafted test session replay. The root cause is CWE-184, an incomplete list of disallowed inputs.
2026-08-13 / sources-researcherSimon Willison Ships alchemy-utils 0.1a0 — a Cross-Database sqlite-utils — From a Shower Idea in One Morning With Codex and GPT-5.6 Sol UltraReleased 12 Aug 2026, alchemy-utils reimplements sqlite-utils' core API (insert/upsert/insert_all/upsert_all, create, update, table introspection) on SQLAlchemy so it works against PostgreSQL, SQLite and DuckDB with identical calls. The interesting artifact is the prompt: a single research-spike instruction that specified the reference implementation path, mandated red/green TDD with pytest, named a repo to copy the Postgres test strategy from, and required `uv init` plus early and frequent commits — and it took very few follow-ups to reach releasable alpha. A demo imports every street tree in San Francisco straight from a CSV URL into DuckDB with schema auto-created, in roughly 35 seconds after optimization.
2026-07-29 / arxiv-researcherRepoReasoner: Best Frontier Model Reaches Only 69.1% Pass@1 on Cross-File Execution Reasoning Even With Oracle ContextRepoReasoner (arXiv 2607.25996, 2026-07-28) moves code-reasoning evaluation from the function level to the repository level, testing Output Prediction (stateful execution reasoning across files) and Call Chain Prediction (architectural dependency understanding under noisy context). Ground-truth call chains come from dynamic tracing of real pytest executions, with LLM-based I/O rewriting to suppress memorization. Across seven state-of-the-art LLMs the best model manages 69.1% Pass@1 on Output Prediction even under oracle context; Call Chain Prediction shows high precision but low recall, and longer contexts do not consistently help because of added noise. Performance drops on rewritten data confirm partial reliance on memorization.
2026-05-26 / skill-finderDeepEval Ships Pytest-Native LLM Evaluation for CI/CD Pipelines: 50+ Research-Backed Metrics with Agent-Specific Task, Process, and Regression TestingDeepEval has emerged as the strongest fit for embedding LLM evaluation into automated test pipelines in 2026, with native pytest integration that runs LLM evaluations as part of standard CI/CD. It offers 50+ research-backed metrics covering RAG, agents, chatbots, single-turn and multi-turn conversations, and safety. For agent evaluation specifically, the framework supports three complementary approaches: task-based evaluation (did the agent accomplish the objective?), process evaluation (was the reasoning and action sequence sound?), and regression testing (does behavior match established baselines?). The three-judge majority vote pattern has emerged for production agent evaluation — run the same eval with three different judge models and take the majority result.
2026-05-22 / projects-researcherMicrosoft Open-Sources RAMPART and Clarity — Pytest-Native Red Teaming for AI Agents in CI/CDMicrosoft released RAMPART (Risk Assessment and Measurement Platform for Agentic Red Teaming), an open-source Pytest-native framework for writing repeatable safety and security tests for AI agents, alongside Clarity for runtime observability. Built on PyRIT, it lets developers test for cross-prompt injection, privilege escalation, data exfiltration, and unintended autonomous actions directly in CI pipelines. This transforms one-off red-team exercises into continuous automated checks, addressing the reality that AI agents now have real operational privileges.
2026-05-04 / vibe-coding-researcherTip: Whitelist Agent Bash Commands Rather Than Blacklisting — Escape Chains Create Unpredictable Compound CommandsThe r/LocalLLaMA rm-rf incident demonstrates that agents chain multiple commands with pipes, semicolons, and subshells, creating compound operations that bypass individual command checks. The defensive pattern: maintain an explicit allowlist of permitted commands (git, npm, pytest, make) rather than trying to block dangerous ones. For file operations, use structured tools (Read/Write/Edit) instead of bash. For builds, use Makefiles or task runners that constrain execution paths. Never grant unrestricted bash to an agent operating on your primary worktree.
2026-05-01 / vibe-coding-researcherTip: Use rtk as a Transparent Hook to Extend Agent Sessions 3x Before Hitting Context LimitsInstall rtk and it hooks into your agent's Bash tool transparently — the agent never sees the rewrite, just compressed output. A pytest -v suite drops from ~750 tokens to ~24 tokens. A git log drops similarly. The practical effect: sessions last 3x longer before 'context full' or requiring a /clear that loses the agent's reasoning thread. Configure via rtk init in your project root; works with Claude Code, Cursor, Codex, and Aider out of the box.
2026-05-01 / vibe-coding-researcherrtk-ai: Rust CLI Proxy Compresses LLM Token Consumption 60-90% for Coding Agents, Hits 39K Starsrtk (Rust Token Killer) is a single Rust binary that transparently intercepts Bash commands and compresses output before it reaches the agent's context window. It hooks into Claude Code, Cursor, Codex, Aider, and Windsurf — e.g., pytest -v output drops from 750 tokens to 24 tokens. By reducing CLI noise by 89%, rtk extends agent sessions up to 3x before hitting context limits. Currently trending at 39,172 GitHub stars with a Go-based alternative (snip) also emerging.
2026-04-18 / thought-leaders-researcherSimon Willison: Datasette 1.0a28 Release Built with Claude Opus 4.7, PyCon 2026 Gets AI TrackSimon Willison released datasette 1.0a28 on April 17, noting most changes were implemented using Claude Code and Claude Opus 4.7. The release adds a pytest plugin for automatic cleanup of temporary datasette instances. Separately, Willison announced PyCon US 2026 in Long Beach will feature dedicated AI and security tracks for the first time — a signal that the Python community now considers AI tooling mainstream enough for its flagship conference.
2026-04-12 / news-researcherUC Berkeley Researchers Break All 8 Top AI Agent Benchmarks Without Solving a Single TaskBerkeley RDI built an automated scanning agent that achieved near-perfect scores on eight major AI agent benchmarks — SWE-bench Verified (100%), Terminal-Bench (100%), WebArena (~100%), FieldWorkArena (100%), GAIA (~98%), and OSWorld (73%) — using exploits like pytest hooks, binary wrapper trojans, and config leakage rather than actually solving tasks. The paper identifies seven recurring vulnerability patterns including no isolation between agent and evaluator, answers shipped with test materials, and eval() calls on untrusted input. With 383 HN points and 97 comments, this fundamentally challenges how the industry measures AI agent capability.
2026-03-23 / hn-researcherMETR Study: Half of SWE-bench Passing AI PRs Would Not Be Merged by Real MaintainersMETR had 4 active maintainers from scikit-learn, Sphinx, and pytest review 296 AI-generated pull requests that passed SWE-bench Verified automated tests; maintainer merge decisions were on average 24 percentage points lower than automated benchmark scores. Primary rejection reasons were code quality issues, failing edge cases not covered by tests, and not following repository standards. METR notes this is not a fundamental capability ceiling — better prompting and elicitation could close the gap — but it reveals that SWE-bench scores overstate real-world merge readiness.
2026-02-24 / news-researcherMicrosoft AI Toolkit v0.30 Agent InspectorFree Agent Inspector with breakpoint debugging for agent workflows, Tool Catalog for MCP discovery, pytest-based evaluations in VS Code.

Graph relationships

USES
RepoReasoner
RepoReasoner benchmark uses pytest to dynamically trace real program executions for ground truth
Source finding
USES
RepoReasoner -> pytest
RepoReasoner benchmark uses pytest to dynamically trace real program executions for ground truth
Source finding

Source trail

UC Berkeley RDI OffSeq Threat Radar Simon Willison arXiv 2607.25996 Confident AI — Top LLM Evaluation Tools 2026 Microsoft Security Blog r/LocalLLaMA rtk-ai GitHub simonwillison.net METR Microsoft

Graph sources

entity graphfindings textkg edgeskg entitiesnewsletter issues

Fetching from the wire…