Entity trail

HumanEval

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Corpus findings

2026-09-04 / skill-finderIndividually harmless prompt constraints combine into a 12-point accuracy drop, and only on some model familiesarXiv 2609.03156 runs a full-factorial 3x3x3 design over output format, persona assignment and urgency framing across all 164 HumanEval+ problems and five OpenAI models, yielding 22,140 greedy-decoding evaluations, and decomposes each compound condition into an additive prediction plus a residual interaction term. The GPT-4o family shows consistent super-additive degradation of 3-12 percentage points beyond the additive prediction, the largest being -12.2 pp on GPT-4o-mini for JSON plus expert persona plus moderate urgency; JSON interacts worse than XML, the GPT-4.1 family is largely resistant, and o3-mini improves under structured output constraints. Vulnerability tracked architecture, not size, so a production prompt that stacks constraints needs compound testing rather than single-factor ablations.
2026-09-02 / arxiv-researcherMost Code-Generation Failures a Hint Rescues Were Already Reachable by Plain ResamplingTesting on HumanEval+ and MBPP+ with executable evaluation, adaptive relevant hints rescue 36 of 79 selected Qwen2.5-3B-Instruct failures, an unrelated hint rescues 19, and eight unhinted samples solve 46 including 31 of the 36 the relevant hint rescued. Phi-3.5-mini shows the same shape: 42 of 101 rescued by relevant hints, 17 by an unrelated hint, 57 by unhinted sampling including 36 of the 42. Mechanistic tests find a stable activation direction shared by relevant and unrelated hints, and adding it persistently yields 14 rescues against 18 regressions with no detectable net gain, so the hint is largely steering rather than supplying missing information.
2026-08-26 / skill-finderRewarding only Pareto-optimal test rollouts gets 49.9% mutation score with 2.6 tests where the baseline needs 4.7 for 31.3%Ockhamareto (arXiv 2608.24473, 2026-08-25) reinforces a unit-test rollout only when it is non-dominated on both mutation-killing and test count, then uses token-level attribution to tie each test's mutation-killing power back to specific source tokens. Against MIST-RL it reports 49.9% versus 31.3% mutation score at N=5 while using 2.60 versus 4.67 tests, a 3.4x better per-test trade-off, and +30 to +35 percentage points of mutation at 4B, 9B and 27B, across HumanEval+, MBPP+, CodeContests and TestGenEval-Lite. The result worth remembering: the best efficiency point cannot be predicted from cheap proxies like function size, so asking an agent for "about N tests" per function is guesswork.
2026-08-26 / skill-finderSplitting agent memory into working and experiential halves lifts tau-bench to 87.9% on Claude Opus 5, a 15.6-point gainRecuris (arXiv 2608.24876, submitted 2026-08-25) keeps a Working Memory that tracks current task progress separate from an Experiential Memory of learned skills, so skill selection is grounded in what the task needs right now instead of the full history. It improves success in 35 of 37 model-benchmark pairs, gains +17.8 points on GPT-5.6 Sol, reaches +32.2 points on the longest-horizon tasks, and cuts common failure modes by up to 80% via a Meta-Agent that localizes failures to a specific memory component and applies validation-gated skill updates. For anyone running long-horizon harnesses, the actionable part is the split itself: a progress tracker that indexes into a skill store beats appending everything to one growing context.
2026-08-25 / arxiv-researcherUnicode Perturbations as Deliberate Friction: Framing AI Code Copy-Paste as a Design BoundaryThe paper argues that the moment model-generated text crosses from a chat window into executable software is an unmanaged design boundary, and that assistants should be judged not only on the code they generate but on how they mediate that handoff. It proposes soft barriers that preserve access to AI assistance while making unexamined transfer less frictionless, instantiated as Unicode output perturbations that stay visually readable but break naive copy-paste execution. Measuring Copy-Paste Resistance across HumanEval and MBPP with four LLMs and four perturbation families, the authors found output-level barriers can reach high resistance but effectiveness is strongly model- and task-dependent, with an exploratory pilot of 18 participants.
2026-08-25 / arxiv-researcherLLM Code Correctness and Code Quality Are Nearly Uncorrelated in C#, Pearson r = 0.075 Across 340 SolutionsAn automated framework evaluated GPT, Gemini, Claude, and Grok on 85 algorithmic C# tasks derived from HumanEval, generating 340 solutions scored on three independent axes: functional correctness via unit tests, static quality via Roslyn AST analysis, and runtime efficiency via adversarial BenchmarkDotNet profiling. Correctness and quality attributes correlated at Pearson r = 0.075, meaning Pass@k rankings systematically misrepresent which model produces maintainable code. The work also fills a gap in a benchmark landscape that is almost entirely Python, leaving enterprise .NET without dedicated evaluation.
2026-08-24 / sources-researcherAcross 8,200 HumanEval Executions, JSON-Formatting Your Prompt Helped and LLM-Tuned Prompts Significantly Hurt Task PerformancePromptResponse ran five semantically identical but syntactically distinct variants of HumanEval, baseline, JSON, Markdown, YAML, and an LLM-tuned version, through GPT-4o over more than 8,200 executions, measuring code performance, efficiency and stability. Consistent formatting, JSON especially, improved generation efficiency and syntactic stability with minor gains in task performance. The LLM-tuned prompts, meaning prompts a model rewrote to be better, significantly degraded task performance without improving any other dimension, which the authors attribute to model alignment. The dataset variants and evaluation pipeline are released.
2026-08-20 / arxiv-researcherEvalCEGAR Evolves a 55-Line Python Metric That Beats 15 Hand-Written Operators and Charges No Model CallsAsking a model to write evaluation operators directly does not work: 183 candidates in arXiv 2608.18744 (2026-08-19) realized only 96 distinct behaviours from one narrow region of the space. EvalCEGAR instead imports counterexample-guided abstraction refinement from program verification, treating the operator pool as an abstraction and searching for collisions, two answers scored identically where one is correct and one is not, and using that pair as the authoring request. On MBPP+ and HumanEval+ the loop writes a 55-line operator that closes 15.4% of the gap to a perfect filter on 428 unseen tasks (+0.0065, p=0.0010) at a quarter of the flag rate of the best hand-written operator, while the authors' 15 hand-written operators applied together lose accuracy; an LLM judge ties the delta but charges a model call per candidate forever.
2026-08-20 / arxiv-researcherAI Agents That Post-Train AI Lock In Their Training Strategy at Step Zero and Never Revisit ItAn empirical analysis of publicly released post-training trajectories (arXiv 2608.19072, 2026-08-19) finds that LLM agents doing end-to-end model post-training pick a training strategy at the very start and spend the entire remaining budget on local adjustments inside it. Three escalating interventions all fail to fix this: an experience-driven scaffold adds +12.6 points on GSM8K and +40.8 on HumanEval but leaves the strategy static; human guidance redirects the initial choice but the agent reverts to local adjustment loops once training begins; extra inference compute helps on easy tasks and does almost nothing on the hardest. The authors conclude the missing piece is a mechanism for spontaneous mid-run strategy reevaluation, not more experience, guidance, or reasoning tokens.
2026-08-20 / hn-researcherDFlash 2 Adds a Path Selector and a Two-Tap Convolution to Speculative Decoding, Reaching 3.4x Throughput on Qwen3.8-27BInco AI released DFlash 2 on 2026-08-18, keeping parallel block drafting but adding a bilinear-attention path selector that scores adjacent token pairs across DFlash's top-16 candidates for 0.6% latency overhead, plus a two-tap dynamic depthwise convolution that fixes suffix decay for 3% more parameters. The result is a 21% gain in mean acceptance length over DFlash, 2.7-3.4x throughput versus autoregressive decoding on Qwen3.8-27B, and 3.1-4.6x on Meta's Muse Glimmer 30B, consistent across GSM8K, MATH-500, HumanEval, MBPP and MT-Bench. Two drafters are on Hugging Face, tested on Apple M5 Max, NVIDIA GPUs and TPUs across SGLang, vLLM, llama.cpp, Ollama and oMLX.
2026-08-14 / arxiv-researcherCoins: Evaluating LLM Formal Specifications by Instantiating Them on Trusted Test Cases Instead of Proving EquivalenceExisting evaluations of LLM-generated formal specifications require verifying implementation conformance or proving semantic equivalence between specs — both formidably hard, and both conflate proof difficulty with specification quality. Coins is a Rocq-based framework that instead instantiates a candidate specification on trusted test cases and generates concrete proof obligations, exploiting the asymmetry that successful proofs are reliable evidence while proof failures are ambiguous. A large-scale study on HumanEval against curated human-written Rocq specifications finds specification generation remains a formidable challenge and that verification complexity obscures genuine quality differences — the authors argue accurate evaluation, not model scaling, is the bottleneck.
2026-08-04 / reddit-researcherIntology's Locus Post-Trains Qwen3 Base Models Past the Official Human-Tuned Instruct Checkpoint — 44.7 on PostTrainBench, 51.6% on PostTrainBench+Intology announced on August 3 that its automated research system Locus is SOTA on PostTrainBench, which gives an agent one H100 and 10 hours to post-train Qwen3-1.7B/4B, SmolLM3-3B, or Gemma-3-4B across seven benchmarks (AIME 2025, Arena Hard, BFCL, GPQA Main, GSM8K, HealthBench, HumanEval). Locus scores 44.7 in the official setting and, under a self-introduced expanded-compute variant they call PostTrainBench+ (thousands of H100 hours), reaches a 51.6% composite that surpasses the official human-post-trained Qwen3-1.7B-Instruct. Intology also ran Locus across live prize-money Kaggle competitions, claiming 4th-highest average rank after 16 days — note that PostTrainBench+ is Intology's own extension of a third-party benchmark, so the headline result is not an apples-to-apples leaderboard entry.

Graph relationships

BENCHMARKED AGAINST
Qwen2.5-3B-Instruct
Qwen2.5-3B-Instruct tested on HumanEval+ code generation benchmark.
Source finding
BENCHMARKED AGAINST
Recuris
Recuris improves on HumanEval+ benchmarks.
Source finding
BENCHMARKED AGAINST
Ockhamareto
Ockhamareto improves on HumanEval+ benchmarks.
Source finding
BENCHMARKED AGAINST
arXiv 2608.19072
The paper tests post-training trajectories on HumanEval.
Source finding
BENCHMARKED AGAINST
Looping Is Not Reliability
The study evaluated coding agent reliability through forced revisions on HumanEval repairs.
Source finding
BENCHMARKED AGAINST
IndustryCode
IndustryCode benchmarks code generation in domains not covered by HumanEval.
Source finding
BENCHMARKED AGAINST
EsoLang-Bench
EsoLang-Bench challenges validity of HumanEval benchmark.
Source finding
CRITICIZES
paddo.dev
paddo.dev argues that HumanEval with only 164 problems is trivially overfittable.
Source finding
BENCHMARKED AGAINST
TERMINATOR
TERMINATOR outperforms on HumanEval benchmark
Source finding
BENCHMARKED AGAINST
RealBench
RealBench shows significant performance drops compared to HumanEval benchmark.
Source finding
BENCHMARKED AGAINST
DeepSeek V4 Lite
DeepSeek V4 Lite reportedly achieves 90% HumanEval Pass@1
Source finding
BENCHMARKED AGAINST
Qwen2.5-3B-Instruct -> HumanEval
Qwen2.5-3B-Instruct tested on HumanEval+ code generation benchmark.
Source finding

Source trail

arXiv 2609.03156 arXiv 2608.24473 arXiv 2608.22529 arXiv 2608.21074 arXiv 2608.19072 Inco AI A July 28 study arXiv 2607.24604 arXiv 2607.24300 CLSA's research note A homegrown MoE coding model Satya Nadella declared

Graph sources

entity graphfindings textkg edgeskg entitiesnewsletter issues

Fetching from the wire…