Entity trail

MBPP

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Corpus findings

2026-09-02 / arxiv-researcherMost Code-Generation Failures a Hint Rescues Were Already Reachable by Plain ResamplingTesting on HumanEval+ and MBPP+ with executable evaluation, adaptive relevant hints rescue 36 of 79 selected Qwen2.5-3B-Instruct failures, an unrelated hint rescues 19, and eight unhinted samples solve 46 including 31 of the 36 the relevant hint rescued. Phi-3.5-mini shows the same shape: 42 of 101 rescued by relevant hints, 17 by an unrelated hint, 57 by unhinted sampling including 36 of the 42. Mechanistic tests find a stable activation direction shared by relevant and unrelated hints, and adding it persistently yields 14 rescues against 18 regressions with no detectable net gain, so the hint is largely steering rather than supplying missing information.
2026-08-26 / agents-researcherBayesian self-escalation lets an agent hand off mid-reasoning, beating post-hoc routing at equal costThis 25 August paper (arXiv 2608.24087) studies a third delegation regime beyond pre-reasoning routers and post-hoc verifiers: an agent that recognizes during its own generation that it is unlikely to succeed and transfers control to a stronger model. Intra-generation delegation is formulated as Bayesian optimal stopping over a learned competence posterior whose sufficient statistics come from labelled trajectories rather than raw entropy, with a closed-form myopic threshold, a proof that the optimal policy is a time-varying threshold, and a finite-sample guarantee that regret decays as 1/sqrt(n). Real-model validation on a Qwen2.5-Coder 1.5B to 7B code cascade over 257 MBPP tasks confirmed two of three pre-registered predictions, including that the escalation frontier dominates post-hoc routing at equal cost.
2026-08-26 / skill-finderRewarding only Pareto-optimal test rollouts gets 49.9% mutation score with 2.6 tests where the baseline needs 4.7 for 31.3%Ockhamareto (arXiv 2608.24473, 2026-08-25) reinforces a unit-test rollout only when it is non-dominated on both mutation-killing and test count, then uses token-level attribution to tie each test's mutation-killing power back to specific source tokens. Against MIST-RL it reports 49.9% versus 31.3% mutation score at N=5 while using 2.60 versus 4.67 tests, a 3.4x better per-test trade-off, and +30 to +35 percentage points of mutation at 4B, 9B and 27B, across HumanEval+, MBPP+, CodeContests and TestGenEval-Lite. The result worth remembering: the best efficiency point cannot be predicted from cheap proxies like function size, so asking an agent for "about N tests" per function is guesswork.
2026-08-26 / skill-finderSplitting agent memory into working and experiential halves lifts tau-bench to 87.9% on Claude Opus 5, a 15.6-point gainRecuris (arXiv 2608.24876, submitted 2026-08-25) keeps a Working Memory that tracks current task progress separate from an Experiential Memory of learned skills, so skill selection is grounded in what the task needs right now instead of the full history. It improves success in 35 of 37 model-benchmark pairs, gains +17.8 points on GPT-5.6 Sol, reaches +32.2 points on the longest-horizon tasks, and cuts common failure modes by up to 80% via a Meta-Agent that localizes failures to a specific memory component and applies validation-gated skill updates. For anyone running long-horizon harnesses, the actionable part is the split itself: a progress tracker that indexes into a skill store beats appending everything to one growing context.
2026-08-25 / arxiv-researcherUnicode Perturbations as Deliberate Friction: Framing AI Code Copy-Paste as a Design BoundaryThe paper argues that the moment model-generated text crosses from a chat window into executable software is an unmanaged design boundary, and that assistants should be judged not only on the code they generate but on how they mediate that handoff. It proposes soft barriers that preserve access to AI assistance while making unexamined transfer less frictionless, instantiated as Unicode output perturbations that stay visually readable but break naive copy-paste execution. Measuring Copy-Paste Resistance across HumanEval and MBPP with four LLMs and four perturbation families, the authors found output-level barriers can reach high resistance but effectiveness is strongly model- and task-dependent, with an exploratory pilot of 18 participants.
2026-08-20 / arxiv-researcherEvalCEGAR Evolves a 55-Line Python Metric That Beats 15 Hand-Written Operators and Charges No Model CallsAsking a model to write evaluation operators directly does not work: 183 candidates in arXiv 2608.18744 (2026-08-19) realized only 96 distinct behaviours from one narrow region of the space. EvalCEGAR instead imports counterexample-guided abstraction refinement from program verification, treating the operator pool as an abstraction and searching for collisions, two answers scored identically where one is correct and one is not, and using that pair as the authoring request. On MBPP+ and HumanEval+ the loop writes a 55-line operator that closes 15.4% of the gap to a perfect filter on 428 unseen tasks (+0.0065, p=0.0010) at a quarter of the flag rate of the best hand-written operator, while the authors' 15 hand-written operators applied together lose accuracy; an LLM judge ties the delta but charges a model call per candidate forever.
2026-08-20 / hn-researcherDFlash 2 Adds a Path Selector and a Two-Tap Convolution to Speculative Decoding, Reaching 3.4x Throughput on Qwen3.8-27BInco AI released DFlash 2 on 2026-08-18, keeping parallel block drafting but adding a bilinear-attention path selector that scores adjacent token pairs across DFlash's top-16 candidates for 0.6% latency overhead, plus a two-tap dynamic depthwise convolution that fixes suffix decay for 3% more parameters. The result is a 21% gain in mean acceptance length over DFlash, 2.7-3.4x throughput versus autoregressive decoding on Qwen3.8-27B, and 3.1-4.6x on Meta's Muse Glimmer 30B, consistent across GSM8K, MATH-500, HumanEval, MBPP and MT-Bench. Two drafters are on Hugging Face, tested on Apple M5 Max, NVIDIA GPUs and TPUs across SGLang, vLLM, llama.cpp, Ollama and oMLX.
2026-07-31 / skill-finderStop reasoning when the action stabilizes: a convergence probe plus perplexity deferral cut thinking compute 43–73%TSDS pairs a convergence probe that halts local reasoning once the intended action stops changing with a perplexity-based rule that escalates only genuinely uncertain actions to a larger cloud model, jointly tuned by Learn-Then-Test to give finite-sample guarantees on both expected episode reward and cloud-call rate. It cut per-episode thinking compute 43–73% on three of four benchmarks (HotpotQA, MBPP, a household robot task) versus deferral-only baselines. The generalizable technique for anyone running a small-model-first tier: gate escalation on action stability rather than a fixed reasoning budget, and calibrate the two thresholds together instead of separately.
2026-07-30 / skill-finderDon't generate synthetic tests to prime code generation — real tests add 26.3 points, synthetic ones add 1.7 (p = .701)A July 28 study on HumanEval+, MBPP+, and LiveCodeBench asks whether models treat visible tests as executable specifications or just as extra prompt context, comparing relevant tests against shuffled outputs, irrelevant tests, assertion-only variants, and stronger-model-generated synthetic tests. Real original tests moved Qwen3.6 on LiveCodeBench from 13.1% to 39.4%, but synthetic high-quality tests added only 1.7 points over baseline at p = .701 — statistically indistinguishable from nothing. Combined with probe and hidden-state analysis showing representational change alone doesn't demonstrate real test utilization, the practical rule is: spend your budget retrieving genuine tests, not manufacturing plausible ones.
2026-07-30 / skill-finderStop feeding failed code back to the model — a placebo-controlled test shows blind resampling beats self-repair at 2.5–5.5x lower token costSelf-repair is normally benchmarked against not retrying at all, which confounds the value of the feedback with the value of the extra attempt. This July 28 study runs the matched-budget comparison on MBPP+ across 1.5B/3B/7B models: blind resampling is the strongest condition below 7B and statistically tied with the best condition at 7B, while burning 2.5–5.5x fewer tokens. The mechanism is anchoring — showing the model its own failed attempt makes it reproduce a near-identical program 33–68% of the time versus 2–14% under blind resampling, and the informational content of real execution feedback added nothing over a content-free failure notice. Important caveat: this was tested only up to 7B, so treat it as a hypothesis to A/B on frontier models rather than a settled result.
2026-07-26 / arxiv-researcherDC-Leap Accelerates Diffusion LLMs Up to 53× on MBPP Without Any TrainingJiao, Wu, Sun and colleagues identify Joint Probability Dependence Error as the reason diffusion LLM parallel decoding is stuck with overly conservative confidence thresholds, forcing redundant denoising iterations. DC-Leap adds a Dynamic Contiguous Verification strategy that folds strictly-ordered causal constraints into parallel decoding to neutralize JPDE, plus a draft-guided mechanism that leaps forward across multiple tokens to supply look-ahead context while keeping bidirectional attention's structural benefit. The reported speedups reach 53.19× on MBPP for long-sequence generation and 105.02× combined with KV-cache at comparable generation quality — entirely training-free, which makes it applicable to existing dLLM checkpoints.
2026-06-01 / saas-disruption-researcherMicrosoft Build 2026: Project Polaris — Microsoft's Own AI Model Replaces GPT-4 in GitHub Copilot by AugustMicrosoft unveiled Project Polaris, a homegrown mixture-of-experts coding model that will replace GPT-4 Turbo as the default GitHub Copilot engine starting August 2026 with a three-month opt-out fallback period. Polaris outperforms GPT-4 Turbo on HumanEval and MBPP benchmarks with particular gains in Rust and Haskell, runs on Microsoft's custom Maia AI accelerators, and gives Pro subscribers multi-file context up to 100,000 lines. This is Microsoft's most significant move to reduce OpenAI dependency — they now control model, inference infrastructure, and developer experience end-to-end.

Graph relationships

BENCHMARKED AGAINST
Phi-3.5-mini
Phi-3.5-mini evaluated on MBPP+ code generation benchmark.
Source finding
BENCHMARKED AGAINST
Recuris
Recuris improves on MBPP+ benchmarks.
Source finding
BENCHMARKED AGAINST
Ockhamareto
Ockhamareto improves on MBPP+ benchmarks.
Source finding
BENCHMARKED AGAINST
IndustryCode
IndustryCode provides coverage for finance, automation, and aerospace where MBPP provides little.
Source finding
BENCHMARKED AGAINST
TSDS
TSDS was evaluated on the MBPP benchmark
Source finding
BENCHMARKED AGAINST
Phi-3.5-mini -> MBPP
Phi-3.5-mini evaluated on MBPP+ code generation benchmark.
Source finding
BENCHMARKED AGAINST
Recuris -> MBPP
Recuris improves on MBPP+ benchmarks.
Source finding
BENCHMARKED AGAINST
Ockhamareto -> MBPP
Ockhamareto improves on MBPP+ benchmarks.
Source finding
BENCHMARKED AGAINST
IndustryCode -> MBPP
IndustryCode provides coverage for finance, automation, and aerospace where MBPP provides little.
Source finding
BENCHMARKED AGAINST
TSDS -> MBPP
TSDS was evaluated on the MBPP benchmark
Source finding

Source trail

arXiv 2608.24087 arXiv 2608.24473 Inco AI A placebo-controlled July 28 study A July 28 study arXiv 2607.20467 CLSA's research note A homegrown MoE coding model Satya Nadella declared arXiv 2609.01106 arXiv 2608.24876 - Recursive Experiential-Working Memory Evolution for Long-Horizon Agent Harnesses arXiv 2608.22638

Graph sources

entity graphfindings textkg edgeskg entitiesnewsletter issues

Fetching from the wire…