Entity trail

GRPO

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Showing the first 40 findings. More graph evidence exists in the corpus.

Corpus findings

2026-09-09 / arxiv-researcherEnriching the Environment's Feedback Beats Warming Up the Agent for Sparse Long-Horizon RLRather than agent-side warm-up via supervised fine-tuning, which is limited by data scarcity and constrained exploration, this work builds Feedback-Enriched Environments that shift from action guidance to observation enrichment in the later stages of both intra-episode exploration and inter-episode evolution. Experiments on SciWorld and BFCL across multiple Qwen3 scales with GRPO, GSPO and DAPO show consistent improvement over standard settings. Analysis reports that FEEs stabilize training by reducing entropy volatility, encourage proactive state-space exploration on hard tasks, internalize environmental guidance into policy weights rather than acting as an inference-time prior, and identify intra-group feedback consistency as the boundary condition for stable optimization.
2026-09-08 / skill-finderSynthesizing tool-call training data by executing the tools first lets a 9B model nearly match a 27B oneKOPA-Bench and EDGE (arXiv 2609.05395, v1 2026-09-04) target multi-step tool chaining against live government APIs, a setting where data-sovereignty rules force on-premise open-source agents and where open models consistently underperform. EDGE builds a graph of how each tool's output can feed another tool's input, keeps only the links that actually succeed when called against the live APIs, and traverses those verified links to synthesize executable multi-step trajectories. Fine-tuned with GRPO on that data, their 9B model nearly matches the untuned 27B model from the same family, improving on both the 145-task KOPA-Bench and on BFCL. The reusable idea is that execution-grounded edge pruning is what makes synthetic tool-call data useful; generating plausible chains without calling the APIs is what produces the usual garbage.
2026-09-06 / arxiv-researcherReplay Selection Alone Matches Full Replay Pipelines and Cuts Agentic-Search Wall-Clock TimeRL post-training is bottlenecked by fresh rollout generation, and in agentic settings environment interaction dominates wall-clock cost, but existing replay methods bury their contribution inside exploration and mixed-policy machinery. Headroom-Drift Replay is a group-level control primitive for GRPO splitting reuse into two decisions: Headroom ranks stored groups by remaining learning value, Drift gates them by compatibility with the current policy, with the fresh on-policy stream untouched and no auxiliary generation added. Across mathematical reasoning, multimodal reasoning and Agentic Search it beats naive replay and matches or exceeds broader replay methods on Avg Mean@32, delivering comparable quality at materially lower wall-clock time where environment calls dominate.
2026-09-04 / arxiv-researcherGRPO Rewards Lucky Guesses: Spurious Advantage Appears Wherever the Answer Space Is BoundedGRPO's advantage estimator assigns each rollout a magnitude from within-group reward statistics, which normally rewards reaching the correct answer through reasoning, but an overlooked case shares the same surface: a rollout that lands on the answer by guessing gets the same high magnitude. The authors name this spurious advantage and locate it in three settings, bounded-answer tasks with a small candidate set, open-answer sets hosting bounded sub-cases, and search agents whose budget opens many paths to the same answer, all of which push the policy toward guess-like behavior. Their fix, SIGNBALANCE, keeps only the verifier sign with a global scale and restores zero-mean balance via stop-gradient per-class rescaling, matching GRPO on open-answer math and improving on bounded-answer math and search agents.
2026-09-04 / arxiv-researcherDRACO Redistributes One Rubric Score Across Agent Steps and Beats Ground-Truth-Reward GRPO by 5.3 Points on AppWorldRL from verifiable rewards works when a task has a programmatic checker, but most long-horizon agent domains have none, and the usual fallback of a multi-criteria rubric scored once per trajectory gives one scalar across tens of steps. DRACO generates rubrics dynamically during training to track the policy's evolving capability, scores them once per completed trajectory, then redistributes that judgment in closed form over the steps responsible for annotated rubrics to produce per-step GRPO advantages, with no trained attribution module. It gains 15.9 points over the base model on AppWorld and 5.3 points over GRPO trained on a sparse ground-truth reward despite using no verifier, and 5.3 points on out-of-domain Tau-Bench without a frontier judge; code is released.
2026-09-03 / rss-researcherHugging Face used GRPO to teach a 35B coding model to paint watercolours, and published the whole environmentA September 3 post trained Qwen/Qwen3.5-35B-A3B with LoRA using GRPO inside OpenEnv, which wraps a JavaScript rendering pipeline (the p5.brush library, headless Chromium, and validation gates rejecting invalid sketches) as an RL environment. Three runs of 60-110 steps, 8 rollouts per step and 240 episodes moved mean group reward from 0.45-0.58 to 0.71-0.82, on one H200 for 18-34 hours per run. The interesting mechanic for anyone designing rewards is the ordering the model learned: it eliminated invalid outputs first, and only improved quality once the pairwise style judge carried enough weight.
2026-09-03 / rss-researcher100 GRPO steps on a free Colab GPU lifted a 350M model's JSON compliance from 18.0% to 31.9%A Hugging Face post published September 3 fine-tuned LiquidAI/LFM2.5-350M with TRL's GRPO on roughly 500 samples from nvidia/Nemotron-RL-instruction_following-structured_outputs, augmented with 40% code-block instructions and 20% top-level-array conversions. Overall IFStruct pass rate went from 22.6% to 29.7% in 100 steps, with JSON format specifically going 18.0% to 31.9%, on a free-tier Colab or Kaggle GPU with 16GB VRAM. The reward is three functions, JSON validity, field count accuracy and schema validation, weighted [1.0, 0.5, 2.0], which is a directly copyable recipe for making a small local model reliable at structured output.
2026-09-03 / sources-researcherCoGR trains LLMs to generate keywords on both the query and item side so generative retrieval drops into existing inverted-index infrastructurearXiv 2609.00638 (2026-09-01, 61 HF upvotes) targets a practical blocker in generative retrieval: LLMs are usually confined to query-side augmentation while a conventional retriever still does the matching, because replacing the index is too expensive. CoGR has separate generators emit a compact keyword set for queries and for items, matched directly through an inverted index, so existing keyword retrieval infrastructure keeps working. Training is supervised fine-tuning to align the keyword space, then co-evolving reinforcement learning that alternately optimizes the two generators with GRPO.
2026-09-03 / sources-researcherAn enterprise consolidated 200+ internal apps onto one self-hosted model by training three separate GRPO experts and merging them with two-stage SLERParXiv 2609.01572 (2026-09-01, 33 HF upvotes) is a rare production write-up rather than a benchmark paper: data-residency rules force self-hosting, and stacking newer models without retiring old ones fragments a finite GPU pool. The team closed quality gaps along instruction following, function calling and internal task distribution, but found that optimizing all three jointly caused cross-domain reward interference, so they trained one GRPO expert per axis and merged via two-stage SLERP. Each reward exposed a distinct failure mode with its own fix (semantic collapse, over-calling, verbosity hacking), and in non-reasoning mode the merged model beats one roughly 7x larger by total parameters.
2026-09-01 / sources-researcherCogEvol reports a caught reward-hacking episode alongside its production numbers, and open-weights the 4BarXiv 2608.30968 (submitted 2026-08-31) trains models to turn a course brief into finished slides or a self-contained interactive HTML page in a single pass, replacing multi-turn agent scaffolding. Across 220k production requests the median slide takes 17 seconds and an interactive page 59; a production-grounded pipeline turned real failures into 53,687 verified SFT samples, and a hybrid rule-plus-VLM reward drives GRPO, hardened after the team caught a reward-hacking episode that produced visually convincing but unplayable games. CogEvol-27B scores 83.7 on slide quality and 63.7 on a 500-case interactive-HTML benchmark with 26.9x fewer parameters than flagship coding models, and CogEvol-4B is Apache 2.0 on GitHub.
2026-08-30 / arxiv-researcherA Compute-Centric Taxonomy of Parallelism for RL Post-Training, Framed With the Work-Depth ModelReasoning-model training via RLVR needs millions of GPU-hours and tightly coupled multi-model pipelines, making it as much a distributed systems problem as an algorithmic one, yet the parallelism literature has treated it as the latter. This work systematizes PPO, GRPO and their variants under a compute-centric analysis, then builds a taxonomy spanning traditional data, tensor, pipeline, sequence, context and expert parallelism alongside RL-specific forms such as disaggregated placement, stage fusion, hybrid parallelism and asynchronous execution. The work-depth model of parallel computing is used to keep the taxonomy rigorous and portable across frameworks, and the paper closes with practical guidelines for building cost-effective RLM training stacks.
2026-08-29 / arxiv-researcherMixture of Roles Matches Multi-Agent Performance in One Turn at 20x Lower Token CostMoRe learns a codebook of steering vectors, each encoding a latent role, and uses a query-aware router to fuse them into a single composed steering vector for single-turn inference, giving multi-perspective specialization without multi-turn agent orchestration. The backbone LLM stays frozen and training runs as a three-stage SFT curriculum plus GRPO post-training. Across reasoning and personality benchmarks it beats single-agent baselines by 2.2% on average and matches multi-agent systems while cutting token cost 20x, which is a direct counter-argument to defaulting to a multi-agent pipeline for role diversity.

Graph relationships

USES
CogEvol
CogEvol training uses GRPO, a hybrid rule-plus-VLM reward method.
Source finding
BENCHMARKED AGAINST
SAPO
SAPO beats GRPO by 12.1 percentage points on agent benchmarks.
Source finding
USES
Amazon Nova Lite 2.0
Amazon Nova Lite 2.0 was trained using GRPO with four-component rewards.
Source finding
USES
TRAVEL
TraVEL applies Group Relative Policy Optimization with ego-trajectory similarity reward.
Source finding
USES
MemHarness
MemHarness is trained end-to-end with GRPO.
Source finding
USES
DeepSeek-R1
GRPO is the algorithm behind DeepSeek R1
Source finding
RELEASED
Unsloth
Unsloth released long-context GRPO with batching algorithms
Source finding
USES
DeepSeek-R1
DeepSeek-R1 uses GRPO algorithm for optimization.
Source finding
CRITICIZES
TRIAGE
TRIAGE addresses GRPO's flaw of uniform advantage that punishes useful exploration.
Source finding
USES
Gradient
Gradient trains agents using GRPO.
Source finding
BENCHMARKED AGAINST
CoPES
CoPES recovers 92% of GRPO's validation-accuracy gains with one-eighth the GPU memory.
Source finding
USES
MIP
GRPO system uses mixed-integer programming formulations as candidate solutions
Source finding

Source trail

arXiv 2609.04063 arXiv 2608.30968 arXiv arXiv arXiv 2608.27266 arXiv 2608.24658 RobertGolds1/Gradient Deep Learning Focus arXiv Ornith LittleLearner AWS

Graph sources

entity graphfindings textkg edgeskg entitiesnewsletter issues

Fetching from the wire…