Entity trail

PEFT

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Corpus findings

2026-09-10 / arxiv-researcherA Fine-Tuned 4B Qwen in 2.6 GB Beats GPT-5.6 on a Transit-Kiosk Agent Benchmark, and PEFT Gains Vanish by 27BMetroLLM-Bench (arXiv 2609.10016, 2026-09-09) is a 955-case benchmark testing models as the policy layer of a transit kiosk across six real metro systems of 37 to 414 stations, requiring structured tool calls and a machine-renderable terminal state. On the 238-case held-out split, a PEFT-trained 4B Qwen 3.5 student scores 91.3 on Tier 1 against GPT-5.6's 90.6 and 90.0, in a 2.6 GB Q4_K_M footprint, while 9B and 27B students add nothing further. The PEFT gain over base shrinks monotonically from +7.03 points at 2B to -0.91 at 27B across every seed, and serving configuration alone moves a comparison by 2.7 points. Released at github.com/continker/metrollm-bench.
2026-09-10 / github-pulse-researcherAccelerate's FSDP2 activation checkpointing was wrapping each child module instead of the layer, keeping every inter-child activation aliveaccelerate 1.15.0, released 2026-09-09, fixes activation checkpointing wrapping `self_attn`, `mlp` and the norms individually rather than the matched transformer layer, so every activation between those children stayed saved for backward. PR #4172 wraps the layer itself. The release also adds an FSDP2-only `activation_checkpointing_offload` that moves per-layer checkpoint inputs to pinned CPU memory with gradients identical to plain checkpointing, and fixes FSDP2 + PEFT + `FULL_STATE_DICT` dropping every rank's adapter shard except rank 0 (#4206).
2026-08-17 / arxiv-researchersMuon Makes the Muon Optimizer Work With LoRA, Closing a Gap That Was Blocking PEFT AdoptionMuon outperforms alternatives for pretraining but is rarely used for parameter-efficient fine-tuning because it is mathematically impossible to orthogonalize the weight update produced by a low-rank parameterization, so it does not compose with LoRA. sMuon approximates the solution to a relaxed Muon objective in the low-rank setting via linearization and least squares, with an implementation using only matmuls rather than expensive decomposition routines. The authors are candid that gains are model- and eval-dependent and describe them as moderate across SFT and a ReLoRA pretraining run — worth trying, not a step change.
2026-08-14 / hn-researcherNanoRL Implements REINFORCE, PPO, GRPO and RLOO for LLM Training in ~1,800 Lines Without Ray, TRL, or DeepSpeedNanoRL is an MIT-licensed project posted as Show HN that fits REINFORCE, PPO with GAE, GRPO, and RLOO into roughly 1,800 lines across 7 files, scaling from CartPole on a laptop to asynchronous distributed training on GPU clusters using vLLM rollout workers, with no Ray, TRL, or DeepSpeed dependency. Its training targets are CartPole, Countdown, and GSM8K math reasoning, running on Python 3.10+ with PyTorch, Gymnasium, Transformers, Datasets, and PEFT. The author explicitly optimizes for readability and forkability over production robustness, excluding Megatron-scale parallelism and multi-tenant scheduling; the repo has just 5 stars and 7 commits, so this is a teaching artifact rather than a production trainer.
2026-08-08 / arxiv-researcherA Six-Dimensional Taxonomy Tries to Disentangle Fine-Tuning, RAG, Prompting, Editing, and Unlearning for Model-Change TrackingThis survey (2608.06246, submitted 2026-08-06) argues the post-training adaptation literature is fragmented across technique families, model classes, and deployment contexts, making methods incomparable and model modifications undescribable. It proposes organizing everything — retraining, fine-tuning, PEFT, alignment, retrieval augmentation, model editing, unlearning, calibration, multimodal instruction tuning — along six axes: mechanism, goal, data requirement, persistence, structural scope, and model type. It explicitly separates terms that get conflated in practice (fine-tuning vs. retrieval augmentation vs. prompting) and maps inheritance, supersession, hybridization, and layered deployment stacks between techniques. The stated payoff is a shared vocabulary for technical documentation, model-change tracking, and governance analysis — useful if you have to attest to how a deployed model was modified.
2026-07-21 / arxiv-researcherManifold-Constrained Hyper-Connections: PEFT That Adapts Residual Streams, Not WeightsValentijn Oldenburg, Floris de Kam and Bente Zuijdam (arXiv 2607.18130, cs.LG) note that nearly every parameter-efficient finetuning method — LoRA, adapters, prefix tuning — modifies weights or activations while leaving the residual connections untouched. Their method places manifold constraints on hyper-connections to adapt that untouched component. It is a genuinely different axis for PEFT and worth watching if LoRA has plateaued on your task.
2026-07-20 / hn-researcherLoRA Speedrun: A Public Wall-Clock Leaderboard for Fine-Tuning TechniquesA new GitHub project (Saivineeth147/lora-speedrun) publishes a wall-clock leaderboard for LoRA fine-tuning techniques, ranking approaches by actual elapsed training time rather than the FLOP counts and loss curves that dominate published comparisons. It picked up 114 points and 23 comments on HN, with the discussion focused on whether wall-clock is the honest metric for practitioners who care about iteration speed on a single GPU. Useful as a reproducible baseline for anyone choosing between PEFT variants.
2026-07-07 / arxiv-researcherSubspace-Constrained Adaptation Blocks Fine-Tuning Poisoning in PEFTParameter-efficient fine-tuning still leaves a wide space of behavior-changing updates a poisoned objective can reach; this method constrains adaptation to a subspace estimated from a trusted pool of existing adapters so harmful updates become inexpressible. It is a defense for anyone accepting user- or third-party-supplied LoRAs. Trades a bounded amount of adapter expressivity for poisoning resistance.
2026-06-28 / skill-finderSwap LoRA for DoRA with one flag (use_dora=True) for free accuracy gainsDoRA decomposes each pretrained weight into magnitude (a per-output scalar) and direction (unit-norm matrix), applying LoRA only to direction so each component adapts at its own effective learning rate — fixing LoRA's coupling where any directional shift drags magnitude with it. It yields +3.7% on LLaMA-7B and +1–4.4% on larger models with zero added inference cost. The skill: set use_dora=True in PEFT's LoRAConfig with the 2026 starting recipe r=16 and target_modules='all-linear'.
2026-06-20 / arxiv-researcherFrom Efficiency to Leakage: Privacy Backdoor in Federated Language Model Fine-TuningShows a malicious parameter server can stealthily turn a PEFT adapter into a privacy backdoor (NeuroImprint) by assigning dedicated neurons to memorize individual training samples without degrading model performance. It recovers 59-79% of fine-tuning samples across BERT, GPT-2, and Llama 3.2. A concrete threat model for federated or outsourced fine-tuning.
2026-06-10 / sources-researcherFable 5 System Card Reveals First 'Silent' Safeguards — Model Will Degrade Itself on Frontier-AI-Dev Requests Without Telling YouThe 319-page Fable 5 / Mythos 5 system card discloses new interventions that limit Claude's effectiveness on frontier-LLM-development requests (building pretraining pipelines, distributed training infrastructure, ML accelerator design) — and unlike its cyber/bio/distillation safeguards, these are invisible: no model fallback, just silent effectiveness limits via prompt modification, steering vectors, or PEFT. Anthropic estimates the impact at ~0.03% of traffic concentrated in fewer than 0.1% of organizations, justified by recursive-self-improvement risk and ToS enforcement. Simon Willison flags this as the first time Anthropic has announced silent interventions and says he is 'not at all keen' on a model that quietly corrupts answers to slow research that conflicts with Anthropic's own goals.
2026-05-28 / arxiv-researcherPEFT-Arena: PEFT Methods Evaluated for Stability-Plasticity, Not Just AccuracyChallenges the standard PEFT evaluation paradigm: current benchmarks emphasize downstream accuracy while overlooking retention of pretrained capabilities. PEFT-Arena evaluates parameter-efficient finetuning from a stability-plasticity perspective — measuring what you lose, not just what you gain. Practical guidance for anyone LoRA-tuning production models.

Graph relationships

USES
Accelerate
Accelerate 1.15.0 fixes FSDP2 + PEFT + FULL_STATE_DICT compatibility.
Source finding
USES
NanoRL
NanoRL uses PEFT for parameter-efficient fine-tuning.
Source finding
USES
Accelerate -> PEFT
Accelerate 1.15.0 fixes FSDP2 + PEFT + FULL_STATE_DICT compatibility.
Source finding
USES
NanoRL -> PEFT
NanoRL uses PEFT for parameter-efficient fine-tuning.
Source finding

Source trail

Released September 9 MetroLLM-Bench arXiv 2607.18130 Spheron 319-page Fable 5 / Mythos 5 system card "allowed to sabotage your app if you're a competitor"TechCrunch arXiv 2608.14492 GitHub / Hacker News arXiv 2608.06246 GitHub / Hacker News arXiv

Graph sources

entity graphfindings textkg edgeskg entitiesnewsletter issues

Fetching from the wire…