ML repos' own validation workflows caught 2 of 23 reproducibility-breaking mutations.

MLReproMutate applies controlled mutations across four classes (random seed, dependency pin, data split, cross-validation fold count) and runs them against the validation workflows the repositories already ship. Of 39 frozen repository-operator cases, 24 were evaluable with 23 confirmed non-equivalent mutations, and existing workflows detected 2. An 8.7% detection proportion. (arXiv 2608.27100) The authors are careful this doesn't make those repos irreproducible, only that green CI tells you almost nothing about whether experimentally important choices are pinned.