Detecting benchmark leakage with code-specific features beat perplexity-only methods on eight code benchmarks

arXiv 2609.09865·medium signalXBlueskyLinkedInCopy link

CGMIA fine-tunes a shadow model on a subset of benchmark samples to build labeled member and non-member data, then extracts expert features per sample (CodeBLEU, edit distance, test pass rate, perplexity) alongside CodeBERT semantic embeddings, and combines them in a learned classifier. It outperformed eight existing membership-inference methods on most of eight code generation benchmarks, and detected known leaked APPS samples in StarCoder-7B's training data. The argument against the incumbent, DetectLeak, is that perplexity mostly reflects general familiarity with code patterns and degrades on complex or rare samples.