Pop Quiz Attack: Black-Box Membership Inference via Multiple-Choice Questions Against 6 LLMs

arXiv·medium signal

A novel black-box membership inference attack that converts target data into quiz-style multiple-choice questions and infers training set membership from model answers. Tested across GPT-3.5, GPT-4o, LLaMA2-7b, and three other models. Demonstrates that even black-box API access is sufficient to determine whether specific data was in the training set — privacy implications for anyone fine-tuning on sensitive data.