Vision-Default, Prior-Override: Causal Mechanisms of Perception-Knowledge Conflict in VLMs

arXiv·medium signal

Niclas Lietzow, Danielle Bitterman, and Carsten Eickhoff probe how vision-language models reconcile visual evidence against memorized world knowledge when the two conflict, identifying a 'vision-default, prior-override' causal mechanism. The finding is directly useful for practitioners debugging VLM hallucinations where the model overrides what it actually sees with baked-in priors. It gives a mechanistic handle on when to trust a VLM's perception versus its memory.