Green Text Alone Flips Vision Model Sentiment Scores

Coloring positive words green pushes vision-language model sentiment predictions positive, even when the same text contains negative words, per the arXiv paper on VLM color bias (2608.14286).

Anyone feeding screenshots, resumes or documents into a VLM inside an automated decision process inherits this blind spot. The effect is strong enough that models stop weighing negative words at all.

The authors trace the effect to color-induced changes in the vision encoder's latent representations. That happens before any language-level reasoning, so a text-level filter has nothing to catch.

Lowering text-background contrast makes it worse. The model leans harder on visually salient cues, and the paper reports more wrong answers on visual question-answering (VQA) tasks as a result.

Code for reproducing the result is public at github.com/KohsukeIde/color-bias-vlm.

The take

This is a prompt-injection vector, not a fairness footnote. Any system that feeds a VLM a rendered screenshot, resume or document has an attack surface that text-based content filters cannot see. The exploit lives in pixels, not tokens.

Watch for whether vendors start shipping color-normalization checks ahead of the vision encoder, since that's the only place this can be caught.