Vision-Default, Prior-Override: Causal Mechanisms of Perception-Knowledge Conflict in Vision-Language Models
arXiv:2606.28273v1 Announce Type: new Abstract: Vision-language models must reconcile visual evidence with memorized world knowledge when the two conflict. How they resolve this conflict shapes the reliability of multimodal systems, yet prior work characterizes it behaviorally without a component-level causal account. We combine activation patching across three granularities (residual stream, attention heads, and MLP sublayers) with model-component ablation studies and mechanistic analysis. Across three VLM families, we find that visual grounding emerges by default, whereas prior grounding dep
This research addresses a critical limitation in current Vision-Language Models (VLMs) as they become more ubiquitous and are deployed in high-stakes applications requiring reliable decision-making.
Understanding how VLMs reconcile conflicting information is crucial for building robust AI systems that can avoid biases and make accurate interpretations in complex real-world scenarios.
The ability to causally analyze and influence VLM behavior regarding perception-knowledge conflicts changes the approach to designing and fine-tuning more reliable multimodal AI.
- · AI developers
- · Multimodal AI research
- · Autonomous systems
- · Developers of unreliable VLMs
- · Applications with high perception-knowledge conflict risks
Improved interpretability and control over VLM decision-making in ambiguous situations.
Faster development and deployment of trusted AI agents and systems in sensitive domains.
Enhanced AI capabilities to operate effectively in environments where visual evidence and pre-existing knowledge may diverge significantly.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.CL