Exploring why vision-language models hallucinate visual details of nonexistent objects to follow prompt-based information
CosmicAI researcher William Rudman explored why large vision-language models describe hallucinate visual details about objects not present in an image, but present in the prompt. Working in a controlled object-counting setting, he studied a failure mode called prompt-induced hallucination (PIH), in which a model that correctly counts the objects in an image, but hallucinates details about nonexistent objects to conform to an input prompt. Through mechanistic analysis, he identified a small set of attention heads whose ablation substantially reduces PIH without additional training.
The team also collaborated with Michal Golovanevsky, Dana Arad, Yonatan Belinkov, and Kyle Mahowald.
Vision-language models (VLMs) often follow textual prompts even when they contradict visual evidence, a growing concern as they are deployed in settings with noisy or inaccurate user inputs. Asking a model to "describe the four waterlilies in the image" when only three are present should elicit a correction. Instead, models frequently describe a nonexistent fourth flower in detail, including its position in the pond. This is unlikely to be driven by limitations in numerical reasoning, since the same model correctly answers the underlying counting question when asked directly. The error appears only when the prompt supplies a competing number.
What the team did
The researchers evaluated three state of the art open sourced VLMs. For each image with ground-truth count N, the team used misaligned prompts asking the model to describe N + k objects, where k denotes the discrepancy distance. To locate causally responsible model components, the team performed attention head knockouts to prevent a given attention head from contributing to the models final output. Heads were ranked by the fraction of samples that switched from the prompted count to the true count, yielding a per-model set referred to as PIH attention heads. To test generalization, the same heads were applied to a color identification where the prompt expresses counterfactual inputs.
What the researchers found
For images containing small numbers of objects (2 to 4), models typically conform to small offsets, but correct users when the offset between the asserted and ground truth count is large. However, beyond four objects, models increasingly conform to the prompt regardless of the discrepancy size, with prompt-matching responses approaching 80 to 90% and ground truth responses dropping to near zero. The effect persists for extreme offsets, with models readily describing, for example, fifty-nine cats in an image containing only nine.
Ablating PIH-heads substantially reduces prompt-following behavior, while random ablations produce only marginal reductions. Prompt match rates drop from ~64% before intervention to below 11% after ablation, and true count match rates rise to ~78%. The same heads reduce prompt-color copying by up to 94.25%, demonstrating that PIH heads identified through counting are not task-specific. Further, performance remains broadly stable on other visual question answering benchmarks indicating the intervention does not disrupt model performance.
Why the work matters
VLMs are increasingly deployed in settings where users supply mistaken or leading input, and a model that silently agrees with a false premise is more harmful than one that visibly fails. This work shows that such hallucinations stem from systematic prompt-based copying rather than failures of visual perception, and that they are mechanistically localized to a small number of early-layer language-model heads shared across architectures. Because the intervention requires no additional data or training and does not degrade general performance, these results highlight PIH as a tractable failure mode that targeted interventions can address.
The team is now studying how multimodal LLMs balance user assertions against counterfactual input on galaxy morphology, using the graded, probabilistic human labels of Galaxy Zoo to measure sycophancy along both human certainty and feature rarity. Preliminary results show that models such as Claude Opus are accurate on salient features under neutral prompts but exhibit a similar prompt-induced hallucination when users assert a counterfactual opinion.
The paper was published in ACL 2026.