Accuracy paradox: addressing epistemic, manipulative, and societal risks of hallucination in AI governance

The rise of generative AI has intensified concerns around AI hallucination, which involves outputs that are fabricated, misleading, oversimplified or untrustworthy. While many technical and policy responses treat hallucination as a failure of factual accuracy, this paper argues that such a narrow lens underestimates the complexity of the problem. AI hallucination is not merely a matter of truth or falsehood, but a multifaceted phenomenon with cognitive, communicative, and societal implications. Overreliance on accuracy has counterproductive effect: the accuracy paradox. We propose a taxonomy and theoretical framework for understanding hallucination risks across three dimensions: epistemic reliability, Human-AI interactive influence, and social impact. Through regulatory analysis, we show that accuracy-driven approaches often overlook harms such as illusion of consensus, subtly persuasive misinformation, and diminished social progression. Current legal regulation, including the EU AI Act, GDPR, and DSA, struggle to address these subtler forms of distortion. We call for regulatory strategies that go beyond static verification, embracing pluralistic, context-aware, and manipulation-resilient approaches to AI trustworthy governance.